petewarden/MLloWorld — pull request #1 by ogrisel (2011-11-01 14:58:28+00:00)
Title: Fixed varioous bugs in the CSV reader + better baseline model

Hi Pete!

I have found a couple of errors in the CSV parsing script. Also as field 6, 7, 8 are word occurrences in text fields it is much better to fit a linear model on a TF-IDF normalization of the occurrence vectors. Kernel SVM is pretty useless on high dim sparse data and much more CPU intensive.

The new baseline algorithm is still very bad (f1-score is around 0.5) but much better than random predictions as previously (because of the feature index shift between train and test).

Also (but maybe too late since the challenge has already started), you should probably have extracted occurrence information of bi-grams rather than individual words. That can improve text classification accuracy by quite a bit.