Splitting the dataset and setting a test set apart

Using a basic TfidfVectorizer and BernoulliNB.

F1-Score: 0.64

A baseline, using 30% of the training set for validation:

             precision    recall  f1-score   support

          0       0.65      0.59      0.62     24001
          1       0.63      0.69      0.66     24000

avg / total       0.64      0.64      0.64     48001
Using 2 and 3-grams as features

F1-Score: 0.72.

As intuitively expected, the f1-score has improved:

             precision    recall  f1-score   support

          0       0.75      0.67      0.71     24001
          1       0.70      0.78      0.74     24000

avg / total       0.73      0.72      0.72     48001
Using sklearn's cross_validation

Instead of doing it by hand.