JohnLangford/vowpal_wabbit — pull request #137 by pierce1987 (2013-07-28 19:52:22+00:00)
Title: Holdout valid

--holdout arg : to use a portion of data as holdout set for testing only. Useful to select the best model across passes. Have a better idea when the model overfits in multiple passes.

arg is between 0 (default) and 1. For example, if arg ==0.3, 30% of the chance one example is used for testing only.

The holdout set is consistent across passes.

can use --random_seed arg to specify the random seed

When holdout is on, the output looks like (--passes 500 --holdout 0.1 --random_seed 7)
```
average since example example current current current
loss last counter weight label predict features
average since example example current current current
loss last counter weight label predict features
5.390109 5.390109 3 3.0 2.0000 0.5267 5
0.209017 0.209017 6 6.0 2.0000 1.1834 5h
0.456507 0.951488 11 11.0 3.0000 2.7605 7h
1.306819 1.944553 22 22.0 3.0000 2.9959 7h
1.960639 2.614460 44 44.0 3.0000 3.0000 7h
2.381127 2.773581 87 87.0 2.0000 2.0000 5h
2.580477 2.779828 174 174.0 2.0000 2.0000 5h
2.680154 2.779831 348 348.0 2.0000 2.0000 5h
2.729993 2.779831 696 696.0 2.0000 2.0000 5h
2.754912 2.779831 1392 1392.0 2.0000 2.0000 5h
```
'h' at the end indicates the reported errors at the same line are from holdout test, instead of progressive validation loss( e.g. in the first line)
in this case the trained model converges and overfits, thus generating bad results on the holdout set. In some cases the holdout test error would go up and down.

We are also working on --save_best_model. Two possible strategies: Strategy 1: maintain a 'best_so_far' for holdout validation loss, save the model whenever a better one is achieved. By the end of user specified passes, the best model along the way is maintained.
Strategy 2: early stop when there is no improvement in k (maybe user specified) passes.

This is like an efficiency vs effectiveness tradeoff, which one do you guys think is better? (or both?)