eHarmony/vowpal_wabbit — pull request #5 by pierce1987 (2013-07-24 02:00:00+00:00)
Title: --holdout arg to use examples for test only

--holdout arg : to use a portion of data as holdout set for testing only. Useful to select the best model across passes.

arg is between 0 (default) and 1. For example, if arg ==0.3, 30% of the chance one example is used for testing only.

The holdout set is consistent across passess.

can use --random_seed arg to specify the random seed

Now after each pass, output like the following: ( e.g. -c --passes 15 --holdout 0.3 --random_seed 1 )

** holdout loss on pass 0 is 1.67465 (overall average 1.67465, best 1.67465 on pass 0) **
** holdout loss on pass 1 is 1.31861 (overall average 1.49663, best 1.31861 on pass 1) **
** holdout loss on pass 2 is 1.24813 (overall average 1.4138, best 1.24813 on pass 2) **
** holdout loss on pass 3 is 1.24562 (overall average 1.37175, best 1.24562 on pass 3) **
** holdout loss on pass 4 is 1.25905 (overall average 1.34921, best 1.24562 on pass 3)
** holdout loss on pass 5 is 1.27308 (overall average 1.33653, best 1.24562 on pass 3)
** holdout loss on pass 6 is 1.28383 (overall average 1.329, best 1.24562 on pass 3)
** holdout loss on pass 7 is 1.29115 (overall average 1.32427, best 1.24562 on pass 3)
** holdout loss on pass 8 is 1.29584 (overall average 1.32111, best 1.24562 on pass 3)
** holdout loss on pass 9 is 1.29876 (overall average 1.31887, best 1.24562 on pass 3)
** holdout loss on pass 10 is 1.30053 (overall average 1.31721, best 1.24562 on pass 3)
** holdout loss on pass 11 is 1.3016 (overall average 1.31591, best 1.24562 on pass 3)
** holdout loss on pass 12 is 1.30224 (overall average 1.31485, best 1.24562 on pass 3)
** holdout loss on pass 13 is 1.30261 (overall average 1.31398, best 1.24562 on pass 3)
** holdout loss on pass 14 is 1.30278 (overall average 1.31323, best 1.24562 on pass 3)