JohnLangford/vowpal_wabbit — pull request #139 by pierce1987 (2013-08-05 22:38:30+00:00)
Title: --save_best_model

--save_best_model file_name
to save best model across passes. This is built upon the holdout validation part.

It is compatible with --save_per_pass and -f (user can have all any combination of them on.)
By default it is implemented by maintaining a best so far holdout loss for each pass. At the end of each pass, whenever a better model is met, a 'best' model is saved.
This guarantees to save the best predictor based on holdout set across user-specified passes.

update 8/3/2013: use --save_no_win arg for early stopping. If the holdout loss does not decrease for arg number of passes, the learner terminates (the best model has already been saved earlier.)

For example:
(1)--passes 1000 --save_best_model model_best: the best model across the 1000 passes would be saved based on holdout loss. The program will run 1000 passes.
(2)--passes 1000 --save_best_model model_best --save_no_win 5 : The best-so-far model might be saved after pass 5. Then for pass 6-10, if the holdout validation loss never decreases, the learner terminates. So vw only runs for 10 passes, instead of 1000.