JohnLangford/vowpal_wabbit — pull request #141 by pierce1987 (2013-08-23 23:50:12+00:00)
Title: --bs arg and --bs_percentile for bootstrapping vw

John, here is an implementation of boostrapping vw predictions by online importance sampling from Poisson distribution suggested by Nikos.

bootstrapping vw to get confidence interval on predictions. Implemented as a reduction to gd.
--bs arg to specify arg numbers of bootstrapping rounds.
--bs_percentile arg (such as 0.95) to get 0.95 percentile in -p.

Example:
in training:
./vowpalwabbit/vw -d multiclass -k -c --passes 2000 --holdout_off -f model --bs 100
for testing:
./vowpalwabbit/vw -d multiclass -k -c -p result.txt -i model --bs 100 -t --bs_percentile 0.95

Then in result.txt we get:
```
1.645815 1.543533 1.725334
2.532156 2.468851 2.575895
3.527597 3.478951 3.574625
4.556210 4.514622 4.596277
5.475436 5.451206 5.500712
6.129002 6.075596 6.164859
6.529974 6.453068 6.593857
6.790787 6.671144 6.866988
6.961917 6.856057 7.052547
7.075695 6.946770 7.191328
```
The first number is final_prediction (mean value of bootstrapped predictions), followed by the lower bound and upper bound of confidence interval of the specified percentile.