JohnLangford/vowpal_wabbit — pull request #136 by pierce1987 (2013-07-25 00:58:30+00:00)
Title: --feature_mask_on

Hi John, here is a new implementation of --feature_mask_on. Hope your concerns for our last impl are addressed. In this implementation, we:
(1)use the 4th position in the weight vector to store mask for the default setting. (when either --adaptive or --normalized is used, or --sgd is used, stride is augmented and index for mask is updated accordingly. )
(2)feature_mask_off is moved to GD. It is added to the templated updating and norm computing inline functions.
(3)Creation of a new vw is avoided and existing save/load functions are re-used.

We test the mask idea on our competition dataset and observe some benefits:
the dataset consists of 13065074 examples.

For single pass (we will try more passes soon), the baseline method (without regularization) has an average loss of 0.2512

In the following we have the average loss, 'original' means l1 regularized model. 'retrain' means a model retrained using only features survived from the 'original' one. We want to beat the baseline model.

l1 value | original | retrain
0.0001 bad result as only constant is survived
0.00001 | 0.3006 | 0.279131
0.000001 | 0.264081 | 0.260696
0.0000001 | 0.25207 | 0.250616 (win)
0.00000001 | 0.250973 (win) | 0.250444(win)
0.000000001 | 0.251165 (win) | 0.251076(win) -this model has most features survived

It seems that the retrained model using mask is indeed consistently better than using l1 only. It can also outperform the model without regularization.

I did not see an noticeable delay when using the mask.