Title: Neural Data Filter for Bootstrapping Stochastic Gradient Descent
Authors: Yang Fan
Fei Tian
Tao Qin
Tie-Yan Liu
Venue: ICLR 2017
Keywords: Reinforcement Learning
Deep learning
Optimization

Abstract: Mini-batch based Stochastic Gradient Descent(SGD) has been widely used to train deep neural networks efficiently. In this paper, we design a general framework to automatically and adaptively select training data for SGD. The framework is based on neural networks and we call it \emph{\textbf{N}eural \textbf{D}ata \textbf{F}ilter} (\textbf{NDF}). In Neural Data Filter, the whole training process of the original neural network is monitored and supervised by a deep reinforcement network, which controls whether to filter some data in sequentially arrived mini-batches so as to maximize future accumulative reward (e.g., validation accuracy). The SGD process accompanied with NDF is able to use less data and converge faster while achieving comparable accuracy as the standard SGD trained on the full dataset. Our experiments show that NDF bootstraps SGD training for different neural network models including Multi Layer Perceptron Network and Recurrent Neural Network trained on various types of tasks including image classification and text understanding.

TL;DR: We propose a reinforcement learning based teacher-student framework for filtering training data to boost SGD convergence.

Review discussion:
Does authors have other baselines that plain SGD? Dropout, Adam, etc..?
Thanks for your questions.
In fact, our designed NDF framework can be applied to any of your mentioned optimization algorithms such as adagrad/adadelta/plain sgd… All these algorithms just act as a `black box’ for updating neural network status (weights), which corresponds the evolution of environment in reinforcement learning. Our strategy aims to provide `useful’ input (i.e., data) to these optimizers.
Therefore, these optimizations algorithms are not `baselines’, but are our `customers’.
We can conduct some additional experiments based on other optimization algorithms, which means `one optimization algorithm, one Figure 2’ and that seems somewhat not necessary at status :)
Thank you again for your question!
This paper's main claim is faster convergence, then in fact it is necessary to compare the learning curve to other common in practice methods such as Adam and not just plain SGD which is never used in practice. It could be the case that e.g. Adam optimizer alone without NDF works better or as well as NDF + Adam optimizer inside the "block-box" which makes the input data filtering obsolete.
Dear Reviewer:
We are sorry that you may neglect/misunderstanding our paper as well as our response.
In fact, we are using Adadelta in our experiments and never use Plain SGD. We've stated it very clearly from the first version of the paper. We will formalize our formal response to give a further explanation.
Thanks!
- You make the comparison with curriculum learning. Is it something you have observed? Could you/did you design a toy experiment showing that? (If not I think it would be an awesome addition to the paper).
- are "Features to represent the combination of both data and model" computed each time when feeding the state representation to the actor? (which is expensive) Or do you use the previous run's computations? (which would mean you need to go over your whole dataset at least once before using your method) Or I'm missing something.
- What is the margin you refer to? The distance between the decision boundary and the point? That seems an odd choice in high dimensions.
- Did you try this approach on simpler, or toy datasets? (like MNIST, or as I said in my first question, some toy dataset where curriculum learning drastically helps)
Thanks!
Thanks for your detailed questions!
1)	For margin value, we follow widely used definition (e.g. ref 1) for multi-class classification, that is for a hypothesis h and training instance (x,y),   the margin is defined as h(x,y) -  \max_{y’\neq y} h(x,y). In our scenario, h(x,y) indicates the conditional probability induced by h, i.e., h(x,y)= P_h(y|x).
2)	Once a mini-batch data D arrives, its state features is computed at once. Note here very tiny extra computations are added, compared with standard feedforward/backpropagation process, since the feedforward results in computing state features can be re-used in backpropagation.
3)	Thank you very much for the suggestions of further analyzing the behaviors of curriculum learning.
In fact, we did observe some interesting results on using CL for DNN training, such as some training stability issues. Due to the emphasis of this paper, we have not included this part. We are creating some toy (and maybe corrupted) dataset to further investigate the behaviors of CL.
4)	We have not obtained enough gains on MNIST dataset yet. Our conjecture is that the noise in MNIST may be less compared with text tasks which includes human judgments, and the room for improvement on such tiny dataset may be limited.
However, another reason might be that more training efforts are necessary and the experiments are still undergoing. We will keep updating the manuscript for these extra experiments.
We have improved the manuscript to clarify some of your questions such as margin value definition and feature computation process. Thank you again!
1.	Cortes C, Mohri M, Rostamizadeh A. Multi-Class Classification with Maximum Margin Multiple Kernel[C]//ICML (3). 2013: 46-54.
This work proposes to augment normal gradient descent algorithms with a "Data Filter", that acts as a curriculum teacher by selecting which examples the trained target network should see to learn optimally. Such a filter is learned simultaneously to the target network, and trained via Reinforcement Learning algorithms receiving rewards based on the state of training with respect to some pseudo-validation set.
Stylistic comment, please use the more common style of "(Author, year)" rather than "Author (year)" when the Author is *not* referred to or used in the sentence.
E.g. "and its variants such as Adagrad Duchi et al. (2011)" should be "such as Adagrad (Duchi et al., 2011)", and  "proposed in Andrychowicz et al. (2016)," should remain so.
I think the paragraph containing "What we need to do is, after seeing the mini-batch Dt of M training instances, we dynamically determine which instances in Dt are used for training and which are filtered." should be clarified. What is "seeing"?
That is, you should mention explicitly that you do the forward-pass first, then compute features from that, and then decide for which examples to perform the backwards pass.
There are a few choices in this work which I do not understand:
Why wait until the end of the episode to update your reinforce policy (algorithm 2), but train your actor critic at each step (algorithm 3)? You say REINFORCE has high variance, which is true, but does not mean it cannot be trained at each step (unless you have some experiments that suggest otherwise, and if so they should be included or mentionned in the paper).
Similarly, why not train REINFORCE with the same reward as your Actor-Critic model? And vice-versa? You claim several times that a limitation of REINFORCE is that you need to wait for the episode to be over, but considering your data is i.i.d., you can make your episode be anything from a single training step, one D_t, to the whole multi-epoch training procedure.
I have a few qualms with the experimental setting:
- is Figure 2 obtained from a single (i.e. one per setup) experiment? From different initial weights? If so, there is no proper way of knowing whether results are chance or not! This is a serious concern for me.
- with most state-of-the-art work using optimization methods such as Adam and RMSProp, is it surprising that they were not experimented with.
- it is not clear what the learning rates are; how fast should the RL part adapt to the SL part? Its not clear that this was experimented with at all.
- the environment, i.e. the target network being trained, is not stationnary at all. It would have been interesting to measure how much the policy changes as a function of time. Figure 3, could both be the result of the policy adapting, or of the policy remaining fixed and the features changing (which could indicate a failure of the policy to adapt).
- in fact it is not really adressed in the paper that the environment is non-stationary, given the current setup, the distribution of features will change as the target network progresses. This has an impact on optimization.
- how is the "pseudo-validation" data, target to the policy, chosen? It should be a subset of the training data. The second paragraph of section 3.2 suggests something of the sort, but then your algorithms suggest that the same data is used to train both the policies and the networks, so I am unsure of which is what.
Overall the idea is novel and interesting, the paper is well written for the most part, but the methodology has some flaws. Clearer explanations and either more justification of the experimental choices or more experiments are needed to make this paper complete.
Unless the authors convince me otherwise, I think it would be worth waiting for more experiments and submitting a very strong paper rather than presenting this (potentially powerful!) idea with weak results.
We thank you so much for your professional and valuable comments, which will definitely help to improve the paper.  We’d like to make some clarifications on this work:
1.	We have updated the paper according to your suggestions on citation formatting, and clarified clearly on the first-step forward pass. Please check the latest version. Thanks again!
2.	Actually, from the first submitted version, we have stated that all the curves are obtained from averaged results of five repeated runs. Please refer to the paragraph right below Figure 3. Maybe the formatting (fairly long distance between this statement and Figure 2) makes you unaware of such statement and we feel very sorry.
3.
As to the usage of optimization algorithms Adam and RMSProp, please refer to points 1) of our reply to reviewer 3.  Basically, there are two justifications of using Adadelta: 1). Adadelta is widely used in RNN-related tasks and there is no evidence showing that it is inferior to Adam/RMSProp; 2) We’re able to use Adadelta to obtain even better classification accuracy on IMDB dataset, compared with the numbers reported in previous works.
Of course, we will test Adam and RMSProp in the future work.
4.	For the difference of REINFORCE and Actor-Critic, such as update frequency and reward form, we would like to make the following clarifications: 1) Our goal is to boost the convergence of the whole training process, from the very beginning to convergence.
Therefore, in our MDP formulation (in the beginning of section 2), we define each episode as the whole training process, aiming to taking `future’ reward into consideration in each training step. 2)  According to our knowledge, standard REINFORCE algorithm uses one-time observation (obtained through Monto-Carlo sampling) to approximate true reward, and usually the reward is observed at the end of each episode.
Thus, REINFORCE in most previous work does not update at each single step, given future rewards have not been obtained yet. 3) Given 1) and 2), we have to use different rewards for REINFORCE and Actor-Critic, since the validation accuracy in each step (used in A-C) reflects the convergence speed, and the A-C policy will get updated immediately in each step.
However, for REINFORCE, the final accuracy, and the sum of accumulated immediate accuracy, are roughly the same for different episodes, which means pure validation accuracy numbers cannot reflect the quality of different policies.   This is exactly the motivation of our designed rewards for REINFORCE, given that it can reflect convergence speed.
5.	We use Adam to optimize the policy. The missing statement has been added in the updated version.
6.	Sorry we are not quite clear about what you mean by `policy adapting’. As we stated in the paper, there are several steps in each of our experiments: in the first step, we trained both the policy network and the RNN classification network; in the second step, we fixed the policy network and trained a RNN classification network using the policy network;
finally, we tested the learnt RNN network on the test set.
7.	For the validation dataset selection, again we thank you very much for pointing out the `four-disjoint’ dataset separation principles. Here are several of our points:
a.	Yes, the validation set is usually used for hyper-parameter search, but not limited to hyper-parameter search.
For example, in neural network training, people frequently use validation accuracy to perform learning rate scheduling, e.g, halving the learning rate when validation accuracy drops. In such a scenario, as far as we know, there is no `extra’ validation set used.
In other words, validation set is used for both TRAINING CONTROL and hyperparameter search.
In fact, in this scenario, all hyper-parameter configurations are still treated equally – they are both under the same training protocols (e.g., halving learning rate when validation accuracy is dropped) and the chosen of the best hyper-parameter is fair.
b.	We in fact follows previous hyper-parameter configuration suggested in [2] as above, such as RNN model size, learning rate scheduling, to make sure the public classification accuracy is obtained.
That is to say, hyper-parameter configuration is fixed in our experiment.
c. For the paper from Google [3], there are several statements such as “Training the network – the ”child network” – on the real data will result in an accuracy on a validation set. Using this accuracy as the reward signal, we can compute the policy gradient to update the controller”, “At convergence, this child network will achieve an accuracy R on a held-out dataset. We can use this accuracy R as the reward signal and use reinforcement learning to train the controller. ”, “On each dataset, we have a separate held-out validation dataset to compute the reward signal.
” and “The validation set has 5,000 examples randomly sampled from the training set, the remaining 45,000 examples are used for training”. Once you dig into details, you can find that from these four statements, they do NOT distinguish the term ‘validation’ and ‘held-out validation’ – in fact, the reward they used in REINFORCE are from such a (held-out) validation dataset with size 5,000, i. e. , [3] and our paper use the validation set in the same way.
Overall thanks again for your detailed suggestions. We are constantly improving the paper and hope the new version and our clarifications can address your concerns.
Hi, thank you for your comments,
I have read the paper again and updated my review score.
Overall I think it is improved, and I still think this is an cool new idea, but I think this work would benefit from a more thorough set of experiments, especially since it has the potential to be a general algorithm, independent of a particular task.
For REINFORCE, you might be missing out on the fact that your data is I.I.D., and as such you can pretty much define an episode arbitrarily, so you don't need to wait to go through your whole dataset. That way you could train your policy much faster.
I still think that the validation setup is somewhat weird; even Google publications can have some faults.
Final review: The writers were very responsive and I agree the reviewer2 that their experimental setup is not wrong after all and increased the score by one.
But I still think there is lack of experiments and the results are not conclusive. As a reader I am interested in two things, either getting a new insight and understanding something better, or learn a method for a better performance. This paper falls in the category two, but fails to prove it with more throughout and rigorous experiments.
In summary the paper lacks experiments and results are inconclusive and I do not believe the proposed method would be quite useful and hence not a conference level publication.
The paper proposes to train a policy network along the main network for selecting subset of data during training for achieving faster convergence with less data.
Pros:
It's well written and straightforward to follow
The algorithm has been explained clearly.
Cons:
Section 2 mentions that the validation accuracy is used as one of the feature vectors for training the NDF. This invalidates the experiments, as the training procedure is using some data from the validation set.
Only one dataset has been tested on. Papers such as this one that claim faster convergence rate should be tested on multiple datasets and network architectures to show consistency of results. Especially larger datasets as the proposed methods is going to use less training data at each iteration, it has to be shown in much larger scaler datasets such as Imagenet.
As discussed more in detail in the pre-reviews question, if the paper is claiming faster convergence then it has to compare the learning curves with other baselines such Adam. Plain SGD is very unfair comparison as it is almost never used in practice. And this is regardless of what is the black box optimizer they use. The case could be that Adam alone as black box optimizer works as well or better than Adam as black box + NDF.
Thanks for your feedbacks. We are afraid you made several misunderstandings about our work.
1) We used Adadelta as the optimizer, rather than plain SGD as you stated. This has been clearly stated from the first version of our submission.
In fact, Adadelta is commonly used in RNN-related tasks, such as the RNN-Search Neural Machine Translation model [1].
In addition, using Adadelta, we can reproduce, or get even better results than previous reported classification accuracy on using LSTM for IMDB dataset (i.e., >88.0%, check Table 2 of [2], and our Figure 2.).
Therefore, it is sufficient to leverage Adadelta as the black-box optimizer for this particular task.
2) We do not agree with you on your doubt of using validation accuracy as features for training.
In fact, as long as the test data is unseen, one may design multiple ways of using validation dataset to control training process,  such as early stopping training, tuning learning rate, etc.  From that sense,  we in fact act in the similar way: validation accuracy is simply used as one of the features for controlling training.
In fact, you may observe other works that leverage validation accuracy as the reward for RL to better design training strategies, such as [3],  also submitted to ICLR 2017.
We agree with you that other tasks are necessary to better demonstrate the effectiveness of our algorithm.
However, some misleading points may lead to your unfair overall judgement of this work.
[1] Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate[J]. arXiv preprint arXiv:1409.0473, 2014.
[2] Dai A M, Le Q V. Semi-supervised sequence learning[C]//Advances in Neural Information Processing Systems. 2015: 3079-3087.
[3] Zoph, Barret, and Quoc V. Le. "Neural architecture search with reinforcement learning." arXiv preprint arXiv:1611.01578 (2016).
Regarding point 2,
I'm sure you know this, but I just wanted to make sure we're using the same words:
In general, and to respect correct training procedure, one uses three disjoint sets: a training, validation and test set. It is commonly accepted that the 'validation' set should *only* be used for hyperparameter search, and that the 'test' set should **really only** be used to test the model that performed best on the validation set.
Therefore, if the training set is intermixed with the validation set (via it being a feature), then the validation set cannot be used to accurately represent accuracy on 'unseen' data, because via that feature its information will be leaked and it will have been 'seen' by the training model.
What I suspect that you are actually doing, but fail to mention properly, is that you in fact have _FOUR_ disjoint sets. The training set, the 'train-valid' set (which is used to extract 'validation-accuracy' features), the validation and test set.
If you are not doing that then your experimental setup is wrong. If you are doing that, it is not apparent enough or clearly explained in your paper.
edit: for example in [3], "At convergence, the accuracy of the network on a held-out validation set is recorded", they do have a fourth set to do in-training validation and out-of-training normal validation (they call that set the held-out validation set).
1. I am referring to the Figure 2 which you are only comparing with plain SGD. Adadelta/Adam/.. without any data filtering is necessary for comparison.
2. Thank you for the pointers regarding the use of validation set accuracy as a reward signal in other works. It is still a strange setup, and I agree with the comment from AnonReviewer1 below. Proper experimental setup would be to choose parameters using a held-out set that training set have no access in anyway.
Oh We are quite sorry for using the term `plain SGD' in the figure, that leads to your misunderstandings. The `plain' here means there is no data filtration used, rather than the `straightforward' SGD.
In fact it is the Adadelta optimizer without any data filtration.
We are updating the paper to remove such a concern.
Dear Reviewer:
Please check our response 7) to AnonReviewer1's review, on validation set up. Thanks!
Best,
Fei
Dear Reviewer,
Please check the new paper version. Basically:
1) Using validation set for reward computation is justifiable and common in the literature. Please refer to our (detailed) response point 7 to the first reviewer.
2) We have removed the term `plain SGD' in the paper to avoid misunderstandings.
In fact we are using Adadelta + No data filtration as baseline.
3) We have added a new experiment on mnist.
Hope all these can relieve your concerns listed in your original reviews and provide a new basis of your view towards our work.
Thank you again for the comments.
Best,
Fei
Paper is easy to follow, Idea is pretty clear and makes sense.
Experimental results are hard to judge, it would be nice to have other baselines.
For faster training convergence, the question is how well tuned SGD is, I didn't
see any mentioning of learning rate schedule.
Also, it would be important to test
this on other data sets. Success with filtering training data could be task dependent.
Dear Reviewer,
Thank you very much for your positive comments and scores!  As stated before, we are using Adadelta as the basic optimizer, rather than simple SGD. Very sorry for the misunderstanding brought by the term `Plain SGD’ in the paper. It just means `Adadelta without any data filtration'.
In addition we are working on new datasets and conducting more qualitative analysis.
Dear All Reviewers,
We have updated a new version, which: 1) adds a new experiment on mnist dataset to further verify the effectiveness of our proposed NDF algorithm; 2) makes a more dedicated and clearer description towards baseline method, REINFORCE v.s. Actor-Critic and validation setup.
We hope the new version can remove your previous concerns towards this work, even it is approaching the end of rebuttal period (we feel it sorry to submit it late).
Thanks again for all of your valuable comments and suggestions to this work!
Best,
Fei

Meta-review:
role: PC
data: comment: The authors propose a meta-learning algorithm which uses an RL agent to selectively filter training examples in order to maximise a validation loss. There was a lot of discussion about proper training/validation/test set practices. The author's setup seems to be correct, but the experiments are quite limited. Pro - interesting idea, very relevant for ICLR. Con - insufficient experiments. This is a cool idea which could be a nice workshop contribution.

Decision: Invite to Workshop Track