Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: - You make the comparison with curriculum learning. Is it something you have observed? Could you/did you design a toy experiment showing that? (If not I think it would be an awesome addition to the paper).
- are "Features to represent the combination of both data and model" computed each time when feeding the state representation to the actor? (which is expensive) Or do you use the previous run's computations? (which would mean you need to go over your whole dataset at least once before using your method) Or I'm missing something.
- What is the margin you refer to? The distance between the decision boundary and the point? That seems an odd choice in high dimensions.
- Did you try this approach on simpler, or toy datasets? (like MNIST, or as I said in my first question, some toy dataset where curriculum learning drastically helps)

Thanks!
review_title: Some questions

final_score: rating: null
review_content: Does authors have other baselines that plain SGD? Dropout, Adam, etc..?
review_title: Are there any other baseline results?

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work proposes to augment normal gradient descent algorithms with a "Data Filter", that acts as a curriculum teacher by selecting which examples the trained target network should see to learn optimally. Such a filter is learned simultaneously to the target network, and trained via Reinforcement Learning algorithms receiving rewards based on the state of training with respect to some pseudo-validation set.


Stylistic comment, please use the more common style of "(Author, year)" rather than "Author (year)" when the Author is *not* referred to or used in the sentence.
E.g. "and its variants such as Adagrad Duchi et al. (2011)" should be "such as Adagrad (Duchi et al., 2011)", and  "proposed in Andrychowicz et al. (2016)," should remain so.

I think the paragraph containing "What we need to do is, after seeing the mini-batch Dt of M training instances, we dynamically determine which instances in Dt are used for training and which are filtered." should be clarified. What is "seeing"? That is, you should mention explicitly that you do the forward-pass first, then compute features from that, and then decide for which examples to perform the backwards pass.


There are a few choices in this work which I do not understand:

Why wait until the end of the episode to update your reinforce policy (algorithm 2), but train your actor critic at each step (algorithm 3)? You say REINFORCE has high variance, which is true, but does not mean it cannot be trained at each step (unless you have some experiments that suggest otherwise, and if so they should be included or mentionned in the paper).

Similarly, why not train REINFORCE with the same reward as your Actor-Critic model? And vice-versa? You claim several times that a limitation of REINFORCE is that you need to wait for the episode to be over, but considering your data is i.i.d., you can make your episode be anything from a single training step, one D_t, to the whole multi-epoch training procedure.


I have a few qualms with the experimental setting:
- is Figure 2 obtained from a single (i.e. one per setup) experiment? From different initial weights? If so, there is no proper way of knowing whether results are chance or not! This is a serious concern for me.
- with most state-of-the-art work using optimization methods such as Adam and RMSProp, is it surprising that they were not experimented with.
- it is not clear what the learning rates are; how fast should the RL part adapt to the SL part? Its not clear that this was experimented with at all.
- the environment, i.e. the target network being trained, is not stationnary at all. It would have been interesting to measure how much the policy changes as a function of time. Figure 3, could both be the result of the policy adapting, or of the policy remaining fixed and the features changing (which could indicate a failure of the policy to adapt).
- in fact it is not really adressed in the paper that the environment is non-stationary, given the current setup, the distribution of features will change as the target network progresses. This has an impact on optimization.
- how is the "pseudo-validation" data, target to the policy, chosen? It should be a subset of the training data. The second paragraph of section 3.2 suggests something of the sort, but then your algorithms suggest that the same data is used to train both the policies and the networks, so I am unsure of which is what.


Overall the idea is novel and interesting, the paper is well written for the most part, but the methodology has some flaws. Clearer explanations and either more justification of the experimental choices or more experiments are needed to make this paper complete. Unless the authors convince me otherwise, I think it would be worth waiting for more experiments and submitting a very strong paper rather than presenting this (potentially powerful!) idea with weak results.

review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: Final review: The writers were very responsive and I agree the reviewer2 that their experimental setup is not wrong after all and increased the score by one.  But I still think there is lack of experiments and the results are not conclusive. As a reader I am interested in two things, either getting a new insight and understanding something better, or learn a method for a better performance. This paper falls in the category two, but fails to prove it with more throughout and rigorous experiments. In summary the paper lacks experiments and results are inconclusive and I do not believe the proposed method would be quite useful and hence not a conference level publication. 

--
The paper proposes to train a policy network along the main network for selecting subset of data during training for achieving faster convergence with less data.

Pros:
It's well written and straightforward to follow
The algorithm has been explained clearly.

Cons:
Section 2 mentions that the validation accuracy is used as one of the feature vectors for training the NDF. This invalidates the experiments, as the training procedure is using some data from the validation set.

Only one dataset has been tested on. Papers such as this one that claim faster convergence rate should be tested on multiple datasets and network architectures to show consistency of results. Especially larger datasets as the proposed methods is going to use less training data at each iteration, it has to be shown in much larger scaler datasets such as Imagenet.

As discussed more in detail in the pre-reviews question, if the paper is claiming faster convergence then it has to compare the learning curves with other baselines such Adam. Plain SGD is very unfair comparison as it is almost never used in practice. And this is regardless of what is the black box optimizer they use. The case could be that Adam alone as black box optimizer works as well or better than Adam as black box + NDF.
review_title: Final Review

final_score: rating: 7: Good paper, accept
review_content: Paper is easy to follow, Idea is pretty clear and makes sense.
Experimental results are hard to judge, it would be nice to have other baselines.
For faster training convergence, the question is how well tuned SGD is, I didn't
see any mentioning of learning rate schedule. Also, it would be important to test
this on other data sets. Success with filtering training data could be task dependent.
review_title: data filtering for faster sgd

Meta-review: The authors propose a meta-learning algorithm which uses an RL agent to selectively filter training examples in order to maximise a validation loss. There was a lot of discussion about proper training/validation/test set practices. The author's setup seems to be correct, but the experiments are quite limited. Pro - interesting idea, very relevant for ICLR. Con - insufficient experiments. This is a cool idea which could be a nice workshop contribution.
Decision: Invite to Workshop Track