Venue: ICLR 2018 conference

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The authors define a deep learning model composed of four components:  a student model, a teacher model, a loss function, and a data set. The student model is a deep learning model (MLP, CNN, and RNN were used in the paper). The teacher model learns via reinforcement learning which items to include in each minibatch of the data set. The student model learns according to a standard stochastic gradient descent technique (Adam for MLP and CNN, Momentum-SGD for RNN), appropriate to the data set (and loss function), but only uses the data items of the minibatch chosen by teacher model. They evaluate that their method can learn to provide learning items in an efficient manner in two situations: (1) the same student model-type on a different part of the same data set, and (2) adapt the teaching model to teach a new model-type for a different data set. In both circumstances, they demonstrate the efficacy of their technique and that it performs better than other reasonable baseline techniques: self-paced learning, no teaching, and a filter created by randomly reordering the data items filtered out from a teaching model.

This is an extremely impressive manuscript and likely to be of great interest to many researchers in the ICLR community. The research itself seems fine, but there are some issues with the discussion of previous work. Most of my comments focuses on this.

The authors write that artificial intelligence has mostly overlooked the role of teaching, but this claim is incorrect. There is a long history of research on teaching in artificial intelligence. Two literatures of note are intelligent tutoring and machine teaching in the computational learnability literature. A good historical hook to intelligent tutoring is Anderson, J. R., Boyle, C. F., & Reiser, B. J. (1985). Intelligent tutoring systems. Science, 228. 456-462. The literature is still healthy today. One offshoot of it has its own society with conferences and a journal devoted to it (The International Artificial intelligence in Education Society: http://iaied.org/about/). 

For the computational learnability literature, complexity analysis for teaching has a subliterature devoted to it (analogous to the learning literature). Here is a hook into that literature: Goldman, S., & Kerns. M. (1995). On the complexity of teaching. Journal of Computer and Systems Sciences, 50(1), 20-31.

One last related literature is pedagogical teaching from computational cognitive science. This one is a more recent development. Here are two articles, one that provides a long and thorough discussion that is a definitive start to the literature, and another that is most relevant to the current paper, on applying pedagogical teaching to inverse reinforcement learning (a talk at NIPS 2016).

Shafto, P., Goodman, N. D., & Griffiths, T. L. (2014). A rational account of pedagogical reasoning: Teaching by, and learning from, examples. Cognitive Psychology, 71, 55-89.

Ho, M. K., Littman, M., MacGlashan, J., Cushman, F., & Austerweil, J. L. (NIPS 2016). 

I hope all of this makes it clear to the authors that it is inappropriate to claim that artificial intelligence has “largely overlooked” or “largely neglected”. 

One other paper of note given that the authors train a MLP is an optimal teaching analysis of a perceptron: (Zhang, Ohannessian, Sen, Alfeld, & Zhu, 2016; NIPS).



review_title: excellent article, though minor revisions to the previous work would strengthen it

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper suggests a "learning to teach" framework. Following a similar intuition as self-paced learning and curriculum learning, the authors suggest to learn a teaching strategy,  corresponding to choices over the data presented to the learner (and potentially other decisions  about the learner, such as the  algorithm used).   The problem is framed as RL problem, where the state space corresponds to learning configurations, and teacher actions change the state.  Supervision is obtained by observing the learner's performance. 

I found it very difficult to understand the evaluation.  
First, there is quite a bit of recent work on learning to teach and curriculum learning.  It would be helpful if there are comparisons to these models, and use similar datasets.  It's not clear if an evaluation on the MNIST data set is particularly meaningful.   The implementation of SPL seems to hurt performance in some cases (slower convergence on the IMDB dataset), can you explain it?  In other text learning task (e.g., [1]) SPL showed improved performance.   The results over the IMDB dataset in the original paper [2] are higher than the ones reported here, using a simple model (BoW). 
Second, in non-convex problems, one can expect curriculum learning approaches to also perform better, not just converge faster.  This aspect is not really discussed.  Finally,  I'm not sure I understand the X axis in Figure 2, the (effective) number of examples is much higher than the size of the dataset. Does it indicate the number of  iterations over the same dataset? 

I would also like to see some analysis of what's actually being learned by the teacher. Some qualitative analysis, or even feature ablation study would be helpful.

[1] Easy Questions First? A Case Study on Curriculum Learning for Question Answering. Sachan et-al.
[2] Learning Word Vectors for Sentiment Analysis. Maas et-al.
review_title: Review for Learning to Teach

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper focuses on the problem of "machine teaching", i.e., how to select a good strategy to select training data points to pass to a machine learning algorithm, for faster learning. The proposed approach leverages reinforcement learning by defining the reward as how fast the learner learns, and use policy gradient to update the teacher parameters. I find the definition of the "state" in this case very interesting. The experimental results seem to show that such a learned teacher strategy makes machine learning algorithms learn faster. 

Overall I think that this paper is decent. The angle the authors took is interesting (essentially replacing one level of the bi-level optimization problem in machine teaching works with a reinforcement learning setup). The problem formulation is mostly reasonable, and the evaluation seems quite convincing. The paper is well-written: I enjoyed the mathematical formulation (Section 3). The authors did a good job of using different experiments (filtration number analysis, and teaching both the same architecture and a different architecture) to intuitively explain what their method actually does. 

At the same time, though, I see several important issues that need to be addressed if this paper is to be accepted. Details below. 

1. As much as I enjoyed reading Section 3, it is very redundant. In some cases it is good to outline a powerful and generic framework (like the authors did here with defining "teaching" in a very broad sense, including selecting good loss functions and hypothesis spaces) and then explain that the current work focuses on one aspect (selecting training data points). However, I do not see it being the case here. In my opinion, selecting good loss functions and hypothesis spaces are much harder problems than data teaching - except maybe when one use a pre-defined set of possible loss functions and select from it. But that is not very interesting (if you can propose new loss functions, that would be way cooler). I also do not see how to define an intuitive set of "states" in that case. Therefore, I think this section should be shortened. I also think that the authors should not discuss the general framework and rather focus on "data teaching", which is the only focus of the current paper. The abstract and introduction should also be modified accordingly to more honestly reflect the current contributions. 
2. The authors should do a better job at explaining the details of the state definition, especially the student model features and the combination of data and current learner model. 
3. There is only one definition of the reward - related to batch number when the accuracy first exceeds a threshold. Is accuracy stable, can it drop back down below the threshold in the next epoch? The accuracy on a held-out test set is not guaranteed to be monotonically increasing, right? Is this a problem in practice (it seems to happen on your curves)? What about other potential reward definitions? And what would they potentially lead to? 
4. Experimental results are averaged over 5 repeated runs - a bit too small in my opinion. 
5. Can the authors show convergence of the teacher parameter \theta? I think it is important to see how fast the teacher model converges, too. 
6. In some of your experiments, every training method converges to the same accuracy after enough training (Fig.2b), while in others, not quite (Fig. 2a and 2c). Why is this the case? Does it mean that you have not run enough iterations for the baseline methods? My intuition is that if the learner algorithm is convex, then ultimately they will all get to the same accuracy level, so the task is just to get there quicker. I understand that since the learner algorithm is an NN, this is not the case - but more explanation is necessary here - does your method also reduces the empirical possibility to get stuck in local minima? 
7. More explanation is needed towards Fig.4c. In this case, using a teacher model trained on a harder task (CIFAR10) leads to much improved student training on a simpler task (MNIST). Why?
8. Although in terms of "effective training data points" the proposed method outperforms the other methods, in terms of time (Fig.5) the difference between it and say, NoTeach, is not that significant (especially at very high desired accuracy). More explanation needed here. 

Read the rebuttal and revision and slightly increased my rating.
review_title: Review of "Learning to Teach"

Meta-review: The paper addresses the problem of learning a teacher model which selects the training samples for the next mini-batch used by the student model. The proposed solution is to learn the teacher model using policy gradient. It is an interesting training setting, and the evaluation demonstrates that the method outperforms the baseline. However, it remains unclear how the method would scale to larger datasets, e.g. ImageNet. I would strongly encourage the authors to extend their evaluation to larger datasets and state-of-the-art models, as well as include better baselines, e.g. from Graves et al.
Decision: Accept (Poster)