Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: *Revision after author response*

I thank the authors for the comments on my questions. 

Unfortunately, I do not feel that these comments addressed my main concerns. For all my experimental analysis questions, the authors promised some analyses for future versions, but I was hoping to see at least a minor preliminary analysis at this point, to see if indeed my concerns are valid or not. 

Moreover, for my question number 1 about the optimization problem, the authors referred me to Corollary 1 from the paper, but that didn't really help me because, as the other reviewers also point out, the writing is quite hard to follow.

Because of all these, I have decided to revise my score to a weak reject. While I believe the paper has merit, it requires revisions at many points in order for a reader to truly understand the method and trust the experimental results.

--------------------------------------------------------------------------------------------------------------
The paper proposes a curriculum learning approach that relies on a new metric, the dynamic instance hardness (DIH). DIH is used to measure the difficulty of each sample while training (in an online fashion), and to decide which samples to train on next. The authors provide extensive experiments on 11 datasets as well as some theoretical motivation for the use of this approach.

---- Overall opinion ----
Overall I believe this paper is an interesting take on curriculum learning that is able to achieve good results. I believe this approach is a combination of core ideas from multiple sources, such as boosting, self-paced learning, continual learning and other curriculum learning approaches, but overall it seems different enough from each one of them individually. Because of the resemblance with these many different methods, the method itself does not surprise through the novelty of a new idea, but the authors seemed to have found something that was missing from these methods and that leads to very good results. The experimental results look great, but I believe the paper is missing some ablation studies to assess the importance of certain components (see details below). I also had some trouble understanding certain arguments, which I hope the authors can clarify. 

---- Major issues ----
1. I find the arguments section 2.1 quite difficult to follow. In particular, under the assumption stated in the paper that r_t(i) = f(i|S_{1:t−1}) =  f(e_i + S_{1:t−1}) − f(S_{1:t−1}) , why does it follow that r_t(i) can be used instead of f in the minimization problem (2). 

2. Based on the method itself, it seems to me that the parameter k_t could would have a lot of influence on how well the method doing.  The authors mention in the experimental section what values they use, but there is no indication on how one would choose this value. Moreover, it would be good to see an analysis of how sensitive the results are to this choice.

3. In Figure 1, it is not clear whether the figure on the right shows the actual loss, or the smooth loss using Equation (1) with instantaneous instance (A). If it is the former, then if the loss is so smooth, why do we need DIH? If it is the latter, then what does the instantaneous loss look like? This actually raises the question of how important the smoothing component is -- could we achieve the same results with an instantaneous loss (i.e. set gamma to 1 in Eq. 1)?

---- Minor issues ----
1. How do you choose T0, gamma and gamma_k?

2. In the conclusions, the authors state that “ The reason [why  MCL and SPL are less stable] is that, compared to the methods that use DIH, both MCL and SPL deploy instantaneous instance hardness (i.e., current loss) as the score to select sample”. Since there are so many other differences in the way training progresses, I think we don’t have enough evidence to attribute this to merely the “instantaneousness” of the loss. In fact, it would be interesting to see how SPL does if you use DIH as a metric (just smoothing the loss over time), but their approach of scheduling samples (easy to hard, and not the opposite and in DIHCL).

3. Appendix C shows some interesting results regarding wall time comparison. I was surprised to see that, despite the extra computations, DHCL is comparable to random mini-batches. This makes me wonder what the stop criteria was, because when you stop matters a lot for run time comparisons. It would also be interesting to see a more ample discussion on this in the main text.

4. In Figure 1, the axes are barely readable.

5. The authors oftentimes reverse the use of \citet and \citep, for example “has been called the “instance hardness” Smith et al. (2014) corresponding to” should have a bracket, whereas “Our paper is also related to (Zhang et al., 2017)” should not have brackets.

6. This is not an issue, but I just wanted to say I appreciated Appendix B.

---- Suggestions ----
1. It would be interesting to make a connection between the DIH and what other papers have discovered about example forgetting (e.g. Toneva et. al, that was mentioned in the paper).

2. Major issues 3 -> a study on the effect of k and how to choose it.

3. While I understand that the models chosen in the experiments are expensive to train, it would be good to report standard deviations in Table 1.

4. Based on Table 1 and Figure 3, there is no concrete winner among the DIHCL methods. It would be good to include some recommendations in your conclusion on which one to choose and when.

---- Questions ----
1. “On average, the dynamics on the hard samples is more consistent with the learning rate schedule, which implies that doing well on these samples can only be achieved at a shared sharp local minima.” -> can you please explain why this is so?

2. See Major issues 3.

3. In Table 1, on some datasets, the authors apply lazier-than-lazy-greedy, and on some not.Why, and how does one decide this for a new dataset?

4. How did you choose T0, gamma and gamma_k, as well as the schedules in Appendix C (page 17)? 
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper presents a technique for curriculum learning using Dynamic Instance Hardness (DIH). DIH is defined as value for each training instance that characterizes how hard that instance is and is updated throughout the training process. The DIH value is used to select the best set of instances to learn. It is shown that instances with high  (low)  DIH values maintain the high (low) value throughout the training process.

The main contributions and pros of the paper are
1. A notion of instance hardness that is persisted and updated throughout the training procedure.
2. An objective function for characterizing instance hardness as a dynamic subset selection problem. Presenting a greedy algorithm for online maximization of this function.
3. Experimental results that show how the property of instance hardness is maintained throughout training and showing how DIH-driven curriculum learning techniques that use random sampling outperforms non-curriculum learning techniques.

Cons
1. The writing of this paper is difficult and in many of the core parts of the paper, the important definitions are not clear. E..g a) the role of the function 'f' that is being maximized in section 3.2 is not clear. To elaborate, it is not obvious what this function is or why should one care about it? 
2. The greedy algorithm has a bound in equation 7 that appears to be quite loose as the value of k is as high as the order of the size of the entire training set (in the experiments, 0.2n <= k < n). Am I misinterpreting it?
Furthermore, the greedy algorithm -- while being focused on the core of the technical contributions -- is not used in experimental comparisons and instead all the results presented use random sampling. At least the result of DIH-greedy should be presented.

3. This is more of a suggestion: one of the claimed advantages of the algorithm (over non curriculum learning and MCL) is that it requires less training examples to train. Given this, the authors should present training time improvements over large datasets.
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: This paper studies the curriculum learning approach that can more effectively utilize the data to train DNNs. It formulates DIH as a curriculum learning problem, and derives theory on the approximation bound. The method is verified by a set of experiments. There are  several concerns raised by the reviewer.

I found the presentation of this paper is rather bad. The structure of the paper is quite strange. The Introduction section contains a lot of stuffs that I believe should be moved to the preliminary or method sections. 

Also, there are a lot of confusions in the descriptions. For example, when defining the curriculum learning problem in eq.2 and eq.3, are the f's the same? If so, why do they have different input arguments?

"In step t, after the model gets trained on S_t, the feedback a_t(i) for i \in S_t is already available": I don't get this.

I am not sure what Theorem 1 tries to tell. If one chooses k large enough, the inequality satisfies trivially. BTW, what is A_{1:T}?

To sump up, there are some interesting ideas in this paper. However, with the current stage of writing, I cannot recommend acceptance.


review_title: Official Blind Review #1

Meta-review: All three reviewers, even after the rebuttal, agreed that the paper did not meet with bar for acceptance. A common complaint was lack of clarity being a major problem. Unfortunately, the paper cannot be accepted in its current form. The authors are encouraged to improve the presentation of their approach  and resubmit to a new venue.  
Decision: Reject