Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary: The paper conducts a large-scale evaluation of the impact of curriculum learning (CL) in image classification. The paper progresses nicely through a sequence of well-thought research questions and experiments, with the key findings stated up front. In particular, the notion of "implicit curriculum" is shown to exist. Prior findings around when CL is helpful (limited training, label noise) are confirmed, which is nice. Overall, this methodical empirical evaluation comes away with a clear set of takeaways, empirically "summarizing" a lot of prior work on CL and the training of deep models. Some discussion about why CL helps when training is limited or data has label noise (or next steps) would strengthen the paper a bit more.

Strengths:
  + Extremely well-written and easy to read. Key findings are summarized and visualized up front.
  + Well-designed large-scale empirical investigation into important open questions for training image classifiers.

Areas for improvement
  - Can't think of too many. I suppose the paper could have included a bit more discussion into why the reduced training or label noise benefits from curriculum learning. I'm also curious to see how these findings compare with a similar study on sequence (especially text) data but as the paper mentions, it's outside the scope of this paper.
  - A pointer up front directing the reader to the appendix where "standard training" is defined would have been nice to have.

Questions:
  * On Page 5, I didn't follow the sentence "Given **these pacing functions**, we can now ask if the explicit curricula enforced by them can change the order in which examples are learned.". I agree that Fig 3-right shows that the difficulty ordering (e2d, rand, d2e) can change the order in which examples are learned when using the step pacing function. What other pacing functions are used in Fig 3-right? Is there a different interpretation of the sentence involving the pacing functions?

Minor comments
  - A few typos
    - Fig 2's caption ("ReseNet50", "EfficeientNet")
    - Page 6 ("CIAFR10")


UPDATE: I thank the authors for their detailed response and updated paper. I'm now more inclined to accept.
review_title: Detailed, methodical, large-scale empirical evaluation of the impact of curriculum learning for image classification

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: ##########################################################################
Summary:
 
The paper provides a comprehensive analysis of the benefits of curriculum learning in different application scenarios. This includes investigating the phenomenon of implicit curricula, showing if the examples are learned in a consistent order across different architectures, and exploring the influences of explicit curricula in the standard and emulation settings. The paper empirically shows that curriculum learning has marginal benefits for standard training, but is helpful when the training time is limited or the training data is noisy.

##########################################################################

Reasons for score: 

I vote for accepting the paper. I believe the analyses presented in the paper can be valuable for the community. I like the implicit curricula experiments, showing that the difficulty of an example is somewhat independent of the training method. Other empirical observations are also interesting. In general, I think the paper provides a satisfactory answer to the question raised in the paper title (when do curricula work?) 

##########################################################################

Pros: 

This paper has extremely comprehensive evaluations, examining the influence of curriculum learning (curriculum/anti-curriculum/random-curriculum) in diverse settings (standard/limited training time budget/noisy data). The methodology for the evaluations is technically sound;

The findings presented in the paper can be valuable for the community: (1) the difficulty of an example is somewhat independent of the training method; (2) curriculum learning provides little benefit for standard training, but help for limited time and noisy training;

The paper is well written. It is a thoroughly enjoyable experience to read the paper.

##########################################################################

Cons:

I found few weaknesses in the paper. I include a question below which I hope could be clarified:

The learned iteration of a sample is defined by the first epoch from which the prediction remains correct for all subsequent epochs. I wonder if there is any sample that is predicted correctly in earlier steps but incorrectly in later steps (e.g.  the forgettable examples). How to handle them in the implicit curricula experiment?



##########################################################################

Minor comments: 

Page 16: two data loader → two data loaders

#########################################################################

Final recommendation:

I have read the authors' responses as well as the comments from my fellow reviewers. I would like to keep my rating of the paper (8).

review_title: A nice paper

final_score: rating: 7: Good paper, accept
review_content: This paper provides a comprehensive empirical study on the effects of the ordering of training samples in curriculum learning. The authors designed extensive experiments and obtained some interesting results: 1. the models with a similar architecture learn samples in a consistent order; 2. with enough iterations curriculum learning has no gain on performance comparing with random ordering; 3. curriculum learning outperforms others when the training time is limited; 4. curriculum learning is more robust with noisy samples. 
In general, I think this is a quite practical work that could be beneficial to the community. The experiments are carefully designed and the results are sound, and the paper is well structured. 

There are just some minor issues that may need some clarification:
1. As the authors mentioned in Sec.3, the random ordering is not as the same as i.i.d. training because it corresponds to dynamic training size. Isn't it more similar with bootstrapping? It would be interesting if the authors can have some discussion in this point of view. 
2. The parameters of pacing function a, b should be introduced with the pacing function in the beginning of Sec.3 , as they were mentioned before Sec. 3.2 without explanation. 
3. Subfigures in Fig.5 should have subcaptions.
4. In the experiments with noisy labels,  the best pacing functions ignore all noisy data, does it mean their gain of performance is simply from filtering out the noisy samples? Is there any other influence from the formulation of the pacing function itself? 
5. In the paragraph under Fig.8, the last sentence 'a trend to start and maintain ...' is a bit confusing to understand, could authors clarify it a bit?

review_title: A comprehensive empirical study with interesting results

Meta-review: This nice paper gives a better understanding of how Curriculum Learning (CL) affects image classification. In particular, it gives insight into cases such as noisy training data and limited training time. It shows that examples can be rated by difficulty to some extent, in that the order in which examples are learned seems to be consistent across runs. The paper is thorough and well-written.
Decision: Accept (Oral)