Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new parameterized data partitioning and weighing scheme, that partitions data into three groups {easy, medium, hard} and determines a curriculum based on relative importance of different samples. They evaluate on three datasets (full and balanced versions) and show improvements over other CL approaches. The curriculum also provides interesting insights about the datasets and scale from smaller datasets to larger datasets. 
main_review: #### Strengths
- Proposed curriculum method encompasses other well known curricula by parameterizing the data partition and weighing schemes.
- Data partitioning is effective, on the three datasets the authors evaluate upon, as is evident from improvements over no curriculum approach.
- Specialized curricula obtained using curriculum discovery improves over no curriculum and other state of the art approaches.
- Sample weight patterns shows the relative importance of samples on different datasets, which is interesting.
- One quite interesting advantage of this approach is that curriculum discovered for smaller, balanced datasets work well on larger datasets.

#### Weaknesses
- The proposed method partitions a dataset into three classes $\textit{easy}$, $\textit{medium}$ and $\textit{hard}$. Will this partitioning scheme work for different types of datasets, where there might be multiple levels of difficulty, multiple sub tasks? How will this method scale? 
- Using human agreement as a measure of entropy might not be applicable to datasets where there is less ambiguity between samples. That will cause majority of samples to be in a particular partition. For those datasets using entropy as a measure to partition samples might not be optimal. 
-  In Section 2.3, the authors describe how the proposed framework encompasses other well known CL approaches. It will be interesting to add more analysis on how often the TPE algorithm(Section 2.4) discovers curriculums where the parameters fall into pruning or sub-sampling strategy space. 
- How does this approach scale to larger datasets in vision like Imagenet, CIFAR etc, that other SOTA methods like SuperLoss evaluate on? Authors evaluate on three datasets, not all of which are well known to the community. I will encourage the authors to also evaluate this approach on more popular datasets.
- How significant are the improvements over other baselines? Showing average over full and balanced datasets(Fig 4) hides some important issues. It seems from Figure 10, that DP method outperforms the proposed CL method on full datasets. We should pay more interest to this number as we would want any approach to work well on full dataset, rather than a subset(balanced) dataset. 
- Hard examples are down weighted more aggressively in this approach. It can be seen that the proposed approach often lags behind other approaches like No-CL, MentorNet, DP when it comes to hard examples. This issue is more prominent when it comes to full datasets. Authors should provide some additional analysis on these hard samples from these datasets for more clarity. What % of these datasets are partitioned into $\textit{hard}$ class?
summary_of_the_review: I like the idea of encompassing different curriculum approaches inside a parameterized function. The authors show improvements over other approaches on three datasets with curriculum discovered by this approach. My reservations are mostly around how well this approach will scale to larger datasets and on unbalanced datasets. Some results show that other methods are better when it comes to full datasets. Due to these reasons, I will am recommending the current score. I encourage authors to address these concerns and I will be happy to bump up my score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes to learn training curricula by considering 3 sigmoids (representing "easy", "mid" and "hard" difficulty levels), that would define instances' weight as a function of time. The sigmoids' parameters are fitted to 3-bucket values of entropy of multi-annotator labels (that need to pre-exist), or of the instances' loss value dynamics (as provided by a baseline, no-curriculum, model that also needs to be trained). The paper emphasizes that the approach can replicate some of existing curriculum heuristics (e.g. easy-to-hard, hard-to-easy etc.), which however so far failed to produce a general recipe of training deep models, -- this diminishes the value of this flexibility. Finally, the approach is illustrated on 3 small to mid-size datasets with largest gaps to no-curriculum achieved on reduced datasets to re-balance classes.


main_review: In general, the approach appears to be rather constrained (only 3 difficulty levels; limited to sigmoids; reliance in part on multi-annotator labels), may be onerous to implement (pre-training a baseline; fitting the sigmoids; making manual decisions on entropy vs loss) and was validated on rather small data (balanced datasets with <10k samples) to be useful in practice.

Strengths:
1. Addresses a hard unsolved problem.
2. Modeling simplicity of the curriculum functional class (simple sigmoids that would cover most of the human-hypothesized heuristics; however, it may not be what we need - see below).

Weaknesses:
1. I find the approach conceptually different to classical curriculum learning, since it reweighs instances in the loss rather than schedules them in a particular order (except for the zero-weight case, equivalent to example skipping, where both ways converge); as far as could tell, SGD still samples data uniformly at any time. Besides, since the loss function changes, the baseline and the weighted tasks optimize different objectives and it's not straight-forward to compare them.
2. Regarding sigmoids, I'm not convinced that practical cases of ML curricula can be covered by such simple monotonic schedules; rather, the functional class should be capable of dynamically changing the scheduling weight depending on the learning stage to mitigate forgetting and revisiting learned difficulty levels (and this encompasses monotonic functions too). In fact, such bandit-based curricula increasing/decreasing importance dynamically have been already proposed (Graves et al, 2017, https://arxiv.org/abs/1704.03003) and even evaluated for NLP tasks beyond classification (Kreutzer et al, 2021, https://arxiv.org/abs/2110.06997). The draft could be completed with a discussion of why the approach of Graves et al. is not considered or, better, a comparison experiment could be added. Note that the bandit approach is free from limitations such as number of difficulty levels.
3. The requirement to pre-train the baseline prior to curriculum-enabled training defeats one of the main purposes of curriculum learning -- saving resources in the large data regime and relieving developer from manual training design. Again, dynamic curricula that are trained on the fly would be a more practical approach, as they don't require pre-training.
4. The entropy-based curriculum relies heavy on assumed human-perceived difficulty, which may not be the same for the network's point of view. The fact, that Zhang et al. (2018) and Kocmi & Bojar 2017 both found that reverse curriculum works similarly well, witnesses that human intuitions of what is difficult for the network may be wrong.
5. It looks like data balancing increases the gap to the no-curriculum approach. This again changes the task (in addition to different objectives) and raises the question of practical relevance for tasks where such balancing is not desired or not possible (seq2seq).

Minor:
- i found Fig. 1 redundant, it is just illustrating what is sigmoid function
- fuzzy wording in a few places ("good metric", "fairly distributed", "significant variance")
- change \citet to \citep in some places, Richards (1959), Shannon (2001)
- sec 3.2: "configurations is" -> "configuration is"
- "in Nie et al (2020), which" -> "by Nie et al (2020), who"
summary_of_the_review: In my opinion, the proposed approach is too limited to be of practical usefulness, and its weaknesses outweigh the strengths so that the current draft is below the ICLR bar.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a curriculum learning approach applied to NLP models. Texts are separated into easy, medium and hard subsets based on a difficulty score that is given a standard NLP model. Hyperparameter tuning is used to determining evolving weights for each batch. The approach is evaluated on three benchmark datasets (SNLI, Alcohol, Cancer).
main_review: Strengths:
+ The authors conducted a braod set of experiments.
+ The paper is easy to follow.

Weaknesses:
- The main contribution of this work is to add changing weights to easy, medium, and hard training subsets, which an incremental development in my opinion. 

- At the beginning of section 2.3, the authors should acknowledge curriculum methods that do not maipulate data samples, such the "model-level" curriculum approaches surveyed in [A].

- The approach divides the training data into 3 parts {easy, medium, hard}, based on a difficulty score. Ablation results with a different number of parts and other difficulty scores should be presented. I think it is particulary interesting to see what happens if th examples are randomly partitioned, i.e. does the difficulty play an important role?

- The introduction leaves the impression that weights for data shards are somehow predicted by the model at each iteration. However, in section 2.4, the reader learns that "curriculum discovery" is based on hyperparameter tuning. The claims from the introduction should be toned down.

- The performance gains shown in Figure 4 are rather small (wintin the standard deviations) compared to competing curriculum methods. The same applies to the supplementary.

Formatting and language mistakes:
- "Without a curriculum learning will be an intractable" => "Without a curriculum, learning will be an intractable";
- "Nie et al. (2020), which also studies" => " Nie et al. (2020), who also study";
- many citations are wrongly formatted, e.g. "logistic functions Richards (1959)" should be "logistic functions (Richards, 1959)".

References:
[A] Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A survey. arXiv preprint arXiv:2101.10382, 2021.
summary_of_the_review: In my opinion, the weaknesses outweigh the strengths of this paper.
review_title: null

Meta-review: The paper proposes a new curriculum learning framework by parameterizing data partitioning and weighting schemes. Extensive experiments are performed on three different datasets to demonstrate the effectiveness of the proposed framework. The reviewers acknowledged that the proposed framework is interesting as it encompasses several existing curriculum learning methods. However, the reviewers pointed out several weaknesses in the paper and shared concerns, including the scalability of the framework to larger datasets and the significance of the improvements over baselines. I want to thank the authors for their detailed responses. Based on the reviewers’ concerns and follow-up discussions, there was a consensus that the work is not ready for publication. The reviewers have provided detailed feedback to the authors. We hope that the authors can incorporate this feedback when preparing future revisions of the paper.
Decision: Reject