Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper is motivated by the gap in our understanding of the importance of curriculum learning in biological models vs machine learning models. The main question the paper tries to address is, when and why curriculum learning helps. 
They consider a teacher, student setup where both teacher and student are shallow 1-layer neural networks of size N and the teacher network is sparse, with only a fraction ρ < 1 of ∼ N (0, 1) non-zero components. The student model has to guess which components should be set to zero and align the relevant weights in the correct direction.
The findings of the paper are as follows:
* Curriculum learning is most effective when a small signal is embedded amidst many irrelevant feature.
* The effect of curriculum or anti-curriculum depends on other learning parameters such as initialisation (anti-curriculum can lead to better results of hyper-parameters are not optimal).
* Curriculum mainly offers a dynamical advantage: it speeds learning, with only minimal impact on asymptotic performance.
* Curriculum-aware algorithms can better exploit curricula.
strengths_and_weaknesses: The topic of the paper is very interesting and inspiring: linking psycological experiments and theories of how curriculum learning works/helps humans/animals to our understanding of effects and benefits of curriculum learning in machine learning. 

The main issue I think is that the emprical experiments are too simple and far away from the state of the art ML techniques. I wonder how, conceptually, the findings could be replicated in scaled up settings.
questions: - One way to explain why it seems that explicit curriculum learning doesn't help in many cases in ML, is that the models could potentially be impacted by some imiplict curriculum, e.g., curriculum  imposed by the architecture of the model or the optimization algorithm. What do you think about this, and how does it fit to the dicussions in this paper?

- What does "curriculum-aware algorithms" mean?
limitations: I don't have any comments on this.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper explores the mechanism of curriculum learning, specifically when will, or will not, the curriculum setting work. The paper investigates two kinds of demos, online learning dynamics, and batch learning, as well as proposes an elastic penalty regularization, which demonstrates that the curriculum setting could only be beneficial when the task is sparse. The paper also validates the observation on CIFAR10, which further illustrates the investigation over real datasets.
strengths_and_weaknesses: Strenght:
1. The paper is well organized, and clearly presented, in which it goes from easy to deep to introduce what curriculum learning is, its limitations, and investigate when curriculum settings will benefit or not.
2. The paper investigates the relations between curriculum learning and sparsity, and demonstrates that the effectiveness of curriculum learning heavily depends on the sparsity.

Weaknesses:
1. Though the paper introduces the experiments over CIFAR10, the evaluation over the real dataset is still somehow limited, it could be better to have more evaluations of curriculum learning on other real data with non-convex settings, or at least make the explanation of the evaluation on CIFAR10 more detailed.
2. It can be better (though not necessarily to have) if some methodologies could be proposed to improve the performance of curriculum learning under non-sparse cases, which could make it more practical.
questions: 1. In Figure 2, the paper presents that under some circumstances, anti-curriculum learning could perform better than curriculum or non-curriculum learning (b,c bottom left part), which is anti-intuitive, can the paper give some explanations of such phenomenon?
2. The paper reveals that with too less easy data ($\alpha_1$), the performance of curriculum learning can be degraded, can it be related to overfitting?


limitations: See Sec 'Strengths And Weaknesses'

In addition, the Appendix is not well related to the main text, please use markers in the main text to direct the appendix list.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work introduces an analytical framework that eases the study of the effect of data curricula in teach-student networks. The authors present a set of formulas that can produce the final accuracy of a model based on the initial parameters of the student and teacher networks without the need to perform the simulation (training). The analytical solution is an iterative method that is applied to the online learning case (optimize on one sample at a time, and each sample is used only once) and batch learning case (train until convergence). They evaluate their methods and study the benefits of curricula using a synthetic dataset of Gaussian samples whose label is determined by a sparse teacher network and difficulty is determined by the variance in irrelevant samples. Additionally, they evaluate on a custom-designed task using CIFAR-10 data, where an irrelevant image is concatenated to the target image, and difficulty is based on the opacity of the irrelevant image. The study the effect of curricula based on model initialization scale, teacher network sparsity level, and variance amount in irrelevant features (SNR).
strengths_and_weaknesses: **Strengths**

- An analytical theorem, with a rigorous proof, which can be of great use to the study of curricula and learning theory.
- The connection of machine learning results with experiments of psychology is interesting, and convincing, and attests to the soundness of the author's methods. 
- Well-written introduction about curriculum learning with references to biological, psychological and artificial learning backgrounds.
- Investigates an important problem and presents a solid problem definition, particularly in the batch learning section.

**Weaknesses**

- The benefits of the analytical solution are states as a) "free of finite size effects and stochastic fluctuation", and b) "their evaluation is very fast". Both of them require further elaboration. I would like to see a comparison of the speed of the analytical solution and full training, and a break-down of how the analytical solution is faster. In the supplementary material I see that the analytical forms of the updates are computed using matrix multiplications and activation functions on a scale comparable to the student network's computations.
- Section two is title "Online dynamical solution in the large input limit", but then the relevance of the large input limit (or even its definition) is not explained. Furthermore, the relevance and effect of high-dimensional input is not clearly shown. 
- Between the analytical theory, experiments on the benefit of CL, introduction of elastic coupling method, and connections between machine learning and psychology, the contributions of the paper seem scattered and unfocused.
- The organization and writing throughout the paper is quite confusing.
  - Figure 1 is based on notations and definitions introduced in section 3 and is not easily understood at the point it is presented.
  - Line 137: "Fig. 5 shows performance as a function of sparsity ρ". Figure 5 does not show that. I think you referenced the wrong figure.
  - Line 140: "low-D tasks .. [6]". The notation low-D is not introduced, not even in [6]. I assumed it means low-dimensional.
  - Line 143: "starting from a large initialisation", I assume it means the random initialization scale of the student or teacher network. It is not entirely clear. Please include more details about this initialization. Does it use a normal random distribution?
  - Figure 1: "The curriculum boundary lies at α = 1/2". What is the curriculum boundary? It is never defined. The abstract talks about "curriculum boundary consolidation", but it is not further elaborated. 
  - What is α in Fig 1c x-axis and other figures? $\alpha_1$ and $\alpha_2$ are defined, but $\alpha$ never is, and I am not able to understand what it means. 
  - Half-way through the paper section numbers stop being used. What I assume are sections 5 and 6 do not have numbers. And I am not sure if the subsections of Section 4 belong there. 
  - Figure 3c "Accuracy hard samples." is not an accurate title. It is accuracy of all samples, easy and hard. It's just that anti-curriculum performs best, which is influenced by hard samples.
  - The figure captions are inconsistent in the uses of (a), (b), (left, center, right), and (top, bottom). For example Figure 3 says (a), (b) and then "the right panel".

  
questions: - *"due to the convex nature of the teacher-student setup [22], the network is bound to converge to a minimum uniquely determined by the final slice of data, with no memory of the progress made at intermediate steps"* ... Is this a property of all student-teacher networks or just the the setup you are using? Because if it isn't, then curriculum learning can have a significant effect to the learning trajectory in the parameter spaces, and the optimum points able to be reached. 

- *"the first stage in the curriculum strategy can only help in the support identification problem"* ... Shouldn't the identification of the support lead to easy, successful learning of the simple remaining task? 

- What do you mean by training trajectories? Is it the accuracy curve over the course of training?
limitations: - I am not sure how generalization the teacher-student network analysis is, as compared to standard learning approaches. In particular, this setup with a convex network, simple optimizer, and simple synthetic dataset. I think the learning dynamics can be significantly different in a deep, non-convex network and using a modern momentum-based optimizer.
review_title: null

Meta-review: metareview: In this paper, authors provide an analytical theory of curriculum learning for an online teacher-student setting where a subset of features are relevant and are used by the teacher while a student might get distracted and use irrelevant features. In such a setting, the difficulty of examples can be captured by the variance in the irrelevant features. The insights from this analysis help explain existing empirical observations reported in prior work for effectiveness of curriculum learning as opposed to random-ordering and anti-curriculum learning in compute-limited regime. Further interesting connections to the literature on cognitive science are discussed in the paper.

Given the lack of theoretical basis for curriculum learning, reviewers are in agreement that this paper is in-time and impactful for the ML community and they all recommended accepting the paper. However, during the discussion period it became clear that this recommendation was based on the promise of authors to revise the paper which never happened during the rebuttal period.

My final recommendation is to accept the paper based on the authors' promise that the final version will include all changes promised by authors in their response to reviewers. Although there is no formal conditional acceptance mechanism, the authors should view this as a conditional acceptance based on taking their word that they will revise accordingly (see the list of changes below). I will check the camera-ready version and call-out anything less than that.

**List of changes promised by authors (quoted from their response)**

**Authors' response to reviewer Cgtb:**

1- "Unfortunately, implicit curricula cannot be directly studied within our framework, since we always assume the hardness information is completely disclosed. Considering a pseudo-labeling step before the actual learning stage would likely make the (already involved) computation unfeasible. Empirically this procedure has been investigated in [e.g. https://arxiv.org/abs/1812.05159, https://arxiv.org/abs/2012.03107] and, limiting our interest in the generalization aspect, this heuristic does not seem to induce a sizeable improvement. We will include this discussion in the revision."
2- "We mean an algorithm that explicitly depends on the curriculum, for instance by changing its objective function when example difficulty changes. That is, a curriculum aware algorithm would adapt the learning process in order to account for different levels of difficulties in the data. A simple way of implementing this is to modify the training loss, as proposed in this paper. Other approaches may involve adapting the optimization algorithm as proposed in https://proceedings.mlr.press/v139/ruiz-garcia21a.html, or possibly modifying the architecture https://www.sciencedirect.com/science/article/pii/0010027793900584. A key message emerging from our work is that standard algorithms do not dramatically benefit from curriculum, and we believe curriculum-aware algorithms may be the way forward. We will include a clear definition and discussion in the revision."

**Authors' response to reviewer S1ip:**

1- "The first point raised by the reviewer is the object of current investigations (see answer below). We will add more details on the CIFAR experiments in the revised version."

2- "We agree that this appears counter-intuitive. We also have hoped for greater intuition on this point, but we do not have a simple explanation for this phenomenon. This result is what appears from solving the equations and it was checked in the numerical simulations. A possible intuition could be that, in some settings, the large amount of noise contained in the hard data will always be too disruptive for effective learning. Thus, leaving the “clean” data for last could allow the model to better exploit the easy data. We will add this possibility to the revision. Even without a clear intuition, our contribution here is to show, in an identical setting and without finite size effects, that both anticurriculum and curriculum can indeed outperform the baseline."

**Authors' response to reviewer S1ip:**

1- "Overall, the analytical solution is between 2 and 6 orders of magnitude faster. We will add this approximate speed-up factor and discussion to the supplement (or if space allows, the revision)."

2- "The large input limit means that the input size N and the dataset size M go to infinity with finite ratio \alpha=M/N. This is an important point that must have gotten lost during the iterations, thank you for catching this. We will reintroduce it in the revised version."

3- "Figure 1 is based on notations and definitions introduced in section 3 and is not easily understood at the point it is presented. Reply: We will replace the image with explicit notation."

4- "Line 143: "starting from a large initialisation", I assume it means the random initialization scale of the student or teacher network. It is not entirely clear. Please include more details about this initialization. Does it use a normal random distribution? Reply: Yes, we use a normal distribution with a fixed variance. When we refer to large/small initialization we mean large/small initial variance. We will clarify this in the revised version."

5- "Figure 1: "The curriculum boundary lies at α = 1/2". What is the curriculum boundary? It is never defined. The abstract talks about "curriculum boundary consolidation", but it is not further elaborated. Reply: We mean the switching point between the two levels of difficulty. We will clarify this."

6- "Half-way through the paper section numbers stop being used. What I assume are sections 5 and 6 do not have numbers. And I am not sure if the subsections of Section 4 belong there. Reply: We will add 5 and 6 to the last two sections."

7- "Figure 3c "Accuracy hard samples." is not an accurate title. It is accuracy of all samples, easy and hard. It's just that anti-curriculum performs best, which is influenced by hard samples. Reply: We will replace it with just accuracy."

8- "The figure captions are inconsistent in the uses of (a), (b), (left, center, right), and (top, bottom). For example Figure 3 says (a), (b) and then "the right panel". Reply: We will use only the a,b,c notation."

9- "As stated in the paper, the answer does depend on the setup and in principle we would expect different behaviours for non-convex models. In particular, the presence of different basins of attraction towards different minima would suggest that initializing close to a good one would produce a performance improvement. However, note that empirical results in the ML field are not showing clear signals in this direction. A possible explanation is that relying on memory effects in the learning dynamics would require one to hit a sweet spot in the learning rate value and in the number of training epochs, and this seems hard to be achieved consistently. For this reason, we speculate that explicitly enforcing this memory by altering the loss with the curriculum information could be useful even in these settings. We will further emphasize that this statement about memorylessness applies only to our setting."

10- "We thank the reviewer for pointing out several elements that were missing in this version: we will make sure to include them in the revised version of this paper."
confidence: Certain
Decision: Accept