Title: When Does Curriculum Learning Help? A Theoretical Perspective
Venue: NeurIPS 2025
Accepted: yes
Keywords: curriculum learning; learning theory; generalization; biased regularization; convex learning problems
Abstract: Curriculum learning has emerged as an effective strategy to enhance the training efficiency and generalization of machine learning models. However, its theoretical underpinnings remain relatively underexplored. In this work, we develop a theoretical framework for curriculum learning based on biased regularized empirical risk minimization (RERM), identifying conditions under which curriculum learning provably improves generalization. We introduce a sufficient condition that characterizes a "good"   curriculum and analyze a multi-task curriculum framework, where solving a sequence of convex tasks can facilitate better generalization. We also demonstrate how these theoretical insights translate to practical benefits when using stochastic gradient descent (SGD) as an optimization method. Beyond convex settings, we explore the utility of curriculum learning for non-convex tasks. Empirical evaluations on synthetic datasets and MNIST validate our theoretical findings and highlight the practical efficacy of curriculum-based training.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents several key contributions to the theoretical understanding of curriculum learning—the strategy of training a model on progressively harder tasks. The authors introduce a formal framework for analyzing multi-task curriculum learning. This framework is centered on Biased Regularized Empirical Risk Minimization (RERM) by using an easy task to provide better inductive biases  to regularize the learning process for the next, more difficult task.\n\nSome theoretical contirbutions of this paper:\nThe (r, α) Condition which connects task similarity and empirical risk. This condition mathematically formalizes the similarity between consecutive tasks. It asserts that if a model performs well on one task, a slightly perturbed version of that model will also perform reasonably well on the subsequent task, so as long as the r is small enough.\n\nThe analysis provides rigorous proofs and performance guarantees for different types of optimization problems.\nFor Convex Problems:\nThe paper provides proofs and excess risk bounds showing that a good curriculum (one with a small r) can significantly reduce the sample complexity required for subsequent tasks.\nThese guarantees are proven for both the RERM framework and the more practical Stochastic Gradient Descent (SGD) method.\nFor Non-Convex Problems:\nThe analysis is extended to non-convex settings, which are highly relevant for modern deep learning.\nThe authors propose an ERM-based algorithm and establish generalization guarantees, demonstrating that a well-designed curriculum improves learning efficiency even in these complex scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths of this paper:\n\n1. This paper, to my knowledge is one of the first to present the notion of (r,α) condition of task similarity to empirical risk. \n2. Authors provides guarantees for both convex and non-convex optimization problems (thm 4.1, 4.2). Since most modern deep learning involves highly non-convex landscapes, this makes the paper's conclusions far more relevant and impactful for practitioners working with neural networks.\n3. The authors extend their analysis from the more abstract Biased RERM to Stochastic Gradient Descent (SGD), which is the optimizer used in the vast majority of real-world machine learning. \n\nWeakness:\nThis work, to my understanding hinges on the (r, α) condition. which in this paper works only when the task sequence have the same domain and target space and the arch. is same. For a more general setting this might not be possible\n\nThe experimentation is a bit weak, for a general setting of sequential how can we identify the rho and alpha, While the experiments are well-designed to support the theoretical claims, they are conducted on relatively small-scale and controlled environments (synthetic data and MNIST).\n\nQUESTIONS:\n1. [Eq 2] how do you define this norm when there are arch changes, tasks need not have errors at the same scale\n2. In your work you mentioned about the scale of loss being different in different subsequent tasks, does that not affect the $\\alpha$ parameter? how if at all any rescaling operation done? if not then despite being a optimal curriclum, the scale difference might indicate a high α\n3. Can you provide some experimental evidence for curricula where the task does not necessarily share the same arch. or target label set, basically increase the complexity of the underlying set of tasks in  the curriculum to show if it is possible to empirically validate the theoretical findings to a more realistic setting.\n\nLIMITATIONS:\nThis work specifies the generalization on the subsequent task given a particular curriculum. The harder and perhaps more important question is what is the optimal curriculum. given the inability to define the (r, α) criterion for a general setting, the applicability of this method seems limited.\n\nHow can we identify the (r, α) criteria for a given curriculum in a general setting?\n\nIt's unclear how these results and the underlying theoretical dynamics would scale to much larger and more complex real-world scenarios. For instance, training massive models like large language models (LLMs) or state-of-the-art vision models on datasets like ImageNet involves far more complex dynamics. While the principles might hold, the practical benefits could be different or harder to achieve, and the cost of exploring different curricula would be prohibitively high.\n\n\nThe process of ordering tasks remains largely a heuristic, domain-specific art that relies on human intuition. For example, in their experiments, the authors rely on common-sense curricula: increasing the difficulty of a classification task by reducing the margin, or increasing the strength of an adversarial attack. The framework doesn't offer a general algorithm for discovering this optimal ordering automatically.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

comments: SUMMARY:
The paper introduces (for the first time?) a theoretical analysis of curriculum learning. It studies a scenario where the learner goes through a sequence of tasks where the parameters found at the end of training for each task is a good initialization for the next task (and not far from a good solution for the next task). This is a scenario that is reminiscent of continuation methods in optimization, but the paper provides a number of theoretical bounds on generalization at each step, given the expected error at the previous step. Several other theoretical assumptions are leveraged to obtain a diversity of results, e.g., if the losses are convex in the parameters, if it is Lipschitz, and if it is smooth. Some of the results apply in the SGD non-convex case typical of neural networks and the paper describes numerical simulations which support the theoretical results.

STRENGTHS AND WEAKNESSES:
ular pre-training followed by successive fine-tuning stages can be seen as instances of curriculum learning that plausibly fit some of the scenarios studied in the paper.

As far as I know, this is the first learning-theoretical analysis of curriculum learning, but I would have liked to see confirmation or not in the paper, along with a review of relevant related work, if only in terms of the theoretical tools being used, as well as non-theoretical work on curriculum learning.

QUESTIONS:
Is there absolutely no previous work on a learning-theoretical analysis of curriculum learning?

How are the proof techniques related to previous work in learning theory?

LIMITATIONS:
In the conclusion, briefly.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the present paper, the authors develop a biased-regularised ERM view of curriculum learning. Given a sequence of $T$ supervised tasks, each task $t$ is learned by minimising the empirical loss plus an $L_2$ penalty that keeps the parameters close to the solution of task $t-1$. The analysis is based on the definition of an $(r, \\alpha)$ “good curriculum” condition, used for proving excess-risk bounds for convex, Lipschitz, and smooth losses. The results are then extended, under some assumptions, to SGD and a non-convex setting. The empirical evaluation is based on experiments on synthetic data and MNIST (including adversarial training).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The work is technically solid: assumptions are carefully stated, proofs are complete, and empirical evaluation seems sufficient (especially considering the long appendix). \n- The idea of introducing the $(r, \\alpha)$ compatibility metric seems to allow a simple formalization of task similarity. \n- Overall, the paper is clearly written. \n\n**Weaknesses**\n- Some relevant references to theoretical works on curriculum learning are missing (e.g., the recent line by Abbe et al). In particular, a similar curriculum protocol with elastic penalties was analysed analytically and experimentally in \"An Analytic Theory of Curriculum Learning\" (Sarao Mannelli et al., 2020), which the paper does not cite.\n- The derived generalisation bounds depend on (potentially unknown) global Lipschitz, smoothness, and local-geometry constants. This makes it hard to judge their tightness or to understand how they can be used in practice (the bound curves are not plotted alongside the learning curves in the empirical experiments). \n- It is a bit unclear whether the analyzed setting is truly connected to curriculum learning or rather represents a generic continual/transfer learning setting.\n- The $(r, \\alpha)$ condition and the regularisation schedule presume task knowledge that a learner rarely has access to.\n\nQUESTIONS:\nOn the tightness of the bounds: Could the authors overlay the analytical excess-risk curves on the experimental plots, so readers can see how loose/tight they are?\n\nOn choosing $(r, \\alpha)$ in practice: Can the authors comment on how one would estimate or tune $r$ when the optimal risks or parameter distances are unknown?\n\nOn the relation to earlier work: Can the authors discuss the connections to Sarao Mannelli et al. (2020) and clarify the novelty of their practical findings?\n\nOn the connection to continual-learning: The drift-per-task setting resembles CL more than classic curriculum. How does this method fare in settings where catastrophic forgetting can take place?\n\nLIMITATIONS:\nThe discussion of the limitations is very limited (a comment in the Conclusions).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to provide a solid theoretical foundation for Curriculum Learning (CL), an effective strategy in machine learning whose success is currently largely empirical and lacks sufficient theoretical understanding. To address this gap, the authors propose a theoretical framework based on biased Regularized Empirical Risk Minimization (RERM). The paper's core contribution is the introduction of a novel (r, a) condition to mathematically characterize a \"good\" curriculum.\nWithin this framework, the authors analyze the performance of CL across different learning tasks:\nFor convex learning tasks, they derive excess risk bounds and demonstrate that satisfying the (r, α) condition can effectively reduce sample complexity. This analysis is also extended to the widely used Stochastic Gradient Descent (SGD) algorithm.\nThe theoretical analysis is extended to non-convex tasks, providing generalization guarantees for the application of CL in modern scenarios such as deep learning.\nFinally, the paper empirically validates its theoretical findings through experiments on synthetic and MNIST datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper successfully identifies the disconnect between the practical application of CL and its theoretical underpinnings. Providing a theoretical basis for this widely used heuristic is a valuable and compelling research direction.\n2. The proposed (r, α) condition is a central highlight of the work. It offers a novel, intuitive, and formal tool for defining what constitutes a \"good\" curriculum, providing a powerful instrument for analyzing CL's effectiveness.\n3. The theoretical work in this paper is solid. The authors provide rigorous excess risk bounds for convex tasks and extend the analysis to the practical SGD optimizer, strengthening the connection between theory and practice.\n\nWeaknesses:\n1. While the theory is elegant, it provides limited insight into how to design a good curriculum in practice, especially for real datasets. There's no principled method to determine task order or difficulty.\n2. Though the paper attempts to extend to nonconvex settings, the treatment is still mainly theoretical, and the methods like ERM over radius-constrained balls are not practical for large-scale deep learning.\n3. Experiments are mainly conducted on synthetic data and MNIST, with relatively simple settings. The generalizability to complex domains (e.g., NLP, vision benchmarks) is not demonstrated.\n\nQUESTIONS:\nHow robust is the model if early tasks are misleading or corrupted?\nWill incorrect prior tasks lead to worse generalization, and how can this be mitigated?\nWhy are the datasets limited to synthetic and MNIST?\n\nLIMITATIONS:\n(r, α)-condition is difficult to verify or enforce in practice. While theoretically elegant, it's unclear when or how this condition holds on real datasets.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}