Title: Gradient-Guided Epsilon Constraint Method for Online Continual Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Online Continual Learning

Multi-Objective Optimization
Abstract: Online Continual Learning (OCL) requires models to learn sequentially from data streams with limited memory. Rehearsal-based methods, particularly Experience Replay (ER), are commonly used in OCL scenarios. This paper revisits ER through the lens of $\epsilon$-constraint optimization, revealing that ER implicitly employs a soft constraint on past task performance, with its weighting parameter post-hoc defining a slack variable. While effective, ER's implicit and fixed slack strategy has limitations: it can inadvertently lead to updates that negatively impact generalization, and its fixed trade-off between plasticity and stability may not optimally balance current streaming with memory retention, potentially overfitting to the memory buffer. To address these shortcomings, we propose the \textbf{G}radient-Guided \textbf{E}psilon \textbf{C}onstraint (\textbf{GEC}) method for online continual learning. GEC explicitly formulates the OCL update as an $\epsilon$-constraint optimization problem, which minimize the loss on the current task data and transform the stability objective as constraints and propose a gradient-guided method to dynamically adjusts the update direction based on whether the performance on memory samples violates a predefined slack tolerance $\bar{\varepsilon}$: if forgetting exceeds this tolerance, GEC prioritizes constraint satisfaction; otherwise, it focuses on the current task while controlling the rate of increase in memory loss. Empirical evaluations on standard OCL benchmarks demonstrate GEC's ability to achieve a superior trade-off, leading to improved overall performance. Code is available at https://github.com/laisong-22004009/GEC_OCL.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the Gradient Guided Epsilon Constraint (GEC), an experience replay-based online continual learning method that formulates single-step parameter updates as an $\\epsilon$-constraint optimization problem. This enables the learning algorithm to dynamically adjust the strength of the replay buffer gradient during optimization to appropriately prioritize stability vs plasticity. The paper attains strong results on various class-incremental benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper tackles the salient problem of dynamically balancing of stability and plasticity during online continual learning, which is important in the field of online continual learning that generally focuses on either stability or plasticity.\n- The paper's $\\epsilon$-constraint perspective of experience replay is not just novel but also useful, as it results in a strong method.\n- The paper is very well-written and has strong empirical results on class-incremental OCL benchmarks.\n- The experiment section is comprehensive, containing various benchmark results, breakdown of GEC's effects on stability and plasticity, computation details, and more.\n\nWeaknesses\n- The main text should refer to various appendix sections, as many useful details such as proofs and algorithms are present there but it is hard to reference them while reading the paper without direct pointers. In addition, a simplified related work section from the appendix should be included in the main text after the experiment section.\n- The paper should discuss and, time permitting, compare against recent work that optimizes the experience replay objectives by formulating a single parameter update step as a constrained proximal update [1,2]. These methods also employ the experience replay loss and slow and/or accelerate optimization in parameter directions that do not interfere with model predictions on the replay buffer data. One advantage of GEC that stands out is that it does not involve preconditioning matrix inversion.\n- MER baseline results are missing for Seq-CIFAR100 and Seq-TinyImageNet experiments in Tables 1 and 2.\n- Nitpicks\n  - The paper refers to the term \"posterior\" on multiple occasions, and it does not seem to be related to the Bayesian posterior. Please define this.\n  - Figure 3c caption should be Seq-TinyImageNet Plasticity, not Seq-TinyImageNet Stability.\n\n\nReferences\n\n[1] Yoo, J., Liu, Y., Wood, F. & Pleiss, G.. (2024). Layerwise Proximal Replay: A Proximal Point Method for Online Continual Learning. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:57199-57216.\n\n[2] Urettini, E., & Carta, A. (2025). Online Curvature-Aware Replay: Leveraging 2nd Order Information for Online Continual Learning. arXiv preprint arXiv:2502.01866.\n\nQUESTIONS:\n- Would the theoretical findings hold when $\\bar{l}_M$ is approximated with Monte Carlo as is the case in practice?\n- What were the values of $\\epsilon$ for the experiments, and how should they be set generally?\n- Do you have any idea why Figure 3a has GEC underperforming ER in plasticity early on for Seq-CIFAR100 while Figure 3c has GEC outperforming ER in plasticity for Seq-TinyImageNet?\n- What do the dashed lines mean in Figure 1a?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper reveals that experience replay (ER) in Online Continual Learning (OCL) implicitly employs a soft constraint with a fixed slack, thereby risking suboptimal updates and overfitting. We analyze ER as an ε-constraint optimization problem, revealing its limitations. We propose Gradient-Guided Epsilon Constraint (GEC), explicitly formulating OCL updates as an ε-constraint problem: minimize current loss while enforcing stability constraints on memory data. GEC dynamically adjusts the update direction using gradients: if memory loss exceeds a slack tolerance, it prioritizes constraint satisfaction; else, it focuses on the current task while controlling memory loss increase. Evaluations show GEC achieves a superior stability-plasticity trade-off, improving overall performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.The paper is well-organized, with clear theoretical derivations, intuitive visualizations and a logical flow from problem formulation to evaluation.\n2.The paper provides a novel perspective by reinterpreting Experience Replay (ER) as an ε-constraint optimization problem. This insight bridges the gap between rehearsal-based methods and constraint-based approaches, offering a unified framework for understanding trade-offs in online continual learning\n3.The proposed Gradient-Guided Epsilon Constraint (GEC) method introduces a dynamic, gradient-aware mechanism to balance plasticity and stability. By adjusting the update direction based on constraint violation, GEC avoids overfitting to the memory buffer while mitigating catastrophic forgetting. This design is both theoretically grounded and empirically validated.\n4.The experiments on standard OCL benchmarks demonstrate consistent improvements over state-of-the-art methods like ER, POCL, and CBA. The ablation studies and analysis of generalization gain vs. forgetting measures strengthen the claims.\n\nWeaknesses:\n1.The quadratic programming (QP) formulation in GEC may introduce computational overhead compared to simpler methods like ER. The paper briefly mentions training time comparisons (Table 3) but does not analyze how this overhead scales with larger datasets or more complex models.\n2.The analysis assumes convexity and differentiability of the loss functions. In real-world non-convex scenarios, the guarantees for constraint adherence and convergence may not hold.\n\nQUESTIONS:\n1.How does the QP formulation scale with increasing task complexity or dataset size? Are there approximations (e.g., closed-form solutions) to reduce computational overhead?\n2.How does GEC perform in non-convex scenarios? What modifications are needed to handle non-convexity?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work investigates the online continual learning (OCL) problem and revisit the classical strategies of Experience Replay (ER), and constraint-based optimization methods such as GEM and A-GEM. The paper first show that ER can be formulated as a soft-constraint optimization problem, revealing its underlying behaviours of why it could achieve a better performance. From there, the authors proposed Gradient Guided Epsilon Constraint (GEC) which dynamically adjusts the slack variable within this optimization framework. Experiments on common OCL benchmark show encouraging results of GEC.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- I like the theoretical analysis in this work. Although not too complex, it is original, easy to follow, and provided useful insights. \n- The proposed GEC algorithm is well-motivated and seems to work well on common benchmarks.\n- The authors conducted extensive ablation studies and also reported the training complexity.\n\nWeaknesses\n- My major concern of this work is the benchmarks considered are quite outdated, and small in scale. It would be more interesting to validate the algorithm on more challenging scenarios, such as the long-tail setting [A], on larger datasets (e.g. ImageNet), or in NLP/RL/Robotics domains.\n- How does GEC interact with the BatchNorm (BN) component in during OCL? Specifically, the original ER mixes current and previous data samples in a mini batch and perform a single forward to update. In contrast, GEC performs two forward passes on the current task and previous task data, which results in different updating dynamic of the BN's global statistics than ER. A prior study [B] showed that this difference may impact OCL. \n\n[A] Kim, Chris Dongjoo, Jinseo Jeong, and Gunhee Kim. \"Imbalanced continual learning with partitioning reservoir sampling.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16. Springer International Publishing, 2020.\n\n[B] Pham, Quang, Chenghao Liu, and Steven Hoi. \"Continual normalization: Rethinking batch normalization for online continual learning.\" ICLR, 2022.\n\nQUESTIONS:\n- How would GEC perform on more challenging benchmarks?\n- Could you clarify the interaction between GEC and BN and compared it to ER?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}