Forget Forgetting: Continual Learning in a World of Abundant Memory

## Abstract

Abstract Continual learning (CL) research typically assumes highly constrained exemplar memory resources. However, in many real-world scenarios—especially in the era of large foundation models—memory is abundant, while GPU computational costs are the primary bottleneck. In this work, we investigate CL in a novel setting where exemplar memory is ample ( i.e. , sufficient exemplar memory). Unlike prior methods designed for strict exemplar memory constraints, we propose a simple yet effective approach that directly operates in the model’s weight space through a combination of weight resetting and averaging techniques. Our method achieves state-of-the-art performance while reducing the computational cost to a quarter or third of existing methods. These findings challenge conventional CL assumptions and provide a practical baseline for computationally efficient CL applications.

## 1 Introduction

Continual learning (CL) has attracted significant attention as a paradigm enabling machine learning models to adapt to sequential tasks while overcoming catastrophic forgetting of previously acquired knowledge (Wang et al., 2024).
A fundamental challenge in CL is the *stability-plasticity dilemma* (Carpenter & Grossberg, 1987; Mermillod et al., 2013), where a model that maintains stability on previous tasks often struggles to integrate new knowledge, and vice versa. Among various approaches, exemplar-based methods that store representative samples from past tasks have been widely adopted, particularly in class-incremental learning (class-IL) (Masana et al., 2022). Traditionally, these methods have operated under strict exemplar memory constraints, assuming that minimizing memory usage is critical for cost efficiency. However, recent studies suggest that memory limitations may not be as critical as once thought, with computational costs—such as GPU usage—emerging as a more dominant factor (Prabhu et al., 2023; Chavan et al., 2023). For instance, an AWS GPU server equipped with 8 A100 GPUs incurs an hourly cost of approximately $32.77, whereas storing 1TB of data on Amazon S3 costs roughly $23.55 per month.

Building on this, we revisit the class-IL setting under more realistic conditions, where exemplar memory constraints are relaxed (i.e., sufficient exemplar memory is available). Our experiments reveal that even the simplest exemplar-based approach—such as naive finetuning (i.e., Replay)—performs comparably to state-of-the-art algorithms while incurring significantly lower training costs (e.g., shorter training time).
Motivated by this, we propose a simple yet effective approach that optimizes computational cost while maintaining strong performance. Our method is designed with two key objectives: (1) outperforming other algorithms including Replay, and also (2) achieving computational efficiency, in the setting with sufficient exemplar memory. The core idea is to lower computational overhead by directly operating in the model’s weight space. Specifically, we integrate weight resetting and averaging techniques to accelerate convergence while enhancing both stability and plasticity during CL. Experimental results show that our method not only surpasses competing approaches but also achieves superior performance while requiring only a quarter to a third of the computational cost of state-of-the-art algorithms. Our contributions are summarized as follows:

- •
We investigate the impact of relaxed exemplar memory constraints on class-IL algorithms and demonstrate that even naive finetuning performs competitively under these conditions while significantly reducing the computational cost.
- •
We propose a novel weight-space manipulation strategy that achives superior performance while drastically reducing computational overhead compared to state-of-the-art algorithms.

## 2 Related Works

Continual learning   Continual learning (CL) has been actively studied in various scenarios and methodological categories. Among the three scenarios of CL (Van de Ven & Tolias, 2019), class-incremental learning (class-IL) has been considered the most challenging and has been the most actively studied scenario (Masana et al., 2022). CL algorithms (including class-IL) can be categorized into regularization-based approaches, which penalize changes to important parameters for past tasks (Kirkpatrick et al., 2017; Aljundi et al., 2018; Cha et al., 2020), rehearsal-based approaches, which store and replay exemplars from past tasks (Rebuffi et al., 2001; Cha et al., 2023), and expansion-based approaches, which expand the model’s capacity to balance the trade-off between stability and plasticity (Yan et al., 2021; Wang et al., 2022).
Additional approaches focus on addressing classifier bias toward recent tasks while using the exemplars (Wu et al., 2019; Zhao et al., 2020).
While exemplar-based methods have demonstrated state-of-the-art performance, they typically rely on strict memory constraints, often limiting memory size to a small percentage of the dataset. Recent studies challenge the necessity of these strict memory constraints, highlighting that the computational cost of maintaining and processing memory—especially GPU usage—can far outweigh storage costs (Prabhu et al., 2023; Chavan et al., 2023). This shift in perspective opens the door to relaxing memory limits in order to reduce training costs, which is the focus of our work.

Weight space operations Weight space operations directly manipulate the model parameters for various goals e.g., multi-task learning (Yu et al., 2024) and continual learning (Marouf et al., 2025; Marczak et al., 2025). For instance, weight averaging is utilized to merge the knowledge of multiple models (Ilharco et al., 2022; Jang et al., 2025), while weight reset is used to enhance plasticity (Dohare et al., 2024) and improve parameter-redundancy (Yadav et al., 2024). For an overview of weight-space operations, please refer to [Section˜A.6](#A1.SS6).

## 3 Problem Formulation

Notation
We generally follow the setting of class-incremental learning (class-IL) (Masana et al., 2022). We consider a sequence of $T$ tasks, each associated with distribution $P_{t}$. Let $\mathcal{D}_{t}=\{(x^{t}_{i},y^{t}_{i})\}_{i=1}^{|\mathcal{D}_{t}|}$ with $(x^{t}_{i},y^{t}_{i})\sim P_{t},$, be the training dataset for task $t$. The tasks are presented in order $t=1,\cdots,T$. The model $F$ (its parameters $\theta$) does not retain explicit access to previous task datasets $\mathcal{D}_{j}$ for $j<t$, except via a exemplar memory buffer $\mathcal{M}$ of a capacity of $K$. Thus, at the training step of task $t$, the model updates its parameters $\theta$ using the data $D_{t}\cup\mathcal{M}_{1:t-1}$ and the cross-entropy loss $\ell(\theta;x,y)$, where $\mathcal{M}_{1:t-1}$ includes selected samples from earlier tasks.

Figure: (a) Acc. graph on CIFAR-100
Refer to caption: /html/2502.07274/assets/x1.png

What changes under a sufficient exemplar memory regime?   We claim that a sufficiently large memory can closely approximate the data distributions of previous tasks, effectively reducing forgetting. For the complete derivation of this claim, please refer to [Section˜A.1](#A1.SS1).
Experimentally, [Figure˜1(a)](#S3.F1.sf1) demonstrates that a significant drop in forgetting occurs when exemplar memory is abundant.
Notably, we observe that the simplest baseline (i.e., Reply) outperforms its competitors with significantly lower cost (see [Figure˜1(b)](#S3.F1.sf2)).
This underscores two key points: First, when the sufficient exemplar memory is available, existing algorithms fall behind even the simplest baseline, making their additional computational cost unnecessary. With these observations in mind, we aim to develop a new algorithm that is (1) cost-effective and (2) addresses the new challenges that arise in a sufficient exemplar memory setting.

## 4 Weight Space Consolidation

Figure: Algorithm 1 Weight-space consolidation for cost-efficient CL

We introduce a simple but cost-effective method that leverages weight-space operations (e.g., weight-reset, weight-average) for continual learning with sufficient exemplar memory.
The grounding idea of our approach is that the sufficient exemplar memory regime allows the model to obtain an effective minimizer of all previous tasks (discussed in [Section˜3](#S3)). Deriving from this, at the beginning of training each $t^{th}$ task, we assume that we already have a good minimizer of the previous tasks, and encourage the model to stay in its proximity in the weight-space.

[Algorithm˜1](#alg1) shows pseudo-code of our method. Prior to the training of the $t^{th}$ task, the model parameters (i.e., minimizer of previous $1:t-1$ tasks) are stored as $\theta_{\textrm{prev}}$ (see Line 6 in [Algorithm˜1](#alg1)).
Then, we begin training on the current $t^{th}$ task using the loss $\ell(\theta;x,y)$ (see Line 8).
After $n_{\textrm{warm}}$ warming epochs, we apply a rank-based weight resetting procedure to reset the parameters that were dormant during training (see Line 11). We rank the dormant parameters by comparing the current value in $\theta$ to the value of the previous task model $\theta_{\textrm{prev}}$. We tested multiple metrics to quantify the dormancy and found that a simple distance $\big{|}\theta[l]-\theta_{\textrm{prev}}[l]\big{|}$ is sufficient to measure the drift for the $l^{th}$ parameter. Please refer to [section˜A.3](#A1.SS3). Specifically, we reset 80% of the parameters, following empirical results of Yadav et al. (2024) (see Lines 12-14). Formally, the $l^{th}$ parameter of $\theta$ is reset using:

$$ $\theta[l]=\alpha\cdot\theta_{\mathrm{prev}}[l]+(1-\alpha)\cdot\theta[l],$ (1) $$

where $\alpha=0.5$ by default. This step is intended to reset parameters that did not contribute to learning the current task $t$, allowing the model $\theta$ to remain in proximity to the previous task model $\theta_{\textrm{prev}}$.
Assuming the reset model is a good (if not optimal) minimizer of the current task $t$, and also retaining the parameters of $\theta_{\textrm{prev}}$, we apply a periodic weight-averaging over the learning trajectory (see Lines 16-17) (Izmailov et al., 2018). The weight-averaging procedure is intended to accelerate convergence and improve generalization by aggregating diverse learning signals. We find this particularly beneficial under sufficient exemplar memory settings, where the variance of data is significant. Please refer to [section˜A.2](#A1.SS2) for a detailed elaboration of our method.

## 5 Experiment

Experimental settings   We evaluate our algorithm on two standard class-incremental learning (class-IL) benchmarks (Masana et al., 2022) using the PyCIL framework (Zhou et al., 2023).
CIFAR100 is an image classification dataset with 100 classes, which is split into 10 sequential tasks, each containing 10 classes. ImageNet100 is another classification dataset with 100 classes, which is also split into 10 sequential tasks, each containing 10 classes.
We compare our approach with six exemplar-based baselines, including iCaRL (Rebuffi et al., 2001), BiC (Wu et al., 2019), WA (Zhao et al., 2020), DER (Yan et al., 2021), and FOSTER (Wang et al., 2022). Additionally, we include Replay, a naive baseline that finetunes the model using both the current dataset and stored exemplars.
All experiments use ResNet-32 for CIFAR-100 and ResNet-18 for ImageNet-100 as the backbone model (He et al., 2016). We conduct experiments across five random seeds and report the final class-IL accuracy, computed as the average classification accuracy across all tasks on the validation datasets after training on the final task (Masana et al., 2022). Lastly, the experimental details are reported in [Section˜A.5](#A1.SS5)

Table: Table 1: Average class-IL accuracy (Acc.) with different exemplar memory sizes. We report experimental results with varying memory sizes, ranging from 20 exemplars per class (a common setting in class-IL) to 500 exemplars per class (storing the entire previous dataset) from previous tasks. The bold indicates the best performance among the non-expansion-based methods.

Figure: (a) Training time spent on ImageNet-100
Refer to caption: /html/2502.07274/assets/x3.png

Experimental results   [Figure˜2](#S5.F2) (and [Figure˜1(a)](#S3.F1.sf1)) presents experimental results on the CIFAR-100 and ImageNet-100 datasets in a 10 task scenario, evaluating various exemplar memory sizes. To ensure a fair comparison, we distinguish expansion-based methods, such as DER and FOSTER, from other algorithms, as these methods increase the model size during class-IL. Based on these results, we make the following key observations: First, in the constrained exemplar memory setting typically considered (e.g., 20 exemplars per class), existing algorithms outperform the naive Replay approach. However, as the number of exemplars per class increases, the performance gap between these algorithms and Replay narrows significantly. Notably, when the exemplar count per class reaches 100 (i.e., using sufficient exemplar memory), Replay achieves competitive performance compared to other methods. Second, [Figure˜1(b)](#S3.F1.sf2) and [Figure˜2(a)](#S5.F2.sf1), which compare training costs (measured by training time), show that as exemplar memory size increases, the training cost also rises. In particular, expansion-based methods achieve superior performance but require 2-3 times the training cost of Replay. In contrast, note that our weight space consolidation method achieves superior performance across both benchmarks, nearly matching the training cost of the simplest Replay method. We also report the results of an ablation study, which demonstrates that each component of our method contributes uniquely and effectively. When combined, these components yield the strongest performance (see Figures [2(b)](#S5.F2.sf2)). We provide more detailed analyses of (1) the ablation study and (2) the effect of weight space operations on continual learning in [Section˜A.4](#A1.SS4).

## 6 Concluding Remarks

Our work shifts the focus in continual learning from memory efficiency to computational cost, demonstrating that with sufficient exemplar memory, exemplar-based methods like Replay can achieve competitive performance at a fraction of the training cost. We introduce a simple, effective method that directly manipulates the model’s weight space, outperforming existing algorithms in terms of both accuracy and efficiency. These results highlight the potential for optimizing computational efficiency in class-incremental learning and provide a promising direction for practical applications, suggesting that exemplar memory constraints may no longer be the primary bottleneck in many real-world scenarios.

## References

- Aljundi et al. (2018)
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars.
Memory aware synapses: Learning what (not) to forget.
In *Proceedings of the European conference on computer vision (ECCV)*, pp.  139–154, 2018.
- Brignac et al. (2023)
Daniel Brignac, Niels Lobo, and Abhijit Mahalanobis.
Improving replay sample selection and storage for less forgetting in continual learning.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pp.  3540–3549, 2023.
- Carpenter & Grossberg (1987)
Gail A Carpenter and Stephen Grossberg.
Art 2: Self-organization of stable category recognition codes for analog input patterns.
*Applied Optics*, 26(23):4919–4930, 1987.
- Cha et al. (2020)
Sungmin Cha, Hsiang Hsu, Taebaek Hwang, Flavio P Calmon, and Taesup Moon.
Cpr: classifier-projection regularization for continual learning.
*arXiv preprint arXiv:2006.07326*, 2020.
- Cha et al. (2023)
Sungmin Cha, Sungjun Cho, Dasol Hwang, Sunwon Hong, Moontae Lee, and Taesup Moon.
Rebalancing batch normalization for exemplar-based class-incremental learning.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp.  20127–20136, 2023.
- Chavan et al. (2023)
Vivek Chavan, Paul Koch, Marian Schlüter, and Clemens Briese.
Towards realistic evaluation of industrial continual learning scenarios with an emphasis on energy consumption and computational footprint.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pp.  11506–11518, 2023.
- Dohare et al. (2024)
Shibhansh Dohare, J Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A Rupam Mahmood, and Richard S Sutton.
Loss of plasticity in deep continual learning.
*Nature*, 632(8026):768–774, 2024.
- Fornasier et al. (2021)
Massimo Fornasier, Lorenzo Pareschi, Hui Huang, and Philippe Sünnen.
Consensus-based optimization on the sphere: Convergence to global minimizers and machine learning.
*Journal of Machine Learning Research*, 22(237):1–55, 2021.
- He et al. (2016)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp.  770–778, 2016.
- Ilharco et al. (2022)
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi.
Editing models with task arithmetic.
*arXiv preprint arXiv:2212.04089*, 2022.
- Izmailov et al. (2018)
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson.
Averaging weights leads to wider optima and better generalization.
*arXiv preprint arXiv:1803.05407*, 2018.
- Jang et al. (2025)
Dong-Hwan Jang, Sangdoo Yun, and Dongyoon Han.
Model stock: All we need is just a few fine-tuned models.
In *European Conference on Computer Vision*, pp.  207–223. Springer, 2025.
- Kaur et al. (2023)
Simran Kaur, Jeremy Cohen, and Zachary Chase Lipton.
On the maximum hessian eigenvalue and generalization.
In *Proceedings on*, pp.  51–65. PMLR, 2023.
- Khromov & Singh (2024)
Grigory Khromov and Sidak Pal Singh.
Some fundamental aspects about lipschitz continuity of neural networks.
In *The Twelfth International Conference on Learning Representations*, 2024.
- Kirkpatrick et al. (2017)
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.
Overcoming catastrophic forgetting in neural networks.
*Proceedings of the national academy of sciences*, 114(13):3521–3526, 2017.
- Kozal et al. (2024)
Jędrzej Kozal, Jan Wasilewski, Bartosz Krawczyk, and Michał Woźniak.
Continual learning with weight interpolation.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp.  4187–4195, 2024.
- Li et al. (2018)
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein.
Visualizing the loss landscape of neural nets.
*Advances in neural information processing systems*, 31, 2018.
- Marczak et al. (2025)
Daniel Marczak, Bartłomiej Twardowski, Tomasz Trzciński, and Sebastian Cygert.
Magmax: Leveraging model merging for seamless continual learning.
In *European Conference on Computer Vision*, pp.  379–395. Springer, 2025.
- Marouf et al. (2025)
Imad Eddine Marouf, Subhankar Roy, Enzo Tartaglione, and Stéphane Lathuilière.
Weighted ensemble models are strong continual learners.
In *European Conference on Computer Vision*, pp.  306–324. Springer, 2025.
- Masana et al. (2022)
Marc Masana, Xialei Liu, Bartłomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer.
Class-incremental learning: survey and performance evaluation on image classification.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 45(5):5513–5533, 2022.
- Merlin et al. (2022)
Gabriele Merlin, Vincenzo Lomonaco, Andrea Cossu, Antonio Carta, and Davide Bacciu.
Practical recommendations for replay-based continual learning methods.
In *International Conference on Image Analysis and Processing*, pp.  548–559. Springer, 2022.
- Mermillod et al. (2013)
Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin.
The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects.
*Frontiers in Psychology*, 4:504, 2013.
- Paszke et al. (2019)
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.
Pytorch: An imperative style, high-performance deep learning library, 2019.
URL [https://arxiv.org/abs/1912.01703](https://arxiv.org/abs/1912.01703).
- Prabhu et al. (2023)
Ameya Prabhu, Hasan Abed Al Kader Hammoud, Puneet K Dokania, Philip HS Torr, Ser-Nam Lim, Bernard Ghanem, and Adel Bibi.
Computationally budgeted continual learning: What does matter?
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp.  3698–3707, 2023.
- Rebuffi et al. (2001)
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert.
Incremental classifier and representation learning.
In *Conference on Computer Vision and Pattern Recognition (CVPR)*, pp.  5533–5542, 2001.
- Van de Ven & Tolias (2019)
Gido M Van de Ven and Andreas S Tolias.
Three scenarios for continual learning.
*arXiv preprint arXiv:1904.07734*, 2019.
- Wang et al. (2022)
Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan.
Foster: Feature boosting and compression for class-incremental learning.
In *European conference on computer vision*, pp.  398–414. Springer, 2022.
- Wang et al. (2024)
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu.
A comprehensive survey of continual learning: theory, method and application.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 2024.
- Wortsman et al. (2022)
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al.
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.
In *International conference on machine learning*, pp.  23965–23998. PMLR, 2022.
- Wu et al. (2019)
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu.
Large scale incremental learning.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp.  374–382, 2019.
- Yadav et al. (2024)
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal.
Ties-merging: Resolving interference when merging models.
*Advances in Neural Information Processing Systems*, 36, 2024.
- Yan et al. (2021)
Shipeng Yan, Jiangwei Xie, and Xuming He.
Der: Dynamically expandable representation for class incremental learning.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp.  3014–3023, 2021.
- Yu et al. (2024)
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li.
Language models are super mario: Absorbing abilities from homologous models as a free lunch.
In *Forty-first International Conference on Machine Learning*, 2024.
- Zhao et al. (2020)
Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia.
Maintaining discrimination and fairness in class incremental learning.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp.  13208–13217, 2020.
- Zhou et al. (2023)
Da-Wei Zhou, Fu-Yun Wang, Han-Jia Ye, and De-Chuan Zhan.
Pycil: A python toolbox for class-incremental learning, 2023.

## Appendix A Appendix

### A.1 An investigation into the sufficient exemplar memory regime

#### Defining the sufficient exemplar memory regime

Most prior work assumes a strict exemplar memory constraint: $K\ll\sum_{t=1}^{T}|\mathcal{D}_{t}|$. Under such limited capacity, the replay buffer $\mathcal{M}$ can only hold a small fraction of each older dataset in $\{\mathcal{D}_{1},\cdots,\mathcal{D}_{t-1}\}$. Consequently, it provides only a partial representation of the task distributions $\{P_{1},\cdots,P_{t-1}\}$, leaving the model prone to catastrophic forgetting (Kirkpatrick et al., 2017) between tasks (i.e., inter-task forgetting)

By contrast, we say the replay buffer $\mathcal{M}$ is sufficient if it can store enough samples to approximate each past distribution $P_{j}$ for $1\leq j<t$. Formally, we let $K\approx\kappa\sum_{j=1}^{t-1}|\mathcal{D}_{j}|,$
where $0<\kappa\leq 1$ determines what fraction of the earlier task data can be retained. A larger $\kappa$ indicates that $\mathcal{M}$ preserves a representative set of samples from past tasks - though not necessarily every sample. As a result of this sufficient exemplar memory setting, the mixture distribution $P_{past}$ of previously encountered tasks at the training step of task $t$ effectively becomes:

$$ $P_{past}(x,y)\approx\frac{\sum_{j=1}^{t-1}\pi_{j}P_{j}(x,y)}{\sum_{j=1}^{t-1}\pi_{j}},$ (2) $$

where $\pi_{j}$ denotes the relative importance (i.e., size) of each past task and $P_{j}(x,y)$ the true data distribution of task $j$. Hence, when learning task $t$, the effective training distribution is

$$ $P_{train}^{(t)}(x,y)\approx\alpha_{t}P_{t}(x,y)+(1-\alpha_{t})P_{past}(x,y),$ (3) $$

where $\alpha_{t}\in[0,1]$ is a balancing factor that weights the newly introduced and replayed data, usually emphasizing the current task.

#### Sufficient exemplar memory mitigates catastrophic forgetting

Figure: (a) CIFAR-100
Refer to caption: /html/2502.07274/assets/x5.png

Next, we discuss how a sufficiently large exemplar memory buffer $\mathcal{M}$ approximates previous tasks’ distributions, thus reducing inter-task forgetting.

Ideally, if we could train jointly on all tasks $1,\cdots,t$, the obtained model parameter $\theta^{*}_{1:t}$ would minimize the risk:

$$ $R_{1:t}(\theta)=\mathbb{\sum}_{j=1}^{t}\mathbb{E}_{(x,y)\sim P_{j}}\bigr{[}\ell(\theta;x,y)\bigr{]},$ (4) $$

which minimizes inter-task forgetting by design. By joint-training on all task data up to task $t$, we would find $\theta^{*}_{1:t}=\operatorname{arg}\operatorname{min_{\theta}R_{1:t}(\theta)},$ with no conventional notion of forgetting between tasks.

Similarly, with a large enough exemplar memory, the replayed data for previous tasks closely approximates their true distributions ($\tilde{P}_{j}\approx P_{j}$ for all $j<t$). Therefore training on $D_{t}\cup\mathcal{M}_{1:t-1}$ yields a risk

$$ $\tilde{R}_{1:t}(\theta)\approx\alpha_{t}\mathbb{E}_{(x,y)\sim P_{t}}\bigr{[}\ell(\theta)\bigr{]}+(1-\alpha_{t})\mathbb{\sum}_{j=1}^{t-1}\hat{\pi}_{j}\mathbb{E}_{(x,y)\sim\tilde{P_{j}}}\bigr{[}\ell(\theta)\bigr{]},$ (5) $$

where we may bound $|\tilde{R}_{1:t}(\theta)-R_{1:t}(\theta)|\leq\epsilon$ under standard assumptions (e.g., Lipschitz continuity in $\theta$ (Khromov & Singh, 2024)), with $\epsilon$ shrinking as the replay buffer size $K$ increases, leading to $\tilde{P_{j}}\approx P_{j}$. A small gap between $\tilde{R}_{1:t}(\theta)$ and $R_{1:t}(\theta)$ implies that $\lVert\tilde{\theta}^{*}_{1:t}-\theta^{*}_{1:t}\rVert$ is also small, where $\tilde{\theta}^{*}_{1:t}=\operatorname{arg}\operatorname{min}_{\theta}\tilde{R}_{1:t}$. Intuitively, if the two risk surfaces are proximate, their minimizes are also close in the parameter space (Fornasier et al., 2021).

In this sense, inter-task forgetting on a previous task $j$ arises when $\tilde{\theta}^{*}_{1:t}$ drastically changes predictions on the previous task distribution $P_{j}$. But if $\tilde{\theta}^{*}_{1:t}$ remains near $\theta^{*}_{1:t}$ (which, by definition, does well on previous tasks by training jointly), it must still perform well on task $j$. Hence, if we measure inter-task forgetting in the parameter space as

$$ $\Delta_{j\rightarrow t}=\mathbb{E}_{x\sim P_{j}}\bigr{[}\ell(\tilde{\theta}^{*}_{1:t};x)-\ell(\theta^{*}_{1:j};x)\bigr{]},$ (6) $$

the measure of how replay-based parameters after task $t$ perform on task $j$ compared to the parameters after task $j$. Here, $\Delta_{j\rightarrow t}$ remains small if $\tilde{\theta}^{*}_{1:t}\approx\theta_{1:t}^{*}$. Naturally, since $\theta^{*}_{1:t}$ is a reliable minimizer on all tasks, the small parameter drift ensures that the performance of the model trained under sufficient exemplar memory does not degrade on earlier tasks - i.e., inter-task forgetting is reduced as exemplar memory becomes larger (Merlin et al., 2022; Brignac et al., 2023).

#### New challenges under sufficient exemplar memory

While the sufficient exemplar memory regime mitigates inter-task forgetting by closely approximating past data distributions, it also creates a hybrid training dataset that combines current task data $D_{t}\cup\mathcal{M}_{1:t-1}$ with replayed samples from previous tasks.

This combined dataset introduces several challenges. First, gradient interference may occur as the gradients computed on current task samples can conflict with those from past tasks, leading to partial cancellation of updates and impeding effective learning across tasks. We view that this potentially could lead to a new form of forgetting that distorts the learned features and ultimately interfering with the model’s learning process. Second, the heterogeneous nature of the hybrid dataset increases the variance of the stochastic gradient estimates, resulting in slower convergence. This heightened variance necessitates either more iterations or more sophisticated optimization techniques to minimize the loss reliably. Furthermore, imbalances in task representation can arise if the replay buffer unevenly captures the diverse distributions of past tasks.

Conventional continual learning methods typically address forgetting by assuming that new data is markedly different from prior data and focusing on preserving performance on previously learned tasks. However, in the sufficient exemplar memory regime, where all tasks are presented simultaneously within a combined dataset, these methods fall short. They are not equipped to handle the multi-task learning dynamics and the associated balancing issues that emerge when the model must integrate and harmonize learning signals from a diverse set of tasks. We believe a more thorough investigation is required, and we set this as a key objective of our future work.

### A.2 Method (continued.)

In this section, we elaborate on our method, highlighting the role of each component. The proposed method is a combination of two weight-space operations. (1) Weight Reset (Yadav et al., 2024) and (2) Weight Averaging (Wortsman et al., 2022).

Figure: Figure 4: Weight Reset Experiment on CIFAR-100. Most parameters are redundant in learning new tasks.
Refer to caption: /html/2502.07274/assets/x7.png

#### Weight Reset

The weight reset step selects and resets parameters that were redundant in adapting to the new task. The aim of this procedure is to find the minimal set of parameters that are capable of adapting to the new $t^{th}$ task, and reverting the redundant parameters to the value of the previous task model $\theta_{prev}$, which helps recover learned features. In this process, we use a simple distance metric to measure a model’s contribution to the new task, formulated as:

$$ $\big{|}\theta[l]-\theta_{prev}[l]\big{|},$ (7) $$

where $\theta[l]$ and $\theta_{prev}[l]$ indicates the $l^{th}$ parameter of the current model $\theta$ and the previous task model $\theta_{prev}$, respectively. Using this metric, we retain the top-$Q\%$ parameters that have largely drifted during the current task and reset the dormant parameters using [eq.˜1](#S4.E1). The idea of resetting model weights is not novel in the continual learning literature, but most focused on improving plasticity (Dohare et al., 2024). Conversely, we improve both plasticity and stability using a mixing technique ([eq.˜1](#S4.E1), integrating previous and present tasks. In [section˜A.4](#A1.SS4), we provide experimental results on the effect of $Q$, the percentage of weight reset on model performance (see [fig.˜4](#A1.F4)).

#### Weight Average

The weight-averaging step aggregates various learned signals into a single model, allowing for faster convergence and improved generalization (Izmailov et al., 2018). The underlying idea is that the model weights can address catastrophic forgetting, functioning as a substitute for replay memory. Using this, we aim to fill the gap between the full exemplar memory setting (i.e., joint-training) and the sufficient exemplar memory setting. Another motive behind this approach is the new challenges (see [section˜A.1](#A1.SS1)) rooted in training with a mixture of datasets $D_{t}\cup\mathcal{M}_{1:t-1}$, which generally requires more training steps for convergence. In [Section˜A.4](#A1.SS4), we empirically show that the sufficient exemplar memory setting complicates training, requiring more training epochs for convergence (see [Figure˜5](#A1.F5)). The weight averaging technique is also an emerging practice in the continual learning literature (Marouf et al., 2025; Kozal et al., 2024). However, such works merge the model weights after training, which requires the storage of multiple model weights (proportional to the number of tasks). On the other hand, our method uses a moving averaged model that is updated during training, reducing the cost of storage.

### A.3 Measuring Parameter-Drift

In our work, we measure a parameter’s contribution to learning a new task by comparing its parameter drift ([eq.˜7](#A1.E7)). However, there are a number of alternative approaches we could take. For instance, we can differentiate between how a parameter changes (1) between tasks (2) within a task.

#### Inter-task Parameter Drift

When transitioning from one task to the next, the change in the $l$-th parameter can be measured as

$$ $\Delta\theta_{\text{inter}}[l]=\bigl{|}\theta_{t}[l]-\theta_{t-1}[l]\bigr{|},$ $$

where $\theta_{t}[l]$ denotes the $l^{th}$ parameter after training on the current task $t$, and $\theta_{t-1}[l]$ represents the $l^{th}$ parameter after training on the previous task $t-1$. A large value indicates that the parameter is highly task-specific, while a small value suggests robustness across tasks.

#### Intra-task Parameter Drift

Alternatively, we can analyze how a parameter evolves during the training process of a single task. Let $\theta^{(i)}[l]$ denote the value of the $l^{th}$ parameter at the $i^{th}$ iteration during training. Then, the intra-task parameter drift is given by

$$ $\Delta\theta^{(i)}_{\text{intra}}[l]=\bigl{|}\theta^{(i+1)}_{t}[l]-\theta^{(i)}_{t}[l]\bigr{|}.$ $$

This measure captures the incremental updates of the parameter as the model optimizes its performance on the current task. By comparing both the inter-task and intra-task parameter changes, we gain a more comprehensive understanding of the role each parameter plays in adapting to new tasks as well as the dynamics of learning within a single task. In our future work, we will seek a more reliable metric to express a parameter’s behavior in the weight space.

### A.4 Experimental Results and Analysis

In this section, we continue our discussion of the experimental results.

Table: Table 2: Ablation study on CIFAR-100

We begin with the ablation study. In this experiment, we perform an ablation study on the two components that our method utilizes, (1) the weight reset and (2) the weight average procedure. In [table˜2](#A1.T2), we report the results of the experiment. Here, w/o reset indicates the replay method combined with weight average (hence without reset), while w/o avg. indicates the replay method added with the reset procedure (i.e., without avg.). The replay method is displayed to show the case when neither weight reset nor weight average is utilized. Our method showed the strongest results compared to its ablations. Notably, we can see that weight reset alone (w/o avg.) does not show strong gains. However, when the weight reset is combined with the weight average method, it allows significant performance gains.

Another experimental result is the weight reset experiment in [fig.˜4](#A1.F4). Here, we can see that approximately only $20\%$ of the model parameters are responsible for the model’s learning process (i.e., resetting $80\%$ of the parameters does not pose significant issues). In contrast, when the percentage drops to $10\%$ or less, the model’s task accuracy drops by a large margin. We further discover that the weight average step helps the model to become even more sparse (i.e., not reliant on individual parameters), improving the model performance under extreme resetting settings.

Table: Table 3: Average Acc. on CIFAR-100 with different exemplar memory sizes (full)

Figure: Figure 5: The training loss under a sufficient exemplar memory continual learning setting. When training with sufficient exemplar memory, convergence becomes difficult.
Refer to caption: /html/2502.07274/assets/x8.png

Lastly, we show that under sufficient exemplar memory, convergence becomes difficult, as discussed in [section˜A.1](#A1.SS1.SSS0.Px3). The results are illustrated in [Figure˜5](#A1.F5). Specifically, we analyze the training loss of the model under two cases. (1) Continual Learning: train the model across 20 tasks sequentially, with sufficient exemplar memory. (2) Joint Training: train individual models for each task using all known task data. For this experiment, we used the CIFAR-100 dataset divided into 20 subtasks. In [Figure˜5](#A1.F5), we can see that under sufficient exemplar memory, converging to the training data becomes difficult, especially for the continual learning model. While a simple solution would be to extend the training epochs for convergence, it would collide with our aim for cost-efficiency. On the other hand, joint-trained models relatively converge better. This result aligns with the results of Dohare et al. (2024) that a model sequentially trained on different tasks (i.e., continual learning) suffers a loss of plasticity (i.e., a model’s ability to learn new tasks). We believe a thorough investigation of this new phenomenon is required.

### A.5 Experimental Details

In this section, we report the experimental details of our experiments. In all our experiments, we have used the PyCIL (Zhou et al., 2023) library, which allows easy replication. We followed the standard training hyperparameters of the PyCIL library, which are fixed across experiments. Regarding unique hyperparameters, the average interval $j$ was set as $5$, and the warming epoch $n_{warm}$ was set as $25\%$ of the total training epochs (default set as $70$ under the PyCIL setting). $j$ and $n_{warm}$ were selected using a grid search. Please refer to [Section˜4](#S4) for a better understanding of each hyperparameter. Lastly, regarding the computing resources, we used a single NVIDIA RTX 6000 GPU for all experiments. For our experiments, we used the 2.2.1 version of Pytorch (Paszke et al., 2019).

### A.6 Related Works (continued.)

#### Weight Space Operations

Recent works have shown that manipulating model parameters directly with weight-space operations (e.g., model merging (Wortsman et al., 2022)) can handle multi-task learning Yu et al. (2024) and continual learning (Marouf et al., 2025; Marczak et al., 2025) in a more principled way. These techniques usually intervene post-training by merging the weights of different models e.g., Yadav et al. (2024) suggested a selective merging algorithm that mitigates the interference between different models, while Ilharco et al. (2022) showed that arithmetic operations on the weight space can edit the model without training. Unlike these post-training interventions, our approach manipulates the model’s weight space amidst training (Izmailov et al., 2018; Jang et al., 2025) without storing multiple model parameters, aiming for cost-effective editing of the continual learner. Another relevant yet overlooked topic is the effect of weight-space operations on model attributes e.g., loss landscape (Li et al., 2018; Kaur et al., 2023) and plasticity (Dohare et al., 2024), that contribute to continual learning and generalization. This work empirically investigates various aspects of the model to study their effect on the model’s ability to handle distribution shifts. In the continual learning literature, several works have adopted weight-space operations to obtain multi-task solutions without joint training. For instance, Kozal et al. (2024) has suggested the use of weight averaging techniques for continual learning, and Marczak et al. (2025) has extended the idea using task arithmetic. However, these approaches merge models post-training and requires the storage of multiple model weights during training. On the other hand, our approach utilizes weight-space operations amidst training, without the redundant storage of multiple model weights. We view this as an important difference in modern settings where the model size is exponentially growing.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: Q1. Can plasticity be measured as a metric on an eval set? Currently, plasticity loss is provided in the graph.
soundness: 2
strengths: Pros:

1. The overall point about gpu compute constraints being more than storage is good. Exploration of such middle-ground strategies makes sense to me. The core idea of reseting certain parameters for plasticity also is interesting.

2. In the high-sample regime, the proposed method almost always outperforms existing baselines by 2-3 points. 

3. The paper has nice ablation experiments such as comparing replay with reset (table 3), reset strategies (table 5). The plasticity loss experiment (Fig2) in particular suggest the issue with lower plasticity with increasing size of past examples. The additional experiments in the supplementary are appreciated.
summary: Brief Summary: The paper tackles the task of continual learning. The authors suggest gpu compute cost is the main constraint and find the problem is with plasticity where bottleneck is on struggling to learn new tasks. To address this, authors propose Weight Space Consolidation involving rank-based parameter resets and weight averaging. Experiments are conducted on both image-classificaiton tasks (cifar-100) and llm instruction tuning (trace) showing improvement at significantly reduced costs.
weaknesses: Cons:

1. My main concern is with the framing of the paper for continual learning and its potential application. The point about GPU cost being dominant with storage is fine, but the main problem is that the data itself might not be available in the first place. So storage was never the real issue, it is access to data. For a practical example, assume we have llama-3.2b instruct model but we don't know what data was used in the base to instruct training. Here, the authors are essentially assuming we have access to the underlying data, which is not the case. A few other points:

(i) While storage cost is not a problem, the cost of downloading setting up s3 buckets high throughput speed for gpu access are all real costs. These need to be highlighted. 

(ii) If the authors are motivating it based on cost, some experiments on exact cost saved for practical reference should be provided.

This is not to say the original point about GPU vs storage cost isn't correct, but that storage is not the only factor, data access itself is a big one. The entire argument essentially dilutes the author's novelty.

2. It seems a naive full-fine-tuning baseline is missing in Table 2 ? Difference between proposed method and full-fine-tuned on the same set would be very helpful. 

3. Authors primarily explore task-based paradigm only, but it seems the method might be more impactful in task-free settings [Ref1]. 

4. For LLM datasets, authors only consider TRACE dataset, so tough to know the generalizability.. Given that authors are exploring image-based and llm-based separately, it might be worth repeating experiments on multi-modal datasets as well such as on VisCOLL [Ref2]. 

5. (Minor) Some qualitative visualizations would be very helpful.

6. (Minor) For Trace experiments, the training hyper-parameters are underspecified. What is the fine-tuning method for llama-3.2-1B? Some experiments using LoRA adaptors would also be interesting. 

---
[Ref1]: Aljundi, Rahaf, Klaas Kelchtermans, and Tinne Tuytelaars. "Task-free continual learning." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11254-11263. 2019.

[Ref2]: Jin, Xisen, Junyi Du, Arka Sadhu, Ram Nevatia, and Xiang Ren. "Visually grounded continual learning of compositional phrases." arXiv preprint arXiv:2005.00785 (2020).


---

Overall Rating: 4/10

The main framing of gpu vs storage cost is not really novel. While the authors get 2-3 points improvement in high-sample regime, it is unclear how interesting that is. Authors miss key experiments such as full-fine-tuning baselines, only considering task-based not task-free, and only one trace dataset for llm experiments.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: See "Weaknesses." What are the technical contributions (s) of this approach, if the A and B components were previously proposed and employed by existing works?
soundness: 3
strengths: + The paper investigates a new paradigm/scenario that challenges the previous assumptions. Such new thinking is always valuable.
+ The mathematical formulations are rigorous and I discovered zero mistakes there.
+ The new paradigm is not just an "assumption," they have evidence (Section 3) to empirically demonstrate that the new assumption is valid.
summary: The authors challenge the assumption of traditional CL. Instead, they argue that, instead of the storage, the GPU time is the main bottleneck. They investigated the scenario where memory is sufficient enough to mitigate forgetting, but full retraining from scratch remains the main challenge. As the authors have discovered that models become biased toward prior tasks and struggle to learn new tasks, they propose Weight Space Consolidation, a lightweight method that combines rank-based parameter resets to restore plasticity with weight averaging to enhance stability.
weaknesses: - While the mathematical formulation is rigorous, the theoretical foundation on why this new approach would work is lacking.
- The proposed approach reads like A (rank-based parameter resets) + B (weight averaging). A + B isn't always necessarily bad, but both A and B are the results of prior work, so what's the technical contribution here with this approach?

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: - Why not compare against better memory-based methods, like for example:
    - Buzzega, Pietro, et al. "Dark experience for general continual learning: a strong, simple baseline." Advances in neural information processing systems 33 (2020): 15920-15930.
        - It has been shown to scale well with the number of tasks.
soundness: 4
strengths: - The paper is well motivated and written. The authors raise the challenge that, in current systems, memory is not the most important constraint, as GPU time is more costly. Presenting references and numbers, the paper encourages researchers not to focus on minimising memory size.
    - However, there are scenarios where having a small memory is required, such as when resource constraints are imposed or when the privacy of previously learned data is an issue. These are not mentioned or explained in the paper.
- Along with raising the challenge and presenting a new scenario, the authors analyse the limitations of naively increasing memory size and propose a new approach to increase plasticity and enhance stability in this new scenario.
- The experiments and results clearly help to demonstrate what is discussed in the text. Experiments across multiple benchmarks and modalities provide robustness to the results, and ablation experiments help understand the method's limitations and provide further insights into how it works.
summary: Memory-based methods in Continual Learning focus on learning a sequence of tasks with help of a buffer, which store samples from previous tasks and use them when learning the current tasks. Normally, this buffer is limited in the amount of samples they can store due to some constraints of the environment. In this paper, the authors challenge this constraint mainly because storage costs much less than training time. However, as we increase the samples we store, models could face a different challenge, and instead of having forgetting (too much plasticity) this shifts to a high stability and not learning the current task enough. Considering all this, this paper proposes Weight Space Consolidation, which combines a reset strategy to improve plasticity and weight averaging to enhance stability. Experiments on multiple modalities show good performance of the proposed method as they increase the memory budget. Also, multiple ablation experiments help understand why and how the method performs.
weaknesses: - A common approach to increase the plasticity of memory-based methods is to concatenate buffer samples with current-task samples at the batch level. This is different from what is shown in the paper, where the full batch is sampled from the intersection of the current data and the buffer. As shown in the paper, this last approach suffers from plasticity because it is least likely to sample data from the current task; however, by concatenating at the batch level, there is no plasticity problem.
    - How does this sampling method affect the scenario, results and the proposed method? 
    - Concatenating at the batch level can be more costly (in terms of GPU time), but it may mean not keeping a copy of the model in memory, which can increase the batch size.
- Figure 2b is unclear. The orange line makes it impossible to see the blue lines and compare them.

---

confidence: 3
contribution: 2
initial_rating: 2
presentation: 4
questions: I have several questions/ideas for improvement
- You use a rank-based approach --> where are the ranks (i.e., matrix ranks) used here? Do you mean "ranking based approach"?
- Equation 2 and 5 both use $\alpha$ for different jobs
- Equation 2: how is alpha annealed (l 232)
- Equation 5: describe that the [l] are the weights to be reset
- Figure 1: are the memory sizes per class or total? Additionally give percentage
- L 460: Q versus q (in the top-q%)
- Figure 3 and 405ff: how is VRAM usage measured? Do not modern frameworks pre-allocate as much GPU memory as possible (i.e., aiming at 100% GPU memory utilization)
soundness: 3
strengths: Strengths:
+ novel take on memory buffers in CL
+ simple and effective method to improve plasticity under performance-stable regimes
+ image and text modalities tested
+ easy-to-follow description of algorithm
summary: Traditionally, continual learning has been studied under the memory-restricted setting (i.e., completely exemplar-free or only limited buffer sizes allowed). Authors argue that this is unrealistic under modern standards: compute, and not memory-costs, are the main drivers for the high costs related to model training. Thus, authors study the setting where larger-than-common buffer sizes are used, and where instead the compute availability is the driving factor for comparison. They find that, as the memory buffer grows and thus allows storing more samples of prior tasks, the challenge in CL shift from forgetting to plasticity. Models are stable wrt old tasks, but not plastic enough to new tasks. To alleviate, they propose using stochastic weight averaging and resetting weights with relatively little gradient activity. They study their method, termed Weight Space Consolidation, under the class-incremental setting on CIFAR100, ImageNet-100, and text-based TRACE benchmark, where competitors are outperformed.
weaknesses: - l 311 to 319: this is a central part of the proposed method, but no experimental results support this thesis
- Paper claims less compute overhead from their method, but no tabular results indicating the overhead versus naive replay are given
- if we can store samples of old tasks, then we could, in turn, learn longer task sequences. Paper should integrate longer task sequences (e.g., TinyImageNet split into 20 or 40 tasks)
- Full re-training is often used as comparison, but missing from the tables
- Its unclear how "from scratch" is interpreted. Is that an entirely fresh model that's trained on increasing unions of memory buffer? Then, no knowledge would be transferred by design
- Hyperparameter tuning on ResNet32 and then using tuned params on ResNet18 seems strange (l 364). Please elaborate on this.
- I do not agree with the definition of post-hoc merging versus in-training merging (l292ff). Post-hoc merging would be, in my understanding, merging after Training on a task, which is thus before a next task -- which is still "during training".


Decision: Accept (Poster)