Revisiting Weight Regularization for Low-Rank Continual Learning
Authors: Yaoyue Zheng
Yin Zhang
Joost van de Weijer
Gido M van de Ven
Shaoyi Du
Xuetao Zhang
Zhiqiang Tian
Venue: ICLR 2026 Poster (2026)
Keywords: Continual Learning
Class-incremental Learning
Weight Regularization
Elastic Weight Consolidation
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: EWC-LoRA revisits weight regularization by constraining shared low-rank updates to mitigate task interference without increasing model size.

Abstract: Continual Learning (CL) with large-scale pre-trained models (PTMs) has recently gained wide attention, shifting the focus from training from scratch to continually adapting PTMs. This has given rise to a promising paradigm: parameter-efficient continual learning (PECL), where task interference is typically mitigated by assigning a task-specific module during training, such as low-rank adapters. However, weight regularization techniques, such as Elastic Weight Consolidation (EWC)-a key strategy in CL-remain underexplored in this new paradigm. In this paper, we revisit weight regularization in low-rank CL as a new perspective for mitigating task interference in PECL. Unlike existing low-rank CL methods, we mitigate task interference by regularizing a shared low-rank update through EWC, thereby keeping the storage requirement and inference costs constant regardless of the number of tasks. Our proposed method EWC-LoRA leverages a low-rank representation to estimate parameter importance over the full-dimensional space. This design offers a practical, computational- and memory-efficient solution for CL with PTMs, and provides insights that may inform the broader application of regularization techniques within PECL. Extensive experiments on various benchmarks demonstrate the effectiveness of EWC-LoRA, achieving a stability-plasticity trade-off superior to existing low-rank CL approaches. These results indicate that, even under low-rank parameterizations, weight regularization remains an effective mechanism for mitigating task interference. Code is available at: https://github.com/yaoyz96/low-rank-cl

Published as a conference paper at ICLR 2026
REVISITING WEIGHT REGULARIZATION FOR Low- RANK CONTINUAL LEARNING
Yaoyue Zheng1.2.3 Shaoyi Du Xuetao Zhang' Zhiqiang Tian 6*
Yin Zhang Joost van de Weijer Gido M van de Ven
1 State Key Laboratory of Hluman-Maching Hybrid Augmented Intelligence
Institute of Artificial Intelligence and Robotics, Xi'an an Jiaotong University, Shaanxi, China
2 Computer Vision Center, Barcelona, Spain
3 Universitat Autônoma de Barcelona, Barcelona, Spain
School of Instrument Science and Engineering, Harbin Institute of Technology, Harbin, China
5 Bernoulli Institute, University of Groningen, the Netherlands
6 School of Software Engineering, Xi an Jiaotong University, Shaanxi, China
ABSTRACT
Continual Learning (CL) with large-scale pre-trained models (PTMs) has recently gained wide attention, shifting the focus from training from scratch continu- ally adapting PTMs. This has given rise to a promising paradigm: parameter- efficient continual learning (PECL), where task interference is typically mitigated by assigning task-specific module during training, such as low-rank adapters. However, weight regularization techniques, such as Elastic Weight Consolidation (EWC)-a key strategy in CL-remain underexplored in this new paradigm. In this paper, we revisit weight regularization in low-rank CL as a new perspective for mitigating task interference in PECL. Unlike existing low-rank CL methods, we mitigate task interference by regularizing a shared low-rank update through EWC, thereby keeping the storage requirement and inference costs constant regardless of the number of tasks. Our proposed method EWC-LoRA leverages a low- rank representation
estimate parameter importance over the full-dimensional space. This design offers practical, computational- and memory-efficien solu- tion for CL with PTMs, and provides insights that may inform the broader applica- tion of regularization techniques within PECL. Extensive experiments on various benchmarks demonstrate the effectiveness of EWC-LoRA, achieving stability- plasticity trade-off superior to existing low-rank CL. approaches. These results indicate that, even under low-rank parameterizations, weight regularization re- mains effective mechanism for mitigating task interference. Code is available at: htttt::://// //ittbu wtt.://www....o.o/////////// rank- cl
INTRODUCTION
Continual Learning (CL) has emerged as rapidly growing research area, aiming to enable machine learning systems acquire new knowledge without forgetting previously learned concepts (Parisi et al. 2019). This ability plays a crucial role in addressing real-world problems (Shaheen et al., 2022 Wang al.] 20224)) where data distributions are constantly changing. Ideally, CL model should be able to maintain stable performance across all previously encountered tasks. A significant decline in performance on previous tasks after learning new ones known as catastrophic forget- ling (McCloskey & Cohen 1989 Ratelifi 1990), which typically arises from task interference.
With the rise of large-scale pre-trained models (PTMs) (Bommasani 202T Awais al., 2025). the research focus in CL has shifted from training models from scratch to continually adapting these powerful models (Ostapenko et al. 20222) Yang et al., 2025). This trend is driven by the impressive transferability and robustness of PTMs, and a growing body of work has shown promising results in PTM-based continual adaptation. A particularly popular paradigm is parameter-efficiee continual
"Corresponding author.

---

Published as a conference paper at ICLR 2026
learning (PECL) (Qiao & Mahdavi 2024). in which the PTM is typically kept frozen and augmented
with lightweight modules such as prompts (Wang et al. 2022c; Smith et al. 2023). adapters (Ermis)
et al. 2022 Gao et al. 2024), or low-rank adaptations (LoRA) Liang & Li 2024 Wu et al. (025).
The predominant strategy in these works is to prevent task interference by assigning task-specific
modules during training- either structurally isolated adapters or LoRA modules, prompts that
provide task-specific conditioning at the feature level.
On the other hand, weight regularization, key continual learning strategy, remains underexplored in the era of continual learning with PTMs. A canonical example is Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017). which has played a central role in combating catastrophic for- getting in small-scale models (Schwarz et al. 2018 Liu et al., 2018 Ehret et al. 2020). Although effective for smaller models, EWC difficult to apply to PTMs, as estimating parameter importance via the Fisher Information Matrix (FIM) is computationally expensive, requiring storage of frozen copy of the old model and Fisher matrix of equal size, resulting in a memory overhead three times that of the original model. Several studies have attempted to apply EWC to the fine-tuning of large language models (Xiang et al. 2023 Sliogeris et al. 2025). However, they typically fine-tune the model with precomputed Fisher matrix that fixed throughout training, making them impractical for CL..
In this paper, we adopt EWC as a canonical example to study weight regularization in low-rank CL, systematically analyzing key considerations for applying EWC within low-rank adaptations and proposing a feasible veighhtregullrizzaton---es regularization in low-rank CL as a new perspective mitigating catastrophic forgetting. Existing low-rank CL methods assign each task an independent LoRA module, constraining updates to sub- spaces that reduce interference with prior tasks. While effective, the addition of LoRA modules incurs storage overhead that scales linearly with the number of tasks. In contrast, we mitigate task interference by regularizing a shared low-rank update through EWC, rather than structurally iso- lating task-specific parameters, thereby keeping the storage requirement constant regardless of the number of tasks. Moreover, we provide the first systematic investigation of EWC in low-rank CL, and we theoretically and empirically demonstrate that a naïve integration of EWC with low-rank adaptation is suboptimal. To address this limitation, we propose a principled method to estimate parameter importance within the low-rank space. Our method leverages full-dimensional FIM to reliably estimate the importance of parameters in low-rank updates. We empirically show that the proposed regularization on low-rank matrices achieves better and more flexible stability-plastiche
solution for low-rank CL. First, we revisit weight
trade-off than existing low-rank methods (Wang et al.) [202cc] Liang & Li 2024: Wu et al. 2025). which are typically constrained to fixed operating point.
Drawing on these insights, we propose EWC-LORA, which updates the model via low-rank adap- tation while leveraging the full-dimensional space FIM for weight regularization. EWC-LoRA does not explicitly fine-tune the full model or store model components for all previous tasks, thereby sig- nificantly reducing computational and memory overhead while enabling effective Fisher estimation, making it a resource-efficient solution for CL with PTMS. The main contributions of this work are as follows:
We revisit weight regularization a new perspective for mitigating catastrophic forgetting in low-rank CL. By exploiting the low-rank structure, we develop an efficient realization of EWC in PTMs. Specifically, by regularizing a shared LoRA module, EWC-LoRA main- tains a constant memory footprint regardless of the number of tasks.
We present the first systematic investigation of EWC in low-rank CL and propose estimat- ing the FIM over the full-dimensional space to accurately capture parameter importance. Our analysis shows that naively integrating EWC with low-rank adaptation is suboptimal, while EWC-LORA effectively regularizes learning and achieves better stability-plasticity trade-off than existing low-rank CL methods.
Extensive experiments across multiple benchmarks demonstrate that EWC-LoRA is effec- tive, improving over vanilla LoRA by average of 8.92%, while achieving comparable or even superior performance to state-of-the-an low-rank CL methods, with better computa- tional and storage efficiency.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORKS
Continual Learning (CL). In contrast to standard supervised learning, which assumes that train- ing data are independent and identically distributed (i.i.d.), CL focuses on training models data streams that exhibit non-stationary and often continuous distribution shifts (Lesort et al., (2021). This departure from the i.i.d. assumption introduces the central challenge of catastrophic forget- ting, where the model experiences significant performance degradation on previously learned tasks as new tasks are introduced (McCloskey & Cohen, 1989, Ratclifi 1990). Following Van de Ven et al. (022), CL can be categorized into three main scenarios: task lusk-incremental (Gao et al. 2023), comain-incrementt
(Wang et al.) 2024b), and glass-incremente learning (Hersche al., 2022). Among these, class-incremente learning can be considered the most challenging. In this work, we adhere to the class- incremental learning setting, where the model must learn to distinguish between all classes encountered across all tasks (Masana et al. 2022).
Crrameter-Efficienn Continual Learning (PECL). PECL QQia & Mahdavi 2024) has recently emerged a promising paradigm in CL. It builds upon the idea of parameter-efficiee tuning (Houlsby et al., 2019 Hu et al., 2022, Jia et al. 2022), where a pre-trained model is kept introduced to adapt to new tasks. To pre-
fine-
frozen and a small number of learnable parameters vent task interference, existing PECL methods can be categorized into two types: (1) prompt-based methods, which typically provide each task with task-specific prompts condition PTMs at fea- ture level (Wang et al. 2022c b Smith et al. (202)). and (2) adapter- or low-rank adaptation-
methods, which typically insert task task- specific hightweihht modules during training, thereby providing isolation at the structural level (Gao et al. 2024 Liang & Li 2024] Dou et al. 2024 Wu et al. (0025) Qian et al., 2025). Existing PECL works thus primarlly focus on introducing task-specific modules to mitigate task interference, which often leads to increased memory and computational costs as the number of tasks grows. contrast, weight regularization techniques have received little attention, and it remains unclear how they can be effectively applied in PECL-a setting that presents unique structural and optimization challenges compared to full-model tuning. For regularization-base low- rank CL methods, O-LoRA (Wang et al., 2023) mitigates task interference by enforcing geometric orthogonalization in the update subspace, thereby preserving the direction of parameter updates. Within the context of low-rank CL, we revisit weight regularization as a means to mitigate task interference. In contrast to O-LoRA, we preserve the magnitude of updates through Fisher-based penalties, providing insights that may inform the broader application of regularization techniques within PECL an area that remains underexplored in the current literature.
Elastic Weight Consolidation (EWC). EWC (Kirkpatrick et al. 2017) mitigates catastrophic for- getting in CL by penalizing changes to parameters that are deemed important for previous tasks, as quantified by the Fisher Information Matrix (FIM). As canonical example of weight regularization techniques, EWC has inspired a series of follow-up up studies that aimed to address its limitations and broaden its applicability. For example, Huszáá (2018) analyzed its behavior beyond two tasks, while van de Ven (2025) investigated strategies for estimating the FIM in the context of CL. In the era of PTMs, et al. 2023) apply EWC during fine-tuning of a large language model (LLM), using a precomputed FIM to protect the knowledge acquired by the original model. Similarly, Šliogeris et al. 2025) employ EWC in the context of LLMs, estimating the FIM on comprehensive benchmark to preserve domain knowledge. However, both studies rely on a precomputed FIM, which is kept fixed throughout training. making them unsuitable for setting. Thede et al. (2024) briefly note the continued value of regularization in the context of PTMs, but they do not examine its detailed effect and do not combine regularization with low-rank adaptation. (Wei et al. (2025) do combine EWC with low-rank adaptation, but they separately regularize each low-rank module, causing inaccurate Fisher estimation and suboptimal performance. In this work, we conduct a focused investigation of EWC in PTMs-based CL. By leveraging a low-rank structure, we propose practical approach for adapting EWC
CL and demonstrate its effectiveness.
METHODOLOGY
In this section, review the necessary preliminaries, then discuss the structural and optimization challenges of applying EWC to low-rank adaptation, and finally present an overview of EWC-LORA, highlighting its learning procedure and differences from existing low-rank CL methods.
3

---

Published as a conference paper at ICLR 2026
3.1 PRELIMINARIES
Notations. In this paper, bold lowercase letters represent vectors, while bold uppercase letters denote matrices. The superseript T indicates the transpose of matrix, and [E[]] stands for the expectation operator. Optimal values of variables are indicated with a superscript
Problem Formulation. We start with pre-trained model parameterized by Wo and fine-tune it sequentially on series of new tasks (Tr)=1 with corresponding datasets (D, (D.J/= For each
task Tr, the model receives batch of samples { v k- drawn from classes, where z and vi denote the input image and its corresponding label, respectively. After completing training on Tr, the model is evaluated on all so-far encountered tasks T1.. The objective to learn parameters W that generalize well across all tasks far, without storing any past data. With the model parameterized by W. the training loss function at task Te is usually defined as:
12d
(W)
bi = (y=
Elastic Weight Consolidation. Following (Huszár 2018). we maintain single penalty term that approximates the combined effect of all previous tasks, preventing the double-counting in- herent in the multi centi-peenaty approach. When learning on task Tú, we approximate the posterior pl (( using the Laplace approximation (MacKay 1992), forming a Gaussian distribution N(W; W (F ) where W; denotes the optimal parameters on task TI:-I, and accumulated Fisher matrix Frum serves the precision matrix, reflecting the importance of each parameter for retaining knowledge from all previous tasks. To improve computational efficiency, EWC assumes parameter independence and retains only the diagonal elements of the Fisher matrix. During training on Tr the loss function in Eq. [ is augmented with a quadratic penalty term that constrains important parameters remain close to their previously learned values:
C,( = (,(W) + (W- -
(W-W? I diag ( (W - WW,1)
(2)
where À is hyperparameter that controls the relative importance of the new task compared to the old one(s). diag(F denotes the diagonal matrix formed from the diagonal elements of the accumulated Fisher matrix. Hereafter, unless otherwise stated, F refers to the diagonal Fisher matrix, with the diag(-) omitted for simplicity.
Low-rank Adaptation (LoRA). When fine-tuning model, LoRA (Hu al. 2022) restricts changes to the parameters to lie in a low-rank subspace. Suppose W, Wo E Rdoxa, are the adapted and pre-trained weight matrix, respectively. LoRA expresses the weight update AW W W. as the product of two learnable matrices A Roo and B E Rexds, with & min do)- Thus, the adapted weight matrix can be represented as W WT W.+WW Wo AB. During fine-tuning, the pre-trained weights Wo are frozen and only the parameters of A and B are trainable.
3.2 EWC WITH LOW-RANK ADAPTATION
The core idea of EWC in Eq. 2 lies in the second term, which measures how far the current pa- rameters deviate from the previously learned ones. Directly applying EWC entails fine-tuning all parameters, as well as preserving both a frozen copy of the old model and a Fisher matrix of the same size. However, in the case large PTMs, this is often not feasible. To reduce the number of trainable parameters and improve efficiency, we represent the weight update as AW W. W AB low-rank decomposition.
To regularize low-rank matrices, straightforward way is to compute individual Fisher matrices for A and B, and apply regularization to each accordingly. This is the approach of Wei et al. 2025). However, focusing solely the individual low-rank matrices ignores the interaction be- tween and B, which is problematic because each element of the update AW depends on their joint product, i.c., AW, i Ei= By In Appendix A.I.1] we mathematically prove that regularization performed separately in the low-rank space generally diverges from that performed in the full-dimensional space. Another way regularize low-rank matrices is to precompute the
4

---

Published as a conference paper at ICLR 2026
A specific branch for task
* Frozen
Trainable
Parameter Importance
O
Outputf
Regularization in Full-dimensional Space
Parameter
Fisher Matrix
Pre-trained Weights
A
Pre-trained Weights
regularize
d
AxB
W, Rond
B
B
W
IR
pm
Input
Input
(a) Multi Task-specific LoRA
(b) Shared LoRA with Weight Regularization (Ours)
Figure 1: Overview of learning task T, at specific layer of the VIT model. (a) Prior low-rank CL methods structurally isolate task-specific LoRA parameters by adding a new LoRA branch for each task. (b) The proposed EWC-LoRA employs a shared LoRA module that is learned across all tasks and regularized according to parameter importance measured by a Fisher Information Matrix, which updated after learning each task.
Fisher matrix on the pre-trained model and then apply the regularization to the update AW AB using this fixed Fisher matrix (Xiang et al. 2023). However, this estimation also includes directions associated with the frozen Wo, u. which introduce noise in measuring sensitivity to the loss. This issue is further illustrated in Appendix A.2 In Table I we can observe that the two naïve methods for regularizing low-rank matrices result suboptimal performance.
To address the above limitations, we propose updating parameters in the low-rank space while regu- larizing over the full-dimensional subspace of W spanned by A and B, illustrated in Figure (b). This ensures that the penalty captures the true sensitivity of the model output to low-rank updates, rather than merely the local gradient magnitudes. Consequently, reformulate Eq. 2 as:
À B), C.(A vec(AB) F vec(AB)
(3)
where vec(A ((BB is flattened AB. This formulation enables regularization in the full-dimensional subspace AW, without requiring explicit storage of vec( (B). After training on Tr. we estimate F, and incorporate into the previously accumulated Fisher matrix to obtain the updated Fisher Fçu..
To estimate the Fisher matrix F, at the optimal parameters W: for task Tr, we follow the definition in (Martens 2020). Specifically, the i-th diagonal element of F, is defined as:
F. EYNDD Bynoww: [
a @ Pw (ylæ) Dw;
2 )'I
w-w?
The outer expectation in Eq. Dis computed over the current task data D,, while the inner expectation approximated using the empirical Fisher for computational efficiency. In practice, this inner expectation can be estimated by taking the squared gradients of the log-likelihood with respect to W, evaluated at W;. Since only the low-rank update W is trainable, the gradient with respect to W and AW are identical. Consequently, the Fisher matrix is effectively computed in the AW- space. The equivalence between estimating the Fisher information the W-space and in the AW- space is established in Appendix A.... A.1.3
3.3 OVERVIEW OF EWC-LORA
Figure I illustrates the difference between EWC-LORA and existing state-of-the-art low-rank CL methods, in the context learning task Tr at a specific layer of the Vision Transformer (ViT). For task Tr, initialize the shared LoRA branch, and the forward computation is given by: f W + ABx. Specifically, A irro-initialized while B is drawn from a uniform distribution. During training Tr, only A and B are updated, while the base weights W.-1 keep frozen. After
5

---

Published as a conference paper at ICLR 2026
Table I: Comparison of different Fisher estimation strategies on CIFAR-100. "+ Mem." indicates the additional memory required for Fisher estimation and regularization during training.
Strategy
A1o
Avg.
Stability
Plasticity
Mem.
w/o F
82.99 (o.s4)
(9.74(0.58)
87 87.56(0.09)
98.86(0.09)
0 GB
Precomputed Fw
83.87 (0.21)
(9.36(0.49)
93.15(a.45)
94.74(0.56)
1 GB
Separate FA, ; FB F FAw (Ours)
86.41 (0.09)
(1.33
94.23
(0.47
4 GB
87.91
92.27 (0.39)
94.45 (0.59)
(7.90
6 GB
completing task Tr. the learned parameters are integrated to the base weight as We Wi-1+AB. For any sample from previous task, the forward computation becomes f = Wxx. During training, the update of the low-rank matrices A and B are regularized using the accumulated Fisher matrix F from step 1. Accordingly, EWC-LoRA maintains only two states after learning step t: (I) the updated model parameters W. obtained from the current task Tr, and (2) the accumulated Fisher matrix Frum, diagonal matrix that aggregates information from all previous tasks Ti:.. The dataset Dr and the task-specific Fisher matrix F, can then be discarded. For clarity, we outline the learning procedure of EWC-LoRA in Algorithm [ provided in (..2.. For computational and memory efficiency, the only additional memory cost of EWC-LoRA comes from storing the FIM. However, due the low-rank parameterization, EWC-LORA incurs only modest memory overhead while achieving higher computational efficiency than existing low-rank CL methods. A detailed efficiency analysis is provided in Section 4.2.
4 EXPERIMENTS
4.1 BENCHMARKS
Datasets.
In line with existing continual learning methods (Wang et al.
2023 Liang & Li 2024
Wu et al. 2025), we evaluate the performance of EWC-LORA under two settings: (1) Four widely used CL vision benchmarks, including CIFAR-100 (Krizhevsky al.)
2009). DomainNet (Peng) ct al. 2019 Wang et al., 2022a). ImageNet-R (Hendrycks et al.) 202 la Wang et al. 20226), and ImageNet-A (Hendrycks et al. 2001bb;; and (2) A standard language CL benchmark with five text classification datasets: AG News, Amazon Reviews, Yelp Reviews, DBpedia, and Yahoo Answers. Following Wang al. (2023), evaluate the models under three different task orders. The detailed task order is provided in Appendix AA.3] CIFAR-100 consists of 100 natural image classes and is the most commonly used dataset continual learning. DomainNet includes 345 classes across six diverse visual domains. making it a challenging multi-domain benchmark. Note that similar class-incremental learning problem. ImageNet-R contains 200 ImageNet classes (Deng et al.) 2009) rendered with various artistic styles, introducing significant distribution shifts. IeageNet-A consists of 200 natural adversarial examples that are frequently misclassified by standard ImageNet-trained models. We follow the mostly used task splits for continual learning benchmarks: CIFAR- 100 divided into 10 tasks (10 classes per task); DomainNet into 5 tasks (69 classes per task); ImageNet-A A into 10 tasks (20 classes each); ImageNet- R into 5, 10, and 20 tasks (with 40, 20, and 10 classes per task, respectively).
Liang & Li (2024). DomainNet is split
Evaluation metries. We adopt accuracy as our evaluation metric, in line with standard prac- tice (Lopez-Paz Ranzato 2017 Chaudhry et al. 2019). Let Ar, denote the classification accuracy on the 2-th task after training on the f-th task. The average accuracy at learning step t is defined as: A, Ei A The overall average accuracy is then computed as the mean of all intermediate averages: Avg. ΣT Ar, where T the total number of tasks.
To further analyze model's stability and plasticity and to enable intuitive comparisons across tasks, we propose measuring each metric on a normalized scale. Specifically, the stability score is defined as one minus the normalized forgetting:
T
max A - AT
Stability = 1 F=1
(5)
T
max A

---

Published as a conference paper at ICLR 2026
Table 2:
Comparison results on CIFAR-100, DomainNet, ImageNet-R, and ImageNet-A (in %).
Bold and underline indicate the highest and accoddhiihees, scores, respectively.
Tasks
CIFAR-100
DomainNet
ImageNNt-R
ImageNet-A
Methods
Aio (f)
Avg. (t)
As (1)
Avg. (?)
A1o (1)
Avg. (1)
A1o (t)
Avg. (*)
Joint Train
(2.82
(0.0)
(6.84
81. 81.25
81.69
(5.01 86.25
74.10(0.44)
Finetune
79.08
88. 38.17 (8.17
65.57 (0.20)
75. 75.12go.12
610.42
13.18
(4.55 $2.86(1.55)
L2P
83. 18 (1.000
87 (.05)
70. (0.26((
75. 75.83 96)
71.26%
76. (6.13
(1.
51 51.40 (1.55
DualPrompt
(1.48
86.41 86.41 (0.68)
(8.26)
08. 08.22..020) (8.84.0.55)
(8.81
(5.49
(1.68
CODA Prompt
86.31 (0.12)
90.67 90.67 (0.22)
(0.53)
76.68%o.40 44)
74.05;0.41)
78. (8.14
(5.36
57.03
InfLoRA
(6.34
91 (3
71.01
77.75/0.08
74.41 (0.63)
80.31
50.75
64. 1.036
SD-LoRA
86.77
90.96
71.27
72.93:2.78) 77.70
76)
(9.80(a.16)
55.23(0.94)
66. 66.10(o.54)
CL-LoRA
87.65
92 22.25 41)
71. (1.0)
77.78 200
78.72
(5.20
57.62
o0.7
BiLoRA
(5.9990.49)
90.62
(9.75))
74.28 73.8660.21
77.38(0.85)
51.05(a.79)
62. (2.88)
Vanilla LoRA
82, (2.99)
89.74
(9.19
77.44.0.08)
64.87,o.73
(5.57
40.01 (1.32)
BWC-LORA
87.91 (0.57)
92.27 (0.58)
73.46 (0.16)
79.58
72.8660.79 79)
(8.95
59.89 (0.26)
(3
CIFAR-100
DomainNet
Im ImageNet-R
ImageNet-A
98
87
92
Accuracy (%)
93-
BL
83
C.ARA Blsa
Accuracy (%)
QuaaRA Bick
Accuracy
84
CLLSAA
92-
79
83
68
89
75
78
60
86
73
2
7
2
5
Number Tasks
Number of Tasks
Number Tasks
Number Tasks
Figure 2: Task-wise performance comparison of different methods across various datasets.
where normalized forgetting F represents the relative drop in task's performance from its peak to the end of continual learning. For each task i, plasticity is defined as the ratio between the model's performance on the task after learning and the corresponding reference performance. The overall plasticity is defined as:
1
T
A
Plasticity
(6)
where 1.f denotes the accuracy obtained by fine-tuning model exclusively on task i. This normal- ization facilitates intuitive comparison across tasks and methods.
Implementation details. Following prior works (Wang et al., 2022c 2023 Smith et al., 2023 Liang & Li. 2024), we adopt the ViT-B/16 backbone (Dosovitskiy et al.) 2020) pretrained on ImageNet-21 K in supervised manner as the initialization for vision models, and use the pre-trained encoder-decoder T5-large and LLaMA-3.2-DD 3.---IBtnttt for language tasks. To facilitate comparison, all methods are implemented within a unified framework. We align the experimental setup with that of InfLoRA (Liang & Li 2024), fine-tuning the model with the Adam optimizer using hyperpa- rameters 3 = 0.9, B2 0.999. For each comparison method, we report the best results using the hyperparameterss provided by the authors whenever available. In cases where such configurations not released, we apply a unified set of hyperparameters that has been validated to perform reliably across all methods, ensuring consistency in our experimental setup. To better contextualize perfor- mance, we report both an upper target (Joint Train) and a lower target (Finetune). The upper target jointly trains on all tasks, while the lower target sequentially trains on tasks without any forgetting mitigation. For all experiments, we perform five runs with different seeds and report the average and standard deviation of the results.
4.2 MAIN RESULTS
Comparison with Various CL Baselines. We benchmark EWC-LoRA against state-of-the-an art
PTM-based continual learning methods, including the prompt-based methods L2P (Wang et al.
2022c). DualPrompt (Wang et al. (0226), and CODA-Prompt (Smith et al. 2023), as well as the
LoRA-based methods InfL.RRA ofR (Liang & Li 2024), SID-LRRA (Wu al. 2025). CL-LoRA (He
7

---

Published as a conference paper at ICLR 2026
et al. 2025), and BiLoRA Zhu et al. 2025). We report results 10 sequential tasks for CIFAR- 100, ImageNet-R, R, and ImageNet-A, and on tasks for DomainNet. The results are summarized in Table 2 From the table, we observe that EWC-LoRA achieves the highest final accuracy three out of four datasets. On average, across all four datasets, EWC-LORA outperforms vanilla LoRA by substantial margin of +8.92%. EWC-LLRA even surpasses other boRe-based methods that use task-specific low-rank modules, highlighting its effectiveness. On DomainNet, EWC-LoRA reduces the gap with the upper target Joint Training significantly from 5.57% of SD-LoRA to 3.38% for EWC-LoRA. On the challenging ImageNet-A. it reduces the gap considerably from 7.39% for CL-LoRA to only 5.12% for EWC-LoRA. Figure ( iillustrates the task performance of LoRA- based methods. We observe that EWC-LORA consistently outperforms other methods throughout the entire task sequence on most datasets, while also exhibiting lower standard deviations, indicating greater stability. We further evaluate EWC-LoRA on L.Mss for language CL. tasks and compare it with the LoRA-based methods O-LoRA (Wang et al. 2023) and TreeLoRA (Qian et al. 2025). The average accuracy is reported in Table From the table, EWC-LoRA achieves comparable or even superior performance across the three task orders, demonstrating its effectiveness and applicability to LLMs.
Table 3: Comparison results (reported as average accuracy (%) on the standard language CL bench- mark with the two pretrained models. Details about the task order are provided in Appendix A.3.I
Backbone
Method
Order-1
Order-2
Order-3
Avg.
O-LoRA
75.69
74.92
74.40
75.01
TS-large
EWC-LoRA
78.01
76.85
74.30
76.39
O-LoRA
56.96
55.74
67.32
60.01
LLaMA-32-IB-Insttun
TreeLoRA
58.54
56.96
65.42
60.30
EWC-LORA
61.17
60.47
67.61
63.08
Analysis on Stability and Plasticity. We evaluated both stability and plasticity for different low- rank CL methods across the four datasets. Stability reflects the ability of the model retain pre- viously learned knowledge, while plasticity measures its ability to adapt to new tasks. The results are reported in Table 4 As expected, Vanilla LoRA typically exhibits the highest plasticity but the lowest stability, since it employs no strategy to mitigate catastrophic forgetting. We observe that InfLoRA emphasizes stability, while SD-LoRA favors plasticity. On CIFAR- -00, EWC-LoRA matches InfLoRA in stability while achieving higher plasticity; ImageNet-A, it matches SD- LoRA in plasticity while providing greater stability. On DomainNet and ImageNet-R, while other methods struggle to balance stability and plasticity, EWC-LoRA can achieve this trade-off effec- tively by tuning the regularization strength A.
Table Stability (f) and plasticity (f) scores different low-rank CL methods, reflecting how well each model retains previous knowledge and adapts to new tasks. We report the normalized form of the two metrics, which independent the absolute performance on the dataset.
Tasks
CIFAR-100
DomainNet
ImageNet-R
ImageNet-A
Methods
Stability
Plasticity
Stability
Plasticity
Stability
Plasticity
Stability
Plasticity
Vanilla LoRA
87.56
08.86ro 09)
81. (1.29
97.9440.0)
78. 63
99. 99.57 (0.20)
85.56
07.
InfLoRA
94.84o.55
95.80
a3.80
97.34(..9)
(2.69(
97. (3..3))))
88.63
(2.69
SD-LoRA
91.85 65)
98. 08.24:0.25
82.45
91.78 98.69
95. 95.61,o.a2)
88.61
92, 12.13
EWC-LORA
94.45go.50)
97.99/0.500
91.51
(3.83
95.62/0.42)
93. 93.23%).a4)
89.52
92.78(1.91
Specifically, for the results in Table and 4 unified regularization strength is used across datasets. With X 107, EWC-LORA consistently achieves a favorable balance between stability and plas- ticity, without requiring dataset-specific tuning. To explore the trade-off between the two metrics, we show cability-plasticit curves for different values of the regularization strength, as illustrated in Figure 3al Unlike other methods that typically exhibit fixed performance, EWC-LoRA provides a clear and controllable trade-off: smaller À promotes plasticity at the cost of stability, while larger values enhance stability but reduce plasticity. This tunability enables EWC-LoRA to achieve com- petitive or superior performance across a wide range of trade-off points, demonstrating its robustness and adaptability. We also note that even when different methods achieve similar average accuracy,
8

---

Published as a conference paper at ICLR 2026
CIFRR1000
Accuracy (%6
68
Vanilla LoRA
Accuracy (%)
Plasticity
86
70
92
ImageNet-R
Domeinnie
SOLORA
68
Vanilla Lo
DomainNet
90
80
84
88 Stability
92
96
164
10
10
À
(a) Cubblltt-Plaasticity curves.
(b) Performance across a range of regularization strengths A.
Figure 3: (a) Stability-Plasticitty curves illustrating the trade-off between retaining previous knowl- edge and learning new tasks. (b) Performance across range of regularization strengths on CIFAR- 100 and DomainNet, showing the effect of X accuracy.
they can differ substantially in terms of stability and plasticity. This suggests that CL model evalua- tion should place greater emphasis on explicitly reporting stability and plasticity metrics.
Memory footprint and training time. The additional memory and computational cost of EWC- LoRA compared Vanilla LoRA comes only from the FIM. After each learning step, the LoRA parameters are merged into the backbone, so the total parameter size matches that of the pre-trained model. As a result, only one shared LoRA is required for new tasks, and the memory footprint remains constant regardless of the number of tasks. In contrast, other low-rank CL methods require maintaining separate LoRA parameters for each task or performing more complex computations, leading to a linear increase in memory usage and training time, or to higher computational cost as the number of tasks grows. EWC-LoRA incurs modest memory overhead during training while improving computational efficiency, achieving comparable or even superior performance.
Comparison of different methods in terms of memory cost and training time. Memory usage is measured on the Quadro RTX 6000 GPU with a batch size of 128. Training time is reported as the average time required to train a single task.
Table 5:
Methods
Memory
CIFAR-100
DomainNet
ImageNNetRR
ImageNet-A
Vanilla LoRA
18 GB
10m22s + 7s
32m8s + 44s
13m32s + 105s
0m55s + 21s
InfLoRA
20 GB
11m9s
42m25s + 68s
14m33s 107s
1m16s 22s
SD-LoRA
37 GB
12m16s +: 86s
34m18s 1 160s
22m533 232s
Im56s 36s
EWC-LORA
~ 24 GB
10m31s + 3s
33m10s +: 50s
13m42s + 104s
0m55s +: 21s
The memory cost and training time are reported Table 5
Memory usage measured on two
Quadro RTX 6000 GPUs with a batch size of 128. Training time is recorded as the average duration to train a single task, presented along with the standard deviation. Notably, the training time of EWC-LoRA nearly identical to that of Vanilla LoRA, demonstrating that the additional computa- tions introduced by Fisher estimation in the low-rank space incur relatively small overhead.
4.3 ABLATION STUDY
Results Across Varied Task Lengths. We further examine the performance of different low-rank CL methods under varying task lengths. As complementary study Table 2 we split CIFAR- 100 and ImageNet-R into 5-task and 20-task sequences, respectively. As reported in Table 6 the improvement most evident on CIFAR- 100, where EWC-LORA achieves the highest overall aceu- racy among all methods. In contrast, the gains on ImageNet-R are more moderate, which may be attributed to the domain shift in ImageNet-R, potentially limiting the effectiveness regularization- based approaches for continual adaptation. Additional results are provided in the Appendix A.3
Ablation Regularization Strength X. We evaluate performance under varying values of the regularization strength X, ranging from 10 to 10 The results are presented in Figures 3b] We empirical Fisher in all experiments. From the figures, we observe that setting À to 10" allows

---

Published as a conference paper at ICLR 2026
Table 6: Final accuracy on CIFAR-100 and ImageNet-R for task lengths of 5 and 20 tasks.
Table Best performance with Exact Fisher (500 random samples) and Empirical Fisher on CIFAR-100.
Tasks
CIFAR-100
ImageNet-R
Estimation
Exact (n=500)
Empirical
Methods
As
A20
As
A20
Vanilla LoRA
87.15
74.78
70.15
56.17
A10
88.28
87.91
InfLoRA
89.45
81.77
77.37
69.63
Avg.
92.76
92.27
SD-LoRA
89.15
83.57
74.90
72.26
Best X
À = 10
X = 10
EWC-LoRA
89.98
85.46
76.36
70.18
Training Time
14m
11m
Memory Cost
20 GB
24GB
EWC-LLRA A to consistently achieve favorable performance in terms of accuracy, stability, and plas- ticity. This suggests that an appropriate balance between stability and plasticity can be effectively maintained without fine-grained tuning. This finding facilitates the use of a unified regularization strength across datasets, eliminating the need for dataset-specific tuning.
Estimation of Fisher Information Matrix. In the context of low-rank adaptation, we examine the trade-off between performance and the computational cost of estimating the Fisher Information Matrix, as discussed by yan de Ven (225).. In general, the Exact Fisher outperforms the Empir- ical Fisher, requiring smaller regularization strength, but higher computational costs. To reduce computational costs, the Exact Fisher can be computed using only a small batch of data, which still yields superior results compared to the Empirical Fisher while significantly reducing computational overhead. We use 500 randomly selected samples to estimate the Exact Fisher, and the comparison results shown in T We report both the training time for single task and the corresponding memory usage. Additional results comparing different strategies for estimating the Fisher matrix are provided in Appendix A.3
Different Fisher Estimation Strategies. We compare three different Fisher estimation strategies, based on the discussion in Section 3.2 "Precomputed Fw" refers to using a precomputed FIM in full parameter space W to preserve prior knowledge. Instead of estimating over a large-scale dataset, we consider a dataset-based Fisher, computed using the entire dataset that will be learned sequen- tially. "SSparrtee F FB"" denotes estimating the Fisher separately for the low-rank matrices and regularizing them accordingly, while "F "FAw" denotes estimating the Fisher in the full-dimensional space W. The results are shown in Table We observe that the precomputed Fisher results the lowest plasticity, which may be caused by undesirable sensitivity arising from the frozen weights. Moreover, applying separate regularization in the low-rank space also improves performance but has noticeable drawbacks in terms of plasticity. This may be due to the joint contribution of the low-rank factors, which imposes stronger constraints on the parameters.
5 CONCLUSION AND DISCUSSION
In this work, we revisit weight regularization in low-rank continual learning (CL.) as a means to mitigate catastrophic forgetting. Using Elastic Weight Consolidation (EWC) as canonical example, we discuss the main considerations about applying regularization in the low-rank space and propose EWC-LORA, a computational- and memory-efficient solution for low-rank CL with large pre-trained models. This work aims to offer insights that may guide the broader application of regularization techniques in parameter-efficien continual learning.
A key limitation of regularization-based low-rank CL methods lies in two aspects. The first is the degradation in the accuracy of Fisher estimation as the task sequence grows longer. This can be alleviated by rehearsal-based Fisher estimation, as discussed in Wu et al. (2024) and our Appendix. The second is their sensitivity to dataset complexity. When combined with regularization techniques, it is important to carefully allocate the low-rank learnable space for each task. This sensitivity also helps explain why, on ImageNet-R, EWC-LoRA shows lower plasticity than methods based on task-specific modules. Consequently, promising direction for future work is to investigate the performance of regularization techniques in continual learning tasks.
settings and on more complex
10

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGMENTS
This paper was supported by European Union's Horizon Europe research and innovation programme under grant agreement number 101214398 (ELLIOT). We acknowledge project Grant AIA2025- 163919-C52 and PID2022-13257NBBU
funded by MICIU/AEV/ 10.1 10.13039/501 10001 1033 and FEDER. Yaoyue Zheng acknowledges the China Scholarship Council (CSC) No. No.202406280387. This work supported by the National Natural Science Foundation of China under Grant Nos. U24A20252 and 62125305, Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China under Grant No. JYB2025XDXM504, and Guangdong Major Project of Basic and Applied Basic Research under Grant No. 2023B0303000009,
REPRODUCIBILITY STATEMENT
The theoretical analysis and complete proofs of the main results are provided Appendix A.
The implementation details of our method, including model architecture, training procedures, and hyperparameters, are described in Section 3 of the main paper and Appendix [A.2] All datasets used in experiments are publicly available. Code is available at: https: //ggttgtttt siithuu... com/ yaoyz96/ low rank- cl
REFERENCES
Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming- Ming-Hsuan Yang, and Fahad Shahbaz Khan. Foundation models defining new era in vision: survey and outlook. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
Rishi Bommasani. On the opportunities and risks of foundation models. arXiv:2108.0m 07.5588 2021.
ar preprint
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv: 190.::::992.100000 10486, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 Icee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 1929, 2020.
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, et al. Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), pp. 1932-1945. 2024.
Benjamin Ehret, Christian Henning, Maria R Cervera, Alexander Meulemans, Johannes Von Os- wald, and Benjamin F Grewe. Continual learning recurrent neural networks. arXiv preprint arXiv:2006.1 121000006.1109, 2020.
Beyza Ermis, Giovanni Zappella, Martin Wistuba, Aditya Rawal, and Cédrie Archambeau. Con- tinual learning with transformers for image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3774-3781 2022.
Qiang Gao, Xiaojun Shan, Yuchen Zhang, and Fan Zhou. Enhancing knowledge transfer for task incremental learning with data-free subnetwork. Advances in Neural Information Processing Systems, 36:68471-68484 2023.

---

Published as a conference paper at ICLR 2026
Xinyuan Gao, Songlin Dong, Yuhang He, Qiang Wang, and Yihong Gong. Beyond prompt learning: Continual adapter for efficient rehearsal-free continual learning. In European Conference on Computer Vision, pp. 89-106. Springer, 2024.
Jiangpeng He. Zhihao Duan, and Fengqing Zhu. CI-lora: Continual low-rank adaptation for rehearsal-free incremenatt learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 30534-30555 2025.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustmess: A criti- cal analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8340-8349, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021b.
Michael Hersche, Geethan Karunaratne, Giovanni Cherubini, Luca Benini, Abu Sebastian, and Ab- bas Rahimi. Constrained few-shot class-incremental learning. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pp. 9057-9067, 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790-2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Ferenc Huszár. Note on the quadratic penalties in elastic weight consolidation. Proceedings of the National Academy of Sciences, 115(11):E2496 E2497, 2018
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. European conference on computer vision, pp. 709-727. Springer, 2022.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Sarbsssk-Braiiska,, al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences. 114(13):351-3352 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Timothée Lesort, Massimo Caccia, and Irina Rish. Understanding continual learning settings with data distribution drift analysis. arXiv preprint arXiv:2104.01678. 2021.
Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference- low-rank adaptation for continual learn- ing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 23638-23647,, 2024.
Xialei Liu, Mare Masana, Luis Herranz, Joost Van de Weijer, Antonio M Lopez, and Andrew D Bagdanov. Rotate your networks: Better weight consolidation and less catastrophic forgetting. In 2018 24th international conference on pattern recognition (ICPR), pp. 2262-2268. IEEE, 2018
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
David JC MacKay. A practical bayesian framework for backpropagation networks. Neural computation, ((3):44-472) 1992.
James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 116146):177, 2020.
12

---

Published as a conference paper at ICLR 2026
Mare Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-incremental learning: survey and performance evaluation on image clas- sification. IEEE Transactions on Patter Analysis and Machine Intelligence, 45(5):5513-5533, 2022.
Michael McCloskey and Neal J Cohen. Catastrophic interference in comnectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Oleksiy Ostapenko, Timothee Lesort, Pau Rodriguez, Md Rifat Arefin, Arthur Douillard, Irina Rish, and Laurent Charlin. Continual learning with foundation models: An empirical study of latent replay. In Conference on lifelong learning agents. pp. 60-91. PMLR, 2022.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural networks, 113:54-71, 2019.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
2019.
Yu-Yang Qian, Yuan-Ze Xu, Zhen-Y Yu Zhang, Peng Zhao, and Zhi-Hua Zhou. Treelora: Efficient continual learning via layer-wiss loras guided by hierarchical gradient-similane tree. arXiv preprint arXiv:2506.1033 2025.
Fuli Qiao and Mehrdad Mahdavi. Learn more, but bother less: parameter efficient continual learning. Advances in Neural Information Processing Systems, 37:97476 97799,, 2024.
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and
forgetting functions. Psychological review, 97(2))285, 1990.
Jonathan Schwarz, Wojeiech Czarnecki, Jelena Luketina, Agnieszka Crabska-Barwinskee Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for contin- ual learning. In International conference on machine learning, pp. 4528 4537. PMLR, 2018.
Khadija Shaheen, Muhammad Abdullah Hanif, Osman Hasan, and Muhammad Shafique. Continual learning for real-world autonomous systems: Algorithms, challenges and frameworks. Journal of Intelligent & Robotic Systems, 105(1):9, 2022.
Vytenis Šliogeris, Povilas Daniušis, and Artüras Nakvosas. Full-parameter continual pretraining of gemma2: Insights into fluency and domain knowledge. arXiv preprint 2.Xi:22505,05946, 2025.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Boniln Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual de- composed aitention-based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11909-11919, 2023.
Lukas Thede, Karsten Roth, Olivier Hénaff, Matthias Bethge, and Zeynep Akata. Reflect- ing on the state of rehearsal-free continual learning with pretrained models. arXiv preprint MtXiv:2406.09384, 2024.
Gido M van de Ven. On the computation of the fisher information in continual learning. arXiv preprint arXiv:2502. 11756 2025.
Gido M Van de Ven, Tinne Tuytelaars, and Andreas S Tolias. Three types of incremental learning. Nature Machine Intelligence, 115) 2022.
Liyuan Wang. Xingxing Zhang, Hang Su. and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE transactions on pattern analysis and machine intelligence, 46(8):5362-53) 2024a.
Qiang Wang, Yuhang He, Songlin Dong, Xinyuan Gao, Shaokun Wang, and Yihong Gong. Non- exemplar domain incremental learning via cross-domain concept integration. In European Conference on Computer Vision, pp. 144-162. Springer, 2024b.
13

---

Published as a conference paper at ICLR 2026
Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuan-Jing Huang. Orthogonal subspace learning for language model continual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10658-10671, 2023.
Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transformers: An occam's 's razor for domain incremental learning. Advances Neural Information Processing Systems, 35:5682-5695, 2022a.
Zifeng Wang. Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European conference computer vision, pp. 631-648 Springer, 2022b.
Zifeng Wang. Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning prompt for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 139- 149, 2022c.
Xiwen Wei, Guihong Li, and Radu Marculescu. Online-lora: Task-free online continual learning low rank adaptation. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 6634-6645, IEEE, 2025.
Yichen Wu, Lai Huang, Renzhen Wang, Deyu Meng, and Ying Wei. Meta continual learn- ing revisited: Implicitly enhancing online hessian approximation via variance reduction. In The Twelfth international conference on learning representations, volume 2, 2024.
Yichen Wu, Hongming Piao, Long- Kai Huang, Renzhen Wang, Wanhua Li, Hanspeter Pfister, Deyu Meng, Kede Ma, and Ying Wei. Sd-lora: Scalable decoupled low-rank adaptation for class incre- mental learning. arXiv preprint arXiv:2501. 13198, 2025.
Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, and Zhiting Hu. Lan- guage models meet world models: Embodied experiences enhance language models. Advances in neural information processing systems, 36:75392-7544 2023.
Yutao Yang, Jie Zhou, Xuanwen Ding, Tianyu Huai, Shunyu Liu, Qin Chen, Yuan Xie, and Liang He. Recent advances of foundation language models-based continual learning: A survey. ACM Computing Surveys, 1-65):1-38, 2025.
Hao Zhu, Yifei Zhang, Junhao Dong, and Piotr Koniusz. Bilora: Almost-orthogonal parameter spaces for continual learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 25613-25622 2025.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 THEORETICAL ANALYSIS
Setup. In low-rank adaptation, the adapted weight matrix W do reparameterized as W = Wo + AW = Wo - AB, where Wo denotes the pre-trained weight, which remains frozen during fine-tuning. The trainable parameters are the low-rank matrices A Rdoxs xT and B E grxd, such that AW AB is the only trainable update to the model. Note that the learned LoRA modules on the previous task are merged back into the base weights before starting the new task. The optimal weight matrix is indicated as W
A.1.1 REGULARIZATION UNDER DIFFERENT PARAMETERIZATIONS
Proposition 1 Let AW E Rdo xd: be model parameter matrix factorized as AW- AB, with A dd B 1xxdo Define the EWC regularization term in the full-space as Rw, and in the low-rank parameter space as RA.B. Under general conditions, Raw # RA.BB
Proof. Let vec(.) denote the vectorization operator, and @ the Kronecker product. The following identity holds: vec (B) (B* 3 Ido) vec( (A) = (Id, @ A) vec(B)
For simplicity, we define a vec( A) E RRdor, vec(B) Red,, Define the Jacobians:
JA B Iio Rdodixdor
JB(A) Ia, @ A Rdody xrd,
By the standard vectorization identity:
vec(AW) = JBBa JB(A)b
The Fisher Information Matrix estimated in the full-dimensional space is defined as:
Fw = Ez-D [vec(V V Vw vec( L)
Although the full-space regularization Rw be expressed equivalently using either the A or B Jacobian, the update directions for and B remain coupled. Thus, we consider a first-order approximation of vec( W) function of both and b:
rec(AW) (B)Aa + (B)DAb
The EWC regularization term in the full-space is defined as:
Rw
(vec( W) vec( W))) (wec((((W) vec(W"))
2
vec(A FFW
FW vec (W)
2
Aa " FBJAa 1
Ab A FW-J)))AAAA(A
2
2
+ Aa T JA(B) F())Ab
If Fisher regularization is applied separately to A and B, we have two Fisher matrices:
FA Ex-D vve (VAC) vec V A
FB Ex-D (vee(V vec CB
By the chain rule, the gradients low-rank factor space are related to the full-space gradient by:
0C oW
VAL
= VwLJa(B)
OW 0A
0L
oW
VLC OW
=JE(A)
ØB
15

---

Published as a conference paper at ICLR 2026
Hence, the Fisher Information Matrices can be expressed as:
FA = (A( Fw (aB). FB = F
The EWC regularization term in the low-rank parameter space is defined
RAB = Aa FAAA + Ab A FBAb
2
The regularization term RAABB captures independent penalties in the factor space. Unless the cross- covariance term FA, explicitly computed and retained, the interaction between A and B is ignored. Consequently, the two regularizations cannot be equal, except in special cases such as when only A or B updated. In contrast, estimating F Fw in the full-dimensional space avoids this issue and captures the true sensitivity of the model to perturbations. Therefore, regularization in the full space better preserves the true geometry and parameter importance induced by the low-rank update, providing more faithful and effective constraint on A and B.
A.1.2 FISHER INFORMATION CONSISTENCY WITH THE TRAINABLE PARAMETER SPACE
Proposition 2 Consider parameterizattin:
= + AB
where Wo is fixed and only A, B are trainable. Let
vee( A)
0 0=
= vec vec(W).
vec B)
Then, the Fisher Information Matrix with respect to satisfies
Fe J FwJ.
where Fw is the Fisher Information Matrix with respect to w, and J 0w/00 is the Jacobian of the reparameterization.
Proof. Let L(W) denote the loss (negative log-likelihood). By the chain rule,
VeL
VW
The Fisher Information Matrix with respect is defined as:
Fg = E V logg Va log T
Substituting the chain rule,
Fg = E [J TV (J V log T]
J E V logp V log J.
The term inside the expectation is precisely the Fisher Information Matrix with respect w, denoted Fw. Hence we have:
F, FwJJ
The Jacobian structure has been illustrated in Proposition fundamentally estimation of the curvature of the loss function with respect to the trainable pa- rameters. Eq. 7 shows that the correct Fisher matrix for 9 is obtained by projecting Fw into the subspace spanned by A and B via J. Computing Fw directly in the full parameter space W in- troduces directions corresponding to the frozen W.. which are irrelevant for training. Thus, Fisher should be consistently defined in the trainable subspace.
The Fisher Information Matrix is
16

---

Published as a conference paper at ICLR 2026
A.1.3 CONSTRAINING LOW-RANK UPDATES VIA FULL-SPACE FISHER REGULARIZATION
Proposition 3 Let the adapted weight matrix be: W = W + AW Wo + AB, where Wo denotes the frozen pre-trained weights and AW AB is the low-rank update. Then, the empirical Fisher Information Matrix F saw is estimated by the squared gradients of the log-likelihood with respect to W. Consequently, the Fisher regularization on AW induces constraints on the update directions of the low-rank factors A and B.
Proof. Since W, is constant and only AW1 is trainable, the loss function L depends on W only through AW. Therefore, the gradients satisfy VwL V VawL, L, because òΔ W /0W I. Let Fw denote the empirical Fisher Information Matrix estimated over weight matrix W:
Fw = EenD vve V LW vee Vw
Using the gradient equivalence above, the Fisher matrix in the AW space identical:
Pww- EnnD
(V Aww) Fw
For notational simplicity, we omit the vec(-) operator and treat W vectorized parameter. The Fisher regularization term can be defined as:
Rw (W FwW- AW 3 AW
The gradients of Rw with respect A and B
VARW VawRW BT FAWAW B VBRw A VAWRW= A FAWAW
with:
VAWRW FAWAW
Therefore, by propagating the gradient through the low-rank decomposition AW AB, the Fisher regularization over AW imposes a constraint on the update directions of the low-rank factors.
A.2 IMPLEMENTATION DETAILS
This section provides additional details on the method described in Section 3 and the experimental setup Section of main text. particular, we present the algorithm of EWC-LoRA and discuss the effects of hyperparameters, accompanied by additional ablation studies.
A.2.1 OPTIMIZATION ALGORIIHMS
Algorithm outlines the learning procedure of EWC-LoRA across tasks Tı Tr. At each task, the model is updated via low-rank adaptation while incorporating EWC regularization computed from the Fisher Information Matrix. Unlike prior methods that assign task-specific modules, EWC-LoRA regularizes a shared LoRA, thereby maintaining constant memory cost. For clarity, the key equation referenced in the main text is restated here.
A B) C. vec(AB) vec (ec(A
where vec(AB) is flattened AB. F.um denotes the accumulated Fisher matrix obtained from task Tr-1- i- A is the regularization strength.
17

---

Published as a conference paper at ICLR 2026
Algorithm 1: The learning procedure of EWC-LORA.
Input:
A sequence of tasks (Tr) with datasets (Dr Y A frozen pre-trained model with
parameters Wo E Rdoxxd;; Low-rank adaptation matrices A R xr and B E Rexd, with rank r; Task decay factor = 0.9, Vtt = 1.....T T.
Output: Adapted model parameters WT E IRdo xd, that generalize across all tasks {T:)=r
Initialize accumulated Fisher Information Matrix: Fo + "pprior I; // if no prior
for 1
Step Initialize < 0, U(a, b)
Step Low-rank adaptation on current task Tr: A", B² Eq.
Step Fisher estimation for Tr
Get gradient VwC on the full-dimensional space;
0 log Pwr (ulwr Owg
2
2. Compute diagonal entries of Fisher matrix: F Dii E E(
3. Update accumulated Fisher Information Matrix: F 9 F Fs;
Step Integrate low-rank parameter: W, W + A
Step Discard task-specific dataset D, and Fisher matrix F..
A.2.2 HYPERPARAMETERS
We follow the training configurations specified by the authors in the original papers. When such configurations are not available, we adopt unified set of hyperparameters that has been validated to perform reliably across methods, thereby ensuring consistency our experimental setup. In all experiments, we use the Adam optimizer with 31 0.9 and B2 0.99. The number of training epochs depends on the specific dataset, and the training batch size set to 128. No shuffling is applied during training; however, verified that enabling shuffling generally leads to improved results. Table & summarizes the hyperparameters used during training, with -seehod-peccficc param- eters highlighted for each respective approach.
Table 8: Hyperparameters for different benchmarks and methods. "Ir" denotes learning rate. For the parameter e, we refer readers to Liang & Li (2024) for details.
Datasets
Methods
optimizer: Adam; schedular: Cosine; batch size: 128; shuffle: False; epochs: 20; rank: Ir: 0.0005; classifier 0.005; decay:
CIFAR-100
Ir: 0.008; classifier Ir: 0.008; Ir decay: 0.1 (SD-LoRA)
e : 0.95 (InlLoRA)
optimizer: Adam; schedular: Cosine; batch size: 128; shuffle: False; epochs: 5; rank: 30
Ir: 0.0005; classifier Ir: 0.005; decay: 0.1
DomainNet
Ir: 0,02; classifier 0.02; decay: 0.0 (SD-LoRA)
: 0.95 (IniLoRA)
optimizer: Adam: schedular: Cosine: batch size: 128; shuffle: False; epochs: 50; rank: 10
Ir: 0.0005; classifier Ir: 0.0005; Ir decay: 0.1
ImageNet-R
Ir: 0.01; classifier Ir: 0.01; Ir decay: 0.0; weight decay: 0.0005 (SD-LoRA)
weight decay: 0.005 (EWC-LORA)
0.98 (InfLoRA)
optimizer: Adam; schedular: Cosine; hatch size: 128; shuffle: False; epochs: 10; rank: 10 Ir: 0.0005; classifier Ir: 0.005: Ir decay: 0.1
ImageNet-A
0.98 (InfLoRA)
For the regularization strength À in Eq. evaluated wide range from 10 to 1000 The trend illustrated in Figure (3b of the main text. We observe that across all datasets, when using the empirical Fisher, the best overall performance is achieved around 10". Moreover, with relatively
18

---

Published as a conference paper at ICLR 2026
smaller values of (e.g.. 10' to 10*). performance tends to decrease initially. For all experiments, the regularization strength À is set to 10".
For the parameter y in Algorithm I we all tasks to be equally important to 7 0.9. To further assess its effect, we evaluate the effect of varying y using a single seed for performance comparison. The results presented in Table 9 and Figure The parameter y controls the accumulation of Fisher information across tasks in continual learning. When ? = O, no Fisher information carried over from previous tasks, meaning that only the Fisher matrix of the current task is used regularize the subsequent task. As increases, past information is accumulated more strongly, which can help preserve knowledge from earlier tasks.
As shown in Table 9 the final accuracy A remains relatively similar across different settings. However, notable differences are observed in terms of stability and plasticity. From Figure 4 the trends associated with different values are clearly visible. We find that there exists broad range of y values that achieve a similar -alatity-ll trade-off. As indicated by the green-shaded region, the method is not overly sensitive to the exact choice of y.
Table 9; Performance comparison under different values y on CIFAR-100.
? 1.0
7 0.9
7
0.7
7
0.5
7
0.3
7
A
87.64
87.47
88.07
88.14
88.22
86.63
Avg.
91.96
92.12
92.42
92.34
92.46
92.04
Stability
94.46
94.45
94.28
94.37
94.51
91.81
Plasticity
97.66
97.99
98.35
98.33
98.29
99.07
99.4
98.0
96.6
95.2
93.8
Stability
92.4
Plasticity
0.0
0.2
0.4
0.6
0.8 1.0
Task decay factor
Figure 4: Stability-Plasticii trade-off with various task decay factor 7. A broad range of (0.3-0.9) yields similar trade-off, indicating that the method is not sensitive to the precise choice of ry.
We further illustrate the effect of 7 using per-task accuracy matrix. In Figure each row rep- resents the performance on all previously encountered tasks (x-axis) after learning the current task (y-axis). The results show that setting to zero causes significant drop in stability, with carlier tasks experiencing more severe forgetting. In contrast, higher y values help preserve performance on previous tasks while still enabling effective learning of new tasks.
100
100
100
Task
w N
Learning on Task
2 u A waw. N
Learning on Task
Learning on
90
90
0 an
10
70
10
70
10
70
Performance
Performance
Performance
7 0.9
(b) 0.5
(c) 7
Figure 5: Task-wise performance on CIFAR-100 under different settings.
19

---

Published as a conference paper at ICLR 2026
Table 10: Task order of the standard CL benchmark for language tasks.
Task
Task 2
Task 3
Task 4
Order Order
dbpedia dbpedia
amazon
yahoo
ag
amazon
yahoo
Order
yahoo
amazon
ag
dbpedia
A.3 ADDITIONAL EXPERIMENTS
A.3.1 COMPARISON RESULTS
Results standard CL benchmark for language tasks. To further evaluate the applicability of EWC-LoRA beyond vision tasks, we extend our experiments to natural language processing scenarios using the T5-large and LLaMA-3.22 1--I-IBttntt pretrained models. Following the standard language CL benchmark and previous work (Wang et al. (023). , we use three task orders composed of four text classification datasets, as summarized in Table I0 We compare EWC-LoRA with O- LoRA and LoR under identical training settings. The task-wise performance is evaluated using average accuracy, and the results are shown in Figures [ and 7 For the larger model T5- large, the effectiveness of EWC-LoRA is more pronounced, as it mitigates forgetting most tasks across all three orders. For the smaller model LLaMA-3.22 2----3..-B--Intruuee EWC-LoRA also exhibits less forgetting compared with the other two methods.
Methos
92
92
Mathoz
Mested
Nocuracy
82
Datwest
Detreet
Datseet
dipoda
dipedi
saloo
54
52
52-
dbood
Inst
twok
Task
Mask
Task
twst
Isst
lask:
Twek
Task
Task
Task
(a) Task Order 1
(b) Task Order 2
(c) Task Order 3
Figure Task-wise wise performance across the three task orders using the TS-large model.
79
80
89
copodie
71
Accuracy
Methou
C-LaRA
Accuracy
agnano
EVC-ASRA
Amaann
46
Methed
46
Mathos
salao
Task
Task
Task
Mathod
Mctted
Muthod
TextafM
TeecutA
80
12
Accuracy
Accuracy
Datapet
Daliset
Dotazel
44
arrazon
dipod's
44-
arazze
46
vthoo
35
35
25
dbaada
t3ck1
tack
tack
task1
tack3
120k4
task
13.6
135044
Taak
Tasskc
Thskk
(a) Task Order 1
(b) Task Order 2
(c) Task Order 3
Figure 7: wask performance across the three task orders using the LLaMA-332-1B-Instmm model.
Results Across Varied Task Length. Table repports the full results of low-rank continual learn- ing methods under varying task lengths on ImageNet-R. For EWC-LoRA, the performance gains
20

---

Published as a conference paper at ICLR 2026
appear relatively modest, likely due the inherent domain shift in this benchmark, which may constrain the effectiveness of regularization baseed methods for continual adaptation. We further ob- serve that InfLoRA achieves better results on shorter sequences, whereas SD-LoRA performs more strongly on longer sequences. Examining the accuracy matrix for longer sequences, we find that although SD-LoRA exhibits lower stability, its higher plasticity often leads superior final perfor- mance. This observation suggests that model evaluation should go beyond reporting only the final accuracy. It is also important to track performance throughout the task sequence and explicitly report both stability and plasticity.
Table 11: Comparison results on ImageNet-R across different task lengths (in %).
Tasks
ImageNet-R (N=5)
ImageNet-R (N=20)
Methods
Ag (1)
Avg. (T)
A20 (1)
Avg. (1)
Joint Train
81.69
85.57
(1.66(0.22))
86.41(0.10)
Finetune
69.26(o.74)
78.88
47.06 (7.06(2.05)
63.01(0.50)
InfLoRA
77.37 (0.30)
82.19 (0.24)
69. (v.662)
76.95
SD-LoRA
74.90
(9.93
72.26
77.81 (o.21)
Vanilla LoRA
70. 70.15(1.00)
(9.76
56. (1.50)
69.51
EWC-LoRA
76.36
81.43 (0.13)
70.18
(7.06
Task-wise Performance. Figure 8 and Figure show the accuracy matrix of the LoRA-basee methods on CIFAR-100 and DomainNet. Fach row corresponds to the performance on all previ- ously encountered tasks after training the current task. The diagonal entries correspond to the most recently trained tasks. As expected. Finetune exhibits severe forgetting on previous tasks while maintaining high performance we observe that InfLoRA better preserves performance on the earliest task (first column). On Do- mainNet, SD-LoRA adapts more effectively to tasks. On both datasets, EWC-LoRA achieves more balanced trade-off between stability and plasticity compared to the other two methods.
the current task, shown in the matrix entries. On CIFAR- 100,
106
100
100
100
Learning on Task
Learning Task
Learning on Täsk
Learning on Task
16
123436 8 910
-70
10
1234366
7
910
-70
910
-70
3 9110
70
Performance
Performance
Performance
Performance
(a) Finetune
(b) InfLoRA
(c) SD-LoRA
(d) EWC-LoRA
Figure 8: Task-wise performance of LoRA- Abased methods on CIFAR- 100. Each row represents the performance on all previously encountered tasks (x-axis) after learning the current task (y-axis).
100
100
100
100
Learning on Task
$7.50
Learma on Task
Cmmmn on Task
$1.9
Leaman on Task
$7.09
13.79
05.32
B7
27 1a $4.26
87
7E-40
87
$2.97 63
87
7335
7t:1 82.02
roa0 02.53
7755
10.19
4353
73
73
13
13
3
60
160
660
Performance
Performance
Performance
Performance
(a) Finetune
(b) InfLoRA
(c) SD-LORA
(d) EWC-LORA
Figure 9: Task-wise performance of LoRA-based methods on DomainNet. Each row represents the performance on all previously encountered tasks (x-axis) after learning the current task (y-axis).
21

---

Published as a conference paper at ICLR 2026
Trade-off between Stability and Plasticity. To better understand how different methods balance stability and plasticity, we introduce trade-ofi off metric that approximates this balance:
P
T
S+P P
where S and P denote the Stability and Plasticity, respectively, defined in Eq. 5 and Eq. 6 Figure Iiilustrats the trade-off between stability and plasticity for different low-rank CL methods. Vanilla LoRA generally achieves the highest plasticity, it does not include mechanisms to prevent forgetting. EWC-LoRA attains stability comparable to InfLoRA, while retaining more plasticity than InfLoRA. Overall, EWC-LoRA achieves the best trade-off among the methods.
100
Vanilla LoRA Info
SD LoRA
95
90
85
80
75
Stability
Plasticity
Trade-off
Figure 10: Trade-off between stability and plasticity.
Ablation on LoRA Ranks. In our main experiments, we used a fixed LoRA rank across tasks to ensure fair comparison. Here, explore whether more complex datasets benefit from different rank settings. Specifically, we conducted an ablation study with different rank (r) values on two benchmarks of varying difficulty, and the results are summarized in 12 We find that increasing or decreasing the rank r does lead to some variation in performance. However, the difference is relatively small, which indicates that the effective parameter space of the model is inherently low- rank, and that the regularization applied within this low-rank space follows the same principle.
Table 12: Ablation study on LoRA ranks r across two benchmarks of varying difficulty.
CIFAR-100 100
ImageNet-R
A10
Avg.
Āιo
Avg.
16
88.31
92.59
72.10
79.18
10
87.91
92.27
72.86
78.95
4
87.79
92.12
72.42
78.78
1
86.77
91.61
70.37
77.27
To exclude the effect of additional regularization, we examine the cubbility-plasticiie trade-off using vanilla LoRA. We observe that vanilla LoRA with lower rank (e.g., r 1) even outperforms higher-rank settings terms of final average accuracy. The corresponding results are presented in Table 13 The reason is that a smaller rank naturally provides the model with stronger stability, but at the cost of reduced plasticity. For CIFAR-100, this cost of plasticity is relatively minor, whereas for pronounced. This indicates that for more challenging bench-
ImageNet-R, it becomes much marks, the model should adopt a larger rank to ensure sufficient plasticity.
A.3.2 FISHER ESTIMATION ANALYSIS
Precomputed Fisher. As illustrated in Xiang et al. 20233 and Šliogeris et al. (025). they use a precomputed Fisher Information Matrix to preserve prior knowledge. Following this approach, we evaluate performance when a Fisher Information Matrix precomputed and used throughout the continual learning process. Unlike these works, do not rely large-scale dataset to compute the Fisher matrix. We consider two settings: (1) Uniform parameter importance: All parameters are assigned equal importance, i.e., Fprior in Algorithm I is constant, and the Fisher matrix is
22

---

Published as a conference paper at ICLR 2026
Table 13: Ablation study on LoRA ranks (r) in the Vanilla LoRA setting for CIFAR-100 and ImageNet-R. Note that the plasticity is computed based on the upper bound at rank 10. When the rank is 16, the plasticity can exceed 100%.
CIFAR-100
ImageNet-R
A10
Avg.
Stability
Plasticity
A1o
Avg.
Stability
Plasticity
16
83.05
89.71
86.25
100.42
66.05
76.26
74.17
101.41
10
82.99
89.74
87.56
98.86
66.32
76.35
75.69
99.86
83.35
90.10
88.70
98.24
67.52
76.33
77.27
97.30
83.72
89.94
89.95
97.36
68.43
77.09
81.95
96.75
identity matrix. (2) Dataset-based Fisher: The Fisher Information Matrix is computed in advance using the entire dataset. The results are shown in Table [4] We observe that the uniform Fisher exhibits lower stability, while the plasticity of both methods is similar, but still much lower than ours. This suggests that using precomputed, fixed Fisher imposes stronger constraints on the weights, thereby limiting plasticity. As expected, the dataset-basee based Fisher achieves higher stability than the uniform Fisher, which is reasonable since it better captures the true importance of the parameters.
Table 14: Regularization using precomputed Fisher on CIFAR-100.
Strategy
A10
Avg.
Stability
Plasticity
Uniform 1
83.02
88.85 89.36
92.26 93.15
94.63 94.74
Dataset-basse F
83.87
Computation on Fisher Information Matrix. As suggested by van de Ven (2025) we investigate different methods for estimating the FIM. The results are presented in Table [5| Here, "Exact" indicates that the inner expectation in Eq. 4 is computed exactly for each training sample. "Exact (n=500)" denotes that the outer expectation is calculated using a subset of 500 samples from the old training data. "Sample" indicates that the inner expectation is computed over a sampled class. The results indicate that the optimal regularization strength varies according to the estimation method. In general, the Exact Fisher outperforms the Empirical Fisher, requiring smaller regularization strength. The Sample method yields slightly better results than the Empirical Fisher.
Table 15: Different ways for estimating the Fisher matrix. Final accuracy of each variant using its optimal strength À CIFAR-100.
Estimation
Aιo
Avg.
Best À
Exact
88.32
92.77
À 10s
Exact (n=500)
88.28
92.76
À = 105
Sample
88.10
92.50
À 107
Empirical
87.91
92.27
À = 107
Accuracy of the Fisher Estimation. rehearsal-free EWC, the Fisher matrix for the current task is computed only once at the end of training and is not updated thereafter, which may result in a stale Fisher estimate when the parameters drift substantially from the task-specific optimum (Wu et al.) 2024h. We further investigate whether similar issue exists in low-rank CL methods. To this end, track the evolution of the Fisher matrix for each task throughout the learning process. Specifically,
for given task i, we first compute its Fisher matrix F!() The model is then sequentially trained on subsequent tasks + 1,
t. We evaluate the accuracy of the Fisher Information Matrix from two perspectives: (1) Scale sensitivity, which indicates whether the parameter importance encoded by the Fisher matrix becomes degraded or inflated. (2) Structural pattern, which reflects whether the set of parameters that EWC aims protect changes over time.
23

---

Published as a conference paper at ICLR 2026
1.80
Spearman Rank Corr.
1.00
1.60
Task
Cosine Similartty
Task Task
Task
Nom Ratio
1.40
Task
0.99
Task
1.20
0.96
1.00
0.95
0.98
0.80
0.94
0.97
1
2
6
4
2
7
6
Task
Figure 11: Evaluation of Fisher accuracy as the task index increases on CIFAR- 100 with 10 tasks.
Task
Cosine Similarity
Task Task
Gearman Rank Corr.
1.00
Task
3.20
0.99
Norm Ratio
Task
0.92
Task
2.40
0.98
0.90
0.88
0.80
0.96
0.00
5
11
13
15
17
19
3
7
11 Task
13
17
19
0.95
3
5
11
15 17
Figure 12: Evaluation of Fisher accuracy as the task index increases on CIFAR- 1-0 with 20 tasks.
To evaluate scale sensitivity, recompute the Fisher matrices F(() using the current model (sub- script t) and the data from task i (superscript (i)) after learning each new task. We use the Norm Ratio quantify the numerical changes in the Fisher matrix. The Norm Ratio defined as:
NR |
(9)
where NR > 1 indicates that the Fisher matrix has become inflated, and NR < indicates that has degraded.
To quantify changes in the structural patterns of the Fisher matrix, we measure the similarity between the Fisher matrix estimated task / and the original Fisher matrix. We consider two settings: (1) Rehearsal-free: we compare the accumulated Fisher matrix F with the original optimal Fisher
compute the optimal Fisher matrix F(I...) for the current task by incorporating data from previous tasks, and compare F(I...) with F(A) We employ Spear- man Rank Correlation to capture the overall consistency and Cosine Similarity assess shifts in the most critical task parameters. The Spearman Rank Correlation evaluates the agreement in rank ordering between two vectors. Given two vectors V and V2, each element first converted to its rank, denoted rı,i rank(v1.s) and 22,2 rank(v2,)). The Spearman correlation coefficient is then computed as:
matrix F and (2) Rehearsal-based,
6E - r
(10)
nnn 1)
where n represents the number of elements in the vectors. The Spearman correlation coefficient P E [],, where p indicates identical rank ordering, indicates correlation, and -1 indicates complete inverse ordering. Spearman correlation captures the overall pattern similarity between vectors regardless of their scale.
Cosine Similarity focuses on the directional alignment between two vectors. A high cosine similarity indicates that the important parameters in one model are largely aligned with those in the other, capturing the consistency of parameter sensitivity patterns across models. The cosine similarity
between Feum and F(() is defined
Feum (
CosSim(F!um)
F(")
(1I)
F
| [
We report the above metrics for the three oldest tasks throughout the learning process, and the results shown in Figure 11 12 and 13 We observe that the Norm Ratio fluctuates around indicating
24

---

Published as a conference paper at ICLR 2026
0.96
0.92
Cosine 5imilarty
Cosine millarity
0.90
0.88
0.94
0.93
Task
0.84
Task
Task2
0.92
0.82
5
11
15 17
Figure 13: Cosine similarity between optimal Fisher matrices under the rehearsal-based setting.
that the strength of the constraint imposed on old tasks changes the model learns new ones. As for the structural pattern the Fisher Information Matrix, we observe that as the model learns more tasks, the Cosine Similarity gradually decreases, whereas the Spearman Rank Correlation shows much milder decline. This indicates that although the absolute directions of parameter importance shift with new tasks, the relative ordering of important parameters remains largely stable.
Wu et al. (2024) has demonstrated the staleness of Fisher estimates in full- uall-parameter EWC. In low- rank continual learning, similar issue also exists for EWC. Under the rehearsal-based setting, how- ever, the Fisher matrix remains relatively stable compared to the rehearsal -ree setting. As the model learns more tasks, the key parameters identified by the Fisher matrix remain largely consistent, and the observed inaccuracies arise primarily from changes in magnitude and directional scaling, rather than from fundamental reordering of parameter importance.
A.4 THE USE OF LARGE LANGUAGE MODELS (LLMM))
In the preparation of this manuscript, the LLM used to refine sentence structures, ensure clarity, and improve the readability of the text.
25


Reviews:
[{"review_id": "MCeWet0Urf", "reviewer": "Reviewer_UxrP", "summary": "This paper introduces EWC-LoRA, a parameter-efficient continual learning method that integrates Elastic Weight Consolidation (EWC) with low-rank adaptation for large-scale pre-trained models. Unlike prior low-rank approaches that rely on task-specific modules, EWC-LoRA regularizes a shared low-rank update using the Fisher Information Matrix, maintaining constant memory while improving the stability–plasticity trade-off. Experiments show that EWC-LoRA outperforms vanilla LoRA by 8.92% on average and matches or surpasses previous low-rank continual learning methods.", "questions": "See Weaknesses above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized and easy to follow.\n\n2. The investigated problem of continual learning using low-rank adaptation is important.", "weaknesses": "1. It would be helpful if the authors could further elaborate on the paper’s main contributions. In particular, clarifying the fundamental challenge in integrating EWC with LoRA would strengthen the work.\n\n2. The experiments are primarily conducted on artificial image datasets. Including additional experiments on large language models (LLMs) would make the evaluation more comprehensive and convincing.\n\n3. The paper seems to overlook an important line of research on continual learning (continual fine-tuning) of LLMs, such as O-LoRA, LoRA-MoE, and TreeLoRA. Discussing how this work relates to or differs from these studies would provide better context and positioning.\n\n4. The performance improvements over previous methods appear relatively modest; as shown in Table 1, the accuracy increases only slightly. A more detailed analysis of these results could help clarify the practical significance of the gains.\n\n[O-LoRA] Orthogonal Subspace Learning for Language Model Continual Learning\n\nLoRA-MoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin\n\nTreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6zh6XASKpS", "reviewer": "Reviewer_ETzF", "summary": "This paper revisits weight regularization in the context of low-rank continual learning (CL), specifically focusing on Elastic Weight Consolidation (EWC) applied to parameter-efficient continual learning (PECL) with pre-trained models. The authors propose EWC-LoRA, a method that regularizes a shared low-rank update using the Fisher Information Matrix (FIM) estimated in the full-dimensional space. This approach avoids the linear growth in memory with the number of tasks by maintaining a constant memory footprint. The authors provide a systematic analysis of EWC in low-rank CL, demonstrate a superior stability-plasticity trade-off, and validate their method across multiple benchmarks (CIFAR-100, DomainNet, ImageNet-R, ImageNet-A), showing an average improvement of 8.92% over vanilla LoRA and competitive or better performance compared to state-of-the-art low-rank CL methods.", "questions": "+ The paper mentions sensitivity to dataset complexity—could you discuss how to automatically select or adapt the rank for different tasks?\n+ The storage overhead of inflora as well as one efficient version of sd-lora is not increased linearly, could you explain your benefits compared with them? Or why your performance can be better?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "+ Strong theoretical grounding and extensive experiments across multiple benchmarks.\n+ Well-written with clear motivation, method, and results.\n+ Provides a memory-efficient, tunable, and high-performing solution for PECL.", "weaknesses": "+ The Fisher estimation, though efficient, still introduces non-negligible memory overhead\n+ EWC-LoRA is indeed not very original, but the reviewer acknowledged that it is meaningful to make existing methods work in PEFT and explain why.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TKDxSiWlYc", "reviewer": "Reviewer_VVBD", "summary": "The paper introduces EWC-LoRA, a parameter-efficient CL method that combines EWC with a shared LoRA. Unlike prior low-rank CL approaches that allocate task-specific LoRA modules, EWC-LoRA regularizes a single low-rank update in the full-dimensional space using an accumulated diagonal Fisher matrix. This yields constant memory overhead, competitive accuracy, and a tunable stability–plasticity trade-off.", "questions": "Overall, I think this paper makes a valuable and insightful exploration by bridging traditional regularization-based CL methods with the PECL framework. However, I still have some concerns, and I hope the authors can address the issues raised in Weaknesses 1 and 2.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents an interesting idea of revisiting weight regularization in the low-rank regime, which effectively bridges traditional CL methods and PECL approaches.\n- The authors provide a comprehensive comparison of computational cost and parameter efficiency, clearly demonstrating how the proposed method performs relative to other LoRA-based continual learning baselines.\n- The proposed approach shows consistent improvement across multiple datasets and experimental setups, indicating good robustness and generalizability.", "weaknesses": "- Accuracy of the Hessian estimation. The paper relies on the empirical Fisher information matrix to estimate the importance of weights for each task. However, as discussed in Meta-CL [1], the Fisher matrix used in EWC-based methods tends to become stale and outdated over time, leading to inaccurate importance estimation. It remains unclear whether a similar issue [1] arises in the PECL setting adopted here.\n- Effectiveness under longer task sequences. When the number of tasks increases, it is uncertain whether the accumulated Fisher matrix can still provide reliable importance estimation. Indeed, the performance drop observed on ImageNet-R with N = 20 tasks suggests that the method’s stability may degrade as the task length grows. A more detailed analysis or explanation from the authors would help clarify this behavior.\n\n[1] \"Meta continual learning revisited: Implicitly Enhancing Online Hessian Approximation via Variance Reduction.\" ICLR, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cLw3NuYNY3", "reviewer": "Reviewer_gUkm", "summary": "This manuscript focused on the problem of continual learning with pre-trained model (CL-PTM), specifically, parameter-efficient continual learning with low-rank adapters. To mitigate the task inferences, this manuscript revisited the weight regularization method, which was less explored under the context of low-rank continual learning. Instead of applying task-specific LoRA modules, this manuscript proposed to adopt a shared low-rank update to keep the storage requirement constant while applying the regularization on the updated parameters with EWC. Specifically, the parameter importance in EWC was estimated over the full-dimensional space. The authors conducted experiments on the proposed method EWC-LoRA, showing that the proposed method achieved state-of-the-art performance compared to existing methods without sacrificing efficiency.", "questions": "1. Regarding the training time shown in Table 3. It seems that the per-task training time seemed almost the same as the vanilla LoRA method. However, it seems that Eq. (4) needs to compute the gradient over the full dataset. I wonder if this procedure took such little time considering it requires the backpropagation on the whole dataset.\n2. The LoRA-based baselines considered in this manuscript were InfLoRA and O-LoRA. However, recently, many studies about PECL have been published, even considering only the interference reduction in LoRA-based PECL. Could the author provide more comparisons with the methods in this stream, including but not limited to [1,2]. \n\nReferences:\n\n[1] CL-LoRA: continual low-rank adaptation for rehearsal-free class-incremental learning. CVPR 2025\n[2] BiLoRA: almost-orthogonal parameter spaces for continual learning. CVPR 2025.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Compared to previous methods with task-independent LoRA modules, the proposed EWC-LoRA continuously merged the task-specific parameters into the previous ones. The model was trained in a parameter-efficient manner, and the maintained parameters for all previous tasks remained constant during the inference after weight merging.\n2. The proposed EWC-LoRA adopted a dynamic way to update the Fisher Information Matrix for EWC, avoiding the fixed precomputed FIM.\n3. The ablation studies were relatively complete.", "weaknesses": "1. The proposed approach is mainly an adaptation of EWC to LoRA; the innovation lies in its parameterization and efficiency rather than a fundamentally new learning principle.\n2. The update of the accumulated FIM seems heuristic. Although the authors provided some discussions about the choices of $\\gamma$ during accumulated FIM update, it seemed that the choice of this hyperparameter was still not principled. \n3. Some recent LoRA-based PECL methods were not involved in the experimental comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)