SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning
Authors: Yichen Wu
Hongming Piao
Long-Kai Huang
Renzhen Wang
Wanhua Li
Hanspeter Pfister
Deyu Meng
Kede Ma
Ying Wei
Venue: ICLR 2025 Oral (2025)
Keywords: Continual learning; Low-rank adaptation
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Continual Learning (CL) with foundation models has recently emerged as a promising paradigm to exploit abundant knowledge acquired during pre-training for tackling sequential tasks. However, existing prompt-based and Low-Rank Adaptation-based (LoRA-based) methods often require expanding a prompt/LoRA pool or retaining samples of previous tasks, which poses significant scalability challenges as the number of tasks grows. 
To address these limitations, we propose Scalable Decoupled LoRA (SD-LoRA) for class incremental learning, which continually separates the learning of the magnitude and direction of LoRA components without rehearsal. Our empirical and theoretical analysis reveals that SD-LoRA tends to follow a low-loss trajectory and converges to an overlapping low-loss region for all learned tasks, resulting in an excellent stability-plasticity trade-off. Building upon these insights, we introduce two variants of SD-LoRA with further improved parameter efficiency. All parameters of SD-LoRAs can be end-to-end optimized for CL objectives. Meanwhile, they support efficient inference by allowing direct evaluation with the finally trained model, obviating the need for component selection. Extensive experiments across multiple CL benchmarks and foundation models consistently validate the effectiveness of SD-LoRA. The code is available at https://github.com/WuYichen-97/SD-Lora-CL.

Published as a conference paper at ICLR 2025
SD-LORA: SCALABLE DECOUPLED LOW-RANK ADAP- TATION FOR CLASS INCREMENTAL LEARNING
Yichen 2 Hongming Piao'; Long-Kai Huang" Renzhen Wang', Wanhua Li,,
Hanspeter Pfister". Deyu Meng Kede Ma'! Ying Weist
'City University of Hong Kong, Harvard University, 'Xi' an Jiaotong University, *Tencent Al Lab, *Zhejiang University, Pengcheng Laboratory
ABSTRACT
Continual Learning (CL) with foundation models has recently emerged as a promis- ing paradigm to exploit abundant knowledge acquired during pre-training for tack- ling sequential tasks. However, existing prompt-based and Low-Rank Adaptation- based -Lose- methods often require expanding a prompt/LoRA pool or retaining samples of previous tasks, which poses significant scalability challenges as the number of tasks grows. To address these limitations, we propose Scalable Decoupled LoRA (SD--LORA) for class incremental learning, which continually separates the learning of the magnitude and direction of LoRA components without rehearsal. Our empirical and theoretical analysis reveals that SD-LoRA tends follow a low-loss trajectory and converges to an overlapping low-loss region for all learned tasks, resulting an excellent saailittypllsttiiny trade-off. Building upon these insights, we introduce two variants of SD-LoRA with further improved parameter efficiency. All parameters of SD-LoRAS can be end-to-end optimized for CL objectives. Meanwhile, they support efficient inference by allowing direct evaluation with the finally trained model, obviating the need for component se- lection. Extensive experiments across multiple CL benchmarks and foundation models consistently validate the effectiveness of SD-LoRA. The code available at https://github. com/ wwto///aicchhnnn//.c
INTRODUCTION
Continual Learning (CL,, Rolnick et al. 2019, Wang et al. 2024b, et al. 2024, Wang et al. 2022b) aims to develop computational learning systems capable of continually adapting to evolving environ- ments while retaining previously acquired knowledge. In contrast standard supervised learning, which assumes that training data are independent and identically distributed (i.i.d.), CL trains models on non-stationary data where tasks are presented sequentially. This departure from the i.i.d. assump- tion introduces the central challenge of catastrophic forgetting (French, 1999; McClelland et al., 1995; McCloskey Cohen, 1989; Kirkpatrick et al., 2017). indicated by a significant performance degradation of previous tasks when new tasks are introduced.
Over the last five years, foundation models moodels-lrgge-see large-scale pre-trained neural networks have proven highly effective at transferring knowledge while exhibiting strong resistance catastrophic forget- ting (Wang et al., 2022b;a; Smith et al., 2023; Huang et al., 2024; Wang et al., 2024a; Liang & Li, 2024) in the context of CL. One prominent approach centers adapting input and intermedi- ate representations (i.e, prompts) of foundation models to accommodate new tasks. For example, L.P (Wang al., 2022b) and DualPrompt (Wang et al., 2022a) incrementally learn prompt pool and selectively insert the most relevant prompts based on their match with incoming test samples. CODA-Prompt (Smith et al., 2023) refines this strategy by end-to-end optimizing the prompt selection module for CL objectives. Despite obviating manual task identifiers, these methods rely heavily on accurately identifying task-relevant prompts from a potentially growing pool, raising concerns about inference scalability.
"Equal contribution.
Corresponning authors.

---

Published as a conference paper at ICLR 2025
Table 1: Comparisons of existing CL methods with foundation models in terms of three desirable properties: 1) Rehearsal-free (iee, without memory for sample storage), 2) inference efficiency (i.e, without additional computational overhead during inference). and 3) end-to-end optimization (of all model parameters for CL objectives).
Method
Rehearsal-free
Inference Efficiency
End-to-end Optimization
L2P (Wang et al., 2022h)
DualPrompt (Wang et al., 2022a)
x
CODA -Poompt (Smith et al., 2023)
X
HiDe-Prompt (Wang al., 2024a)
X
X
InfLoRA (Liang & Li, 2024)
X
SD-LoRA(Ours)
Another line of work has taken memory-intensive route by retaining samples of previous tasks to bolster performance. Building upon CODA-Prompt, HiDe-Prompt (Wang et al., 2024a) continues learn prompts incrementally but stores large quantities of samples. Likewise, InfLoRA (Liang & Li, 2024) leverages Low-Rank Adaptation (LoRA) (Hu et al., 2022) to remain parameter- efficient, yet similarly rehearses extensive samples during incremental LoRA optimization. This reliance on large-scale memory makes them less scalable in real-world deployments, particularly in conttreeco large-scale scale CL settings.
Table 1 outlines three desirable properties for an ideal CL method with foundation models:
The method should eliminate the need to store samples from previous tasks, thereby ensuring learning scalability;
Rehearsal-free:
Inference Efficiency: The method should maintain computational efficiency during inference, preferably without added computational costs, thereby ensuring inference scalability; End-to-end Optimization: All method parameters should be end-to-end optimized for CL objectives, rather than through segmented and separate optimization stages, thereby maximizing CL performance.
To achieve these properties, we propose Scalable and Decoupled LoRA (SD-LoRA), which incre- mentally adds LoRA components by separating the magnitude and direction learning. By directly employing the finally trained model for testing-without task-specific component selection-S9 LoRA supports computationally efficient inference (Huang et al., 2024), while being rehearsal-free. Through an in-depth empirical and theoretical analysis, we show that SD-LoRA learns to follow a low-loss path that converges to overlapping low-loss region for all learned tasks, thus achieving an excellent
trade-off. Meanwhile, the importance of the incremenaally learned LoRA directions diminishes CL progresses. Building upon these observations, of SD-LoRA with improved parameter efficiency through rank reduction and knowledge distillation, respectively. All SD-LoRA parameters be end-to-end optimized for CL objectives.
introduce two variants
In summary, the principal contributions of our work include
A CL method with foundation Moodlls-SD-Low offering rehearsal-free, inference-efficeen and end-to-end optimized solution. We additionally include two SD-LoRA variants that improve parameter efficiency;
An empirical and theoretical analysis of SD-LoRA, elucidating its plausible working mechanism that climinates task-specific component selection;
A comprehensive experimental evaluation of SD-LoRAs, demonstrating their effectiveness across multiple CL benchmarks and foundation models.
RELATED WORK
Continual Learning (CL), CL seeks to sequentially learn from new tasks while retaining previously acquired knowledge, with the central goal of mitigating catastrophic forgetting. Broadly, existing CL methods can be categorized according to three main design philosophies: Rehearsal-basee regularization-based and architecture- approaches. Rehearsal-basee methods (Riemer et al.,
2

---

Published as a conference paper at ICLR 2025
2018; Chaudhry et al., 2019; Tiwari et al., 2022) selectively retain and replay samples from previous tasks to alleviate catastrophic forgetting. Regularization based methods (Kirkpatrick et al., 2017; Li & Hoiem, 2017: Lee et al., 2019) introduce penalty terms into the training objective constrain updates on parameters deemed important for learned tasks. Architecture- Ebrahimi et al., 2020; Ramesh & Chaudhari, 2021) expand or adapt the model architecture account for new tasks. By allocating additional task-specific parameters or modules, these methods prevent the overwriting of learned important weights. Among various CL settings, this paper focuses on particularly challenging and practical class- incremental learning, in which the model must perform all learned tasks with no access task identity at test time. Conventional glass-inerementee learning methods often require expensive training from scratch parameter-intensive tuning, which can lead to overfitting and interference among tasks.
methods (Mallya et al., 201 20;8
CL with Foundation Models. Foundation models have recently demonstrated their effectiveness in CL by facilitating knowledge transfer across tasks and reducing catastrophic forgetting (Wang et al., 2022b;a; Smith et al., 2023; Wang et al., 2024a; Liang & Li, 2024). Specifically, methods like L2P (Wang et al., 2022b), DualPrompt (Wang et al., 2022a), and CODA Promppt (Smith et al., 2023) integrate Vision Transformers (ViTs) with prompt-tuning strategies (Lester et al., 2021; Jia et al., 2022), thereby improving knowledge retention as new tasks are introduced. Building on these, HiDe-Prompt (Wang et al., 2024a) further stores large number of samples boost performance. Rather than prompt-tuning. LoRA (Liang & Li, 2024) adopts a LoRA-based Aarameter-Ffficiiee Fine-Tuning (PEFT) approach. which similarly necessitates substantial sample storage. Despite their promise, none of the existing CL methods with foundation models simultancously satisfy the three desirable properties outlined in Table To fill this gap, we introduce SD-LoRA, which can also be viewed as form of model -mering techniques (Chitale et al., 2023; Ilharco et al., 2023), developed in parallel with, yet complementary to ongoing CL research.
Parameter-Efficient Fine- Mune-TTning (PEFT). The integration of PEFT methods with CL with founda- tion models is essential because full fine-tuning for each individual task is prohibitive in terms of computation and storage requirements. Representative PEFT methods include adapters (Houlsby et al., 2019), which insert lightweight learnable modules into Transformer layers: prompt-tuning (Qin & Eisner, 2021; Jia et al., 2022) and prefix-tuning (Li & Liang, 2021), which introduce learnable input representations into Transformer layers; and LoRA (Hu et al., 2022), which adds and tunes low-rank branches updates the pre-trained weights. While these techniques have proven effec- tive in single-task and multi-task offline learning settings, their performance boundaries in CL with foundation models remain insufficiently explored. The proposed SD-LoRA adopts rehearsal-free, A-bsed PEFT approach for CL with foundation models.
3 PROPOSED METHOD: SD-LORA
In this section, we first present the necessary preliminaries. We then present in detail the SD-LoRA method for CL with foundation models, accompanied by an empirical and theoretical analysis. Finally, we describe two SD-LoRA variants with improved parameter efficiency.
3.1 PRELIMINARIES
Problem Formulation. Let {1, .2.....To. be N sequential classification tasks. The training (5)
split of Tr, denoted as D, = (-/ M1 (1) [P.) comprises of [P:] training example pairs, where a; representing the input image and y its corresponding label. We consider classification model
(1)
fe. parameterized by 0. When training on D,, no data from previous tasks {T+)==] is accessible. Accordingly, the training objective is given by
1
e (D.; 0) = D1 Σ (fo (x
where -(:-) per-sample loss function such as cross-entropy. For model evaluation, we may compute the average loss of fe across all tasks encountered so far: 7 Ei E(Ve: 0), where V/ denotes the test split of Tg. That is, the overarching goal is ensure that fe performs well on both the current task and all previous tasks. We generally follow the class Blas-inccrmmnntal learning setting described in (Wang al., 2022b; Liang & Li, 2024).
3

---

Published as a conference paper at ICLR 2025
Frozen
Trainable
Decoupled Low-rank Matrix
{
Learnable Magnitude
B
B
Wo E mm
+
A
+
12
AW AB E m
AW a A,A,B, + a2A,
Pro trannd Wights
12
(a)
Illustration of the parameter update in (a) Vanilla LoRA and (b) the proposed SD-LoRA, where the current task index t and r, T1, 72 min(m,
Figure I:
Low-Rank Adaptation (LoRA). As illustrated in Fig. I(a), LoRA (Hu et al., 2022) constrains the parameter updates during fine-tuning lie in a low-rank subspace. Coneretely, let Wo E mxx denote the original weight matrix of layer in the classifier fo- LoRA expresses the parameter update AW Rmxx as the product of two learnable matrices A Rm and B " i.e., AW AB, with < mi, n.. For given layer of so. the LoRA-updated output is
h'
oo + AWa (Wo AB)æ.
(2)
Throughout fine-tuning, the original weight matrix Wo remains fixed.
3.2 SD-L.ORA
In LoRA, the parameter update W can decomposed as follows:
AB
AW
|AB
AB
|AB|
(3)
|AB||
This decomposition highlights two crucial elements of the update: The magnitude (i.e, the Frobenius norm |AB[F) and direction (i.e, the normalized matrix AB). Recent work (Liu al., 2024) has demonstrated that compared to full fine-tuning, LoRA exhibits limited flexibility in precisely adjusting these two elements. This drawback hinders its performance on complex tasks that demand fine-grained control over both magnitude and direction. Furthermore, Qiu et al. (2023) highlighted a more critical role of the direction in fine-tuning than the magnitude.
Motivated by these observations, we describe SD-LoRA for CL with foundation models. In a nutshell, SD-LoRA incrementally decouples the learning of the magnitude and direction of LoRA components, while fixing the directions learned from previous tasks as CL progresses. Concretely, let M = {ax denote the learnable LoRA magnitudes, and W {A Je represent the previously learned directions. As illustrated in Fig. I(b), during learning on Tr. SD-LoRA computes the output of given layer of the classifier fo by
1
h' (Wo + a A, + 02 A, ..... a,
where the color-highlightes terms {au H/= and A,B, are learnable. The original weight matrix Wo 1-1 and the previously learned directions B k= remain fixed.
3.3 EMPIRICAL ANALYSIS OF SD-LORA
By incrementally decoupling the magnitude and direction LoRA components while preserving the directions learned from previous tasks, observe substantial performance gains across various CL benchmarks (see Sec. 4). Nevertheless, the underlying working mechanism- cecchnismm-particulaat) how SD-LoRA mitigates catastrophic forgetting- remaains poorly understood. To shed light this, conduct a series of experiments and distill our observations into three key findings.
When fine-tuning the foundation model directly different downstream tasks, the resulting task-specific weights end up closer to each other than the original model weights. To illustrate this, consider five tasks drawn from ImageNet-R (Boschini al., 2022), and fine-tune
Finding 1:
4

---

Published as a conference paper at ICLR 2025
2x((W: -I] Wiils Wille
ImageNet-R (N 5)
ImageNet-R (N= 10)
Average Accuracy
36
LofA with Fixed Direction
Average Accuracy
LoRA with Fixed Direction
Task Index
24
500
80
17.5
15.0
123
Index
Number of
Nurbee of Tasks
(a)
(b)
(c)
Figure 2: (a) Distances between the five optimal weights (W;) on ImageNet-R (N 5) relative to the foundation model weights Wo. All relative distances are much smaller than one, indicating that } are closer to each other than to Wa. (b) and (c) Performance comparison of Vanilla LoRA versus LoRA with the first learned direction fixed, on ImageNet-R across five and ten tasks, respectively. Shaded regions indicate standard error.
the ViT-B-16 model (Dosovitskiy et al., 2020) for each task, thereby obtaining five sets of optimal task-specific weights WW As shown in Fig. 2(a), measuring the relative distances in parameter space reveals that these task-specific weights cluster more closely with one another than the original weights of the foundation model Wo.
We further conduct CL experiment in which only the LoRA magnitude is continually optimized, while the direction remains fixed after the first task. Consequently, the updated output for given layer at the current T. becomes h' = (Wu + a As shown in Figs. 2(b) and (c), the average accuracy up the current task consistently surpasses that of the vanilla LoRA baseline, i.e, h' (wo + B Aligning well with (Entezari et al., 2022; Gueta et al., 2023), our results further indicate that the fine-tuned weights for different tasks lie in close proximity, enabling relatively strong performance even when fixing a single learned direction.
ImageNet-R (N 5)
ImageNet-R (N= 10)
After training on
16
Residual
Standard arror
Magnitude
2.0
1.8
training
training
Magnitude
After training on
1.4
122
01
Iteration
200
9
82
0n
01
24
(a)
(b)
(c)
Figure 3: Analysis of the learning process of SD-LoRA. (a) Least squares fitting residual between the newly learned direction A,B, and all previous directions {A, 14-1 over time. (b) and (c) Learned magnitudes {ou J ImageNet-R across five and ten tasks, respectively.
Finding 2: The directions preserved from previous tasks (i.e,, (AB play significant role in the CL process-particulat
those learned in the initial tasks. To reveal this, we first compute the least squares fitting residual between A, and [A,Ba)t 1--1 which increases over time (see Fig. 3(a)). Initially, the newly learned direction strongly aligns with earlier ones, enabling learned direction reuse. As training continues, A, B, gradually diverges, incorporating subtle variations that distinguish it from previous directions.
Further analysis of the learned magnitudes. all initialized to reveals that at values corresponding to earlier tasks rise rapidly, while those for later tasks exhibit general decline trend (see Figs. 3(b) and (c) well Appendix A.2). This pattern suggests that the classifier increasingly relies directions learned from the earlier tasks, whereas the recently introduced directions serve primarily slight adjustments accommodate the specific requirements of the later tasks.
Finding 3: SD-LoRd LoRA effectively uncovers a low-loss path by leveraging the fixed directions from previous tasks with the learned magnitudes (av, ) k= loward low-loss region shared by all tasks. To
N
5

---

Published as a conference paper at ICLR 2025
Wo
Low-loss region for Ti /3/
Vanilla LoRA
SD-LoRA
63.0
Accuracy
Tank Accuracy
Task Accuracy
Task Accur acy
W/-ORA
wish
W.
12
Task Yask
$2.0
64.
89.0
20.3
60.4
WgD
60.0
Interpolation
Interpolation
(a)
(b)
(c)
Figure 4: Learning trajectory comparison of vanilla LoRA and SD-LORA. (a) Toy illustration of the learning trajectories for vanilla LoRA (W, - WL + 2 and SD-LoRA (Wo - W - W across two sequential tasks. (b) Classification accuracy along the vanilla LoRA path. The improvement on Tn but degradation on Ti indicates that vanilla LoRA suffers from catastrophic forgetting. (c) Classification accuracy along the SD-LoRA path, showing that it successfully lands on an overlapping low-loss region.
verify this, we conduct weight interpolation experiments to examine the linear path between the two sets of model weights for two sequential tasks. As depicted in Fig. 4, along the linear path from WSD to D learned by SD-LoRA, the performance on T/ steadily improves without loss in accuracy on Ti. However, this is not the case for vanilla LoRA, where the performance improvement of Tz is at the expense of Ti, indicative of catastrophic forgetting. These observations suggest that SD-LoRA selectively scales the parameter update along the previously learned directions, effectively enabling overlapping low-loss region for all
the classifier to trace low-loss path that ultimately settles tasks (see Fig. 4 (a)).
These findings elucidate why SD-LoRA excels in CL with foundation models. Initially, it identifies critical LoRA directions during learning earlier tasks, and relies heavily on these directions to guide the classifier toward an overlapping low-loss region for learned tasks. Subsequently, by progressively incorporating LoRA components, SD-LoRA refines these directions to converge on the shared low-loss region for both earlier and later tasks. This mechanism of tracing a low-loss trajectory eliminates the need to store samples from previous tasks for task-specific component selection, making SD-LoRA strong and rehearsal-free.
3.4 THEORETICAL ANALYSIS OF SD-LORA
In this subsection, based on the results in (Jiang et al., 2023), we present a theoretical analysis to explain why the initially learned LoRA directions are so critical (as in Finding 2).
Let AW* E R=*xn be the optimal update matrix lying in the overlapping low-loss region for all N sequential tasks. Additionally, let (AWi)N represent the optimal update matrices in their respective low-loss regions. Denote the singular values of AW; 0i 2 2 mmn 2 0. The matrices A E Raxr and B E R are updated iteratively. starting from initial values given by 3/m-n+7 (Ag Bo), where the entries Ag and B, are i.i.d. according to (0 o1), and p is the initialization scaling factor. For integer j in the range {0, I,.., min(r, m, n)), define the j-th condition number as K, d. Finally, let llop denote the operator norm.
2
Theorem 1. Suppose the assumptions stated in Appendix A.I hold, where e1 is a small constant. Let E (0,1) be such that 5 & minge(L....) GL-Ok+1 Fix any tolerance level 62 satisfying 62 S m+n+t. Let n denote the learning rate for updating the matrices and B, and define AWli as the rank-i approximation of AW* obtained by retaining the top-i principal components.
Then, there exist some numerical constants e and c, and a sequence of iteration indices:
1 -. S d
Kj
log
8no
8Gz

---

Published as a conference paper at ICLR 2025
such that, with high probability, gradient descent with step size S cmin 1 8] and initializa- tion scaling factor p S (c0e2) ) ensures that the approximation error satisfies
Ni
A BiL -
S 6201 + c1,
4k 1.2.....
(5)
In Theorem 1, we formulate the learning process of SD-LoRA prove that gradient descent with small initialization drives the learned product AB to approximate the principal components of AW", i.e, A AWI1, A AW ...., A AWlll sequentially. This theoretical insight explains the observed decreasing trend in the learned magnitudes, and further supports the feasibility the subsequent parameter-efficient variants of SD-LoRA.
matrix factorization problem, and
3.5 TWO VARIANTS OF SD-LORA
Our analysis in Sec. 3.3 reveals an interesting behavior of SD-LoRA: LoRA directions acquired during learning on earlier tasks are heavily reused and contribute substantially to classification accuracy. In contrast, directions learned through later tasks primarily serve minor refinements, exhibiting a diminishing utility. We leverage this behavior and propose two variants of SD-LoRA with improved parameter efficiency, which integrate rank reduction and knowledge distillation, termed SD-LoRA-RR and SD-LLRA-KD,, respectively.
SD-LORA-RR. To mitigate incremental parameter expansion, implement empirical rank- reduction strategy for the learnable matrices A, E Rmxr and B, E R associated with later tasks. Specifically, the rank Tt is reduced in a stepwise manner:
r1 = = > rr Tp+1
= T2+1 = TN:
(6)
where 1 and v are predefined task indices. The set of hyperparameters include {p. V4 F1, Tus rv). This stepwise reduction ensures that later tasks, which contribute less, are encoded with lower-rank approximations, thereby curbing computational and memory overhead.
SD-LORA-KD. While the rank-reduction strategy limits parameter expansion, each new task still grows parameters incrementally. To fully address this issue, we propose knowledge distillation approach based on least squares. Our method evaluates whether newly introduced LoRA direc- tion A,B, can be linearly represented by the subspace spanned by previously learned directions Bx k==" If sufficient linear approximation exists, the fitting coefficients will be absorbed into existing learned magnitudes M rather than expanding the direction W. Formally, after training on Tú we solve the least squares optimization problem:
k=1
arg min
A,B,
Σ a k=1
where Day denotes the optimal coefficient for the k-th learned direction. If the fitting residual is less than a predefined threshold r, we assimilate the new direction by updating the first 1 learned magnitudes from as
h = (Wo + (an Ao) A B, + (az Aoz) (o2) + (a.-1 + Aor A, BB x, (8)
which prevents parameter expansion while preserving knowledge through coefficient fusion. The complete implementations of SD-LoRAs are detailed in Algorithm
EXPERIMENTS
In this section, we first present the experimental setups, and then compare SD-LoRAs with state-of- the-art CL methods across multiple benchmarks and foundation models.
4.1 EXPERIMENTAL SETUPS
Evaluation Benchmarks and Protocols. Following (Gao et al., 2023; Liang & Li, 2024), evaluate SD-LoRAs on three standard CL benchmarks: ImageNet-R (Boschini et al., 2022), ImageNet- (Hendrycks et al., 2021), and NomainNet (Peng et al., 2019). Specifically, ImageNet-R consists
7

---

Published as a conference paper at ICLR 2025
Algorithm 1 SD-LoRA and its Variants on the Current Task Tr
Input:
Weight matrix from the foundation model Wo, current task Tr, learned LoRA directions
from previous tasks W = (A k= rank parameters {u, u, 11111p "p rut in Eqn. (6), residual threshold for Eqn. (7), and maximum number of iterations Maxiter.
Output: Sets of learned LoRA magnitudes M and directions W.
I: Initialize A, E mmxr. B, Ruxx and M {ou)k=1
2: t jl then
/ Only for SD-LORA-RR
3: Reduce the lower dimension of A, and B, to Ty or Tv
4: end if
5: for Iter = 0 to MaxIter do
6: Compute the cross-entropy loss the current task To using Eqn. (4)
7: Update {o=)k=1 and A, by minimizing Eqn. (1) using some stochastic optimizer
8: end for
9: W + (U(A,B.)
10: Solve Problem (7) to obtain the optimal fitting coefficients (A / Only for SD-LORA-KD
11: if the fitting residual A, E Aoa, B
then
M fax + 1 and W + [A 1-1
Table 2: Performance comparison ImageNet-R across different task lengths.
ImageNet-R (N 5)
ImageNet-R (N
10)
ImageNet-R (N 20)
Method
Acc f
AAAF
Acc
AAA 1
Acc 1
AAA 1
Full Fine-Tuning
75.57 (4.92(0.77)
60.57
72.31
49.95(1.31)
65.32(0.84)
1.2P
(3.04
(6.94(0.41)
(1.26(0.44)
(6.13
68.97 (0.51)
74.16(0.32)
DualPrompt
(9.99)).77)
72. (o.44
(8.22
73.81 (0.39)
65.23 (3.43
71.30(0.16)
KODA-Prompt
76.63 (6.27
80.30
78.14 (8.14 (0.05
(0.33)
73.95
HiDe-Prompt
74.77 (0.25)
78. (8.15
(4.65(0.14)
(8.46)
73. (9.59(0.19)
77 77.93(0.19)
InfLoRA
(6.95
81.81 (0.14)
74.75(0.64)
80.67 (0.55)
69. 19.899,..ok (0.56)
76.68
SD-LoRA
79.15 79.15 (0.20)
(3.01.0.42) (0.42)
77.34 (0.35)
82.04 (0.24)
(5.26
(0.22
SD-LORA-RR
79.01
82.50
(7.18
(1.74
74.05
80.65 (0.35)
SD-LORA-KD
(8.
82.47
77.03
81.52
74.12
80.11 (0.75)
of 200 ImageNet classes (Deng et al., 2009) rendered in artistic styles. ImageNet-A features 200 classes with natural adversarial examples, often misclassified by standard ImageNetttrained models. DomainNet includes 345 classes across six distinct domains. As common practices (Liang & Li, 2024; Huang et al., 2024), split ImageNet-R into 5/10/20 tasks (40/20/10 classes per task), ImageNet-A into 10 tasks (20 classes each), and DomainNet into 5 tasks (69 classes each). Additionally, we include CIFAR100 (Krizhevsky, 2009) and CUB200 (Wah et al., 201 1) results in Appendix A.3.
We adopt two standard and widely used CL metrics: Average accuracy (Acc) and average anytime accuracy (AAA). The Acc metric measures the overall performance by computing the average accuracy across all N tasks upon the completion of CL. AAA further accumulates the average accuracy of all encountered tasks after training on each new task.
Competing Methods and Implementation Details. We compare SD-LoRAs against state-of-the-an ViT-based CL methods, including L2P (Wang et al., 2022b), DualPrompt (Wang et al., 2022a), CODA-Prompt (Smith et al., 2023), HiDe- PrDDe-Prompt (Wang et al., 2024a), and InfLoRA (Liang & Li, 2024). We also incorporate full fine-tuning as a form of performance lower bound. Following prior work (Gao et al.. 2023; Huang et al., 2024), employ ViT-B/16 (Dosovitskiy et al.. 2020), pre- trained on ImageNet-211K and fine-tuned on ImageNet- IK as the foundation model for classification. We also experiment with self-supervised V/T-B/16 from DINO (Caron et al., 2021). The SD-LLRR components are inserted into the attention layers of all Transformer blocks, modifying the query and value projections, with fixed rank of r 10. It noteworthy that we utilize shared set of LoRA magnitudes for all projections, each with different LoRA directions. For SD-LORA-RR, we set the additional rank parameters as μ = set the threshold for the fitting residual to
Fμ 8, and r, For SD-LORA-KD, we 10 For all methods, training is carried out by
8

---

Published as a conference paper at ICLR 2025
Table 3: Performance comparison on ImageNet-A and DomainNet across different task lengths.
ImageNet-A (N = 10)
DomainNet (N 5)
Method
Acc
AAAT f
Acc
AAA 7
Full Fine-Tuning L2P (Wang et al., 2022b)
(7.88)
30.04 13.118
51.466o..5
67.08 (1.13)
(2.94(1.27)
51.40(1.95)
70. 25.
75. (0.83
DualPrompt (Wang ct al., 2022a)
45.49 45.49/0.90)
(4.68(1.24)
68. 26,0.90
73. (3.84(0.45)
CODA-Prompt (Smith ct al., 2023)
45.30 45.36(0.78)
57. (0.94)
70.58to.5a)
76.68 (0.44)
HiDe-Prompt (Wang et al., 2024a)
42. 12.70(0.00)
56.32 (0.4)
72. (0.00)
77.01 (0.04)
InfLoRA (Liang & Li, 2024)
49, (9.20(1.12)
60.92.0.61)
(0.59
78.29 (8.29
SDLoRA
(5.96
64.95
(2.82
78.89 (0.50)
SD-LORA-RE
55.59 (1.8)
64.59(1.91)
72.58 (2.58
78. 79
SD-LORA-KD
54.24(1.12)
(3.8)
72. (5
78. (8.44))
ImageNet-A (N 10)
ImageNet-R (N 10)
ImageNet-R - with DINO
80
Dus inrrpt
colx Pronig::
Accuracy
200
Accuracy
05
Fronce
Accuracy
60
50 Lond-et
50- L0-0000D
cook-rumpt HDWwrrmm
Inturt
50
60-
5o
50
40
8
6
Number of Tasks
Number of Tasks
Number of Tasks
(a)
(b)
(c)
Figure 5: Average accuracy during sequential training on (a) ImageNet-A (N = 10), (b) ImageNet-R
(N =10,, and (c) ImageNet-R (N 5) using ViT-B/16 from DINO (Caron et al., 2021).
Adam (Kingma & Ba, 2014) with a learning rate of 0.008 and a minibatch size of 128 for 30 epochs on ImageNet-R, 10 epochs on DomainNet, and 20 epochs on all other datasets. We report mean results across five runs with standard errors.
4.2 EXPERIMENTAL RESULTTS
Results on Different CL benchmarks Using Different Backbones. Tables 2 and 3, is clear that SD-LORA achieves significant improvements over existing methods. Specifically, on ImageNet- R (N = 20), SD-LoRA surpasses InfLoRA by margins 7.68% in Acc and 4.62% in AAA. Similarly, on ImageNet-A. SD-LoRA outperforms HiDe-prompt by approximately 31.05% in Acc and 15.32% in AAA. Even on the more complex DomainNet, comprising six distinct domains, SD-LoRA consistently attains the best performance. To demonstrate the generality of SD-LoRA, we also evaluate it using the self-supervised ViT-B/16 from DINO. The results in Fig. 5(c) show that SD-LoRA continues to deliver superior performance under both Acc and AAA using different backbones. Finally, we observe that the two variants SD-LoRA-PE RRR and SD-LoRA- KD, exhibit only marginal performance degradations relative the full SD-LORA model, confirming the effectiveness of their parfieter-efffcieet designs.
Results across Varied Task Lengths. To evaluate the scalability and generalizability of SD-LORAS under different task lengths, we follow (Liang & Li, 2024; Huang et al., 2024) and partition ImageNet- into 5, 10, and 20 sequential tasks containing 40, 20, and 10 classes per task, respectively. As shown in Table 2, SD-LoRA, As demonstrate consistent superiority over existing methods, with performance margins growing as the number of tasks. This underscores the suitability of SD-LoRAs for scenarios requiring resource-efficient CL without compromising accuracy.
Ablation Studies. To validate the contributions of design choices in the proposed SD-LoRA, we conduct series of ablation experiments, with quantitative results summarized in Table 4. First, we fix the singly learned LoRA direction while allowing its magnitude to adapt during training. This simplified configuration already achieves nontrivial performance, highlighting the critical role of the initial LoRA direction in CL. Second, we decouple the magnitude and direction learning, but restrict the classifier to single LoRA component. The inferior performance relative to SD-LoRA

---

Published as a conference paper at ICLR 2025
Table 4: Ablation analysis the proposed SD-LoRA. Trainable parameters are highlighted in orange.
ImageNet-R (N = 5)
ImageNet-RR (N = 10)
Training Strategy
Acc
AAA f
Acc
AAA f
Wo + o,
78. 78.17 (0.27)
(1.93
(4.82
80.63
Wo +aAB
73.24
(8.80(o.13)
70.62 (0.78)
76.32(o.16)
Wo A A, + B,
78. 28 (0.59)
82.02
74.29 (0.32)
79.74
Wo + 0. A
B, (SD-LoRA)
79.15 (0.20)
83.01 (0.42)
77.34 77.34 (0.35)
82.04(0.24)
Table Comparison on ImageNet-R (N 20) terms of computation (GFL.OPs), parameter, and storage efficiency.
Learnable Parameters (M)
Stored
Method
GFLOPs
Features (M)
L2P (Wang et al., 2022b)
70.14
0.48
0
DualPrompt (Wang et al., 2022a)
70.26
0.06
0
CODA-Promes (Smith et al., 2023)
70.61
0.38
0
HiDe-Prompt (Wang et al., 2024a)
70.36
0.08
0.15
Infl.oRA (Liang & Li, 2024)
35.12
0.37
0.10
SD-LoRA
35.12
0.37
0
SD-LoRA-RE
35.12
0.23
suggests that the performance gains SD-LoRA cannot attributed solely to decoupling. Instead, the synergistic effect of training multiple decoupled LoRA components is essential for achieving satisfactory results. Last, we fix learned LoRA components without magnitude rescaling, and also observe noticeable performance decline. This suggests that the rescaling mechanism enables SD-LoRA to navigate low-loss paths by reweighting contributions from earlier components.
Analysis of Computation, Parameter, and Storage Efficiency. As presented in Table 5, compare the inference computations in terms of GFLOPs, trainable parameters, and feature storage requirements of various CL methods. Notably, InfLoRA (Liang & Li, 2024) and the proposed SD-LoRA eliminate the need for task-specific prompt selection during inference, thereby enjoying the highest inference efficiency. Moreover, our proposed SD-LORA-RO capable of further reducing the number of LoRA parameters without reliance on sample rehearsal, making it an ideal choice for resource-constrainedd CL scenarios.
CONCLUSION AND DISCUSSION
In this paper, have introduced SD-LoRA, computational method designed to address scalability challenges class-incremental learning with foundation models. By decoupling the learning of magnitude and direction of LoRA components, SD-LoRA provides rehearsal-free, inference- efficient, and end-to-end optimized solution. Our empirical and theoretical analysis demonstrates that SD-LoRA uncovers low-loss trajectory that converges an overlapping low-loss region for all learned tasks, effectively balancing stability and plasticity. Extensive experiments confirmed the effectiveness of SD-LoRA in mitigating catastrophic forgetting while maintaining adaptability to new tasks. Additionally, our two parameter-efficient variants, SD-LoR L-RA-RR and SD-LoRA-KE --R,DD, further enhance practicality for resource-constrained applications.
While SD-LoRA has shown promise, several avenues for future research warrant exploration. First, extending SD-LoRA A to other foundation models beyond ViTs could provide valuable insights into its generality and effectiveness across different backbone architectures. Second, integrating SD-LoRA with other PEFT techniques, such as adapters or prefix-tuning may further enhance its performance and scalability. Finally, developing more theoretically grounded strategies for rank reduction and knowledge distillation within SD-LoRA could lead to additional improvements in parameter efficiency and overall performance.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We would like to thank Ziye Ma and Xinyuan Song for helping formalize the proofs, and Xuelin Liu for assistance with the plots and diagrams. This work was supported in part by the National Key R&D Program of China (2020YFA071 (AAA0011900) 13900), the Hong Kong RGC General Research Fund (1 (11220224), the CityU Applied Research Grant (9667264), the National Natural Science Foundation of China under the Tianyuan Fund for Mathematics (12426105) and under Grant 62306233, and the Major Key Project of PCL (PCL2024A06).
REFERENCES
Matteo Boschini, Lorenzo Bonicelli, Angelo Porrello, Giovanni Bellitto, Matteo Pennisi, Simone Palazzo, Concetto Spampinato, and Simone Calderara. Transfer without forgetting. In European Conference on Computer Vision, pp. 692-709, 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision Transformers. IEEE/CVE International Conference Computer Vision, pp. 9650-9660, 2021.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv: 1012.10486 10486, 2019.
Rajas Chitale, Ankit Vaidya, Aditya Kane, and Archana Ghotkar. Task arithmetic with LoRA for continual learning. arXiv preprint arXiv:2311.022: 1.00488, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16 x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adver- sarial continual learning. In European Conference on Computer Vision, pp. 386-402, 2020.
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance linear mode connectivity neural networks. International Conference on Learning Representations, 2022.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, (4):128-135, 1999.
Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang unified continual learning framework with general parameter-effiiiet tuning. In IEEE/CVF International Conference Computer Vision, pp. 1 11483-11 11493, 2023.
Almog Gueta, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Knowl- edge is region in weight space for fine-tun tune language models. In Findings of the Association for Computational Linguistics, pp. 1350- 2023.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Christina Kwon, and Jacob Stein- hardt. Natural adversarial examples. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262-15271, 2021.
Neil Houlsby, Andrei Giurgiu. Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficientt transfer learning for NLP. In International Conference Machine Learning, pp. 2790-2799, 2019.
11

---

Published as a conference paper at ICLR 2025
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Yuanzhi Li, Shean Wang, and Weizhu
Wei-Cheng Huang, Chun-Fu Chen, and Hsiang Hsu. OVOR: OnePrompt with virtual outlier regu- larization for rehearsal-free class-incremental learning. In International Conference on Learning Representations, 2024.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference on Learning Representations, 2023.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727, 2022.
Liwei Jiang, Yudong Chen, and Lijun Ding. Algorithmic regularization model-free over- parametrized asymmetric matrix factorization. SIAM Journal on Mathematics of Data Science, pp. 723-744, 2023.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXhv:1412.6980, 2014.
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinnka, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic for- getting in neural networks. Proceedings of the National Academy of Sciences, 1 11<(13)3521-3326, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical re- port, University of Toronto, 2009. URL https://www.cr cs toronto. wew.ootto.duuuoa.eo ~krzz/ Bearning-features 200- TR.
Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee. Overcoming catastrophic forgetting with unlabeled data in the wild. In IEEE/CVF International Conference Computer Vision, pp. 312-321, 2019.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Conference Empirical Methods in Natural Language Processing, pp. 3045-3059 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2 100.00/99 2021.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pallern Analysis and Machine Intelligence, 40( (1)):2935-2247, 2017.
Yan-Shuo Liang and Wu-Jun Li. InfLoRA: Interference-free low-rank adaptation for continual learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23638 23647, 2024.
Shih-yang Liu, Chien-Y Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang F Wang, Kwang- Ting Cheng, and Min- Min-Hung Chen. DoRA: Weight-decomposed low-rank adaptation. In International Conference on Machine Leurning, pp. 32100-32121, 2024.
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback:
Adapting single network to multiple tasks by learning to mask weights. In European Conference on Computer Vision, pp. 67-82, 2018
James L McClelland, Bruce L McNaughton, and Randall C O'Reilly. Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102(3) 1995.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, pp. 109-165, 1989.
12

---

Published as a conference paper at ICLR 2025
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In JEEE/CVF International Conference on Computer Vision, pp. 1406-1415, 2019.
Guanghui Qin and Jason Eisner. Learning how ask: Querying LMs with mixtures of soft prompts. arXiv preprini Siiv:2104.06599, 2021.
Zeju Qiu, Weiyang Liu, Haiwen Feng. Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. In Advances in Neural Information Processing Systems, pp. 79320-79362, 2023.
Rahul Ramesh and Pratik Chaudhari. Model Zoo: A growing "brain" that learns continually. arXiv preprint arXiv:2 0ww.06603022. 2021.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro, Learning learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:I810. 11900, 2018.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems, pp. 350-360, 2019.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante- Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and 7solt Kira. CODA-Prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11999-11919, 2023.
Rishabh Tiwari, Krishnateja Killamsetty, Rishabh Iyer, and Pradeep Shenoy, GCR: Gradient coreset based replay buffer selection for continual learning. /EEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 90-108, 2022.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech- UCSD Birds-200-2011 Dataset. Technical Report ENS-TR-2011-40 1-001, California Institute of Technology, 2011, 1. URL http::ttt/// /atts////uttss../ alibrrryy ry ca I tech. edu/ 27 27452/.
Liyuan Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical decomposition of prompt-base continual learning: Rethinking obscured sub-optimality. In Advances in Neural Information Processing Systems. pp. 69054-69076, 2024a.
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Puttern Analysis and Machine Intelligence, 46(8):5362-5385 2024b.
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, and Jennifer Dy. DualPrompt: Complementary prompting for rehearsal- free continual learning. In European Conference on Computer Vision, pp. 631-648, 2022a.
Zifeng Wang, Zizhao Zhang, Chen Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 139-149, 2022b.
Da-Wei Zhou, Qi- Wang, Zhi Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Class- incremental learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):9851-987) 2024.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A. PROOF OF THEOREM 1
In this section, we prove Theorem 1 as specific case of Theorem 2.
Assumption 1. The optimal updates A Wit E {1, 2, : , lie within a small neighborhood in the loss landscape, meaning there exists small constant E| > 0 such that | AWW < <11.
Assumption 2. The first j + 1 singular values of AW; are distinct, i.e, 0i > >... >jj > jjtl- Theorem 2. Fix any S r. Suppose 0= < Jg choose any ? E (0,1) such that Ø+1
o, S 7 Pick
any stepsize n S min { 70, (1->)=2] }. For any cp < 1, let the initialization size p satisfy
600m?
200;
1-7
10
p S min
24
12(m m + + r
1-7
Vo
24
and
2017
(1 (1-7)) 7)
fo
y0,v 2J
p S min
1200(m + n + r)jor
16000r07
160, v m + 12
Define
log(
12(m+n+rr+ / vor
24 log( V
I-7
T
+
T2
+1,
log(1 + 1+7 ,
log(1 + 001111,))
1-x log(p का+न /p)
log(p ア,क5 /3)
T3
+ 11
T
log(1 n1,,
log(1 + mm,o))
Define To = Ti + T2 + Ts, then we have
To
(3 - (1
1
T
6(3 +
Furthermore, there exists universal constant C such that with probability at least (Ccp) C' Cexp(-r/O),fo for all To < i T, we have
A B A AW loo I 8p on Ap 22jor +
(A.9)
Proof. The sequential training of LoRA can be conceptualized as matrix approximation prob- lem (Jiang et al., 2023):
1 |AB - AW; P2
((A, B)
(A.10)
2
where A W denotes the optimal update matrix for the current 1-th task. We compute the gradients of l with respect to A and B, respectively:
VAl= (AB AW;
Vg = AT - AW?).
Then, the gradient descent updates with a step size of 1) are
A = V + (AW; BT)
B, B+=B B B B AT AW?). By performing the singular value decomposition (SVD) of AW;,
AW; क E, V.
14

---

Published as a conference paper at ICLR 2025
and exploiting the rotational invariance of the Frobenius norm in Eqn. (A.10), i.e, by substituting A + Φ A and B + B,B, we may assume without loss of generality that AW; diagonal.
We then rewrite A AW7
E 0 AW;
0 Σ
with Σ, dia.. og)
Rixj and Σ (mm-))x(n-) be diagonal matrix with jj+1:-... min(m,n) on the diagonals. We next introduce the following block partitions:
U (
A =
J
and == (V K)
where
U
J E ((m-j)xr j)xr VERMA K KER' R'x((())))
In this way, we can decompose
AB AW? as
AB AW; UV E.
UK
JV
JK E/
and we are ready to bound the difference AB AW
||B AW*l llop
|AB AW, + AW; - A W opp
< AB
AW7 olopp WAW; AW*
op
UV Exllop |UK op JV
op
JJK
Eyllop + 61,
where we note that E, = Aw! [5]] (after diagonalization of A W To ensure convergence, it suffices
to show that the dominant term UV approaches E, while the error terms (J, K) remain small. Let us first extract the gradient descent update of the top left block U:
U+ =U E. UV)VT (-UK) KT
-U -U+ EE VT U KKT)).
Similarly, we have
V, V + UT E. (UTU +TJ+)
J + EKKT -VVV + KKT)
K+ K + JE JJJ+K
To account for the potential imbalance of U and V, we introduce the following quantities,
U+
2 and G V
VT
F
so that
=
G
and
VT F F G.
Then, the updates for and are given by
F- U- +V
= U VT
E, VT + E.U) (UU(VVT+ KKT) + (T(UT
F + n Σ, F - /
(F + G) (VVV+VV+T + (F (UUU+
similar computation gives
G (+
[ (F (VVT +KT) - - -)U(UU+JT))) (VV-KKi)--CCG((OUU---]
= G GG
15

---

Published as a conference paper at ICLR 2025
It is now natural to introduce the following equations:
P Ex FFT GGT,
Q == GFT,
so that one can verify that
P Q - Σ, UV.
According the Proposition B.2 and B.5 of Jiang et al. (2023), it holds with high probability that for any Ti + 22 + T3 sisT,
||U,K /lop S 3p 01, J Viloo S 23p 01,
JJKKillop S pitioo 1Qillop I 4paliti) / 22001.
P Popp 1 (1 - -0.79h997
| P + 80mp ++; jo1
$ (1 3/101
P
Pi-illop + 80mpti7
i-7
01 + 80p - jof
< 2
1
3110,
2
Given that T3- = Ing( オ( /3)
1, it follows that for all satisfying Ti + T2 + T33 Sis T, we
log(1 11r,)
have PPollop S p2(14950 01.
IU.V Eploop = IP, + Qillop S [PPllop + QQillop S p p σ1 + Ap 2jor
By combining these parts, we can have
|. - Aw I 8p + 4p 2jo1 +1 (Sp + 4p v 22))o +
A.2 ADDITIONAL RESULTS FOR SEC. 3.3
To further investigate the temporal evolution of learned LoRA magnitudes M = fox IN k=1 in SD- LoRA, we conduct additional experiments on ImageNet-R (Boschini al., 2022) with an extended task length of N = 20 and a more challenging DomainNet dataset (Peng et al., 2019). As visualized in Fig. 6, both experimental configurations reveal systematic decrease in ag values throughout the training process. These results corroborate the descending trend observed in our main experiments, demonstrate the consistent behaviors of SD-LoRA across extended task horizons and diverse domain distributions, and align with the theoretical analysis presented in Sec. 3.4.
ImageNet-R 20)
DomainNet
1.3
After training
2.25
After training on 5
Magnitude
1.2
Magnitza
After training
2.00
training on
1.75
1.50
1.0
1.25
(b)
Figure 6: Learned LoRA magnitudes M = (ax J.N
in SD-LORA on (a) ImageNet-R (N -20) and
(b) DomainNet (N = 5).
16

---

Published as a conference paper at ICLR 2025
DomainNet
ImageNet-R (N = 10) with DINO
ImageNet-R (N= 10)
Task Index]
Task Index
Task
18-
Task Index /
Task Index
Task Index
(a)
(b)
(c)
Figure 7: Relative distances computed on (a) DomainNet (N 5), (b) ImageNet-R (N 5) using ViT-B/16 from DINO, and (c) ImageNet-R(N = 10), respectively.
A.3 RESULTS ON OTHER CL BENCHMARKS
In addition to ImageNet-R, ImageNet-A, and DomainNet, we evaluate the proposed SD-LoRA on two other widely recognized benchmarks: CIFAR-100 (Krizhevsky, 2009) and CUB-200 (Wah et al., 2011). CIFAR- 100 is standard dataset for image classification, comprising 60, 000 images evenly distributed across 100 classes, with 600 images per class. For our experiments, we split CIFAR-100 100 into ten tasks, each containing ten classes. Similarly, CUB-200 a fine-grained dataset specifically designed for bird classification, which consists of 11, 788 images across 200 classes. We divide this dataset into ten tasks, with each task encompassing 20 species. As shown in Table the proposed SD-LoRAs consistently deliver outstanding performance on both datasets.
Additionally, we provide supplementary results to those in Fig. 2(a) by analyzing the relative distances between fie-tuned and pre-trained weights across different benchmarks, backbones, and task lengths. As shown in Fig. 7, the observed trends remain consistent with Finding 1.
Table 6: Performance comparison on CIFAR and CUB200.
CIFAR100
CUB200
Method
Acc f
AAA f
Acc
AAA
Full Fine-Tuning
(0.50)).50)
80.35xo.s7)
51.41
69.74(0.93)
L2P (Wang et al., 2022b)
83.18
87.69(1.05)
65. 65.18(2.49)
76. 76.12(1.2)
DualPrompt (Wang et al., 2022a)
81.48
86. 41 (0.68)
68.00(1.06)
79. 79.40(o.8s)
CODA-Prompt (Smith al., 2023)
86.31 (2)
(0.22)
71 71.92 (0.33))
78.76(0.65)
onfl (Liang I., 2024)
86.75
(1.72(0.15)
70.82
(1.39
SD-LoRA
88.01 (0.31)
92.54
77.48 (0.20)
85.59
SD-LORA-RR
87. (0.2)
92.05ço.31)
76.35
83.89(0.35)
SD-LORA-KD
87.09
92.01 (0.33)
(5.95
83.21 (0.31)
17


Reviews:
[{"review_id": "OlHt9730Bj", "reviewer": "Reviewer_6nN3", "summary": "The authors propose to incrementally learn how to merge different LoRA models by keeping old LoRA modules fixed and only update their weights and a new LoRA module in the memory-free CL scenario. The authors compare their method S-LoRA as well as variants of it on multiple benchmarks and different benchmarks against multiple baselines.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors present a simply, but principled idea. It makes a lot of sense to learn to fuse models in the context of CL.\n\nBesides the choices mentioned in \"Weaknesses\", the empirically evaluation is convincing.", "weaknesses": "The presented work uses ideas from model merging/fusing, e.g., \"Editing Models with Task Arithmetic\" by Ilharco et al., but doesn't discuss these works in the related work. Adoption of these ideas to CL, such as \"Task Arithmetic with LoRA for Continual Learning\", are not discussed or not compared to.\n\nThe authors compare apples with oranges. Most baselines use inferior PEFT methods (prompt tuning) while the authors use LoRA which naturally will give them an edge. It would be great if the authors could show that the improvement over the baseline is not coming from the PEFT method. See a discussion on this topic in \"Choice of PEFT Technique in Continual Learning: Prompt Tuning is Not All You Need\".\n\nTable 1 mixes desired properties of CL methods with general properties of methods. End-to-end optimization is no desired property, high predictive performance is.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L5Y9BA1xQ0", "reviewer": "Reviewer_yo9h", "summary": "Existing methods depend on the precision of the selection mechanism. This paper propose a class incremental learning (CLS) method called S-LoRA to decouple the learning of the direction and magnitude of LoRA parameters. It can incrementally learn task-specific LoRA and importance parameters while preserving the optimization directions of previously tasks. The experimental results show their superiority on multiple benchmarks.", "questions": "See the Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The ablation study is interesting and validate the effectiveness of the learned importance factors $a$.\n2. The paper is well organization and the proposed approach is easy followed.", "weaknesses": "1. My main concern is that the idea of this paper is just like a MOE-LoRA, which has been studied in [1-2] in continual learning. The learned importance parameters are just like the gates of MOE.  We expect the author to discuss more differences between the proposed \"reweighting LoRA\" and \"MOE LoRA\".\n2. The experiments are insufficient. There are some other methods in CLS that should be compared, like RanPAC[3], and ADAM[4]. To the best of our knowledge, the RanPAC currently performs best among CIL methods, where the details can be found in this survey[8] about PTM-based CL.  Additionally, some SOTA LoRA methods for continual learning (e.g., O-LoRA[6] and N-LoRA[7]) should be also compared. These two works are very related to LoRA for continual learning, even though they are performed on language process tasks.\n3. The two efficient versions of S-LoRA seem unrelated to the core idea of this paper, and the effect is not significant. The author should discuss more details about the ES-LoRA1 and ES-LoRA2. \n\nIf the author can answer my issues well, I am willing to improve my score. \n\n\nReference\n\n[1] Liu J, Wu J, Liu J, et al. Learning Attentional Mixture of LoRAs for Language Model Continual Learning[J]. arXiv preprint arXiv:2409.19611, 2024.\n\n[2] Dou S, Zhou E, Liu Y, et al. LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin[C]//Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024: 1932-1945.\n\n[3] McDonnell, M. D.; Gong, D.; Parvaneh, A.; Abbasnejad, E.; and van den Hengel, A. 2024. Ranpac: Random projections and pre-trained models for continual learning. Advances in Neural Information Processing Systems, 36.\n\n[4] Zhou, D.-W.; Ye, H.-J.; Zhan, D.-C.; and Liu, Z. 2023b. Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need. arXiv preprint arXiv:2303.07338.\n\n[6] Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. 2023a. Orthogonal subspace learning for language model continual learning. arXiv preprint arXiv:2310.14152.\n\n[7] Yang S, Ning K P, Liu Y Y, et al. Is Parameter Collision Hindering Continual Learning in LLMs?[J]. arXiv preprint arXiv:2410.10179, 2024.\n\n[8] Continual Learning with Pre-Trained Models: A Survey.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S7Qd4fsFqH", "reviewer": "Reviewer_on1i", "summary": "This paper presents a new method called S-LoRA for addressing the Class Incremental Learning (CIL) problem. Currently, prompt-based methods are the most effective method that does not rely on memory-buffer methods in CIL. However, as highlighted by the authors, these approaches have some limitations, particularly regarding prompt selection: they could be more efficient since they require double passes through the model to identify similar prompts and depend heavily on choosing the correct prompt. The proposed method (S-LoRA) aims to alleviate these issues by enabling each task to learn a set of LoRA weights over a pre-trained model, along with a separate set of coefficients that determine the importance of each LoRA-weight. The authors provide theoretical and empirical evidence demonstrating that their proposed method achieves strong results across multiple benchmarks.", "questions": "- I agree with the first finding shown in section 4.2. However, this can change drastically when the input data distribution changes abruptly.\n    - I understand that this may be outside the scope of the paper, but did you measure the relative distance with other benchmarks? It would be helpful to compare elements outside the ViT pre-training distribution.\n- Can you explain the intuition behind the lack of forgetting in the proposed method?\n    - For example, Figure 3.A shows how the alpha values change as different tasks are trained. I imagine this may strongly affect the representation generated for the first task examples, which are only expected to be affected by the weights of A_1 and B_1.\n    - Do you have forgetting results?\n- Some implementation details are not clearly explained:\n    - How are alphas initialised? In most routing methods, the initialisation of the alphas plays a critical role; I can imagine that this can also happen here.\n    - Does any regularisation apply? Or do you train them freely?\n- Can you explain how the values in Table 5 were calculated?\n    - Specifically, I have doubts about the comparison of L2P's trainable parameters versus ES-LoRA's. In L2P, only the prompts pool is trained, which should be smaller than LoRA's A and B.\n    - On the other hand, in ES-LoRA, the A and B values are accumulated, which is not accounted for in the table.\n    - Can you add the S-LoRA values for comparison?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper is very well written. It presents the constraints and motivations the authors want to work with and clearly explains a simple but effective method. The experiments shown support the majority of what is presented in the paper.\n- The S-LoRA method represents an advancement over current methods that use pre-trained models. It mitigates the issues introduced by prompt-based approaches and reduces the inference time. The idea of splitting the LoRA trainable weights into direction and magnitude is interesting, as it helps each task have the plasticity to add new information while encouraging the reuse of weights learned in past tasks.\n    - The authors mentioned that an ideal CL method needs to be rehearsal-free, which I may partially agree with as this depends strongly on the context and scenario of the problem. However, I do agree that it needs to be more efficient at the time of inference.\n- The experiments performed in Section 4 help to understand the proposed method's behaviour. It is also interesting to find limitations in the model and propose alternatives that mitigate these problems.\n    - Moreover, they are well accompanied by a suitable experimental ablation.", "weaknesses": "- Some sections of the paper mention that the proposed method improves up to 31.05% across various CL benchmarks. It could be helpful to change this as it can present confusion, especially as this number does not necessarily relate to the current best method.\n- Algorithm 1 needs to be more precise. In lines 1 and 2, you assume you are working in a Task_j, but then iterate over the T task in lines 6 to 8. This iteration may confuse readers and make them think that a task identifier conditions parameters alpha, A_j and B_j.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "im35iyQFVK", "reviewer": "Reviewer_Uver", "summary": "This manuscript focused on continual learning with the pre-trained model (CL-PTM). By indicating that the existing prompt-based methods rely on an unreliable prompt selection mechanism which can lead to the scalability issue, this manuscript proposed Scalable Low-Rank Adaptation (S-LoRA). Specifically, S-LoRA incrementally decouples the learning of the direction and magnitude of Low-Rank Adaptation (LoRA) parameters. The theoretical and empirical analysis indicated that the proposed method tends to follow a low-loss trajectory that converges to an overlapped low-loss region. Experiments on standard benchmarks were conducted to support the proposed method.", "questions": "1. Regarding the magnitude and the direction of the LoRA parameters. In Section 4.1, the decomposition of magnitude and direction of LoRA's parameter updates $\\Delta \\mathbf{W}$ was only casually mentioned. However, I wonder how the magnitude and direction of such a matrix were defined in the authors' derivation. I believe they should be clearly defined to help the readers to understand the core idea of your method.\n2. In Findings 3, the concept of \"low-loss path\", \"linear low-loss path\", and \"low-loss region\" need to be explained with more intuitive expressions.\n3. In the first equation after Algorithm 1, I wonder if there is a typo for the definition of $r$. Should the first term in RHS be $\\mathbf{A}_j \\mathbf{B}_j$ rather than $\\mathbf{A}_i \\mathbf{B}_i$? If not, please provide further explanations about this point.\n4. Still with the same equation. Since the RHS consists of a linear combination of \"directions\" mentioned by the authors, I wonder how \"the residual\" of a direction approximation should be defined. In my understanding, $r$ should also be a direction analog to the operations on vectors. However, the authors also had some expressions like \"the residual $r$ is less than the threshold $\\tau$\". I didn't get how to control a residual direction by a scalar threshold.\n5. Before Theorem 1 in Section 5, the authors mentioned that $\\|\\cdot\\|$ refers to the operator norm. Do all $\\|\\cdot\\|$ notations within the manuscript stand for the operator norm?\n6. I noticed that the proposed S-LoRA method was mainly compared to the prompt-based baselines except for InfoLoRA. However, S-LoRA is not an improvement of the prompt-based methods. It would be better if the authors provide comparisons with a wide range of recent studies like EASE [1], or at least some recent prompt-based methods like [2-5].\n\nI will accordingly change my rating if my concerns are addressed.\n\nReferences:\n[1] Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning. CVPR 2024.\n[2] RCS-prompt: learning prompt to rearrange class space for prompt-based continual learning. ECCV 2024.\n[3] Prompt Gradient Projection For Continual Learning. ICLR 2024.\n[4] PromptFusion: Decoupling Stability and Plasticity for Continual Learning. ECCV 2024.\n[5] Consistent Prompting for Rehearsal-Free Continual Learning. CVPR 2024.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of decoupling the learning of LoRA’s direction and magnitude is novel to me. This method addresses issues with scalability and efficiency in class-incremental learning, providing a valuable contribution to the field.\n2. The motivation of this manuscript was driven by extensive empirical support. Some interesting findings were observed through the experiments, which were not investigated before.\n3. Theoretical analyses were conducted to support the empirical findings. The explanation of the shared low-loss region and how S-LoRA converges to it effectively supports the results presented.", "weaknesses": "1. Some core ideas were not explained clearly. The description of how the decoupling of magnitude and direction need to be better defined.\n2. While S-LoRA shows strong performance, some newer methods outside the scope of the prompt-based methods need to be compared for further validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)