SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splitting
Authors: Haomiao Qiu
Miao Zhang
Ziyue Qiao
Weili Guan
Min Zhang
Liqiang Nie
Venue: ICLR 2026 Poster (2026)
Keywords: Continual Learning
Low-Rank Adaptation
Gradient orthogonal projection
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: This paper presents SplitLoRA, a method for continual learning that combines orthogonal projection with LoRA. It improves the balance between plasticity and stability by effectively mitigating interference between new and old tasks.

Abstract: Continual Learning (CL) requires a model to learn multiple tasks in sequence while maintaining both stability—preserving knowledge from previously learned tasks, and plasticity—effectively learning new tasks. Orthogonal projection has emerged as an effective and popular paradigm in CL, where it partitions the gradient space of previously learned tasks into two orthogonal subspaces: a primary subspace and a minor subspace. New tasks are learned effectively within the minor subspace, thereby reducing interference with previously acquired knowledge. However, existing orthogonal projection methods struggle to achieve an optimal balance between plasticity and stability, as it is hard to appropriately partition the gradient space. In this work, we consider a continual learning paradigm based on Low-Rank Adaptation (LoRA), which has gained considerable attention due to its efficiency and wide applicability, and propose a novel approach for continual learning, called SplitLoRA. We first provide a theoretical analysis of how subspace partitioning affects model stability and plasticity. Informed by this analysis, we then introduce an effective method that derives the optimal partition of the gradient space for previously learned tasks. This approach effectively balances stability and plasticity in continual learning. Experimental results on multiple datasets demonstrate that the proposed method achieves state-of-the-art performance. The code is available at https://github.com/qhmiao/SplitLoRA.

Published as a conference paper at ICLR 2026
SPLITLORA: BALANCING STABILITY AND PLASTICITY IN CONTINUAL LEARNING THROUGH GRADIENT SPACE SPLITTING
Haomiao Qiu Miaa Zhang' Ziyue Qiao²*, Weili Guan', Min Zhang', Liqiang Nie
Harbin Institute of Technology (Shenzhen)
2 Great Bay University
24B951058@stu, hit. edu. en, zhangmiao@hit. www.:hhhttteeecooommoo en, zyqiao8gbu. edu on honeyguan@gmail com, zhangmin2021ehit..dd edu. mieliqiang@gmail.cmmm
ABSTRACT
Continual Learning (CL) requires a model learn multiple tasks in sequence while maintaining both stability knowledge from previously learned tasks, and plastiitt-effeettvvee emerged as an effective and popular paradigm in CL, where it partitions the gradient space of previously learned tasks into two orthogonal subspaces: primary subspace and minor subspace. New tasks are learned effectively within the minor subspace, thereby reducing interference with previously acquired knowledge. However, existing Gradient Projection methods struggle to achieve optimal balance between plasticity and stability, as is hard to appropriately partition the gradient space. In this work, consider continual learning paradigm based Low-Rank Adaptation (LoRA), which has gained considerable attention due to its efficiency and wide applicability. and propose novel approach for continual learning, called SplitLoRA. We first provide a theoretical analysis of how subspace partitioning affects model stability and plasticity. Informed by this analysis, we then introduce an effective method that derives the optimal partition of the gradient space for previously learned tasks. This approach effectively balances stability and plasticity in continual learning. Experimental results on multiple datasets demonstrate that the proposed method achieves state-of-the-art performance. The code is available at httpp::://...... /ttp::///gthbb com wwwm/hhmaao/ppptt....t..
learning new tasks. Gradient projection has
INTRODUCTION
Continual Learning (CL) refers to a model's ability to sequentially learn new tasks while retaining knowledge from previously learned tasks (Parisi al. 2019). This contrasts with traditional machine learning paradigms, which assume that models are trained on fixed dataset where all data is available once. In the CL setting, the challenge lies in maintaining performance on previous tasks while adapting to new ones, necessitating a balance between stability and plasticity. In recent years, orthogonal projection methods have demonstrated strong performance in continual learning tasks. These methods require storing the subspace spanned by the gradients of previous tasks in memory. During new task training, the gradient of the current task projected onto the minor subspace of the previous task's gradient subspace, reducing the interference of new task updates with previously learned knowledge.
Parameter- Eframmter-Efficient Fine-Tuning (PEFT) (Hu al. 2022 Houlsby et al. 2019 Jia et al. 2022) enables efficient fine-tuning for new tasks by keeping the pre-trreiin trained model parameters unchanged while introducing small subset of trainable parameters. Due to its advantages in computational efficiency and performance, PEFT methods have gained increasing popularity in continual learning (Wang et al.) 2022c Smith et 2202] Gao et al. 22023 Liang & Li 2024 Lu et al. (2024). Combining orthogonal projection with PEFT can better leverage the knowledge of the pre-trained model, allowing for faster adaptation to new tasks.
"Co-corresponding Authors.

---

Published as a conference paper at ICLR 2026
However, existing methods (Liang & Li 2024, Lu et al. 2024) typically determine the subspace dimension for each module within the model by using a predefined threshold based on the cumulative sum of squared singular values. This approach enforces uniform partitioning rule across all modules, ignoring the fact that different modules contribute unequally to knowledge retention (Jiang et al. 2024, Dai et al.] 202T Geva et al. 2020). As result, it fails to achieve an optimal trade-off between stability and plasticity.
In this paper, we theoretically analyze the relationship between the size of the minor gradient subspace of previous tasks and the upper bound of loss increments across all tasks. Furthermore, we model its impact on both stability and plasticity. practice, we build upon the LoRA framework and propose a novel method called SplitLoRA for CL tasks. Specifically, to minimize the upper bound of the total task loss growth, we construct an optimization problem to determine the optimal size of minor subspace and derive approximate solution to balance stability and plasticity.
Our contributions are summarized follows:
We theoretically model the impact of the gradient subspace size of previous tasks on stability and plasticity in orthogonal projection based continual learning in Theorem 4.2 and derive approximate optimal minor subspace in CL.
We introduce SplitLoRA, a novel PEFT framework. By projecting the minor subspace onto the LoRA dimension reduction matrix A, via a random projection and optimizing only B, SplitLoRRA ensures that updates remain confined to the minor subspace, thereby achieving an effective balance between stability and plasticity.
Our method achieves state-of-the-ard performance across multiple datasets, surpassing existing CL methods by 2%-5% on different datasets.
RELATED WORK
2.1
PARAMETER EFFICIENT FINE-TUNING
Parameter-efffcieet fine-tuning modifies pre-trained models by introducing small set of trainable parameters while keeping the original model frozen, significantly reducing computational costs while maintaining strong performance. Adapter (Houlsby et al. 2019) fine-tunes small modules added to
multiple layers, while Prompt-tuning (Lester et al. 2021| 1) and Prefix-tuning (Li & Liang] (021) inject trainable tokens into Transformer layers. LoRA (Hu et al. 2022) decomposes weight updates into low-rank matrices, tuning only these structures. Despite training fewer parameters, PEFT methods often achieve comparable
superior performance (Zaken al.) 2022 Fu et al.) 2022; Hu al.)
2022 Mahabadi et al. (22II. Initially developed for NLP, PEFT has been extended to vision tasks, with methods such as Visual Prompt Tuning (VPT) (Jia et al. 2022 and AdapterFormer (Chen et al.) achieving performance on par with full fine-tuning.
2022
2.2
CONTINUAL LEARNING
Continual learning methods fall into three main categories: regularization-basel memory-based, and expansion- Regularization-base approaches (Zenke et al. 2017 Jung et al. 2020 Aljundi ct al. 2018: Kirkpatrick et al.) 2017) constrain significant changes to key parameters to mitigate catastrophic forgetting. Memory-based methods (Aljundi al., 2019a Sun et al. 2022 Liang
& Li| 2023) retain prior task information in buffer, allowing models to revisit past knowledge. Expansion- techniques (Rusu al. 6016)/Hung al. 2019
al. 2019) dynamically expand the model architecture to accommodate new tasks while preserving learned representations.
Gradient Projection in CL. Gradient projection (Zeng et al. 2019 Farajtabar et al. 2020 Saha et al..
2021b) mitigates task interference by constraining updates to directions orthogonal to previous tasks. Orthogonal Weight Modulation (Zeng et al. 2019) learns a projector matrix to prevent new gradients from overwriting prior knowledge. Orthogonal Gradient Descent (Farajtabar et al. 2020) projects new gradients onto the orthogonal complement of previous task gradients. Gradient Projection Memory (GPM) (Saha et al., (0211) stores subspace bases of old task data and projects new gradients onto their orthogonal complement. Trust Region Gradient (Lin et al. 2022a) enhances forward knowledge transfer by leveraging task-related representations.
2

---

Published as a conference paper at ICLR 2026
How split gradient space old tasks
Update LoRA the minor subspace
InfLoRA
SpliLLoRA
Pretrsined Weights
argmin Lows LLo
W,, E R
Major Subspace
Major Subspace
Random Projection
B
Minor
Singular Value
Sabspace
Minor
Ae
Subqpare
An overview of our proposed SplitLoRA. During the learning of the I-th task, the gradient space of tasks to is decomposed into major and minor subspaces. InfL.RA determines k* solely based on a predefined threshold, whereas SplitLoRA balances stability loss and plasticity loss determine k*. Then the minor subspaces are randomly projected onto the low-dimensional matrix A of LoRA and fixed, while only B is trained. Specifically, W;- Wo + 2 A,B, where Wo represents the pre-trained model weights, and & denotes the size of the minor subspace.
Figure 1:
PEFT in CL. With the rise of pre-trained models (He et al. 2022 Dosovitskiy et al. 2021 Devlin et al. 2019), continual learning has shifted toward leveraging them rather than training from scratch. While some approaches (Boschini et al. 2022 Zheng et al.) 2023) fine-tune pre-trained models fully, this is often inefficient. To address this, PEFT methods have been explored in continual learning, with studies (Smith et al., 2023 Wang et al. 2022c: Khan et al. 2023 Yu et al., 2024) integrating prompt-tuning to improve class -ncemmnna learning. unified framework (Gao et al. (023) further combines various PEFT techniques, including prompt-tuning, LoRA, and Adapter, into continual learning.
3 PRELIMINARY
3.1 CONTINUAL LEARNING FORMULATION
In CL, there are T' tasks ..., Tr. each task includes data: D, {(x, the input and yf E IR is the label. The goal of CL is to achieve an overall optimal performance across all tasks. Let Wt represent the model parameters after training on the last task T. The loss on task t, denoted as L,( (Wr), measures the performance of WT on task t. Let Wy. represent the optimal parameters. Then, the objective is to minimize the total loss across tasks:
"h where x Rd is
W = argmin Latt (Wr) argmin WT
(WW.
W
3.2 LORA-BASED CONTINUAL LEARNING
LoRA (Hu et al. 2022) is a low-rank based PEFT method that reduces the number of parameters by decomposing the weight matrix into the product of two low-rank matrices. Specifically, for linear layer, the extra weight matrix A W is decomposed into two low-rank matrices A and as AW AB, where A E Rdixr and B Rrxd2, and is the dimension of the low-rank. In this way, the number of parameters of the weight matrix is reduced from d to 2dr. During training, we learn A and B by minimizing the loss function of the current task. During testing, we recover W Wo + AB and use it for forward propagation.
CL generally initializes an additional LoRA for the new task while the LoRA of old tasks also participates in the forward process (Liang & Li 2024 Wang et al., (023b). In the current task t, the forward process of the linear layer
Y W.X (W.- A,B,),
and only A, and B, are trained during training.
3

---

Published as a conference paper at ICLR 2026
4 SPLITLORA
In this section, we introduce SplitLoRA, a PEFT method for CL that mitigates catastrophic forgetting by partitioning the gradient space. Unlike existing Gradient Projection- Projection-based based methods (Liang & 2024; Lu et al., 2024; Jin et al. 2021), which typically define the minor subspace solely based on the sum of squared singular values being below a predefined threshold, SplitLoRA determines the optimal subspace size by analyzing the impact of the minor subspace stability loss and plasticity loss. We first introduce gradient projection, then model the effect subspace partitioning on learning dynamics and formulate optimization problem to derive the approximate optimal size of the minor subspace. Finally, we present how to construct the low-rank projection matrix within this subspace to enhance CL. The entire process of SplitLoRA is illustrated in Figure T
4.1 ORTHOGONAL. DECOMPOSITION BASED GRADIENT PROJECTION
Empirically, training tasks due to interference between the gradients of new and old tasks, resulting in stability loss. To address this issue, GPM (Saha et al. (2021b) orthogonally decomposes the gradient space of previous tasks into a major subspace and a minor subspace, and constrains the update direction of the new task within the minor subspace. Our work is also built upon orthogonal decomposition.
tasks often leads performance degradation previously learned
In CL. we aim to maintain an average gradient space Gold for all previous tasks. Specifically, after training the task t 1, we re-feed the data from this task into the model and calculate the average gradient G of W throughout this process. Finally, we compute the average gradient space G tasks:
for the previous
Go 1 (11 G + G
For the first task, G1 is equal to zero. Next, we receive the data from task t. We perform a Singular Value Decomposition (SVD) on the gradient G Larger singular values correspond to singular vectors that dominate in describing the vector's importance. We select the last k left singular vectors of Ú, as the minor subspace:
0, ΣS V SVD(G:
0 Ü. -k
The gradient of previous tasks has a much smaller component in the minor subspace compared to the major subspace. Therefore, projecting the gradient of the new task onto the minor subspace will result in minimal interference.
A common projection method is to construct projection transformation matrix. For simplicity, we consider a linear layer W in a model. As the model update A AW is determined by the gradient, projecting the model update onto the minor subspace is equivalent to constraining the gradient direction, which helps mitigate interference. Specifically, we project the model update AW onto the minor subspace. The projection result is given by:
W = projoon(0;) (W) = T.Ü.TAW.
(5)
where Üž is the projection subspace. Since the gradients of previous tasks are distributed primarily in the major subspace, therefore the projection ensures that the updates primarily benefit the new task while minimally affecting the performance of old tasks.
4.2 MINOR SPACE SETTING FOR OLD TASKS
Projecting the gradient onto the minor subspace mitigate interference with previous tasks. And the larger the minor subspace size k, the larger the learning space for the new task, leading to better plasticity. However, as the gradient components of previous tasks in the minor subspace increase, stability deteriorates.
Previous methods (Jin et al. 2021 Lin et al. 2022a Liang & Li 2024 Lu et al. 2024) compute the sum of the squared singular values corresponding to the minor subspace, ensuring that it remains below a predefined threshold r. Among all values of l that satisfy this condition, they select the
4

---

Published as a conference paper at ICLR 2026
largest one:
d-k ot
===1 i= Σ 0²
k- max
kl
(6)
The size of the minor subspace, denoted as k, is crucial parameter that affects both model stability and plasticity. However, previous methods determine & based on a predefined threshold т, which is merely a hyperparameter and does not effectively balance stability and plasticity. Thus, proceed to analyze how subspace selection impacts the loss across all tasks. Based on the smoothness of the loss function L, we can derive an upper bound on the loss incurred due to parameter updates in CL.. Proposition 4.1 (Upper Bound on Loss Increase). Consider a model with a linear layer updated from Wi-1 to W, = Wi-1 A ,, Assume the loss function is L-smooth and that the first t-1 tasks were trained with updates constrained to be orthogonal to the gradients of previous tasks. Then, the total loss change over tasks 1....,t ,, is bounded by:
(t-1)L 2
Σ (C.(W)) C.W,, S (t-1) Go
(AW, G.))
A.. (7)
Stubility Loss
Plasticity Loxs
where G2 (W..) is the gradient for task and Goldd of previous tasks. (:-)) denotes the Frobenius inner product.
- E G. is the average gradient
This result shows that parameter updates affect both the current and past tasks. The term (WW G captures interference with past tasks (stability loss), while (AW,, G.) reflects progress on the current task (plasticity gain). The squared norm Il acts as regularization term controlled by the smoothness constant L.
Next, we discuss how to choose the minor subspace in gradient projection to minimize the combined stability and plasticity losses. Building on Proposition 1, we can theoretically analyze how stability loss and plasticity loss vary as a function of k. From Eq. 77. after replacing A AW, with A AW, t, where A W. U AW, the projected update onto the minor subspace, we can express the stability loss CS (W.) and the plasticity loss C. (W.) as follows:
(W.) (1 -1- (1 (AW., C
(8)
(WW,) (AW..G.).
The stability loss is proportional to the alignment between the projected update A AW, and the gradient of old tasks Gold, while the plasticity loss depends the alignment of AW, with the gradient of the new task G, Then we define the error function e(k), which quantifies the proportion of these minor directions and is given by:
Eind
<(k)
(10)
(k) measures the interference error caused by updating the model within the minor subspace.
To analyze the effect of subspace partitioning. we need to handle the fact that the new task's 's update direction A A, is unpredictable. Since have no prior knowledge about its distribution, we adopt neutral assumption: A AW is treated as having equal expected projection across all feature directions of G This assumption does not imply that gradients are truly uniform in practice, but rather provides a distribution- free baseline that allows us to derive general bounds. Under the above assumption, we obtain the following theorem:
Theorem 4.2. Let W,-1 denote the weight matrix of linear layer in the model, updated as
W.- W. AW,- W.-+ UKKKk, AW, We provide the expected values of the stability loss:
E E|CS = (t (kx) (AW,,
G.
(11)
and the plasticity loss:
E E|C, (W.)- = d (AW G.)
(12)
5

---

k = argmin Published as a conference paper at ICLR 2026
Algorithm SplitLoRA
1: Input: Datasets Dr = ((x), M= for T tasks Ti..., Tr, a pre-trained ViT model fe(+) with layers.
2: Output: The optimized W for each layer
3: Initialization: G
4: for 1. 1 to T do
5: Train LoRA on task Te using dataset Dr
6: Update Goll using Eq. (3) for each layer
See Algorithm
2 Step 1
7: Compute K using Eq. (15) for each LoRA module
b See Algorithm
Step
8: Initialize A, using Eq. for each LoRA module
See Algorithm
Step
9: end for
The proof of this theorem can be found in Appendix A.3 Therefore, achieving an optimal balance between these two objectives requires solving the following optimization problem:
argmin (E(Ci(W (w,)) + E[C, (W.))).
(13)
Noting that A AW, and G, gradually change training progresses, solving this optimization problem challenging. To simplify this, we introduce ratio parameter a:
(AW, G:) AAW, "
(14)
This substitution reformulates the optimization problem into the following form:
(t (t- ())) - ke
(15)
Since W, benefits new tasks, it often interferes with previous task knowledge, leading to:
(AW,, G.) <0, (AW,, G. > O.
(16)
From Eq. [5 it is evident that increasing ky leads higher e (Ree), which increases stability loss CF, while expanding the learning space and thus reducing plasticity loss L." However, since both AW, and G, evolve during training, the ratio a in Eq. (14) also varies dynamically, whereas the update subspace must be determined before training begins for task l. Therefore, we explicitly treat as fixed hyperparameter, serving strategic control knob for the stability-plastien Importantly, our experiments (Tab, 5] Fig. 3) demonstrate that SplitLoRA is highly robust to the choice of a; performance remains consistently superior to baselines across a wide range of values.
trade-off.
In the simplified optimization problem of Fg.(15). the parameter ha is restricted to integer values within the range [1, d]. The optimal solution this equation can be obtained by evaluating the objective function for all possible values of k, and selecting the one that minimizes it as k For example, in ViT-B/16 (Dosovitskiy et al. (202T). embedding dimension 768. and can be selected as any integer between I and 768. is worth noting that we compute k separately for each LoRA module, as weights at different layers and positions capture substantially different knowledge. A fixed threshold, as used in InfLoRA (Liang & Li 2024), cannot effectively account for such variation across modules.
4.3 LORA UPDATES IN THE MINOR SUBSPACE
To ensure LoRA updates remain within the minor subspace, we fix the projection matrix A, and only optimize B, LoRA parameterizes the weight update as: AW, = A, When A, is fixed, the update is confined to its column space (Liang Li 2024 Wang al. 2023b). To restrict this space to the minor subspace of previous tasks, we construct
A A
(17)
where Ú e d is an orthonormal basis of the minor subspace and R E R is a random Gaussian matrix. Importantly, this constraint only holds if A, remains fixed during training, otherwise, the
k

---

Published as a conference paper at ICLR 2026
Table 1: We present FAA (%) and CAA(%) on ImageNet-R under three incremental learning settings: "5-task," "I "10-task," and "20-task." All backbone networks are pre-trained on ImageNet-2IK.
S-lask
10-task
20-ass
Method
Pub.
FAA (f)
CAA
FAA
CAA
FA)
CAA
Upper hound
84.09 1: 0,21
84.09 1: 0.21
84,09 0.21
FT
18.74 0.44
10.12 = 0.51 35.23 0.92
4.75 0.40
22.8
FT++
60.42 = 0.87 71.59 1
48.93 : 1.15 66.79 1: 0.92 35.98 1: 1.38 59.68 : 0.55
1.38
L2P++ Wang et al.
2022c
CVPR22
70.83 1: 0,58 78.34
69.29 1 0.73 78.30 1 0.69 63.89
1:30 77.15 0.65
Deep L2P++ Wang cl
2022e
CVPR22
73.93 8..14:0.54
0..6
79.63
68.42 78.68 1.03
DualPrompt (Wang et al.
20226
ECCV22
73.05
7.44
0.222
7.79
1.39 77.42 0.80
CODA.P (Smith
CVPR23
76.51 0.38 8.50
75.45
81.59
72.37
DiDe-Prompt Wang et ml
A023a
NeurIPS23
76.29 + 0.10 0.177
7.174
78.76
76.46
78.76 0.11
EvePrumpl Kurmiawan
2024
AAA124
77.16
0.08
82.09
InfLoRA Lang T1
CVPR24
1:
78.10
73.11
ProPrompt
NeurIP224
79.23 + 0.65
0.50
VPT-NSP
NeurIPS24 79.71
0.19
84,92
76.72
SD-LoRA
Yichen
2025
ICLR25
0.72
SplitLoRA
This work 81.92 0.29
85.83 0.55
81.00
0.17
85.84
78.82
0.28
84.57 0.44
95
95
SpiiLoRA
SainLalA InfORA
SpiALaRA INTLARA
tafladk VITE-MI
VPT-NSP
VPT-NS"
98
20
75
10
16
15
26
Sandoct of Tasks (NllaggNNee-i & Tadkss
Number of Tesks NeltnogeeNe-- R. 10 Tasks
Numberr ofTads
(olmageNett-k 20 Tonks
Figure 2: Variation of the performance of different methods during the learning of ImageNet-R.
update direction may drift out of the subspace. Fortunately, prior works (Zhag al. 2023b Liang Li 2024) verify that fixing A, maintains sufficient model capacity while controlling interference. This design ensures that task updates are constrained to low-interference directions Ú, balancing stability and plasticity without additional memory computational cost. The full procedure of SplitLoRA is summarized in Algorithm
5 EXPERIMENT
5.1 EXPERIMENTAL, SETTINGS
Datasets. We conducted experiments three standard datasets: ImageNet-R Hendrycks al. (2021a), CIFAR-100 Krizhevsky (200)), and DomainNet |Peng et al.) (019). ImageNet-R is a variant of ImageNet with 200 classes. CIFAR-100 consists of 100 classes, each containing 600 images. DomainNet contains images from diverse domains, posing a challenge for cross-domain generalization. Following Gao al. (2023); Wang et al. (022cc) Liang Li 2024). we divided ImageNet-R into 5, 10, and 20 tasks, with each task comprising 40, 20, and 10 classes, respectively. CIFAR-100 was split into 10 tasks, each containing 10 classes, while DomainNet was uniformly partitioned into 5 tasks.
Baselines and Evaluation Metrics. We compare our method with several state-of-the-art continual learning approaches, including L,P++ (Wang et al.] 2022c). Deep L2P++ (Wang et al. 2022c).
DualPrompt (Wang et al. 2022b). CODA-P (Smith et al. 2023). HiDe-Prompt (Wang et al. 2023a
EvoPrompt KKrnnawan et al. 2024 VQ-Prompt (Jiao al. 2024), VPT-NSP2. and InfLoRA (Liang
Li 2024). The "Upper bound" represents the performance achieved by jointly training on all classes in one go. Results are averaged over three runs with different random seeds. Following Jiao ct al. (2024), we report Final Average Accuracy (FAA) and Cumulative Average Accuracy (CAA).
For details, please refer the appendix A.4
7

---

Published as a conference paper at ICLR 2026
Table 2: We present FAA (%) and CAA(%) on CIFARI00: 10 tasks and DomainNet: 5 tasks. We report results over 3 trials. All backbone networks are pre-trained on ImageNee-22KK,
CIFARI00
DomainNet
Method
Pub.
FAA
C1A
FAA (T)
(1)
Upper-bound
91.92 1 0.05
90.12 1 0.13
DualPrompt Wang al. 2022b
ECCV22
84,42 = 0.30
90.06 + 0.07
72.14 0.05
77.71 + 0.06
CODA-Prompt Smith
2023
CVPR23
86.62 + 0.11
91.08 + 0.28
0.13
78.72 0.07
GAF et al.
2023
ICCV23
84.15 0.16
89.84 0.03
66.85 +.85++0.0
75.01
0.17
C-LoRA Smith et al.
2024a
TMLR24
82.97 + 0.47
88.81 0.34
0.16
75.25 0.11
InfLoRA Liang
2024
CVPR24
87.06 = 0.25
0.50
78.82 + 0.34
VPT-NSP* et
2024
NeurIPS24
1.10
92.25 0.80
8319
88.63 0.10
CoSO [Cheng
2025
2025
88.76 -: 0.16
74.27
80.05 0.04
SplitLooKA
This work
90.33 + 0.73
93.70 + 0.32
84.31 + 0.23
88.99 0.57
Implementation Details. As the mainstream of recent continual learning research has primarily centered on parameter-efficien fine-tuning with pre-trained ViT models, also adopt this setting for our experiments. We follow prior works (Wang et al., [202cc bj Smith et al. 22033 Wang et al.
2024) and adopt ViT- (Dosovitskiy et al. 2021 pre-trained on ImageNet-21K 21K (Ridnik et al. 2021) as the backbone. The LoRA rank is set to 10. and the embedding dimension is D= 768, matching the feature dimension of BossossDDsovvitskiy et al. (2021). Following Liang & Li (2024), we insert SplitLoRA modules into the key and value projections multi-head attention. Unless otherwise specified, we fix a 20, Our method is optimized using AdamW W (Loshchilov & Hutter 2018) with initial learning rate of 1e-3 for LoRA and 2 for the classification head. We use batch size of 256 across all datasets, and each task is trained for 10 epochs. All experiments conducted single NVIDIA GeForce L40SS GPU. All results are reported as mean + standard deviation over three random seeds.
2023a Zhang al. 2023a, Kurniawan al.,
5.2 EXPERIMENTAL RESULTS
Results on ImageNet-R, CIFAR100 and Domainet. presents the results of different methods evaluated on ImageNet-R with varying numbers of tasks. highlights how our proposed method, SplitLoRA, achieves consistently higher accuracy compared to existing continual learning methods across different task setups. Additionally, Table 2 shows the results of these methods on CIFAR and DomainNet datasets. Across both tables, SplitLoRA outperforms other methods in FAA and CAA. Figure ( shows the accuracy trends of various CL methods on ImageNet-R, Our method achieves the highest accuracy the end and outperforms others throughout the learning curve.
Initialization strategies A,, Table Table compares different initialization strategies for A₁ SplitLoRA achieves consistently better performance projected minor subspace over random or InfLoRA.
task splits, demonstrating the effectiveness of using
Table 3: Impact of different A, initialization strategies on ImageNet-R.
Table 4: Efficiency LoRA variants on ImageNet-
R (10 tasks).
Init of A;
5 tasks
10 tasks
20 tasks
Method
Extra Fwd
Mem
Time
Random
76.57
76.13
72.30
LoRA
None
22.80 GB
1h 37m
InfLoRA
78.92
78.10
73.81
InfLoRA
2/task
23.06 GB
1h 48m
SplitLoRA
81.92
81.00
78.82
SplitLoRA
1/task
23.03 GB
1h 43m
Memory and Time Cost. Table 4 shows that SplitLoRA achieves a favorable trade-off between performance and efficiency. It introduces only 1 extra forward pass per task while maintaining similar memory and runtime overheads compared to InfLoRA, and its memory cost remains fixed without growing with the number of tasks.
8

---

Published as a conference paper at ICLR 2026
Table Evaluation of model performance under different values of on ImageNet-R. A higher may improve plasticity but could impact stability.
5-task
10-task
20-task
Method
FAA (1)
CAA
FAA (*) CAA (1)
FAA (1)
CAA (1)
InfLoRA
79.82
84.07
78,10
83.47
73.81
81.02
SplitLoRAA(a 30)
82.15
85.60
81.03
85.56
78.73
84.06
SplitLoRRAee 20)
81.92
85.83
81.00
85.84
78.82
84.57
SplitLoRA((a 10)
82.35
85.82
81.03
85.67
77.89
83.27
SplitLoRA(e 5)
82.52
85.89
81,38
85.89
78.15
84.19
SplitLoRA(aa
82.40
85.86
80.89
85.22
78.59
84.20
Ralatee Maoticiyhh Relariv Forgering!)))
Namber of Tacls
Number Tisks
Namber of Tanks
Number Tecks
Number of Tanks (bilmageNet R. tasks
Namberr of Tasks (cilmaggNNe R. tasks
(OlmmngeNee toiks
Figure 3: The impact of a on the stability and plasticity of the model in continual learning. As increases, stability decreases (higher forgetting) while plasticity improves, illustrating the trade-off between retaining past knowledge and adapting new tasks.
5.3 HYPERPARAMETES ANALYSIS AND DISCUSSION
We study the effect of the hyperparameter changing a has limited impact on final accuracy, and settings consistently outperform InfLoRA. Model stability is measured by forgetting, defined the average gap between each task's 's best historical accuracy and its current accuracy. Lower forgetting indicates better knowledge retention. For clarity, we define relative forgetting the difference from the setting where 1. Plasticity is evaluated by the model's accuracy on the current task. Similarly, relative plasticity is defined as the difference from the plasticity when
continual learning performance. As shown in Table 5
1.
presents results on 5-, 10-, and 20-task splits of ImageNet-R. As a increases, forgetting grows (lower stability) while plasticity improves. These results show that a effectively controls the trade-off between retaining past knowledge and adapting to new tasks, while consistently maintaining better performance than InfLoRA across all settings.
Figure 3
5.4 EFFECTIVENESS OF OPTIMIZING SUBSPACE DIMENSIONS THRESHOLD TUNING
central question whether introducing new hyperparameter for optimization offers real advantage over directly tuning the threshold as done in InfLoRA. Different layers in model serve distinct roles (Zheng et al. 2025). Prior methods such as InfLoRA employ fixed empirical threshold to determine subspace dimensions. However, this fails to accommodate the heterogeneous learning needs across layers, often resulting in shallow layers being overly constrained and deep layers retaining excessive capacity for past tasks.
Layer-wise Subspace Allocation. SplitLoRA replaces the single heuristic threshold with a principled optimization framework. Our objective function (Eq. 13) is derived from the total loss and allows us to analytically compute an optimal subspace dimension for each LoRA module within each task. This ensures that each layer selects gradient space partitioning strategy that minimizes the overall loss. To illustrate, Tab. ?? compares the subspace dimensions selected by InfLoRA and SplitLoRA on the 10th task in the 20-task ImageNet-R setting. While InfLoRA under-allocates capacity in shallow

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of major subspace dimensions selected at different layers the 10th task (ImageNet-R, 20-task setting).
Layer
0
1
2
6
9
11
InfLoRA
14
13
30
52
105
85
SplitLoRA
65
48
58
58
68
70
Table Strategy transplantation results on ImageNet-R with 5, 10, and 20 tasks. Replacing layer-O
strategy of InfLoRA with SplitLoRA significantly improves performance, especially in reducing forgetting.
5-task
Method
FAA (f)
CAA (t)
Forget. ())
PAA (?)
CAA (?)
Forget. (L)
FAA (f)
CAA (f)
Forget. (1)
InfLoRA
79.82
84,07
10.03
78.10
83.47
9.40
73.81
81.02
12.34
81.59
85.57
4.00
80.35
84.68
4.11
76.68
82.39
5.48
SplitLoRA
81.92
85.83
4.45
81.00
85.84
4.68
78.82
84.57
4.66
layers (layers 0-2), SplitLoRA learns to preserve significantly larger subspaces, implicitly protecting critical shallow representations from catastrophic forgetting.
Strategy Transplantation. To validate this observation explicitly, we designed a strutegy transplan- tation experiment. In the 20-task ImageNet-R setting, replaced the subspace selection strategy of layer 0 in InfLoRA with that of SplitLoRA. This modification significantly improved InfLoRA's performance, particularly in terms of reduced forgetting. Nonetheless, the modified InfLoRA still lagged behind SplitLoRA in new-task performance, suggesting that managing knowledge retention in shallow layers is a key factor, but not the sole determinant, of SplitLoRA's superiority.TTbbl repeo results across ImageNet-R benchmarks with 5, 10, and 20 tasks. Compared to InfLoRA, transplanting only the layer-0 strategy from SplitLoRA already yields notable gains: forgetting halved (e.g.. from 10.03 to 4.00 in 5-task setting) and both FAA and CAA improve consistently. Nevertheless, SplitLoRA achieves the best overall balance between forward accuracy and forgetting. highlight- ing that its advantage lies not only in shallow layer protection but also in its holistic optimization framework. These results clearly demonstrate that SplitL.RRR LoRA's learned subspace allocation strategy provides substantial advantages over direct threshold tuning. Protecting shallow layers plays pivotal role mitigating forgetting, while the full optimization framework ensures superior overall accuracy across continual learning tasks.
CONCLUSION
In this paper, we investigate the problem of continual learning based on pre-trained ViT models and propose novel method, SplitLoRRA. Specifically, analyze the gradient space of previously learned tasks and partition into two complementary components: a major subspace, which captures the most informative and stable directions, and a minor subspace, which contains the remaining, less significant directions. We then provide theoretical formulation to model the effect of the minor subspace size the trade-off between stability (the ability retain past knowledge) and plasticity (the ability to acquire new knowledge). To make the method efficient and scalable, we employ random projection map the minor subspace onto the now-dimesiional matrices of LoRA. Extensive experiments multiple benchmark datasets demonstrate that our approach achieves state-of-the-art performance, highlighting both its theoretical soundness and practical effectiveness.
ACKNOWLEDGMENT
Miao Zhang was partially sponsored by the National Natural Science Foundation of China under Grant 62306084 and U23B2051, Shenzhen College Stability Support Plan under Grant and Shenzhen Science and Technology Program under Grant Weili Guan was partially sponsored
1NND2022112810224422 ZDSYS200306260112022 and 1 by the National Natural Science Foundation of China under Grant 62476071 and U24A20328.
10

---

Published as a conference paper at ICLR 2026
The work of Ziyue Qiao partially supported by the National Natural Science Foundation of China (No. 62406056) and the Guangdong Basic and Applied Basic Research Foundation (No. 2024A1515140114).
REFERENCES
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In ECCV, pp. 139-154, 2018.
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems, pp. 1849-11860, 2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning, In Advances in Neural Information Processing Systems, pp. 11816- 11825, 2019b.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. NeurIPS, 2019.
Matteo Boschini, Lorenzo Bonicelli, Angelo Porrello, Giovanni Bellitto, Matteo Pennisi, Simone Palazzo, Concetto Spampinato, and Simone Calderara. Transfer without forgetting. In Proceedings of the European Conference on Computer Vision, pp. 692-709, 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Arslan Chaudhry, Naeemullah Khan, Puneet Dokania, and Philip Torr. Continual learning in low-rank orthogonal subspaces. NeurIPS, 33:9900-999 9911, 2020.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang. Yibing Song. Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition, Advances in Neural Information Processing Systems, pp. 16664-16678, 2022.
Quan Cheng, Yuanyu Wan, Lingyu Wu, Chenping Hou, and Lijun Zhang. Continuous subspace optimization for continual learning. arXiv preprint arXiv:2505.1 1816, 2025.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint S00iv:2204.080600 2021.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, pp. 4171-4186, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In AISTATS, pp. 3762-3773, 2020.
Chin-Lun Fu, Zih-Ching Chen, Yun-Ru Lee, and Hung-Yi Yi Lee. Adapterbias: Parameter-efficient token-dependent representation shift for adapters in nlp tasks. In Findings of the Association for Computational Linguistics, pp. 2608-2621, 2022.
Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang. A unified continual learning framework with general parameter-efficient tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11449-11459 2023.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forwart layers are key-value memories. arXiv preprint 20X1::00/2./49133 2020.

---

Published as a conference paper at ICLR 2026
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Dan Hendryeks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribttion generalization. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. $340-8349 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adver- sarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 122221157,, 2021b.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In Proceedings of the International Conference on Machine Learning, pp. 2790-2799, 2019.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Steven C. Y. Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen. Compacting, picking and growing for unforgetting continual learning. In Advances in Neural Information Processing Systems, pp. 13647-13657, 2019.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge J. Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. Proceedings of the European Conference on Computer Vision, pp. 709-727, 2022.
Shuyang Jiang, Yusheng Liao, Ya Zhang, Yanfeng Wang, and Yu Wang. Taia: Large language models are -usto--disttibution data learners. arXiv preprint 2xXv:22405.20122 2024.
Li Jiao, Qiuxia Lai, Yu Li, and Qiang Xu. Vector quantization prompting for continual learning. arXiv preprint arXiv:2410.20444 2024.
Xisen Jin, Arka Sadhu, Junyi Du, and Xiang Ren. Gradient-based editing of memory examples for online task-free continual learning. NeurlPS, 34:0PPS,,4:29999 34:29193-29922 2021.
Dahuin Jung, Dongyoon Han, Jihwan Bang, and Hwanjun Song. Generating instance-level prompts for rehearsal-free continual learning. In ICCV, pp. 11847-11857, 2023.
Sangwon Jung, Hongjoon Ahn, Sungmin Cha, and Taesup Moon. Continual learning with node- importance based adaptive group sparse regularization. Advances in Neural Information Processing Systems, pp. 3647-3658, 2020.
Muhammad Gul Zain Ali Khan, Muhammad Ferjad Naeem, Lue Van Gool, Didier Stricker, Federico Tombari, and Muhammad Zeshan Afzal. Introducing language guidance in prompt-basee continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11463-11473, 2023.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Baree Barwinska, et al. Overcoming catastrophic forgetting in neural networks. PNAS. 114(13):3521-33 2017.
A Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
Muhammad Rifki Kurniawan, Xiang Song, Zhiheng Ma, Yuhang He, Yihong Gong, Yang Qi, and Xing Wei. Evolving parameterized prompt memory for continual learning. In AAAI, volume 38, pp. 13301-13309, 2024.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 3045- 3059, 2021.
12

---

Published as a conference paper at ICLR 2026
Xiang Lisa Li and Perey Liang. Prefix- Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pP-
4582-4597, 2021. 2021.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: continual structure learning framework for overcoming catastrophic forgetting. In Proceedings of the International Conference on Machine Learning, pp. 3925-3934, 2019.
Yan-Shuo Liang and Wu-Jun Li. Loss decoupling for task-agnostie continual learning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems, 2023.
Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference-free low-rank adaptation for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2024.
Yan-Shuo Liang, Jiarui Chen, and Wu-Jun Li, Gated integration of low-rank adaptation for continual learning of large language models. In The Thirty-nini Anmual Conference on Neural Information Processing Systems.
Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. Trgp: Trust region gradient projection for continual learning. arXiv preprint arXiv:220.002931, 2022a.
Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. Trgp: Trust region gradient projection for
continual learning. In International Conference on Learning Representations (ICLR). ICLR, 2022b.
lya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In ICLR, 2018.
Yue Lu, Shizhou Zhang, De Cheng, Yinghui Xing. Nannan Wang, Peng Wang. and Yanning Zhang. Visual prompt tuning in null space for continual learning, arXiv preprint arXiv:2406.05665, 2024.
Rabech Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low- rank hypercomplex adapter layers. In Advances in Neural Information Processing Systems, pp.
1035-1035, 2021.
Mark D McDonnell, Dong Gong, Amin Parveneh, Ehsan Abbasnejad, and Anton van den Hengel. Ranpac: Random projections and pre-trained models for continual learning. arXiv preprint arXiv:2307.02251, 2023.
German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review, Neural Networks, 11::5477,, 2019.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. Proceedings of the IEEE/CVE International Conference on Computer Vision, pp.
2019.
Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. Progressive prompts: Continual learning for language models. In International Conference on Learning Representations, 2023.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Mand Imagenet-21k. pretraining for the masses. In NeurlPS, 2021.
Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 3XXi:1606.046771 2016.
Gobinda Saha and Kaushik Roy. Continual learning with scaled gradient projection. In AAAI Conference on Artificial Intelligence (AAAI), volume 37, pp. 9677-9685. AAAI, 2023.
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In International Conference Learning Representations (ICIR). ICLR, 2021a.
13

---

Published as a conference paper at ICLR 2026
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning, In ICLR, 2021b.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilly Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal- continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11909-1191 11919, 2023.
James Seale Smith, Yen-Chang Hsu, Lingyu Zhang. Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization of texaeto-imaae diffusion with c-lora. arrhiveEprint: 2304.06027, 2024a. httph:ttt///tttt org/ abs 230..00027.77..000
James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization of text-to-image diffusion with c-lora. Transactions on Machine Learning Research, 2024b.
Qing Sun, Fan Lyu, Fanhua Shang, Wei Feng, and Liang Wan. Exploring example influence in continual learning. Advances in Neurul Information Processing Systems, pp. 27075-27086, 2022.
C. Wah, S. Branson, P. Welinder, Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-201 1-001, California Institute of Technology, 2011.
Liyuan Wang. Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality. arXiv preprint arXiv:2310.07234 2023a.
Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning. arXiv preprint arXiv:2310. /4152, 2023b. URL httt::tt///xxxiv/ee
abs/ /2310. 14152]
Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning. arXiv preprint arXiv:2310. 14152, 2023c.
Yabin Wang. Zhiheng Ma, Zhiwu Huang, Yaowei Wang. Zhou Su, and Xiaopeng Hong. Isolation and impartial aggregation: A paradigm of incremental learning without interference. In AAAI, volume 37, pp. 10209-10217, 2023d.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasckaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A. Sumanta Patro, Tanay Dixit, and Xudong Shen. Suppe-naturrllinstretiioo
Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 5085-5100 2022a.
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In ECCV, pp. 631-648, 2022b.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 139-149, 2022c
Chengyi Yang, Mingda Dong, Xiaoyue Zhang, Jiayin Qi, and Aimin Zhou. Introducing common null space of gradients for gradient projection methods in continual learning. In Proceedings of the ACM International Conference on Multimedia (ACM MM). ACM, 2024.
Wu Yichen, Piao Hongming, Huang Long-Kai. Wang Renzhen, Li Wanhua, Pfister Hanspeter, Meng Deyu, Ma Kede, and Ying Wei. S-lora: Scalable low-rank adaptation for class incremental learning. Under review at International Conference on Learning Representations (ICLR), 2025. URL https://opeereete F r1xx68
14

---

Published as a conference paper at ICLR 2026
Hao Yu, Xin Yang, Xin Gao, Yan Kang, Hao Wang, Junbo Zhang, and Tianrui Li. Personalized federated continual learning via multi- granularity prompt. Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 4023 4034,434,440 2024.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Short Papers), pp. 1-9, 2022.
Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, 188):364-37,, 2019..
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In ICML, pp. 3987-3995, 2017.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto. Maxim Neumann, Alexey Dosovitskiy, al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: (rX1::/90000466,, 2019.
Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slea: Slow learner with classifier alignment for continual learning pre-trained model. In ICCV, 2023a.
Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning, arXiv preprint arXiv:2308.03303 2023b. URL
/bs/ 2308 03303
Yuanhan Zhang, Zhenfei Yin, Jing Shao, and Ziwei Liu. Benchmarking omni-vision representation through the lens of visual realms. In ECCV, pp. 594-611. Springer, 2022.
Junhao Zheng, Xidi Cai, Shengjie Qiu, and Qianli Ma. Spurious forgetting in continual learning of language models. arXiv preprint arXiv:2501. 13453, 2025.
Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19068-19079, 2023.
Da- Wei Zhou, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Revisiting class-incremental learn- ing with pre-trained models: Generalizability and adaptivity are all you need. arXiv preprint 23030v2303.073338 2023.
Da- Wei Zhou, Hai-Long Sun, Jingyi Ning, Han-Jia Ye, and De-Chuan Zhan. Continual learning with pre-trained models: A survey, arXiv preprint arXiv:2401.163386 2024a.
Da- Wei Zhou, Hai-Long Sun, Han-Jia Ye, and De-Chuan Zhan. Expandable subspace ensemble for pre-trained model-based dlas-incrrmental learning. In CVPR, 2024b.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image bert pre- trainiiiing with online tokenizer. In ICLR, 2022.
15

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1
AUTHOR CONTRIBUTIONS AND THE USE OF LARGE LANGUAGE MODELS
The authors confirm that all research contributions, encompassing the conceptualization, method- ology, data analysis, and derivation of scientific conclusions, were performed entirely by the human authors without assistance from large language models. The fundamental intellectual content and technical execution of this work are original human efforts. Large Language Models (LLMs) were used only for editorial purposes refine the presentation of the final manuscript, specifically for improving the grammar, clarity, and flow of the written text.
A.2 PROOF OF PROPOSITION
Before proving Proposition 4.1 we first establish a supporting lemma.
Lemma A.1 (Gradient Preservation under Orthogonal Updates), Let Cj : RR + R be a twice- differentiable loss function corresponding
task and let W be the model parameters after completing j. j. Suppose at step t >> j. the update direction Dt for task satisfies (VL, j).9: The updated parameter is given by: W. = W, nğt- Further assume that the second-order term H in the Taylor expansion of V L can be ignored. Then, the gradient of task j remains unchanged:
V,, = VL)(W,).
Proof. Since Lj is twice-differeeniable, we apply the first-order Taylor expansion of the gradient at point W, in the direction of 9x:
(W,) (W, ng:) = VL,(W, nH3H + =(n)-
Now, under the assumption that n H is negligihle (i.e., small learning rate and low curvature), we ignore the second-order term:
VW,(W.) VL,
Based the Lemma A.I we denote VW, G; Next, we provide the proof of Proposition 4.1
Proposition 4.1. Assume the loss L, (W) I-smmooh for E (I....,I). Let the model update
be W, Wi-1 + AW, Then:
G. + (AW, G.) (t-1)L
(C, - (.WW.1))
AW,
2
Proof. By L-smoothness of each L,, we have:
f,(W,) s C.(W (VC.W..))..W. I. + lw..I.
Summing over i i= to
((w)) C.W. --1 Σ VL.W.-)) Et AW.
1 Let G T-T E VW,(W...).. then:
VCC = (t-1))
Substituting back gives:
(C.(W.) - ( G. (AW,, G.) + (1-1)L, (1-1) IAW.1. 2
16

---

Published as a conference paper at ICLR 2026
A.3 PROOF OF THEOREM 4.2
Let W denote the weight matrix of a linear layer in the model, updated as W. = W.-
W W-1 UK Uk, AW, We provide the expected values of the stability loss:
EIC (t (k,) ( (A G.
(18)
and the plasticity loss:
R (W.)) = (AW,, G.).
(19)
Proof. Stability Loss:
By definition, the projected update is:
AW, U. AW,
Thus, the expected stability loss is:
E[C = (AW, G
(U T AW, G
-(--))[[T(AWW U G.
Let G = -Ea= y.u.v be the SVD. Then,
Σ
a,u.y,
i=d-k
So:
E[C,11 (t 1)
Σ or BIW,, u..))..
imd-ke+1
Under the uniform distribution assumption, all expected projections are equal:
(,))al Vi.
Then:
E[C (1
Also,
AWL G Σ os u.. = -.
1=1
c (AW L: G
Σ 0i
Thus,
B[C = (1 - 1) (k.) (AW,, G
Plasticity Loss:
The plasticity loss is:
ECC B(AW, G.)) = T AW, G.)).
17

---

Published as a conference paper at ICLR 2026
Let a; (AW,, u.), 3== (G,, u.). Then:
Nx
E|c"] =
Under the uniform assumption, the expected contribution over any direction is 1 2. hence:
B(C hit (AW, G.).
A.4 EVALUATION METRICS
To evaluate continual learning performance, track the average classification accuracy all classes encountered so far at the end each task's s training following (Jiao et al. 2024). We denote by Aú the average accuracy the i-th task after training the j-th task. Below, provide formal definitions for two key metrics: FAA and CAA.
(i) Final Average Accuracy (FAA). FAA measures the overall performance after learning all tasks, defined as:
T
FAA Σ AT,
(20)
where T the total number of tasks and Ayr is the accuracy for task i after completing task T. A larger FAA indicates stronger ability to learn while minimizing forgetting. In some literature, FAA also referred to as "Last-Ace."
(ii) Cumulative Average Accuracy (CAA). CAA the average of the FAA values computed after each task is learned, given by:
1
CAA
Ar
(21)
T
captures the overall performance at every incremental step. This metric sometimes referred to as "Inc-Ace."
A.5 THE SIZE OF MINOR SUBSPACE EVOLVES DURING TRAINING.
We tracked the evolution of the minor subspace size throughout training ImageNet-R with 20 tasks. As shown in the Fig. 4 as the number of tasks increases, model stability becomes more critical, leading to progressively smaller minor subspace. Furthermore, when comparing different layers of ViT, the minor subspace larger in shallower layers and gradually decreases as the layer depth increases. This suggests that changes in the deep-layer parameters have greater impact model stability.
When the value of changes, the learning space of the model varies significantly. Nevertheless, the model is still able to learn tasks effectively under different settings of a. Furthermore, as the number of tasks approaches infinity, the size of the minor subspace gradually shrinks toward zero, raising the question of whether the model can still maintain its learning ability in this limit. To provide intuition, we conduct simple experiment to explain this phenomenon.
Table 8: Average accuracy tasks 2-20 under different fine-tuning strategies.
Method
Avg. Acc (Tasks 2 2-20)
Only head
66.08
Only head and the first task
74.75
SplitLoRA
81.47
18

---

Published as a conference paper at ICLR 2026
Table 9: We present FAA (%) and CAA(%) on ImageNet-R: 10-tasks. Backbones are with different self-supervised pre-training paradigms: iBOT-IK and DINO-IK.
iBOT-IK
DINO-IK
Method
Pul.
FAA
CAAT
CAA
Upper-bound
84.09 0.21
81,88 0.07
DualPrompe Wang al. 2022b
ECCV22
61.51 +: 1.05
67.11 0.08
58.57 = 0.45
64.89 = 0.15
RODA-Promph Smith et al 2023
CVPR23
66.56 :: 0.68
0.57
63.15 = 0.39
69.73 = 0.25
HiDe-Prompt Wang et al 27238
NeurIPS23
7 0.21
0.13 73.62
68.11
71.70 = 0.01
InfLoRA Linng &
CVPR24
71.84
0.09
78.29
0.009
68.31 + 0.28
76.15 0.05
VPT-NSP Lu al.
2024
NeurIPS24
73.85
0.23
80.34
0.60
69.45 0.74
76.38 = 0,50
VQ-Prom Prompi Jiao et al. SplitLoRA
2024
NeurIPS24
71.68
+ 0.72
76.66 0.40
68.42 0.28
74.43 + 0.58
This work
74.58 1.05
81.45 1.72
70.49 0.31
78.15 1.13
As Tab. R shows, "Only head" means training only the classifier head for each task, which can be regarded as the extreme case where the minor subspace is nearly zero. "Only head and the first task" means training the first task's LoRA together with the classifiers for all tasks. SplitLoRA is used for comparison. It can be observed that even when the minor subspace is extremely small (or effectively zero, as in the "Only head" setting), the model can still learn tasks to a reasonable extent. Moreover, when subsequent tasks are fine-tuned on the first task's LoRA, performance further improves. This suggests that the knowledge contained in the pre-trained model and the beneficial knowledge retained from old tasks provide sufficient support for learning new tasks, even under highly restricted learning spaces.
775
100
532 of intooe sahopace
-00
225
IrE.ARA
660
IntaaB
Spi
SpitaRkie100
of of mine allquue
Sp18 2/1
Sp1t (
Sp5d BR
Sp
SpiLaRAie 9)
6.00
460
SpitoRAier1)
575-
Number of tocks
of talks
Number of tadks layer
layer 0
layer 2
0te of mitorr sitegace
of bnboor 6ab0000
720
Gie of miner caleguac
640
600
590
SpliLoBAic
3.50
Spadd alt
SpiLoRAie
0000
720
SpiioRAAe-11
240
Number of tooks
Numberof of taoks
Number todes layer
layer
Figure 4: We recorded the evolution of the minor subspace size during training ImageNet-R with 20 tasks.
A.6 VARIANT PRE-TRAINED MODELS.
Tab. 9 provides summary of experimental results the 10-task ImageNet-R dataset using different self-supervised pre-training paradigms. Specifically, we evaluate our method with iBOT-IK (Zhou et al. 2022) and DINO-IK (Caron al., (021) pre-training frameworks. These results clearly demonstrate that SplitLoRA consistently outperforms state-of-the-ans continual learning methods, irrespective of the pre-training paradigm used. This robustness underscores the generalizability and effectiveness of SplitLoRRA leveraging self-supervised pre-training for continual learning tasks.
A.7 MORE RESULTS ON OTHER BENCHMARK.
We follow (hou et al. 2024a) to evaluate the performance on CIFAR100 (Krizhevsky 2009). CUB200 (Wah [ll.[200] 1h, ImageNet-R (Hendrycks al. 2021a), ImageNet-A (Hendrycks et al.
19

---

Published as a conference paper at ICLR 2026
Table 10: Comparison with state-of-the-arr methods multiple benchmarks. We report CAA and
FAA (%) on base and incremental sessions.
Method
CIFAR DO Inc5
CUB Do Inc10
IN-R DO Inc5
IN-A BO Inc20
Obj BO Inc10
Omai BO Inc30
VTAB Inc10
Average
85, 79
67.05
25
06.53/
50
78
77. 77717771
65.30
DualPrompt
87.87/
77.47/
63.317
/41677
06.66/1
67.11
CODA-Prompl
89,11/81.5 81.96
$4.00/ 73.37
64,42/550
53.54
66.07 53.29
77.03 68.09
83.90/10 83.02
69.68
DAP
94,52/ 90.02
94.76/ 94.03
80.61 / 74.76
54.391 46.32
72.08/39.31 /:9.51
S6.44/ 80.65
54.65 84.64
78.47
DAP wio BI
68.07/58.0 55.16
65.27/52.0
50,401 57.99
/2..8/21.84
50.47 /37.55
65.43 65.43/52..53
79.63 79.
53.83
SimpleCT.
87.57/811 81 1.3
92.211 / 96.73
62.55 / 54.55
99.77 / 48.91
65.45/53.55
79. 73.15
85. 99 ( 58
72.55
ADAM VPTD
88.46 82.17
91.02
68.79
774.47
83.00
ADAM SSF
91.72
61.30/50
/7400
85.66/
ADAM -Adapoer
85.15
92.21 86.73
72.35
50.47
74.37
RamPAC
/ 89 30
93.13/ 89
75.74 68.75
64.16 /2.86
71.67 66.08
85.95 79.55
00.56/91.88
79.17
EASE
91.51/ 85 30
92.25 86.81
78.51 70.58
55. 7.55.04
70.84 70.84/57.8 /7..86
S1.11/ 74.85
93.61/95.
79.39
HiDe Prompt
91.22/ 89.92
$9.75 89946
76.20 74.56
51.41 7 49.27
70.13 / 62.88
76.60/ 77.01
91.24/92.75
78,02
ESN
87.15/ 80.37
65.69 / /6.30
60,69/ / 55.13
44. 06. 31.07
63.73 52.55
75. 32/ 66.57
81.522/2.15
63.50
SpliLORA
93.11/90.84
91.52/87.46
81.74 74.81
66.11
66.60/61.33
$2.30/ 76.86
91.39/91.95
79.95
2021b),
ObjectNet (Barbu et al., 2019), Omnibenchmark (Zhang et al.
2022) and VTAB (Zhai
et al.
2019) comparing with nine methods: L2P (Wang al.] (202cc).
DualPrompt (Wang al.
2022b).
CODA-Prompt (Smith et al. (023). DAP (Jung et al. 2023). SimpleCIL (Zhou et al.
2023).
ADAM (Zhou al.] 2023), RanPAC (McDonelll al.] 2023). EASE (Zhou et al. 2024b),
ESN
(Wang et al. 2023d) and HiDe-Promy (Wang et al. 2023a
Tab. 10 compares SplitLoRA with state-of-the-ar continual learning methods on seven benchmarks. SplitLoRA achieves the best average performance (79.95%) and consistently ranks top in individual tasks. In particular, it excels on ImageNet-R, Omni, and VTAB, demonstrating strong generalization and knowledge retention. This confirms that SplitLoRA effectively balances stability and plasticity across diverse scenarios.
A.8 LONG TASK SEQUENCES OF 50 AND 100 TASKS ON IMAGENET-R AND DOMAINNET.
To systematically evaluate the capacity of SplitLoRA, we further introduce long task sequences of 50 and 100 tasks on ImageNet-R and DomainNet. Results are shown below:
Table 11: Performance of SplitLoRA on long task sequences (50 and 100 tasks) on ImageNet-R and Net.
Method
S0.INR FAA
50.INR Furget
50-DN FAA
50-DN Forget
100-INR FAA
100-INR Forget
100-DN FAA
100-DN Forget
IntLoRA
59.00
11.02
69.96
9.51
38.16
15.11
44.32
17.85
NSP
69.48
6.1
71.28
11.36
62:23
12.13
$7.35
13.82
SpliLLRA
71.75
7.59
70.53
10.18
63,30
12.29
61.85
9.27
From these 50-task and 100-task results, we observe that SplitLoRA maintains stable FAA even in extremely long sequences. When the number of tasks reaches 100, the model still retains strong continual learning ability.
A.9 MORE RESULTS ON CNN.
Our experiments is based on GPM (Saha et al. 202 and its extensions: TRGP (Ln et al. 2022b SGP (Saha & Roy| 2023), and GPCNS (Yang al., 2024 To ensure a fair comparison, we the same network architectures as those employed in the baselines. For GPM-based experiments, we utilize 5-layer AlexNet-like network as the backbone for the 10-Split and 20-Split CIFAR-100 datasets, and reduced version of ResNet-18 for the 20-Split MinilmageNet experiment.
A. MORE FINE-GRAINES COMPUTATION OVERHEAD ANALYSIS.
We have added detailed training and evaluation (T&E) time, as well as the projection-updan (PU) time, for the 5-tasks ImageNet-R setting. The table below reports all training and projection-update times in minutes.
Both InfLoRA and SplitLoRA the fixed LoRA matrix A as the projection basis, meaning no additional forward computation is introduced during training. However, SplitLoRA requires significantly less projection-update overhead than Anf,,RR, as reflected by the PU times above.
20

---

Published as a conference paper at ICLR 2026
Table 12: Performance compaarion of GPM-based experiments.
20-Split CIFAR-100
10-Split CIFAR-100
20-Split MinilmageNet
Method
ACC(%))
BWT(%)T
ACC(%)
BWT(%)
ACC(%)
BWT(%)
FS-DGPM
80.50
-3.30
74.33
-3,00
GPM
77.34
0.01
71.81
-0.11
63,90
-1.3
TRGP
81.68
-0.13
75.01
-0.01
62.68
-1004
SGP
80.21
-0.88
74.97
-0,98
66.99
-2.46
GPCNS
78.63
-1.93
71.84
-3.44
62.85
-1.90
SplitLoRA
81.93
-1.06
75.03
-0.50
66.17
0.20
Table 13: Time costs ImageNet-R tasks.
Method
Teski (T8L))
Todk1 PU
Task3 (T
Tisk3 PU
Task3 (T&1))
Task3
Task4 (T&8)
Task4 PU
Tadk5 (E&E)
PU Total
SpliLoRA
20,70
18.24
A.11 MORE RESULTS ON LANGUAGE MODELS.
Following existing CL methods (Razdaibiedina et al., 2023). we evaluate different methods on SuperNI (Wang al. 2022a) and Long Sequence (Razdaibiedina al. 2023) benchmarks. SuperNI benchmark includes various types of NLP tasks, including dialogue generation, information extraction, question answering, summarization, and sentiment analysis. Three tasks are selected from each type, resulting in 15 tasks. These tasks are arranged into two different task sequences with different orders, referred as Order and Order 2.
Table 14: Details of different tasks in SuperNI Benchmark.
Dataset name
Task Type
Metric
Task639 multi salll_ooo_ss___ moo__serr_uterancee geterratinnnnnnn
summarization
Rouge-L
Task1590 Task15900diplomacyy diplomacy_tese _eetttt___t___eee generation
summarization
Rouge-L
next
summarization
Rouge-L
Task181_outcome citicoomm_xttaaatinn
information extraction
Rouge-L
Task748 fask748gglucose @elllcoos_eeverreeeeeeeee cetrrseecuusseeeeeee cause_event detection
information extraction
Rouge-L
Task1510 Task15100evalution wtw.lettio_rrelattiiie celation_extracttee
information extraction
Rouge-L
fask002_quoref_eee annswrrgeneeration
dialogue generation
Rouge-L
Task073_commonneescqqa.. wttpc/cnssss_aaaeeeee_eeee
dialogue generation
Rouge-L
Tusk591_sciq_answerr__ answerr_asww__eeeeeee__ wtw.aaseweerggetttiioooo
Task591
dialogue generation
Rouge-L
Task511_reddit leel_looggggg__geeeeeeeeeeeeelee
tonn_ttex_summarrzatioo
question answering
Rouge-L
Task 10000_ssm__s_0 assu_ssmmarizattio
question answering
Rouge-L
Task1572_ seamsum_uummary
question answering
Rouge-L
Task363_ss sst2___pp__p__ oosa_poolaitt...... classification
sentiment analysis
Accuracy
Task875 emotion Task875_emotion classification
sentiment analysis
Accuracy
Task1687 sentiment 40 classification
sentiment analysis
Accuracy
We compare our method with sate-offtheean CL methods, including IncLoRA Wang al. 2023c
C-LoRA (Smith et al.] 2024b). O-LoRA (Wang al.) (2023c), InfLoRA (Liang & Li| (2024), and GainLoRA (Liang et al.). Additionally, we introduce a simple baseline called ScqLoRA, which does not expand new LoRA branches but sequentially updates old LoRA parameters for new tasks and lacks mechanism to mitigate forgetting.
Evaluation Metric. We Ay.: denote the model's performance the i-th task once the model learns the j-th task. Specifically, Aj.: represents accuracy for classification tasks and Rouge-L for other types of tasks. Following traditional CL. works (Chaudhry al. 2020), employ average
21

---

Published as a conference paper at ICLR 2026
Table 15: The order of different task sequences for experiments.
Benchmark
Order
Task Sequence
task1572 task363 1290 - task181 + task002
SuperNI Benchmark
+ task1510 - task639 + task1729 + task073 +
task task748 task511 task591 task
task875
2
task748 - task073 + task 1390 - task639 - task
+ task1687 + task591 + task363 task1510 +
task 179 - task181 + task511 1 - task002 - task
- task875
performance (AP) and forgetting (FT) to evaluate the model's performance, The formulas for these two metrics are defined as
T AP Σ
AT
FT
(maxie(1.....T111)A)) AT,)),
(22)
1=1
where T denotes the total number of tasks in the task sequence. AP evaluates the model's final performance, and FT' quantifies the forgetting.
Table 16:
T5-Large Results.
Method
Order APP
Order FTІ
Order APT
Order 2 FT)
SegLoRA
7.30
47.60
7.03
47.97
IncLoRA
12.33
41.93
16.65
36.56
C-LoRA
22.69
24.25
32.81
11.60
O-LoRA
26.37
19.15
32.83
11.99
Gain-OLoRA
47.84
2.26
46.84
2.91
InfLoRA
39.78
7.64
39.57
8.93
Gain-Infl.of oRRA
46.21
2.40
46.44
2.61
SplitLoRA
51.64
1.14
52.89
0.73
Table 17:
LLaMA2-7B Results.
Method
Order APt
Order FT]
Order 2 APf
Order 2 FT)
O-LoRA
39.37
15.84
37.55
20.23
Goon-
$1.10
4.96
$1.14
5.57
InfLoRA
42.93
11.23
39.94
15.00
Gain-In/LoRA
51.27
2.84
50.17
4.71
SplitLoRA
53.18
3.01
$2.43
3.72
These results demonstrate that the core idea of SplitLoRA continues to hold even for large language models, further validating generality and scalability of our framework.
22

---

Published as a conference paper at ICLR 2026
Algorithm 2 Projection update between tasks for SplitLoRA
1: Input: datasets (Dr}í=1- current weights Wr, previous average gradient G (Eq. (3)
2: Output: updated G minor-subspace bases {Ū
++1 }. and LoRA bases {A(+1) } for task +++1 3: // Step 1: Update average gradient of old tasks (GPM-stylle using LoRA representations)
4: Initialize per-layer accumulators
(t) G. 0, c()) for all layers
5: for each mini-batch B C DD do
6: Forward pass through fw, obtain LoRA inputs (1) for all LoRA modules
7: for each layer I with LoRA module do
8:
Accumulate: Gr cl (0)
(Cow((t) G,ow,(() x (1)
9:
10: end for
end for
12:
for each layer do
neww,()) G. Gew...()) (0) fcl9)
13:
14: end for
Update the running average of old-task gradients using Eq. 3 Gold = ((t-1)): + G.ew)
16: // Step 2: Compute minor subspace and optimal dimension k++1
17: for each layer I with LoRA module do
18:
Perform SVD on G
((1))
141 () V
6+1 SVD(G (.ld.()) (1)
19:
For each candidate k, compute 4 (k) and the objective in Eq. (15)
20:
Select (1)
2+1 arg min& ((t) e 4 (k)) a :
21:
Form the mubsor-ubsspace basis:
I+1 ( k
22:
end for
// Step Initialize LoRA projection bases A in the minor subspace 24: for each layer with LoRA module do
25: Sample a random Gaussian matrix R!)) E R C 1- xr
(1) 26: Construct A 41 using Eq. I17
A U (0
++ (!)
end for
28: return Gold, (ū 41 ). (A
23


Reviews:
[{"review_id": "5xwC7DQUqn", "reviewer": "Reviewer_Ceq4", "summary": "This paper proposes SplitLoRA, a Gradient Projection method for continual learning. Specifically, the authors present a theoretical analysis of how subspace projection influences the stability-plasticity trade-off, and then introduce a LoRA-based method to determine a better minor subspace for learning new tasks.", "questions": "1. In Eq. 7, should the gradient $G_t$ be $\\nabla \\mathcal{L}_t({W_t})$ or $\\nabla \\mathcal{L}_t (W_{t-1})$? \n\nAnd in Eq. 16, should $<\\Delta W_t, G_t>$ be negative, as the optimization follows gradient descent? \n\nAdditionally, should $\\hat{G}_t$ be $G_t^{old}$​? Please correct me if I misunderstand.\n\n2. As stated in Line 304, the ratio $\\alpha$ can vary dynamically. Does the mechanism of relacing it with a fixed hyperparameter have any theoretical justification, such as a bound or convergence guarantee?\n\n3. I recommend the authors provide more fine-grained time cost analysis in Table 4, such as time per iteration or per epoch. I suppose the currently reported time reflects the whole training process, which may be affected by other factors like data loading and evaluation.\n\n4. The authors should double-check their reference. There is a paper labeled \"under review at ICLR 2025\".", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper establishes a theoretical foundation for the proposed method by analyzing gradient projection and minor subspace, and by formulating a corresponding optimization problem.\n2. SplitLoRA attains state-of-the-art performance across three benchmarks under different task configurations.", "weaknesses": "1. A major concern is the absence of evaluation metrics. In continual learning, it is critical to assess the stability-plasticity trade-off using both average accuracy (like FAA and CAA reported by authors) and backward transfer (BWT). The omission of BWT weakens the credibility of the empirical results and raises doubts about the robustness of the performance evaluation.\n2. Another issue is the limited selection of baselines in experimental comparison. As mentioned in Section 2.2, Gradient Projection methods in continual learning are closely related to the proposed method. However, I do not notice any Gradient Projection-based methods are included in experiments. Instead, most baselines seem to be prompt-based approaches. Could authors give any clarification on this baseline selection and include latest Gradient Projection approaches for a more comprehensive evaluation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y2p2QCrFAO", "reviewer": "Reviewer_kyb6", "summary": "This paper addresses the continual learning problem and the gradient projection method. Due to the balance between plasticity and stability, the paper explores low-rank adaptation within the continual learning paradigm. The experimental results also demonstrate the effectiveness of the proposed methods.", "questions": "see the weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper utilizes LoRA in the continual learning framework and uses the gradient to update the process. An upper bound analysis on the loss increase is provided. Although I have not checked the entire proof, based on previous results, Equation 7 is more or less correct. The SplitLoRA algorithm is also presented.”", "weaknesses": "The overall paper is not well written. The main motivation for balancing plasticity and stability is addressed through the use of LoRA. However, LoRA may reduce information, and it is unclear why this method has this effect. Please provide more explanation to clarify the motivation and core contribution of the approach.\n\nThe modified method uses LoRA to replace the corresponding update, which is more akin to a report.\n\nRegarding the dataset and model, they are outdated and not suitable for the current problem. More recent data and new open-source models should be incorporated to demonstrate the effectiveness of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Arfzj04uj", "reviewer": "Reviewer_1gUU", "summary": "The paper adopts the concept of Orthogonal projection in the gradient (sub-)space in continual learning (CL). It aims to balance two important aspects in CL, i.e., stability and plasticity. The proposed SplitLoRA partitions the gradient space into two complementary  subspaces via SVD, i.e., a major subspace capturing the most informative and stable directions of previously learned tasks, and a minor subspace containing the remaining, less significant directions, where the learning process takes place for the new task. The paper provides a theoretical analysis of how such subspace partitioning affects model stability and plasticity for CL, then proposes the SplitLoRA method.", "questions": "Can the authors investigate the impact of the SplitLoRA method onto the capacity of a specific CL setting? It'd be more convincing to justify the gradient space partition strategy beyond the concern of balancing stability and plasticity.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper well articulated its core idea of splitting orthogonally the gradient space into two complementary subspace via SVD, one for previously learned task and the other for the new task. The motivation is simple, but the authors provided a theoretical analysis on the impact of subspace partitioning on model stability and plasticity for CL. The proposed SplitLoRA methods demonstrated consistent good performance against the SOTA.", "weaknesses": "It is arguable that the orthogonal projection might not be the ultimate solution for the catastrophic forgetting, although the proposed SplitLoRA could still match some of our current development of CL. Capacity of the proposed SplitLoRA in terms of the number of well learned tasks is not sufficiently discussed. It seems that the partitioning of the gradient space into previously learned tasks and the new task doesn't impact on the number of tasks to be learned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aliLChjY6l", "reviewer": "Reviewer_swCo", "summary": "The paper studies the stability–plasticity trade-off in continual learning through the gradient projection method, which is a popular approach in this field. Existing projection methods heuristically choose the number of singular vectors, k, but the authors theoretically show that the upper bound of the loss increase can be expressed in terms of stability and plasticity losses. They further demonstrate that these two losses can be written as functions of k and propose a way to find the optimal value of k. The effectiveness of the proposed method, SplitLoRA, is demonstrated on CIFAR-100, ImageNet-R, and DomainNet, where it outperforms the baselines.", "questions": "1. Does the method work without LoRA?\n\n2. The authors should include pseudo-code for the complete end-to-end model training process. For example:\n\n    1. After training task $t-1$ and before training task $t$, obtain the average gradient of the model on the training data for task $t-1$. This average gradient is denoted $G_t^{old}$\n\n    2. Obtain the minor subspace $\\hat{U}_t$ of the gradient $G_t^{old}$.\n\n    3. During training for task t, map the gradient $\\nabla{W}$ onto the minor subspace of the previous task $t-1$ as $\\hat{U}_t^k \\hat{U}_t^{kT} \\nabla{W}$\n\n    4. ...", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper provides solid work on a fundamental issue in continual learning, the balance between stability and plasticity. Expressing the upper bound as a function of stability and plasticity losses, and further relating this to the number of singular vectors, is an interesting and meaningful contribution.", "weaknesses": "- Although expressing the upper bound in terms of k and proposing a method to find the optimal $k$ is interesting, the paper lacks depth, as this is the only component studied in detail. The authors argue that existing methods are limited because they require selecting $k$, yet their own method for finding the optimal $k$ also depends on the hyperparameter $\\alpha$.\n\n- The experimental section lacks thoroughness, as it only presents basic results such as comparisons with baselines and limited robustness analysis over different hyperparameter settings.\n\n- Additionally, the title does not accurately reflect the content. The proposed method does not fundamentally rely on LoRA and could be applied to other architectures. It is unclear why the authors emphasize LoRA in naming the method SplitLoRA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)