KeepLoRA: Continual Learning with Residual Gradient Adaptation
Authors: Mao-Lin Luo
Zi-Hao Zhou
Yi-Lin Zhang
Yuanyu Wan
Min-Ling Zhang
Tong Wei
Venue: ICLR 2026 Poster (2026)
Keywords: continual learning
parameter-efficient fine-tuning
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Continual learning for pre-trained vision-language models requires balancing three competing objectives: retaining pre-trained knowledge, preserving knowledge from a sequence of learned tasks, and maintaining the plasticity to acquire new knowledge. This paper presents a simple but effective approach called KeepLoRA to effectively balance these objectives. We first analyze the knowledge retention mechanism within the model parameter space and find that general knowledge is mainly encoded in the principal subspace, while task-specific knowledge is encoded in the residual subspace. Motivated by this finding, KeepLoRA learns new tasks by restricting LoRA parameter updates in the residual subspace to prevent interfering with previously learned capabilities. Specifically, we infuse knowledge for a new task by projecting its gradient onto a subspace orthogonal to both the principal subspace of pre-trained model and the dominant directions of previous task features. Our theoretical and empirical analyses confirm that KeepLoRA balances the three objectives and achieves state-of-the-art performance. The implementation code is available at https://github.com/MaolinLuo/KeepLoRA.

Published as a conference paper at ICLR 2026
KEEPLORA: CONTINUAL LEARNING WITH RESIDUAL GRADIENT ADAPTATION
Mao-Lin Luo Zi-Hao Zhou!? Tong Wei
Yi-Lin Zhang!?. Yuanyu Wan',
Min-Ling Zhang!
"School of Computer Science and Engineering, Southeast University, Nanjing 210096, China
KKey Laboratory of Computer Network and Information Integration (Southeast University),
Ministry of Education, China
"ccoool of Software Technology, Zhejiang University, Ningbo, China
ABSTRACT
Continual learning for pre-trained vision-language models requires balaneing three competing objectives: retaining pre-trained knowledge, preserving knowl- edge from a sequence of learned tasks, and maintaining the plasticity to ac- quire new knowledge. This paper presents simple but effective approach called KeepLoRA to effectively balance these objectives. We first analyze the knowledge retention mechanism within the model parameter space and find that general knowledge is mainly encoded in the principal subspace, while task- specific knowledge is encoded in the residual subspace. Motivated by this finding, KeepLoRA learns new tasks by restricting LoRA parameter updates in the residual subspace to prevent interfering with previously learned capabil- ities. Specifically, we infuse knowledge for a new task by projecting its gra- dient onto subspace orthogonal both the principal subspace of pre-trained model and the dominant directions of previous task features. Our theoretical and empirical analyses confirm that KeepLoRA balances these three objectives and achieves state-of-the-an performance. The implementation code is available at https//gtttubbboom//aaaaaatt/let Kocpo
INTRODUCTION
Cabech301
DTD
KeepLoRA KeopLoNk:
Int GRA
Performanc Pletersion
1PC
Rereegen
DI0
+18
CIAA100
SUNe97
La La
Plavers
see
Lef
Pommtt
Dackbore StanfrdCars
Backbone 100% topo
Backbone 90% west
Baciborn west Bacibone west
L2P
Backbore 90% cest
Backbone 65%4 lest
Duckbore 95%
Tonder
(a) General-domain tasks
(b) Specific-domain tasks
(c) Performance comparison
Figure 1: Analysis of model parameter subspaces and overall CL performance. In Fig. la and lb we measure zero-shot performance after reconstructing attention weights using only the top prin- cipal singular components. While performance on performance most specific-domain datasets degrades sharply as more low-energy components are removed. In Fig. Ic. the Last metric measures the accuracy gain on the final learned task relative zero-shot baseline, while Transfer measures the accuracy degradation unseen tasks.
datasets remains highly robust,
CCoresponning author.

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of LoRA-based approaches continual learning. When learning the t-th task, O-LoRA regularizes the down-projection matrix to be orthogonal to that of previous tasks to improve stability; InfLoRA constrains the optimization of task features H, to be orthogonal to previous dominant directions M;-1 to jointly improve plasticity and stability; SD-LoRA optimizes decoupled LoRA to improve stability and re-scales the magnitudes {ar) of parameters from previous tasks to improve plasticity. By contrast, our method constrains optimization to a subspace orthogonal to both the principal weight subspace W and previous task directions M, preserve stability, and initializes the update in an optimal gradient space derived from G, to boost plasticity.
Initialization
Method
Training Objective
LoRA
LoRA
O-LoRA Wang et al.
2023a
A - N (0,02)
B 0
LailA + I1 A.I
USV = SVD
HH. M. MM
InfLoRA Liang
2024
La((B)
A A- U.
B 8-0
SD-LoRA Wu et al.
2025b
A- (0,0)
0
Lallaa)ii 1--1 A,
SVD(G.
W, G. M M,, A, B S,V.
KeepLoRRA (This paper)
(ai(Bi)
Vision-language models (VLMs) have demonstrated remarkable zero-shot transfer capabilities, making them cornerstones of many downstream applications (Comanici et al. 2025 Achiam et al.) 2023 Radford et al. (021). Despite this success, their performance on certain datasets can be insuf- ficient, motivating the need for continual learning (CL). An effective CL method requires the balance of three competing objectives: maintaining the ability to learn new knowledge (plasticity), prevent- ing the forgetting of previously learned tasks (backward stability), and crucially, preserving the general pre-trained knowledge that guarantees general transferability (forward stability) (Mukhoti) et al. 2024] /Zheng et al.] (023). Degradation of this pre-trained knowledge is particularly detrimen- tal, erodes the core value of VLMs. Therefore, the central challenge is how effectively learn new knowledge without undermining these critical stability constraints.
A straightforward solution, replaying pre-training data, rarely viable due to prohibitive computa- tional costs and the frequent unavailability of proprietary training corpora (Wang et al. [024] Zhou al., 2024; Rolnick et al., 2019) The current alternatives largely follow two paths. The first is (i) reference-data regularization, which uses reference data to anchor the model parameters and
retain stability (Wu et al., 2025a Zheng et al. 2023). However, the success of these approaches is highly sensitive to the choice of reference data with additional training costs (Luo et al.) 2025 Zheng et al. 2023). The second path involves (ii) architecture extension, such as prompt-pool (Fu
al. 2025 Wang et al. 2022b) or MoE-adapters Yu et al. 2024; Dou et al. 2024) while freezing the model backbone. Although effective in preventing forgetting, these modules increase inference costs (Nayak et al., (025) and complicate the deployment (Cai et al.] [2025] Zadouri et al.) 2023) Since trained weights are compact representations of data (Deletang et 22
et al.)
2024], an ideal CL method should infuse new knowledge directly into the existing parameter space, leveraging its inherent redundancy rather than accumulating external modules (Sharma et al.) 2024).
To identify where new knowledge can be infused without disrupting existing abilities, we analyze the parameter space of the backbone attention weights via singular value decomposition (SVD). As shown in Fig. [a and Tb our analysis reveals that the principal subspace, spanned by components with large singular values, predominantly encodes general knowledge, while the residual subspace, knowledge. This observation
associated with small singular values, encodes domain--seccin dicates that performance on specialized datasets is highly sensitive to alterations in the residual subspace, whereas general datasets remain robust to such changes. This insight forms the basis of approach: to coherently achieve forwerd stability, backward stability, and plasticity, CL updates should be constrained to the residual subspace, enabling the acquisition of new domain-specine knowledge without affecting the principal subspace that encodes general knowledge.
We implement our method whose updates can be merged into the original weights post-training, thus incurring no inference overhead. Existing low-rank CL methods, such as O-LoRA (Wang et al.] (023a), InfLoRA (Liang & Li 2024), and SD-LoRA (Wu et al. 2025b), lack consideration of the transfer capacity of
updates using low-rank adaptation, a parameter-e efficcient
2

---

Published as a conference paper at ICLR 2026
the pre-trained model and update directions within suboptimal subspaces, which limit their plasticity and stability as shown in Tab. [ To overcome this, we initialize the low-rank update using the gradient from the first training step. This ensures that the update direction closely approximates the full-parameter tuning gradient to boost plasticity. To keep stability, we explicitly project this update into the residual subspace of the pre-trained weights, which constrains learning on directions that do not interfere with the model's ss core transferable knowledge. Building on this, we construct a unified subspace that stores both the principal subspace of the model parameters and the dominant feature directions of each learned task. The unified principal subspace, with size not exceeding the square of feature dimension, preserves both pre-trained and newly acquired knowledge.
Our contributions are summarized as follows:
We empirically analyze the parameter space of pre-trained models and find that general knowledge is primarily encoded in the parameter principal subspace, while domain-specific adaptations are better captured by the residual subspace.
We propose KeepLoRA, a novel method that leverages residual subspace constraints for parameter updates, validated by theoretical analysis showing how it optimally balances plasticity and stability through orthogonal projections.
Experiments on dual-encoder (CLIP) and encoder-decoder (LLaVA) models validate that KeepLoRRA effectively balances the three core challenges of plasticity, backward stability, and forward stability, establishing new state-of-the-an results on benchmark datasets.
RELATED WORKS
In continual learning, forward stability is typically preserved by using reference-data regulariza- tion and architecture extension techniques. In addition, gradient projection methods are commonly employed to address backward stability and plasticity. In this section, we review these lines of work.
Reference-Data Regularization. Continual learning narrow task distributions can cause the model feature space to collapse, degrading its pre-trained zero-shot transfer capabilities (Zheng al. 2023). Reference-data methods aim to counteract this by anchoring the model representa- tions. ZSCL (Zheng et al.] 2023) uses the ImageNet (Deng et al. (009) and Conceptual Captions preserve the structure of the feature space. However, the effectiveness of this approach is sensitive to the choice of reference data and the teacher model, with performance degrading when fewer images or classes are used (Zheng et al. 2023). Yu et al. (2024) propose MoE-Adapters by training selector on the Tinylma- geNet (Deng et al. 2009) dataset to identify out-of-distribution data, which is then processed by the original frozen model. Wu et al. (2025a) leverage the generative model Stable Diffusion (Rombach) ct al. 2022) to create synthetic reference data for distillation. These methods inherently increase computational overhead and depend on external reference data, limiting their practical feasibility.
(Sharma al. | 2018]) datasets as reference data, employing distillation
Architecture Extension. Architecture extension methods freeze the pre-trained model and extend it with new parameters for each task. 1.2P (Wang et al., (022c) selects the most relevant prompts from a prompt pool, while DualPrompt (Wang et al.] 2022b) uses task-sharing and task-specific prompts. CODA-Prompt (Smith et al.) (2023) proposes end-to-end prompt selection to increase plas- ticity. MoE-Adapters et 2024) inserts a mixture of adapters into the image encoder, activating a subset for each task. DIKI (Tang et al. 2024) calibrates knowledge integration by determining the likelihood that test sample belongs to a task. IAP (Fu et al. 2025) introduces Instance-Awer Gated Prompting to improve the effectiveness of prompt selection. However, these methods often suffer from parameter selection errors or suboptimal activation coefficients. Moreover, adding external parameters does not truly infuse new knowledge into the base model.
Gradient Projection. Gradient projection methods mitigate catastrophic forgetting by constraining parameter updates into specific subspaces, thereby preventing interference with previously acquired knowledge Qiao et al. (0024). In the context of full fine-tuning, methods such as Gradient Projection Memory (GPM) (Saha et al. 2021) enforce orthogonality between the gradients of a new task and stored basis of principal gradient directions from previous tasks. To improve the efficiency of full fine-tuning, CoSo (Cheng et al. 2025) utilizes Sask-Specific Subspace Estimation and updates an orthogonal basis matrix. This thought has also been adapted to parameter-efffcient techniques. For
3

---

Published as a conference paper at ICLR 2026
example, O-LoRA (Wang et al. 2023a) constrains the LoRA subspaces of new tasks to be orthogonal to those of previous tasks, ensuring that learning occurs novel directions. InfLoRA (Liang Lip 2024 applies constraint where the LoRA down-projection matrix A is orthogonal to GPM (Saha et al. 2021) or DualGPM (Liang & Lij 2023) to prevent interference. However, these existing methods primarily focus mitigating backward forgetting, the loss of knowledge from previously learned sequential tasks. They do not explicitly address the preservation of pre-trained knowledge, which is crucial for maintaining the model's transferability and preventing forward forgetting.
METHOD
3.1 PRELIMINARY
Problem Formulation. We adopt the multi-domain task incremental learning (MTIL) setting (Zheng et al. 2023), where the model encounters a sequence of n tasks (T,,T2... ; T"). Each task T CD for E (1, n) comprises dataset DD' and corresponding class vocabulary C. The dataset D {(x),, د contains N, training examples, where each æ; denotes input image and yf represents the corresponding one-hot encoded ground truth label. The class vocabulary C = {c] )ー= establishes the mapping between categorical labels and semantic class names, with my denoting the total number of distinet classes for task T During inference, the model classifies an input image a within C² The goal of continual learning is to maintain performance on pre-trained knowledge and all previously encountered tasks while adapting to new ones.
Vanilla LoRA. Low-rank adaptation (LoRA) (Hu et al. 2022 decomposes weight updates into two low-rank matrices A E Rd, and B E R where remains frozen while only A and B are fine-tuned. The matrices are initialized with A ~ NN(0, of) and B = O. For input Ra the forward pass becomes:
min(d dour). During training, W
== x W
ạ AB T
(1)
where a is scaling factor.
3.2 KEEPLORA: GRADIENT PROJECTION ADAPTATION
Continual learning for pre-trained vision- languae models demands a balance between plasticity, the ability to acquire new knowledge, and learning stability, which comprises both forward stability to preserve general pre-trained knowledge and backward stability to retain knowledge from previously learned tasks. To address this problem, we propose KeepLoRA, a method built upon LoRA that employs residual subspace constraints unify stability preservation and new knowledge infusion.
Stability:
Preserving Pre-trained and Previous Task Knowledge. KeepLoRA retains stability by projecting the subspaces of pre-trained knowledge and previous task knowledge onto unified principal subspace. Subsequent adaptations for new tasks are then confined to the residual subspace orthogonal to this principal subspace, thereby minimizing interference with the learned knowledge.
Pre-trained Knowledge Subspace: We analyze the parameters of the pre-trained model to understand how the model stores general knowledge. Specifically, we decompose each weight matrix W e Rd dout as W = USVT. The decomposition produces a subspace Wp = U.1:pp and the subspace is constrained such that:
W 2 e l
(2)
where e (0, 1) controls the energy ratio retained in W,.
Previous Task Knowledge Subspace: To mitigate forgetting of learned tasks, the LoRA module updating matrix W for new tasks should minimize interference with features from previous tasks. Specifically, goal is to make Y LoRA, (X) as close to 0 possible for any input X from previous tasks {-] Since no real or synthetic samples from previous tasks are available for replay, we propose to extract the dominant singular vectors of previous tasks as the dominant feature directions. This approach enables to continuously compress task task-specific information and enforce matrix A to be orthogonal to the dominant singular vectors on LoRA initialization. After training for task I, extract and store the dominant feature directions for this task. These directions
4

---

Published as a conference paper at ICLR 2026
chosen be orthogonal to the subspace jointly defined by the principal weights and the dominant feature directions of all t 1 tasks. We then define the feature space for the t-th task as:
X. X; W, W X. - M M
where M E Rd.x x represents the accumulated direction matrix containing the dominant singu- and & denotes the total number of stored singular vectors. We initialize Mo = 0 an empty matrix. The number of stored vectors k is dynamically determined by an energy threshold es E (0, 1). Specifically, we retain the minimum number of dominant directions required to satisfy:
lar vectors from tasks .1.......t- t-
-),,
X 1|2 + W,W,W x,11 + || M, X ey X
We perform SVD the features X, U. and extract the top-m dominant singular vectors to update our subspace matrix: M, = M Vx(, where m determined by a threshold ey-
Unified Principal Subspace. Since both W, and M, consist of orthogonal direction vectors op- erating within the same din-dimensional feature space, and the total number of orthogonal vec- mathematically unify them into single projection subspace: M W M,]- The unified subspace leverages the theoretical foundation that predictive models can be transformed into lossless compressors (Deletang al. 2024) and model weights embody compressed representation of the training data (Franceschelli et al. 2024). Under this perspective, Wp captures the essential feature representation space of the pre- vee-training data, while M, preserves the dominant feature directions during continual learning. Both components represent compressed knowledge from their respective data distributions.
tors is upper- bound-b by din.
To ensure the new 1-th task updates never interfere with M, KeepLoRA achieves this through modified LoRA approach, where matrix A is initialized within {M, and frozen throughout training. while only B is optimized.
Gradient-Informeed LoRA Initialization in Residual Subspace. While the unified principal subspace ensures learning stability, KeepLoRA also requires maintaining plasticity to adapt to new tasks. We achieve it by initializing the LoRA module using task ssec-speiific gradient in- formation, aligning adaptation directions with full fine-tuning while confining updates to (M, 1. Specially, we utilize gradient information to guide the initialization within the constrained residual space. Let G, WWC(W Dt) denotes the gradient of the weight matrix W of the I-th task at the first training step. We project this gradient onto the residual subspace:
Plasticity:
-.
= G. -
W, G, M, M,,
(5)
plasticity
forward backward stability
We perform SVD on the projected gradient G USVT and initialize the LoRA matrices with top-r singular vectors as:
A= A= B= S VT. :1r
(6)
where 1rr denotes the first r columns of U, and r is the rank parameter. This gradient-informed initialization directly simulates the update direction of full fine-tuning while operating within the residual subspace, enabling effective adaptation without undermining these critical stability con- straints. Since the initial product AB is non-zero, the frozen parameter W can be adjusted to maintain the initial parameter values unchanged. Specifically, we replace the original parameter W with W W & AB to ensure that the initial forward pass behavior remains identical with the initial model. Algorithm summarizes the proposed KeepLoRA method.
3.3 DISCUSSION OF KEEPLORA
6q. as the core formulas of KeepLoRA, enabling its balance of plasticity and stabil- ity: G. enhances plasticity by identifying new task adaptation directions, while the subtracted terms remove gradients that interfere with pre-trained and previous task knowledge, ensuring stability. To verify these core designs, first establish the equivalence between KeepLoRA parameter update rule and gradient projection learning, defining the necessary properties of the subspace spanned by A, We then demonstrate that the initialization of A, meets these properties.
Eq.5 and Blnd
5

---

Published as a conference paper at ICLR 2026
Algorithm KeepLoRRA for Continual Learning
1: Input: Pre-trained model So with updatable parameters (B.). task sequence (T')== hyper-
parameters Ew, ff. a
2: Oufput: Updated model for with merged LoRA adapters
3: for task f. = to do
4: Initialize KeepL.oRA through Eq. 5 and Eq. 6
5: Replace the parameter W with the modified frozen parameter W' W A.
6: Compute the loss and optimize the KeepLoRA parameters B,
7: Merge KeepLoRA and current model by W W + A,
8: Extract dominant feature directions through Eq 3 and Eq
9: end for
Analyzes of Frozen A, LoRA Updates. The KeepLoRA parameter update method involves freez- ing A, and optimizing only B,. The following proposition demonstrates that this update rule is equivalent to gradient descent constrained within the subspace span(A.).
Proposition 3.1. (LoRA with frozen down-pme projection A, is equivalent gradient projection up- date.) Let L(W; D') denote the loss function for the t-th task T where: W = W + + A,, G. = V L(W; D'). Optimizing only B₁ through gradient descent with learning rate 17 is equiva- lent performing gradient descent the orthogonal projection of G2 onto span At). The weight update of W satisfies:
AW -AAB. ACA,A Gt,
where
na² 22
is positive constant integrating the learning rate and LoRA scaling effects.
Remark. Proposition 3.1 reveals that frozen A, updates are inherently subspace constrained: all changes to W are confined to span( At), as A, acts as an orthogonal projection operator on this subspace. Furthermore, span(At) is required satisfy the following two properties in continual learning: (i) Orthogonal to knowledge subspaces: span (A.) needs to be orthogonal to subspaces en- coding pre-trained knowledge and previously learned tasks, ensuring updates to W do not interfere with existing knowledge, preventing both forward and backward forgetting. (ii) Adaptation to the current task: span(A;, AA:) needs to capture the dominant directions of Gt, approximating the gradient of full- parameter fine-tuning maintain plasticity.
Validation of KeepLoRA A, Initialization. The preceding proposition outlines the required prop- erties of span At). The key question is whether the KeepLoRA initialization of A, meets the two properties. We validate it by connecting the initialization to constrained optimization problem. Proposition 3.2. KeepLoRA initialization of A. through Eq. and Eq. 6 is the solution to the following constrained optimization problem:
min
||: - A,A, G.P.P.
A A.=1
(8)
s.1 W A, M,
where G3 is the current task gradient w.r.t the base model W, W, is the principal subspace of pre-trained parameters, and M,-1 is the dominant feature directions from previous tasks.
Remark. Proposition 3.2 directly connects KeepL.oRA's initialization technique to the two proper- ties of Proposition 3.1 verifying its optimality: (i) Satisfying orthogonality (via constraints): The cquality constraints p A, = 0 and M, A, = O explicitly enforce span(A:) 1 span Ww)) and span( A.) 1 ().. It guarantees that span (A.) is orthogonal to both the principal subspace of the model parameters and the dominant feature directions to preserve stability. (ii) Optimal adap- tation (via objective): The objective function minimizes the Frobenius norn of G2 A,A, G., the residual component of G, that lies outside span span(A)). (A)) By the Pythagorean theorem for the Frobenius norms I = A,A, G.ll? + ||G A, G. minimizing this residual is equivalent to maximizing the norm of the projected gradient A,A, G, It ensures span captures the domi- nant gradient directions for the current task, preserving plasticity.
In summary, Propositions 3.1 and 3.2 form a complete theoretical loop: Proposition 3.1 defines
the necessary properties of span A,) for stable-plastic continual learning. Proposition
proves

---

Published as a conference paper at ICLR 2026
Table Comparison different methods on MTIL for each classification task terms of Transfer. Averuge, and Last scores (%). The best results are in bold.
wo Extra Data Aireraft
Arch. Kept
Caltec1011
CIFAR100
100
EuroSAT
OxfordPet
Flowers
MNIST
Sun397
DTD
Food
Cars
Method
Avg.
Zero-shot
24.8
88.4
68.2
44.6
54.9
71.0
88.5
59.4
89.0
64.7
65.4
Transfer
LwF &i& Hoiem 2017
X
74.5
56.9
39.1
51.1
52.6
72.8
60.6
75.1
30.3
55.9
56.9
iCaRL (Rebulii
2017
X
56,6
44.6
32.7
39.3
46.6
68.0
46.0
77.4
31.9
60.5
50.4
LwF-VR
Dang
X
77.1
61.0
40.5
45.3
54.4
74.6
47.9
76.7
36.3
58.6
57.2
WiSE-FT
Wortsman et
2022
73.5
55.6
35.6
41.5
47.0
68.3
53.9
69.3
26.8
52.3
ZSCL
Zhing et al 2023
86.0
67.4
45.4
50.4
69.1
87.6
61.8
86.8
60.1
68.1
O-LoRA
Wang al.
203
V
80.8
68.0
44.5
49.8
67.5
86.7
59.3
88.7
56.1
63.6
66.5
InfLoRA
84.3
67.4
44.3
50.6
68.2
87.1
62.7
88.7
57.8
62.8
67.4
SD-LoRA
V
82.3
67.5
44.4
51.0
67.9
87.2
61.1
88.4
58.2
63.4
67.1
KeepLoRA
84.6
68.7
45.9
54.3
70.1
87.7
90.3
59.5
64.1
L2P Wang
X
V
65.6
50.9
30.4
41.4
49.3
71.8
36.3
77.5
55.3
53.4
53.2
DualPrompt
Wang
20226
V 56.7
51.4
28.7
33.7
45.6
70.9
59.5
77.7
49.5
52.4
S S-Prompts
Wang otall
2022b
X V 67.3
49.4
26.7
39.7
47.1
70,2
34.3
78.9
56.7
52.2
52.2
DIKI Tang
2024
X V 92.9
69.1
43.2
43.9
65.4
85.3
56.0
88.4
64.0
65.6
67.4
MoB
Adapters
X 87.9
68.2
44.4
49.9
70.7
88.7
59.7
89.1
64.5
65.5
68.9
IAP
V
93.0
68.7
44.0
47.0
70.4
85.9
63.5
89.7
66.2
63.3
69.2
Keepl.oRAA
85.9
69.9
44.6
53.7
70.9
88.9
65.4
90.8
63.0
66.1
69.9
Average
LwF Hoiem
36.3
86.9
72.0
59.0
73.7
60.0
73.6
74.8
80.0
37.3
58.1
64.7
iCaRL RRbulli et
2017
35.5
89.2
72.2
60.6
68.8
70.0
78.2
62.3
81.8
41.2
62.5
65.7
LwF-V VR
Ding el
2022
29.6
87.7
74.4
59.5
72.4
63.6
77.0
66.7
81.2
43.7
60.7
65.1
WiSE-FT Warlsman
2022
X
26.7
86.5
64.3
57.1
65.7
58.7
71.1
70.5
75.8
36.9
54.6
60.7
ZSCL Zheng
MZ2
X
45.1
92.0
80.1
64.3
79.5
81.6
89.6
75.2
88.9
64.7
68.0
75.4
O-LoRA
Wang
V
39.8
93.2
78.3
61.7
78.9
76.3
88.5
73.9
90.1
60.2
65.2
73.3
InfLoRA
Liong &
2024
53.6
95.6
82.8
65.0
80.9
79.6
89.1
76.1
90.2
62.3
SD-LoRA
Wi
2025o
V 36.7
92.2
80.2
55.9
77.5
73.2
89.2
74.9
89.8
62.5
65.0
72.5
KeepLoRA
V
55.6
95.7
83.2
65.6
82.2
82.0
89.5
77.4
91.5
63.9
65.8
77.5
L2P Wang
2022c
X
V
38.0
85.2
78.2
61.3
72.9
74.9
79.7
59.1
82.0
59.7
55.4
67.9
DualPromph Wane
20226
X
V
37.8
84.3
78.6
60.1
71.1
73.2
79.1
73.9
82.3
55.1
52.8
68.0
S- S-Prompts
Wang
2002
X
V
37.5
92.5
77.5
58.2
76.4
74.1
78.8
57.9
83.0
60.8
54.4
68.3
DIKI Tang
V
45.4
95.7
83.0
65.0
78.2
82.5
87.1
71.7
90.0
67.2
66.6
75.7
MoR-Adapters
Yo at 2024
50.2
91.9
83.1
69.4
78.9
84.0
89.1
73.7
89.3
67.7
66.9
76.7
IAP
Fu ct 2025
V
45.9
95.8
83.3
66.5
79.5
84.8
87.5
76.6
91.0
69.2
64.5
76.8
KoepLoRA+
x
V
58.4
96.5
84.4
67.8
82.1
84.5
90.7
77.8
91.9
67.5
67.6
79.0
LwF
Li & Hoiem
2017
x 26.3
87.5
71.9
66.6
79.9
66.9
83.8
99.6
92.1
66.1
80.4
74.6
iCaRL Rebuthh
2017
X
35.8
93.0
77.0
70.2
83.3
88.5
90.4
86.7
93.2
81.2
81.9
80.1
LwF-VR Ding
2022
X
20.5
89.8
72.3
67.6
85.5
73.8
85.7
99.6
93.1
73.3
80.9
76.6
WiSE-FT
27.2
90.8
68.0
68.9
86.9
74.0
87.6
99.6
92.6
77.8
81.3
77.7
ZSCL Zheng et sl.
2021
40.6
92.2
81.3
70.5
94.8
90.5
91.9
98.7
93.9
85.3
O-LoRA
Wang et
2023g
V
31.4
91.8
75.7
61.1
89.0
76.0
88.9
99.1
92.3
74.8
81.3
78.3
InfLoRA
Liung &
2024
V
51.1
96.5
85.1
70.7
98.1
87.7
91.3
99.4
94.2
82.0
81.4
85.2
SD-LORA
Waa
V
31.1
92.3
79.8
57.4
88.7
76.1
90.6
99.0
92.9
81.3
81.6
79.2
Keepl.oRA
V
53.2
96.8
85.7
71.4
98.1
90.8
91.4
99.6
94.5
83.1
82.0
86.1
1.2P
Wing
X
V
38.0
87.1
84.2
72.9
86.0
96.1
89.2
99.0
94.1
79.6
76.0
82.0
Dual Prompt
Wine
2022h
V
37.8
87.1
84.6
71.8
89.2
96.3
89.1
99.1
94.5
79.9
76.5
82.3
S-Prompts
Wane cl.l. 20276
V
37.5
95.1
83.7
70.2
97.5
96.5
89.0
99.1
94.0
79.5
DIKI
Tang
20ZA
X
V
45.4
95.9
86.0
73.0
97.8
96.8
89.3
99.3
94.4
81.8
76.4
85.1
MoE- Ade--ees
2024
X
X
49.8
92.2
86.1
78.1
95.7
94.3
89.5
98.1
89.9
81.6
80.0
85.0
IAP
X
V
46.8
96.1
86.7
75.2
98.1
97.0
89.6
99.4
94.7
82.8
76.7
85.7
LocpLoRA+
V 57.3
97.6
87.2
76.5
98.4
95.7
92.6
99.5
94.7
87.2
83.2
88.2

---

Published as a conference paper at ICLR 2026
that the initialization technique of A, in KeepLoRA satisfies these properties, which ensures that span( AA,) orthogonal to the principal subspace of the model parameters W and dominant feature directions of each learned task - to maintain stability, while being adaptive to the current task gradient to improve plasticity.
EXPERIMENTS
We conduct experiments on various benchmarks to validate the effectiveness of KeepLoRA in bal- ancing three core objectives of continual learning: forward stability, backward stability, and plas- ticity. (i) To quantify forward forgetting, we calculate the average accuracy on tasks 1. + 1....,
after training task i, which is defined the Transfer metric, presented in Tab. 2 and 4 Fig. further analyzes how KeepLoRA maintains the transferabilityy (ii) The Last metric, shown in Tab.
3 and assesses model performance after continual training has completed, capturing both plas- licity and backward stability. (iii) To further analyze plasticity, Fig. 2 compares our method with an unconstrained LoRA, demonstrating that KeepLoRA preserves stability with minimal sacrifice to its adaptive capability. The Average metric represents the mean accuracy across all learned tasks, offering holistic measure of the balance between stability and plasticity.
Table 3: Comparison of different continual learning methods on MLLM-DCL benchmark for VQA
tasks in terms of Transfer, Average, and Last scores (%), The best results are in bold.
Sensing
Medical
Driving
Science
Finance
Method
Avg.
Zero-shot
32.29
28.28
15.59
35.55
62.56
Transfer
LoRA-FT Ho ot
2022
28.10
17.44
34.03
50.19
O-LoRA Wang
20236
28.37
18.37
33.72
52.53
33.25
CL-MoE
28.25
19.38
34.08
48.56
32.57
SEFE
28.10
19.63
33.85
52.36
33.49
KeepLoRA
28.49
16.63
34.13
55.61
33.71
Average
LoRA-FT
2003
73.34
44.94
31.38
38.79
57.84
49.26
O-LoRA Wune
75.04
45.71
32.62
38.54
59.64
50.31
CL-MoE
Huni et al.
2025
74.19
45.60
32.08
38.88
56.68
49.49
SEFE
Chen et al
2025
77.71
47.69
35.35
38.99
59.57
51.86
KecpLoRA
79.55
50.80
37.53
40.70
62.35
54.19
Last
LORA-FT Edn et al.
2023
69.34
44.30
29.10
41.44
88.43
54.52
O-LoRA Wang et
2023
72.30
46.89
31.59
41.50
88.06
56.07
CL-MoE
Tlun al
2025
71.83
47.36
29.49
41.48
89.16
55,86
SEFE
77.05
50.86
40.27
42.98
88,40
59.91
KeepLoRA
78.76
54.34
50.19
49.48
89.30
64.41
4.1
MAIN RESULTS
We evaluate our method the dual-encoder model CLIP (Radford al. 2021 and encoder-decoder model LLaVA (Liu et al. 2023). For CLIP, the experiments are conducted on the MTIL (Zheng al. 2023) benchmark. presenting results for alphabetical (Tab. ( and random (Tab. 6 task or- ders in two settings, with and without architecture extension. KeepLoRA+ is structure extension variant with a prototype vector for a class name help classification, detailed in Appendix B.3 For LLaVA, the experiments (Tab. and 4 are conducted on MLLM-DCL. (Guo et al.) 2025b) and UCIT (Guo et al.) 2025b) benchmarks, including various instruction formats such as image captioning, visual question-ansee answer, and multiple-choice questions. Detailed information iment settings and benchmarks is presented in Appendices B.4 and B.I separately. KeepLoRA and KeepLoRA+ achieve state-of-the of-the-art performance the Transfer, Average, and Last metrics in each these settings. This demonstrates that our approach consistently addresses the challenges of forward stability, backward stability, and plasticity in continual learning.
exper-
8

---

Published as a conference paper at ICLR 2026
Table Comparison of different continual learning methods UCIT benchmark for VQA tasks in terms of Transfer, Average, and Last scores (%) The best results are in bold.
ImgNet-R
ArxiQQA
VizWiz
IconQA
CLEVR
Flickr30k
Method
Avg.
Zero-shot
16.27
53.73
38.39
19.20
20.63
41.88
Transfer
LoRA-FT Hi
20122
52.63
18.30
6.02
16.97
40.29
26.84
O-LoRA Wang 0023:
52.87
19.57
4.42
16.85
41.04
26.95
CL-MoE
2025
52.00
19.32
7.37
17.81
41.28
27.56
SEFE Chen
53.33
18.68
7.48
17.03
40.90
27.48
KeepLoRA
52.83
20.39
9.18
18.12
41.50
28.40
Average
LoRA-FT HE 2022
75.98
77.78
41.56
38.83
34,56
43.25
51.99
O-LoRA Wang 3023
82.43
80.06
41.73
35.87
33.94
43.74
52.96
CL-MoE
80.16
77.10
40.43
30.33
33.10
43.95
50.85
SEFE Chen al
2023
85.49
78.55
42.92
40.33
34.80
43.64
54.29
KeepLoKA
86.50
83.63
42.66
40.08
35.24
44.11
55.37
Last
LORA-FT Ifu el
2022
58.60
76.73
45.72
67.43
61.57
58.03
61.35
O-LoRA Wang et al.
20036
74.17
80.93
45.30
62.87
63.83
57.24
64.06
CL-MMoE
Humi 2025
67.17
75.77
44.38
52.63
54.40
57.28
58.61
SEFE Chen
80.23
79.13
47.11
69.40
65.70
57.33
66.48
KeepLoRA
82.43
86.70
46.54
67.80
66.40
57.18
67.84
4.2 ANALYSIS OF MODEL PLASTICITY
Plasticity assesses the ability to effec- tively acquire new knowledge follow- ing sequence of continual learning tasks. We evaluate two performance metries for each task: (i) the accuracy achieved by training the task in iso- lation, serving as an upper bound, and (ii) the accuracy measured immediately after the task is learned within the con- tinual sequence. Our analysis in Fig. compares KeepLoRA with a standard LoRA baseline. In the isolation-task setting, KeepLoRA performs compara- bly to LoRA, as pradient-informer ini- tialization of the frozen down-propf projection matrix effectively captures the essen- tial learning direction, maintaining high learning capacity. Furthermore, when switching the continual learning see- nario, KeepLoRA exhibits consistently smaller performance drop to the residual subspace and avoiding interference with previously learned knowledge, our method enhances the model's plasticity for subsequent tasks.
Decresse LIFAA
ChepladA
Caleentss
Dr
frowrr
Card
S18207
D712
Rovers
(a) Param. 0.49
(b) # Param. 0.98
Figure 2: Comparison of plasticity between KeepLoRA and the LoRA baseline under the same learnable pa- rameter budgets: Fig. 2a 0.49 million parameters and Fig. 2b(0.98 million parameters. Each bar represents the performance drop for task. measured the difference between accuracy from isolated training and accuracy after sequential learning and immediate testing.
tasks compared to LoRA. This suggests that, by confining updates
4.3 ANALYSIS OF MODEL STABILITY
We analyze stability by visualizing the interference of the LoRA module between multiple tasks in Fig. 3 In these heatmaps, the off-diagonal cells represent inter-task interference, while the vertical bar the left indicates the overall impact the backbone. The standard LoRA (Fig. (aa and LoRA with a frozen matrix A (Fig. 3bp both exhibit significant interference. The bright patterns in their heatmaps and vertical bars show that training current task heavily interferes with the

---

Published as a conference paper at ICLR 2026
(a) LoRA
(b) LoRA frozen A
(c) LoRA frozen grad A
(d) KeepLoRA
Figure 3: Visualization the average L2 norm of the output magnitude from the learned LoRA across multiple tasks. Each heatmap cell at row and column displays the normalized average L2 norm of the LoRA's output when the model, trained up to task i, is tested on task j's data. The vertical bar the left of each heatmap indicates the mean output norm across all test tasks after each training stage, with darker colors signifying a lower norm and thus a reduced impact on the stability.
representations of other tasks, leading to poor stability. Although gradient-infonee initialization (Fig. [c) reduces off-diagonal interference, the overall impact the backbone remains high, as shown by its bright vertical bar. In contrast, KeepLoRA (Fig. 3d) shows a desirable pattern: a bright diagonal with dark off-diagonal cells. This indicates that the updates of the model focus the current task, causing minimal interference with others. The dark vertical bar further confirms that the overall impact on the backbone consistently low. By minimizing interference with previously learned tasks, KeepLoRA ensures backward stability. Furthermore, its minimal interference with unseen tasks, indicated by the low norm, is critical to preserve forward stability.
4.4 ABLATION STUDY
To analyze the contribution of each component, we conduct ablation study starting from stan- dard LoRA baseline. As shown in Tab. 5) our modifications progressively improve performance. Freezing the down-projection matrix A even with random initialization, enhances stability in the continual learning setting by mitigating destructive interference with the backbone weights. Sub- sequently, employing (i) gradient-informer initialization further improves plasticity, leading 5.9% increase on the Last metric and indicating a more effective adaptation. After constraining the updates be orthogonal (ii) the principal subspace W, and (iii) the dominant feature directions M, gains of 4.0% on Transfer, 7.3% on Average, and 10.7% 7% on Last, demonstrating the critical role of subspace projection balaneing stability and plasticity.
Table 5: Ablation Study of KeepLoRA on MTIL.
Training Strategy
Transfer
A
Average
Last
LoRA (rank 8, # param. 0.98 M)
58.3
0.0
61.5
0.0
59.4
0.0
LoRA frozen A (rank 16, param. 0.98
63.9
+5.6
68.2
+6.7
69.5
+10.1
(i) Replace Eq.
by G; = G;
65.0
+6.7
70.2
+8.7
75.4
+16.0
(ii) Replace Eq.
by G, G, W, WTW.G. W.W.G. G.
+7.6
71.5
+10.0
+17.1
(ii) Replace Eg
by G. G. M, IM,
68.1
+9.8
77.2
+15.7
86.1
+26.7
KeepLoRA (Eq.))
69.0
+10.7
77.5
+16.0
86.1
+26.7
CONCLUSION
In this work, we reveal that the principal subspace of parameters encodes general knowledge and the residual subspace captures domain-specific adaptations. Leveraging this insight, we propose KeepLoRA, parameter-efffcient fine-tuning method that can effectively achieve balance among the competing objectives of plasticity, backward stability, and forward stability. Our theoretical anal- ysis confirms that constraining parameter updates to the residual subspace an optimal strategy, maximizing plasticity for the current task while maintaining orthogonality to subspaces encoding general and previously learned knowledge. Empirically, KeepLoRA matches the learning capacity of unconstrained LoRA on isolated tasks while significantly mitigating forgetting in continual learn- ing scenarios. As a simple and effective method, KeepLoRA provides a principled approach for continual learning that is applicable larger models and more diverse tasks.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported by the National Science Foundation of China (62576092, 62225602). and the Big Data Computing Center of Southeast University. We would like to thank mnonymous reviewers for their constructive suggestions.
ETHICS STATEMENT
All authors have read and adhered to the ICLR Code of Ethics. This paper presents an algorithmic contribution, KeepLoRA, aimed at advancing the field of continual learning. Our empirical valida- tion conducted exclusively on publicly available and widely used academic benchmarks, such as CIFARION and Caltech101, which do not contain personally identifiable sensitive information. While we acknowledge that advancements in machine learning have broad societal consequences, work does not introduce foreseeable negative applications exacerbate biases beyond those potentially present in the general pre-trained models.
REPRODUCIBILITY STATEMENT
To ensure full reproducibility, we have open-sourced the code for our method. Also, the method is detailed in Sec. B] with the core framework summarized in Algorithm We specify all hyperparam- eters used for our method and the baselines, including learning rates, batch size, and the preservation ratios cu and cy in Appendix B
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.
discriminative compo- nents with random forests. In European conference of computer vision (ECCV), 2014.
Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts in large language models. IEEE Transactions Knowledge and Data Engi- neering, 2025.
Shuaichen Chang, David Palzer, Jialin Li, Eric Losler-Lussier, and Ningchuan Xiao. Mapqa: A dataset for question answering choropleth maps. In arXiv preprint arXiv:2211.08545. 2022.
Jinpeng Chen, Runmin Cong, Yuzhi Zhao, Hongzheng Yang, Guangneng Hu, Horace Ho Shing Ip, and Sam Kwong. Sefe: Superficial and essential forgetting eliminator for multimodal contin- ual instruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2025.
Quan Cheng, Yuanyu Wan, Lingyu Wu, Chenping Hou, and Lijun Zhang. Continuous subspace optimization for continual learning. arXiv preprint arXiv:2505./18// 2025.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2014.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Oni Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.0626/ 2025.
Gregoire Deletang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christo- pher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. Language modeling is compression. The Tweifth International Conference on Learning Rep- resentations, 2024.
11

---

Published as a conference paper at ICLR 2026
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A scage hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 2012.
Yuxuan Ding, Lingqiao Liu, Chunna Tian, Jingyuan Yang, and Haoxuan Ding. Don't stop learning: Towards continual learning for the clip model. arXiv preprint arXiv:2207.09248, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations (ICLR), 2021.
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, et al. Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1932-1945, 2024.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- frika, 1936.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In Conference computer vision and pattern recognition workshop. 2004.
Giorgio Franceschelli, Claudia Cevenini, and Mirco Musolesi. Training foundation models as data compression: On information, model weights and copyright law. arXiv preprint arXiv:2407. 13493, 2024.
Hao Fu, Hanbin Zhao, Jiahua Dong, Chao Zhang, and Hui Qian. lap: Improving continual learn- ing of vasion-language models via instance-aware prompting. arXiv preprint arX1v:2503.20612, 2025.
H. Guo, F. Zeng. Z. Xiang, et al. Hide-llava: Hierarchical decoupling for continual instruction tuning of multimodal large language model. In arXiv preprint 2.NI::2503.1294 2025a.
Haiyang Guo, Fei Zhu, Hongbo Zhao, Fanhu Zeng, Wenzhuo Liu, Shijie Ma, Da-Han Wang, and Xu- Yao Zhang. Mcitlib: Multimodal continual instruction tuning library and benchmark. arXiv preprint ar 2508 Xfv:22508.07007, 2025b.
Ziyu Guo, Ray Zhang, Hao Chen, Jialin Gao, Dongzhi Jiang, Jiaze Wang, and Pheng-Am Ann Heng. Sciverse: Unveiling the knowledge comprehension and visual reasoning of Imms on multi-modal scientific problems. In Findings of the Association for Computational Linguistics: ACL 2025, pp- 19783-19704, 2025c.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 3608-3617, 2018.
Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. In arXiv preprint arXiv:2003. 10286, 2020.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of selected topics in applied earth observations and remole sensing, 2019.
Dan Hendrycks. Steven Basart, Norman Mu, al. The many faces of robustness: A critical analysis of cuittf-distributionn generalization. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV) pp. 8340-8349, 2021.
12

---

Published as a conference paper at ICLR 2026
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):, 2022.
Tianyu Huai, Jie Zhou, Xingjiao Wu, Qin Chen, Qingchun Bai, Ze Zhou, and Liang He. Cl-moe: Enhancing multimodal large language model with dual momentum mixtuer-of-experts for contin- ual visual question answering. In Proceedings of the IEEE/CVF Conference Computer Viston and Puttern Recognition, pp. 19608-19617, 2025.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In European Conference on Computer Vision (ECCV), pp. 235 251- Springer, 2016.
Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4999-5007, 2017.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Lei Li, Yuqi Wang, Runxin Xu, et al. Multimodal arxiv: A dataset for improving scientific compre- hension of large vision-language models. In arXiv preprint arXiv:2403.00231, 2024.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40( (0(11)):33559944 2017.
Yan-Shuo Liang and Wu-Jun Li. Adaptive plasticity improvement for continual learning. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7816- 7825, 2023.
Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference- low-rank adaptation for continual learn- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23638-23647) 2024.
Adam Dahlgren Lindström and Savitha Sam Abraham. Clevr-math: A dataset for compositional language, visual and mathematical reasoning. In arXiv preprint arX5v:2208.05355, 2022.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
Sylvain Lobry, Diego Marcos, Jesse Murray, et al. Rsvqa: Visual question answering for remote sensing data. In IEEE Transactions on Geoscience and Remote Sensing, volume 58, pp. 8555- 8566, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019.
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2021.
Mao-Lin Luo, Zi-Hao Zhou, Tong Wei, and Min-Ling Zhang. Lada: Scalable label-specific clip adapter for continual learning. In Forty-secome second International Conference on Machine Learning, 2025.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- Fine-grained visual classification of aircraft. arXiv preprint arXiv: 1306.5/5, 2013.
13

---

Published as a conference paper at ICLR 2026
Jishnu Mukhoti, Yarin Gal, Philip Torr, and Puneet K. Dokania. Fine-tuning cripple your founda- tion model: preserving features may be the solution. Transactions on Machine Learning Research, 2024. ISSN 2835-8856, Featured Certification.
Nikhil Shivakumar Nayak, Krishnateja Killamsetty. Ligong Han, Abhishek Bhandwaldar, Pra- teek Chanda, Kai Xu, Hao Wang. Aldo Pareja, Oleg Silkin, Mustafa Eyceoz, et al. Sculpt- ing subspaces: Constrained full fine-tuning in Ilms for continual learning. arXiv preprint arXiv:2504.07007, 2025.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference computer vision, graphics & image processing, 2008.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition (CVPR), 2012.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image- to-sentence models. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2641-2649, 2015.
Jingyang Qiao, zhizhong zhang. Xin Tan, Chengwei Chen, Yanyun Qu, Yong Peng, and Yuan Xie. Prompt gradient projection for continual learning. The Twelfth International Conference on Learning Representations, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PmLR, 2021.
Sylvestre-Alviss Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances neural information processing systems, 32, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In International Conference on Learning Representations, 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 2556-2565, 2018
Pratyusha Sharma, Jordan T. Ash, and Dipendra Misra. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. In The Twelfth International Conference on Learning Representations, 2024.
Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph vi- sual question answering. In European Conference on Computer Vision (ECCV), pp. 256-274. Springer, 2024.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante- Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual de- composed antention-basse prompting for rehearsal frrer continual learning. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 11909-11999 2023.
14

---

Published as a conference paper at ICLR 2026
Longxiang Tang, Zhuotao Tian, Kai Li, Chunming He, Hantao Zhou, Hengshuang Zhao, Xiu Li, and Jiaya Jia. Mind the interference: Retaining pre-trained knowledge in parameter efficient continual learning of vision-language models. In European conference on computer vision, pp. 346-365. Springer, 2024.
Liyuan Wang. Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE transactions on pattern analysis and machine intelligence, 46(8):5362-3383, 2024.
Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuan-Jing Huang. Orthogonal subspace learning for language model continual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10658-10671, 2023a.
Yabin Wang. Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transformers: An occam's 'ss razor for domain incremental learning. Advances in Neural Information Processing Systems, 35:5682-5695, 2022a.
Ziao Wang, Yuhang Li, Junda Wu, Jachyeon Soon, and Xiaofeng Zhang. Finvis-gpt: A multimodal large language model for financial chart analysis. arXiv preprint arXiv:2308.014:: 2023b.
Zifeng Wang. Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European conference computer vision, pp. 631-648 Springer, 2022b.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 139-149, 2022c.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 7959-7971, 2022.
Bin Wu, Wuxuan Shi, Jinqiao Wang, and Mang Ye. Synthetic data is an elegant gift for continual vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Confer- ence, pp. 2813-2823, 2025a.
Yichen Wu, Hongming Piao, Long-Kai Huang, Renzhen Wang, Wanhua Li, Hanspeter Pfister, Deyu Meng, Kede Ma, and Ying Wei. Sd-lora: Scalable decoupled low-rank adaptation for class incre- mental learning. arXiv preprint arXiv:2501.13/1 2025b.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In IEEE computer society conference on com- puter vision and pattern recognition, 2010.
Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vasion-language models via mixture-of-exxeer adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23219-23230 2024.
Ted Zadouri, Ahmet Üstüi, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, and Sara Hooker. Push- ing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. arXiv preprint arXiv:2309.05444 2023.
Hongbo Zhao, Fei Zhu, Rundong Wang, Gaofeng Meng, and Zhaoxiang Zhang. Milm-el: Continual learning for multimodal large language models. In arXiv preprint urXiv:2506.05455. 2025.
Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. In Proceedings of the IEFE/CVF international conference on computer vision, pp. 191 19125-19136, 2023.
15

---

Published as a conference paper at ICLR 2026
Pai Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Class- incremental learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12)9881199883,2000 2024.
16

---

Published as a conference paper at ICLR 2026
A PROOFS OF PROPOSITIONS
A. PROOFS OF PROPOSITION
Proof. Suppose for loss function L for task T and a linear layer with y xW, where y is the output of the layer and æ is the input. We can compute the gradient of B, directly as follows:
Oc ow
dc a G.
0B. O
OW
In a gradient descent iteration, the change of B, is represented by negative gradient: AB, no A,G,, G,. Therefore, when A, frozen to only update B, in each iteration, we can obtain the variation of W in one iteration to complete the proof:
na² y2
AW A,AB,
A.A, G..
(10)
A.2 PROOFS OF PROPOSITION 3.2
Proof. We proceed by transforming the constrained optimization problem, leveraging subspace properties, and applying the Eokart-Young-irsky Theorem (Eckart & Young 1936) to confirm the optimal solution.
Step 1: Equivalent Transformation of the Objective Function. For an orthonormal matrix As sat- isfying A, A, = I, the orthogonal projection operator PA. = A,A, satisfies the Pythagorean theorem for the Frobenius norm:
|| G.ll + |G G.I
Since ||G; 1| is a constant independent A,, minimizing the original objective |G,- PA. G equivalent to maximizing the projected norm PA G. C... The optimization problem thus be rewritten as:
A.A, G
max A A,=1
(1I)
s.t W P A,
Step 2: Substitute G. and Simplify Using Constraints. Recall from Eq. 5 that the projected gradient G, is defined as:
G. = G. W,W,W G. M, M,-,M,,,C G..
Rearranging gives G. = GG+ W, G. + MM G.. Substitute this into the objective:
|A:AT (G. C,W,,,,,T W. G. M.- MM-MM, 1|.
For any feasible A., we use W. A. = M,,,A, Ar 0 to simplify: A WW, G.) (W, T T(W G.) = OW G.) = 0. Similarly, G.) =.
Thus, A,A, (WA,WWWW.. WT + M,IMM,
0, and the objective reduces to maximizing || G.l. The optimization problem simplifies to:
A.A G.ll?. A A,
(12)
S.t A, - M,,,M,L. A,=
Step 3: Optimal A, via Eckart- Yoorr-Yuun---ee Miisky Theorem. The Eckart- Eckart-Young Goung-Mirsky Theorem (Eckart & Young 1936) states that for any matrix X mxx and integer k S min(m, n). the d-dimensional subspace that maximizes |P where P is the orthogonal projection onto the subspace, is spanned by the top-r left singular vectors of X.
17

---

Published as a conference paper at ICLR 2026
Here, X X=G, and seek an r-dimensional subspace spanned by A, to maximize | Gilla. By the theorem, the optimal A, consists of the top-r left singular vectors of G.
Step 4: Verify Feasibility of the Optimal Ar. We confirm the optimal A, satisfies the constraints W, A, 0 and M,- A
By the definition of G. in Eq.
we have:
w
G. WT
O, M, = 0.
(13)
Substituting SVD of & = USV in Eq. full rank (singular values non-negative, and V is orthonormal), W U must be the zero matrix.
13
W G. W UUST = 0. Since SVT is column-
Thus, W. U = 0, hence p A. W U.r = 0. The same logic applies to M M, 0, hence A, 0.
M, M, O implies M,
Thus, the optimal solution to Eq. is exactly the top-r left singular vectors of G. which matches KeepLoRA A, initialization. The proof is completed.
B EXPERIMENT DETAILS
B.1
BENCHMARK
MTIL benchmark (heng al. (023) consists of image classification datasets:
Aircraft (Maji
et al.
2013).
Caltech101 (Fei-Fei et al. 2004), Cifarl00 100 (Krizhevsky et al. 2009).
DTD (Cimpoi
al.
2014).
EuroSAT (Helber et 2019) Flowers (Nilsback Zisserman, 2008), Food (Bossard
et al.
2014), MNIST (eng 2012). OxfordPet (Parkhi et al., (012). StanfordCars (Krause et al.
2013
and SUN397 (Xiao
2010). Each dataset treated as a task.
2025) consists of multiple downstream VQA datasets: RSVQA (Lobry et al. 2020), Vat (He et 2020). Drivel.M (Sima et 2024). Fin (Wang et al. 2023b), A12D (Kembhavi et al.] (0166), Sciverse (Guo et 2022cc,, MapQA (Chang et al. 2022). and TQA (Kembhavi et al. 20. covers specialized areas: Remote Sensing, Medical, Driving, Finance, and Science. Each area is treated task.
MLLM-DCL benchmark (hao et al.
UCTT benchmark (Guo et al. 2025a) consists 6 VQA datasets: ArxivQA (Li et al., 2024). CLEVR-M Math (Lindström & Abraham, 2022), IconQA (Lu et al. (02I). ImageNet-R (Hendrycks) ct al., 2021). VizWiz-Caption (Gurari et al. 2018), and Flickr30k (Plummer et al. 2015). Each dataset is treated as task.
B.2 EVALUATION METRICS
We define the Transfer, Average, and Last metries to evaluate model performance under continual learning scenarios. Let at represent the accuracy of the model on task after training on task with total of tasks. The Transfer, Average, and Last metrics for task are computed as follows:
Transfer,
t= 2.3...,,
(14)
iml
Average,
1
Σ
n
(1)
== 12,
(15)
i=1
Last, = a
t 1,2...., n.
(16)
The Transfer metric evaluates forward stability by measuring the performance of unseen tasks throughout (i + 1, + 2,..., after training on the task i. The Last metric measures the final performance each task after completing all training steps, quantifying both plasticity and back- ward stability. The Average metric represents the mean accuracy across all time steps, offering a holistic measure of stability and plasticity.
18

---

Published as a conference paper at ICLR 2026
B.3 IMPLEMENTATION DETAILS OF KEEPLORA+
We extend KeepLoRA with a structure variant, termed KeepLoRA+, which incorporates prototype vector for each class ized using the mean feature extracted by the vision encoder from the corresponding class samples. During the training stage, we jointy optimize the prototype vectors alongside the KeepLoRA pa- rameters. In the inference stage, the logits derived from the similarity of the prototype vectors are averaged with the logits calculated from the text-side contrast.
improve classification performance. Each prototype vector is initial-
B.4 ADDITIONAL IMPLEMENTATION DETAILS
CLIP Experiments. We adopt the CLIP (Radford et al. 2021) model with ViT-B/16 (Dosovitskiy
al. 202T image encoder. The training process carried out using the A (Loshchilov & Hutter 2019) optimizer, with learning rate of 10-3 and batch size of 64 across all tasks with
more than 10 epochs. For the primary experiments, we set the hyperparameters as Eurvision) = 0.85 and (wetex)) 0.2 in vision encoder and text encoder separately and e 0.99. KeepLoRA+ is structure extension variant with an extension prototype vector for a classname to help classification. conducted on single NVIDIA 4090 GPU. For the reproduced methods, we performed careful hyperparameter tuning. For O-LoRA (Wang et al. 2023a) the learning rate x 10-4 with a regularization coefficient of 0.1. For InfLoRA (Liang Li 2024). the learning rate is 10 with Ey = 0.99. The learning rate for SD-LoRA (Wu et al. 2025b is set to x 10-3
All experiments of KeepLoRA
2023) model for multimodal contin- ual instruction tuning experiments. The training is conducted on 4x NVIDIA H100 GPUs using the AdamW W optimizer. For the MLLM-DCL benchmark, we the learning for no more than 3 epochs task. For the UCIT benchmark, the learning rate is set to 2 x 10 for all tasks except Flickr30k, which uses 1 x 10 and train epoch for each task. The hyperparameters for subspace constraints are configured as 0.6 and E = 0.99.
LLaVA VA Experiments. We adopt the LLaVA-1.5-70 el.
to 2 10-5 and train
C SUPPLEMENTARY EXPERIMENTS
C.I COMPARISON ON MTIL WITH ORDER II.
We compare different methods MTIL in random order: StanfordCars, Food, MNIST, OxfordPet, Flowers, SUN397, Aircraft, 101, DTD, EuroSAT and CIFAR100. As shown in Tab. 6 KeepLoRA consistently outperforms previous methods across all metrics.
C.2 HYPERPARAMETER ANALYSIS
36.4
602
66.2
690
Thandfer AccurSO)
552
62.3
33.0
454
354
600
652
25.0
0
apsion, WtV wivslon stonl
eostion, selsect
(a)
(b)
Figure 4: Effects of hyperparameters e and (w((ex)) on Transfer and Last, respectively.
We examine the effects of hyperparameters (vviiiio)) and Ew(tex)) on Transfer and Last. For the image encoder, Last fluctuates slightly, with minor decline when ew(visio) is larger. Between 0.75 and 0.85, Transfer shows clear increase. For the text encoder, which uses only class names and thus has much less training data than images, the coefficient (ai(te)) exhibits low performance sensitivity. Datasets with image-text pairs or VQA tasks, which include substantial text data, warrant further study in this regard.
19

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of different continual learning methods MTIL for each task with order-11 in terms of Transfer, Average, and Last scores (%). The best results are highlighted with bold style.
Inf. Efficiency w/o Extra Data Cars
MNIST
OxfordPet
Flowers
Sun397
Aircraft
Caltech101
EaroSAT
CIFAR100
Food
DTD
Method
Avg.
Zero-shot
64.7
88.5
59.4
89.0
71.0
65.4
24.8
88.4
44.6
54.9
68.2
Transfer
LwF Hoiem
87.8
58.5
71.9
46.6
57.3
12.8
81.4
34.5
34.5
46.8
53.2
iCaRL
RRbuufi et al
2017
86.1
51.8
67.6
50.4
57.9
72.3
31.2
32.7
48.1
50,9
LwF-M VR
Ding
2022
X
88.2
57.0
71.4
50.0
58.0
13.0
82.0
34.4
29.3
47.6
53.1
WISE-FI FT
Warlsman
2022
X
87.2
57.6
67.0
45.0
54.0
12.9
78.6
35.5
28.4
44.3
51.1
7SCL.
M22
X
88.3
57.5
84.7
68.1
64.8
21.1
88.2
45.3
55.2
68.2
64.1
O-LoRA
Wang
87.8
56.7
90.1
71.4
64.0
20.7
87.4
43.9
46.3
65.9
63.4
Infl.oRA
Line &
2024
88.2
56.7
90.2
71.3
65.0
22.2
88.2
43.8
47.3
SD-LoRA
Wa al. 20250
V 88.0
56.4
90.5
71.0
64.6
22.0
87.8
43.7
47.1
66.4
63.7
KeepLoRA
V
88.7
S7.7
91.2
72.1
65.8
23.4
88.8
45.4
48.5
68.2
65.0
L2P
Wang et
XIZ2c
X V
70.6
30.7
78.3
42.8
38.3
17.4
75.3
27.4
23.1
20.7
42.5
Dual Prompt Wane
20226
X V 79.9
46.9
85.2
51.3
45.1
9.3
82.7
29.9
42.9
47.2
52.1
S- S-Prompts Wang
X V 59.8
46.2
67.7
47.5
43.8
13.5
76.8
31.4
22.6
43.5
45.3
DIKI Tanger
2024
V 85.8
59.8
89.1
71.8
62.6
24.3
93.3
42.7
46.8
67.8
64.4
MoE Adapters
Ya eu
2024
X 88.8
59.5
89.1
69.9
64.4
18.1
86.9
43.7
54.6
68.2
64.3
IAP Fu ct 2025
V 85.7
59.4
89_1
71.3
62.7
24.4
94.0
43.8
49.0
68.6
64.9
KeepLoRAA+
X V
89.1
58.1
90.7
72.4
65.4
24.0
88.9
44.0
52.7
70.2
65.5
Average
LwF Li Heiee 2017
X 49.0
77.0
92.1
85.9
66.5
67.2
20.9
84.7
44.6
45.5
50.5
62.2
iCaRL (Rbooti
5017
X
52.0
75.9
77.4
74.6
58.4
59.3
11.7
79.6
42.1
43.2
51.7
56.9
LwF-VR
2022
X
44.9
75.8
91.8
85.3
63.5
67.6
16.9
84.9
44.0
40.6
51.3
60.6
WiSE-FT Wortsman
52.6
79.3
91.9
83.9
63.4
65.2
23.3
83.7
45.4
40.0
48.2
61.5
ZSCL Zheng et al.
202
81.7
91.3
91.9
91.0
82.9
72.5
33.6
89.7
53.3
62.8
69.9
O-LoRA Wang et
20230
V
78.5
91.0
91.3
92.3
77.7
73.0
33.5
90.5
50.7
55.1
67.8
72.9
InfLoRA
2024
V
84.0
92.1
91.7
93.2
81.6
74.3
34.3
91.3
51.5
56.6
69.0
74.5
SD-LoRA
Wa
V
76.8
91.1
90.8
92.5
76.5
73.1
34.0
90.7
49.1
56.2
68.2
72.6
KecpLoRA
V
85.2
92.3
92.0
93.7
84.8
74.8
35.9
91.8
53.1
57.5
70.0
75.6
L2P
Wing
X
V
80.1
87.4
86.7
89.6
76.8
59.1
27.7
79.5
39.9
34,6
26.5
62.5
Dual DromPrompt Wang
20226
V 78.6
88.4
89.7
91.7
80.0
62.4
23.2
85.0
41.3
50.7
67.5
S-Prompts
Wang ctal.
2022a
V 79.2
86.5
89.5
87.0
78.2
61.5
25.5
83.6
41.9
36.3
47.2
65.1
DIKI Tang
3024
X V 81.9
88.9
92.1
92.8
87.7
70.3
34.3
94.2
51.5
56.1
69.5
74.5
MoF-Adapter
Ya
2024
X
X
84.9
89.9
89.3
91.4
86.2
72.2
33.4
89.4
53.3
61.4
69.9
74.7
IAP
V
82.5
89.2
92.3
93.2
88.0
70.4
34.3
94.4
52.4
57.9
70.2
75.1
KoepLoRA+
88.0
92.4
91.9
93.9
87.4
75.2
39.2
92.0
52.8
60.9
71.8
76.9
Last
LwF
Holem
x
34.6
69.6
99.3
88.7
61.1
72.5
32.5
88.1
65.6
90.9
87.9
71.9
iCaRL
46.0
81.5
91.3
82.8
66.5
72.2
16.3
91.6
68.1
83.2
87.8
71.6
LwF-VE
Ding et
2022
27.4
61.2
99.4
86.3
60.6
70.7
23.4
88.0
61.3
84.3
WiSE-FT
Wartsman et
2022
X
35.6
76.9
99.5
89.1
62.1
71.8
27.8
90.8
67.0
85.6
87.6
72.2
ZSCL Zhong al
272
X
78.2
91.1
97.6
92.5
87.4
78.2
45.0
92.3
72.7
96.2
86.3
83.4
O-LoRA
Wang
V
70.3
89.8
97.8
92.9
73.8
79.8
44.4
95.3
66.3
91.5
85.9
80.7
InfLoRA
V
82.4
92.0
99.3
93.9
85.4
81.2
46.1
96.5
70.0
97.6
87.2
84.7
SD-LoRA
Who ebettal a 20250
V
72.3
89.7
97.3
92.4
76.1
78.9
45.3
95.2
61.6
96.9
86.1
81.1
KeepLoRA
V
83.7
92.2
99.5
94.4
90.7
81.3
49.0
96.9
72.3
98.0
87.3
85.9
L.2
Wing otal.
2022
V
80.1
89.1
99.1
93.8
96.2
76.5
40.1
86.9
73.5
86.3
84.2
82.3
Dual Prompt Wang
2022h
X
V
78.6
89.3
99.2
94.1
96.5
76.8
39.8
89.0
71.6
90.7
84.9
82.8
S S-Prompts Wane 20234
X
V
79.2
89.1
99.1
94.3
95.8
76.3
39.9
95.5
70.1
97.6
84.4
83.8
DIKI Tang
V
81.9
89.2
99.4
94.3
96.8
76.7
46.3
95.9
74.8
98.3
86.6
85.5
MoE- AdEp-esss
Baetal 2024
84.1
88.5
94.0
91.8
94.1
77.8
50.4
93.3
77.1
87.7
84.1
IAP Fu 2025
V
82.5
88.6
99.4
94.9
97.7
76.9
46.1
96.1
74.7
98.0
86.6
85.9
KocpLoRA+
X
V
87.4
92.5
99.3
95.0
96.0
83.2
56.9
97.5
76.9
98.0
88.0
88.2
20

---

Published as a conference paper at ICLR 2026
C.3 WER-TRAINIGG-SAE SEENING-STEP RESULTS
We present the detailed per-training-step accuracies through all training steps in Tab. 7
8 10
These results demonstrate strong performance in terms of both learning plasticity and stability.
11 and 12
Table 7: Accuracy of KeepLoRA on the MTIL benchmark with order-I. Each row represents the per- formance on every dataset of the model trained after the corresponding task. Transfer Average
and Last metries shown.
11
CIFARI00
OxfordPet
Aircraft
EuroSAT
Flowers
MNIST
Sun397
DTD
Food
Cars
Transfer
84.6
68.7
45.9
54.3
70.1
87.7
64.8
90.3
59.5
64.1
69.0
Aircraft
59.0
84.6
68.4
45,4
52.2
71.9
89.0
63.8
91.1
60.6
63.6
Caltech101
58.1
97.0
69.1
45,4
50.8
71.1
88.7
61.8
91.1
60.1
64.8
CIFARI00
56.0
96.8
87.6
46.8
56.3
68.9
87.3
66.3
90.1
59.6
64.7
DTD
55.9
96.7
87.5
75.0
57.9
69.6
87.1
64.7
90.3
59.5
64.6
EuroSAT
55.7
96.7
87.0
74.8
98.4
69.3
87.0
65.2
90.2
59.1
64.6
Flowers
55.6
97.0
86.9
74.4
98.4
93.3
86.9
65.0
90.3
59.4
64.3
Food
54.7
96.8
86.2
72.6
98.3
92.2
91.8
66.7
89.8
59.0
63.8
MNIST
96.7
72.4
98.1
91.8
91.8
99.5
59.3
63.8
OxfordPet
54.6
96.7
85.7
72.0
98.2
91.8
91.8
99.5
94.7
59.2
63.8
Cars
54,2
96.7
85.7
71.9
98.1
91.5
91.7
99.5
94.4
84.3
63.7
SUN397
53.2
96.8
85.7
71.4
98.1
90.8
91.4
99.6
94.5
83.1
82.0
86.
Average
55.6
95.7
83.2
65.6
82.2
82.0
89.5
77.4
91.5
63.9
65.8
77.5
Table 8: Accuracy of KeepLoRA on the MTIL benchmark with order-II. Each row represents the performance every dataset of the model trained after the corresponding task. Transfer Average and Last metrics are shown.
OxfordPet
Caltech101
EuroSAT
CIFAR100
MNIST
Flowers
Sun397
Aircraft
Cars
Food
DTD
Transfer
88.7
57.7
91.2
72.1
65.8
23.4
88.8
45,4
48.5
68.2
65.0
Cars
86.2
88.7
57.1
91.3
71.7
65.5
23.5
87.4
46.6
50.7
69.5
Food
85.9
92.9
58.3
91.1
72.3
66.0
23.9
88.3
45.3
49.8
70.5
MNIST
85.8
92.8
91.2
71.9
66.2
23.0
88.6
46.4
50.4
68.1
OxfordPet
85.7
92.8
99.6
94.8
72.4
65.9
23.0
89.3
46.0
48.2
67.8
Flowers
85.6
92.8
99.6
94.8
92.4
65.7
23.0
89.3
46.2
46,9
67.5
Sun397
85.2
92.7
99.6
94.6
92.2
82.7
24.0
89.6
44.2
47.0
68.0
Aircraft
84.8
92.7
99.6
94.6
92.1
82.7
51.6
89.3
44,2
46.3
68.0
Caltech101
92.7
94.6
92.2
82.6
97.1
44.5
48.7
DTD
84.8
92.6
99.6
94.8
92.2
82.6
51.3
96.9
74.5
48.2
68.2
EuroSAT
84.6
92.7
99.6
94.6
92.1
82.2
51.1
97.0
74.5
98.6
66.6
CIFAR100
83.7
92.3
99.5
94.4
90.8
81.3
49.0
96.9
72.3
98.0
87.3
85.9
Average
85.2
92.3
92.0
93.7
84.8
74.8
35.9
91.8
53.1
57.5
70.0
75.6
USE OF LARGE LANGUAGE MODELS
We use the large language model to polish text and check grammar. All outputs were reviewed by the authors, who take full responsibility for the final content.
21

---

Published as a conference paper at ICLR 2026
Table 9: Accuracy of KeepL.RRA+ on the MTH, benchmark with order-I. Each row represents the performance on every dataset the model trained after the corresponding task. Transfer, Average and Last metrics shown.
Caltech101
CIFARI00
OxfordPet
Aircraft
EuroSAT
Flowers
MNIST
Sun397
DTD
Food
Cars
Transfer
85.9
69.9
44.6
53.7
70.9
88.9
65.4
90.8
63.0
66.1
69.9
Aircraft
85.9
44.4
54.3
72.4
62.7
91.2
63.8
64.5
Caltech101
59.2
97.5
70.2
44.3
53.3
71.4
89.5
62.7
91.4
63.8
65.1
CIFAR100
59.0
97.8
88.2
45.1
52.7
70.3
88.7
67.8
90.8
63.3
66.3
DTD
59.0
97.8
88.0
76.4
54.4
70.5
88.6
66.6
90.7
63.3
66.3
EuroSAT
58.6
97.6
87.8
76.2
98.5
70.2
66.0
66.5
Flowers
58.8
97.6
87.9
76.1
98.5
95.8
88.4
66.4
90.8
63.1
66.4
Food
58.4
97.6
87.6
76.6
98.4
95.8
92.9
65.4
90.4
62.5
66.5
MNIST
57.8
97.6
87.3
76.8
98.4
95.9
92.9
99.5
90.2
62.2
66.5
OxfordPet
57.8
97.6
87.2
76.5
98.4
95.8
92.9
99.5
94.8
62.2
66.4
Cars
57.7
97.5
87.3
76.7
98.4
95.6
99.5
94.8
87.7
66.3
SUN397
57.3
97.6
87.2
76.5
98.4
95.7
92.6
99.5
94.7
87.2
83.2
88.2
Average
96.5
84.4
67.8
82.1
84.5
90.7
77.8
91.9
67.5
67.6
79.0
Table 10: Accuracy of KeepLoRA+ on the MTIL benehmark with order-II. Each row represents the performance on every dataset the model trained after the corresponding task. Transfer, Average and Last metrics are shown.
OxfordPet
Caltech101
CIFAR100
MNIST
Flowers
Sun397
Aircaft
EuroSAT
Cars
Food
DTD
Transfer
89.1
58.1
90.7
72.4
65.4
24.0
88.9
44.0
52.7
70.2
65.5
Cars
88.4
89.1
91.6
72.3
65.3
24.2
88.0
53.9
70.1
Food
88.3
92.8
57.2
90.1
72.4
65.4
24.2
88.5
44.5
52.5
70.8
MNIST
88.1
92.8
99.4
90.4
72.1
65.5
24.1
88.5
44.4
52.9
70.1
OxfordPet
88.3
92.7
99.4
95.2
72.8
65.3
24.1
88.7
44.2
52.2
70.2
Flowers
88.3
92.8
99.5
95.2
65.3
23.9
88.7
44.4
51.7
69.9
Sun397
87.9
92.7
99.4
95.0
96.1
83.5
89.9
43.4
52.8
Aircraft
87.9
92.7
99.4
95.0
96.1
83.5
57.8
90.1
43.1
52.8
70.2
Caltech101
87.7
92.7
99.4
95.0
96.0
83.5
57.5
97.4
43.0
52.6
70.4
DTD
87.7
92.7
99.4
95.1
96.0
83.4
57.4
97.4
76.3
52.7
70.1
EuroSAT
87.6
92.7
99.5
95.0
83.4
97.4
76.3
98.3
70.3
CIFARI00
87.4
92.5
99.3
95.0
96.0
83.3
56.9
97.5
76.9
98.0
88.0
88.2
Average
88.0
92.4
91.9
93.9
87.4
75.2
39.2
92.0
52.8
60.9
71.8
76.9
22

---

Published as a conference paper at ICLR 2026
Table 11: Accuracy of LoRA-FT, O-LoRA, CL-MoE, SEFE, KeepLoRA on the MLLM-DCL benchmark. Each row represents the performance every dataset of the model trained after the corresponding task. Transfer
Average
and
Last metrics are shown.
LORA-FT
(b) O-LoRA
Sensing
Medical
Driving
Science
Finance
Sensing
Medical
Driving
Science
Finance
Transfer
28.1
17.4
34.0
50.2
32.4
Transfer
28.4
18,4
33.7
52.5
33.3
Sensing
78.8
28.1
17.3
34.8
55.6
Sensing
79.4
28.4
17.6
34.9
56.1
Medical
75.5
58.4
17.5
32.7
54.8
Medical
74.3
58.5
19.2
33.2
56.0
Driving
70,0
47.5
52.3
34.6
40.9
Driving
74.7
48.3
52.6
33.1
45.2
Science
73.2
46.4
40.6
50.4
49.5
Science
74.6
46.5
42.2
50.1
52.8
Finance
69.3
44.3
29.1
41.4
88.4
54.5
Finance
72.3
46.9
31.6
41.5
88.1
56.1
Average
73.3
44.9
31.4
38.8
57.8 49.3
Average
75.0 45.7
32.6
38.5
59.6 50.3
(c) CL-MoE
(d) SEFE
Sensing
Medical
Driving
Science
Finance
Sensing
Medical
Driving
Science
Finance
Transfer
28.3
19.4
34.1
48.6 32.6
Transfer
28.1
19.6
33.9
52.4
33.5
Sensing
79.4
28.3
18.7
35.2
56.4
Sensing
78.8
28.1
18.6
35.1
56.2
Medical
74.8
60.7
20.1
32.4
54.9
Medical
77.1
59.5
20.7
33.0
55.7
Driving
74.0
44.3
52.1
34.7
39.6
Driving
77.8
51.6
52.5
33.5
47.4
Science
71.0
47.4
40.0
50.7
43.3
Science
77.9
48.4
44.7
50.4
50.1
Finance
71.8
47.4
29.5
41.5
89.2 55.9
Finance
77.1
50.9
40.3
43.0
88.4
59.9
Average
74.2
45.6
32.1
38.9
56.7
49.5
Average
77.7
47.7
35.4
39.0
59.6 51.9
(e) KeepLoRA
Sensing
Medical
Driving
Science
Finance
Transfer
28.5
16.6
34.1
55.6 33.7
Sensing
80.0
28.5
17.0
35.1
55.1
Medical
79.9
58.6
16.3
33.7
55.6
Driving
79.8
57.7
53.1
33.7
54.6
Science
79.2
54.9
51.1
51.6 57.2
Finance
78.8
54.3
50.2
49.5
89.3
64.4
Average
79.6
50.8
37.5
40.7
62.4
54.2
23

---

Published as a conference paper at ICLR 2026
Table 12: Accuracy of LoRA-FT, O-LoRA, CL-MoE, SEFE, KeepLoRA on the UCTT benchmark. Each row represents the performance on every dataset of the model trained after the corresponding
task. Transfer,
Average and
Last metrics are shown.
(a) LORA-FT
(b) O-LoRA
ImgNet-R
ArxivQA
VizWiz
IconQA
CLEVR
Flick330k
ImgNet-R
ArxivQA
VizWiz
IconQA
CLEVR
Flikk33kk
Transfer
52.6 18.3
6.0 17.0 40.3 26.8
Transfer
52.9 19.6 4.4 16.9 41.0 27.0
ImgNet-R
91.7 52.6 23.5
11.8
17.2 36.5
ImgNet-R
91.5 52.9 24.7
13.3
17.3 36.5
ArxivQA
90.5 92.1 13.1
2.1
14.2 21.5
ArxivQA
89.7 94.2 14.5 0.0 12.9 25.0
VizWiz
73.6 90.7 61.0 4.2 19.0 49.7
VizWiz
80.9 91.7 59.8 0.0 19.6 49.0
IconQA
72.7 77.1 53.7 79.7
17.4 47.8
IconQA
80.2 80.3 54.5 75.9 17.6 48.6
CLEVR
68.8 77.4 52.3
67.8
77.9 46.1
CLEVR
78.1 80.4 51.6 63.2
72.4 46.0
Flickr30k
58.6 76.7 45.7 67.4
61.6
58.0 61.4
Flickr30k
74.2 80.9 45.3 62.9
63.8 57.2 64.1
Average
76.0 77.8 41.6 38.8 34.6
43.3 52.0
Average
82.4 80.1 41.7 35.9
33.9
43.7 53.0
(c) CL-MoE
(d) SEFE
ImgNet-R
ArxivQA
VizWiz
IconQA
CLEVR
Flickr30k
ImgNet-R
ArxávQA
VizWiz
IconQA
CLEVR
Flickr30k
Transfer
52.0 19.3 7.4 17.8 41.3 27.6
Transfer
53.3
18.7
7.5 17.0 40.9 27.5
ImgNet-R
91.2 52.0 23,9 5.2 15.6 36.9
ImgNet-R
91.6
53.3
23.7
12.1
16.9 36.4
ArxivQA
89.2 92.5 14.8 10.0 15.7 26.2
ArxivQA
90.4
92.8 13.7 5.0 16.4 21.1
VizWiz
77.2 90.7 60.4 6.9 20.6 49.5
VizWiz
83.6 89.3 61.4 5.3 18.6 49.8
IconQA
79.5 76.2 51.0 54.7 19.4 47.9
IconQA
84.3 78.1 57.4 79.6 16.2 50.6
CLEVR
76.7 75.4 48.1 52.6 73.0 45.9
CLEVR
82.8 78.6 54.2 70.6 75.0) 46.5
Flickr30k
61.2 75.8 44.4 52.6 54.4 57.3 58.6
Flickr30k
80.2 79.1 47.1 69.4 65.7
(57.3 66.5
Average
80.2 77.1 40.4 30.3
33.1 44.0 50.9
Average
85.5 78.6 42.9 40.3
34.8 43.6 54.3
(e) KeepLoRA
ImgNet-R
ArxivQA
Flickr30k
VizWiz
IconQA
CLEVR
Transfer
52.8 20.4 9.2 18.1 41.5
28.4
ImgNet-R
91.5 52.8 25.6
13.4
17.1 36.7
ArxivQA
90.4 94.5 15.2 4.0 17.2 21.5
VizWiz
85.5 92.4 61.5 10.1 21.0 50.6
IconQA
85.1 86.0 55.7 76.9 17.1 50.9
CLEVR
84.1 89.3 51.5 68.3 72.6 47.8
Flickr30k
82.4 86.7 46.6 67.8 66.4 (5.2 67.8
Average
86.5 83.6 42.7 40.1 35.2 44.1 55.4
24


Reviews:
[{"review_id": "jYNDAnqjSZ", "reviewer": "Reviewer_zp4o", "summary": "The paper addresses continual learning (CL) in pre-trained vision-language models (VLMs) by balancing plasticity and stability. It posits that general knowledge resides in the principal subspace of model weights, while task-specific knowledge lies in the residual subspace. Building on this idea, KeepLoRA confines new task updates to the residual subspace by projecting gradients orthogonally to a unified principal subspace and initializing a frozen LoRA adapter. Supported by theoretical analysis, KeepLoRA achieves comparable performance on the MTIL benchmark.", "questions": "Refer to Weaknesses for related questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clear  idea of separating general (principal) and specific (residual) knowledge in the parameter space.\n\n\n2. Provides strong theoretical justification connecting the method to optimal, constrained gradient descent.\n\n3. Good practicality as it adds no inference overhead, unlike architecture-extension methods.", "weaknesses": "1. Unclear if the unified principal subspace, which accumulates past task directions, can scale to a large number of tasks without prohibitive cost.\n\n2. The method requires expensive per-task full-gradient computation and a one-time full-model SVD, which are not fully benchmarked.\n\n3. Relies on crucial hyperparameters (e.g., $\\epsilon_w$, $\\epsilon_f$) whose robustness and sensitivity are not deeply analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rmUxmHpzDj", "reviewer": "Reviewer_FhjQ", "summary": "This paper analyses the parameter space using SVD and finds that the principal subspace encodes general knowledge and the residual subspace encodes domain-specific knowledge. The paper proposes KeepLoRa which projects the gradients onto a subspace\northogonal to both the principal subspace of pre-trained model and the dominant directions of previous task features. Theoretical analysis and empirical results on MTIL are provided.", "questions": "1. What is the computational and memory overhead KeepLoRa and keepLoRa+?\n2. What is the sensitivity of all hyper parameters?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. KeepLoRa works with pre-trained models addressing three competing objectives: maintaining the ability to learn new knowledge (plasticity), preventing the forgetting of previously learned tasks (backward stability), and preserving the general pre-trained knowledge that guarantees general transferability (forward stability).\n2. KeepLora can be implemented in a relatively straightforward manner.\n3. Sensible theoretical analysis and strong empirical performance.", "weaknesses": "1. KeepLora+ outperforms KeepLora but the first mention of KeepLora+ is in Table 2 and $4.1.\n2. KeepLora requires storing dominant singular vectors from tasks M but this is not listed in Table 2 nor analysed elsewhere.\n3. KeepLora introduces epsilon_w, epsilon_f, r, alpha hyper parameters but only the sensitivity of epsilon_w(vision) and epsilon_w(text) is presented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JVemjI1pRH", "reviewer": "Reviewer_1c8k", "summary": "This paper introduces KeepLoRA, a novel method for continual learning (CL) in pre-trained vision-language models designed to balance plasticity, backward stability, and forward stability. The authors of this study found a key insight by looking at how a model stores information, discovering that general, pre-trained knowledge is stored in the principal subspace, while specific skills are stored in the residual subspace. Motivated by this, KeepLoRA learns new tasks by restricting its Low-Rank Adaptation (LoRA) parameter updates to this residual subspace, where it essentially keeps the parameters from the principal subspace and everything it learned from past tasks, ensuring the new information is written only into the residual subspace. This approach prevents interference with existing knowledge, and the paper's theoretical and empirical analyses confirm that KeepLoRA effectively balances the three CL objectives to achieve state-of-the-art performance.", "questions": "see weaknesses", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. State of the Art performance: The paper demonstrates that KeepLoRA and its variant, KeepLoRA+, achieve state-of-the-art results on the MTIL benchmark, outperforming previous methods on all key metrics\n\n2. Strong Empirical Analysis: The paper's core hypothesis is based on a clear, intuitive analysis of the model's parameter space. The finding that general knowledge resides in the principal subspace while task-specific knowledge is in the residual subspace provides a solid foundation for the method.\n\n3.Theoretically Grounded: The authors provide a strong theoretical justification for their approach. They prove that their method of initializing and freezing the LoRA matrix At is an optimal solution (Proposition 3.2) to the problem of maximizing adaptation to the new task while remaining perfectly orthogonal to all previously learned knowledge.", "weaknesses": "1. Limited Evaluation on Language: The paper focuses on Vision-Language Models (VLMs), but the evaluation is performed on a benchmark of 11 image classification datasets. The \"language\" aspect is only used for zero-shot classification via class names. The method's effectiveness on more complex, language-heavy VLM tasks is unproven.\n\n2. Your method, by design, projects new task gradients to be orthogonal to the principal subspace and all previous task directions to ensure stability. Does this strict orthogonality constraint inadvertently prevent positive forward transfer, where the model should be re-using and building upon those similar, previously-learned features, rather than being forced to find a completely new, orthogonal direction to learn?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gnZJQs8ksw", "reviewer": "Reviewer_gbxX", "summary": "This paper focuses on balancing three core objectives in continual learning (CL) for pre-trained vision-language models (VLMs): preserving pre-trained general knowledge, retaining sequential task knowledge, and maintaining new task plasticity. Via SVD analysis, it finds general knowledge resides in the parameter principal subspace, while task-specific knowledge is in the residual subspace. Based on this, KeepLoRA restricts LoRA updates to the residual subspace—by projecting new task gradients onto the subspace orthogonal to the pre-trained principal subspace and previous task dominant directions—and initializes LoRA via purified gradients. Theoretical proofs and MTIL benchmark experiments (11 image classification tasks) confirm its state-of-the-art performance on Transfer, Average, and Last metrics.", "questions": "1. Some typographical errors. For example, \"KeppLoRA\" in table 2 and row 816.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Insightful subspace analysis: The separation of general/task-specific knowledge across principal/residual subspaces provides a clear, interpretable basis for CL design.\n\n- Theoretical rigor: Propositions 3.1 and 3.2 mathematically validate that KeepLoRA’s updates ensure stability and align with optimal plasticity.\n\n- Practicality: Parameter-efficient (only updates LoRA’s B matrix, no inference overhead), data-independent (no replay/reference data), and robust across task orders.\n\n- Comprehensive evaluation: Uses well-defined metrics (Transfer/Average/Last) and ablation studies to verify component contributions.", "weaknesses": "1. KeepLoRA’s design assumes that \"residual subspace updates for new tasks do not interfere with old tasks,\" but it does not analyze how task similarity affects this assumption: For highly similar tasks (e.g., two fine-grained classification tasks like StanfordCars and Aircraft), their residual subspaces may overlap, potentially leading to interference. \n\n2. Scalability gaps: Unaddressed feasibility of SVD and subspace scaling for large VLMs (e.g., CLIP-L) with massive parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)