Continuous Subspace Optimization for Continual Learning
Authors: Quan Cheng
Yuanyu Wan
Lingyu Wu
Chenping Hou
Lijun Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Subspace Optimization
Continual Learning
Primary area: deep_learning
TL;DR: we propose Continuous Subspace Optimization for Continual Learning (CoSO) to fine-tune the pre-trained models in a series of subspaces rather than a single one.

Abstract: Continual learning aims to learn multiple tasks sequentially while preserving prior knowledge, but faces the challenge of catastrophic forgetting when adapting to new tasks. Recently, approaches leveraging pre-trained models have gained increasing popularity in mitigating this issue, due to the strong generalization ability of foundation models. To adjust pre-trained models for new tasks, existing methods usually employ low-rank adaptation, which restricts parameter updates to a fixed low-rank subspace. However, constraining the optimization space inherently compromises the model's learning capacity, resulting in inferior performance. To address this limitation, we propose Continuous Subspace Optimization for Continual Learning (CoSO) to fine-tune the model in a series of subspaces rather than a single one. These sequential subspaces are dynamically determined through the singular value decomposition of the gradients. CoSO updates the model by projecting gradients onto these subspaces, ensuring memory-efficient optimization. To mitigate forgetting, the optimization subspace of each task is constrained to be orthogonal to the historical task subspace. During task learning, CoSO maintains a task-specific component that captures the critical update directions for the current task. Upon completing a task, this component is used to update the historical task subspace, laying the groundwork for subsequent learning. Extensive experiments on multiple datasets demonstrate that CoSO significantly outperforms state-of-the-art methods, especially in challenging scenarios with long task sequences.

Continuous Subspace Optimization for Continual Learning
Quan Cheng!?, Yuanyu Wan Lingyu Wu! Chenping Hou', Lijun Zhangl22
National Key Laboratory for Novel Software Technology. Nanjing University, Nanjing, China
School of Artificial Intelligence, Nanjing University, Nanjing, China Schooo of Software Technology, Zhejjang University, Ningbo, China
A Hangzhou High-Tech Rech Zone (Binjiang) Institute Blockchain and Data Security, Hangzhou, China $ College of Science, National University of Defense Technology, Changsha, China {chengq, wuly, zhanglj) @ladaa. .jju edu. cn wanyyēzju. edu. en, hepnudt.hhotaain com
Abstract
Continual learning aims to learn multiple tasks sequentially while preserving prior knowledge, but faces the challenge of catastrophic forgetting when adapting new tasks. Recently, approaches leveraging pre-trained models have gained in- creasing popularity in mitigating this issue, due the strong generalization ability of foundation models. To adjust pre-trained models for new tasks, existing meth- ods usually employ low-rank adaptation, which restricts parameter updates to a fixed low-rank subspace. However, constraining the optimization space inherently compromises the model's learning capacity, resulting inferior performance. To address this limitation, we propose Continuous Subspace Optimization for Con- tinual Learning (CoSO) fine-tune the model series of subspaces rather than single one. These sequential subspaces are dynamically determined through the singular value decomposition of the gradients. CoSO updates the model by pro- jecting gradients onto these subspaces, ensuring memory -fffcieef optimization. To mitigate forgetting, the optimization subspace of each task is constrained be orthogonal to the historical task subspace. During task learning, CoSO maintains a task-specific component that captures the critical update directions for the current task. Upon completing a task, this component is used to update the historical task subspace, laying the groundwork for subsequent learning. Extensive experiments multiple datasets demonstrate that CoSO significantly outperforms state-of-the- art methods, especially in challenging scenarios with long task sequences.
Introduction
Deep neural networks have achieved remarkable success when trained on large-scale offline data under the assumption of independent and identically distributed (i.i.d)) samples [He all 2016. Vaswani et all 2017 Dosovitskiy et all 2021]] However, real-world applications often require mod- els to learn from a sequence tasks with different data distributions, scenario known as continual learning [De Lange et all, 2025, Van de Ven all 2222, Masana cla 2022 Wang et al) 2024, Zhou et 2024b]. The major challenge in continual learning is catastrophic forgetting [McCloskey and Cohen. 198%, where the model's performance previously learned tasks deteriorates significantly as it adapts to new tasks.
In recent years, pre-trained models especially vision transformers (ViTs) [Dosovitskiy et all 2021] have demonstrated exceptional performance across various downstream tasks through their robust
"ijjun Zhang is the corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

generalization ability. This property makes pre-trained models highly promising in mitigating catas- trophic forgetting, leading to growing research focus on continual learning with foundation models [Smith et all, 2023, Lu et all R024, Liang and Li,, 2024 Zhou all, 20244. Wu et all 2025]. To efficiently fine-tune pre-trained ViTs, existing continual learning methods [Gao et all, 2023, Liang and 2022, et 20255 employ low-rank adaptation (LoRA) [[uu et all, 2022) to optimize the models, which confine parameter updates to a specific low-rank subspace to reduce the interference between tasks. However, this rigid constraint on update directions inherently limits the model's learning capacity, leading to inferior performance.
To address this issue, we propose Continuous Subspace Optimization for Continual Learning (CoSO), which achieves enhanced adaptability by optimizing the model within multiple subspaces rather than a fixed one. These sequential subspaces are derived from the singular value decompo- sition of the gradients. By projecting gradients onto these low-dimensional subspaces for Adam [Kingma, 2014 optimization and then projecting back for parameter updates, CoSO achieves memory-efficiett learning. To prevent forgetting, we enforce orthogonality between the optimiza- tion subspaces of current and historical tasks during training. While learning task, CoSO leverages Frequent Directions (FD) [Ghashami ata 2016, Wan and Zhang 20TS 21PeI to maintain a com- pact task-specific component, which captures critical update directions of the current task with neg- ligible computational cost. After completing the current task, we use this dedicated component to estimate the task task-specific subspace, which is then integrated into the historical task subspace, laying the groundwork for subsequent learning.
Experimental results on CIFAR 100, ImageNet-R, and DomainNet show that CoSO consistently out- performs sfate-of-the-art methods by significant margin across diverse continual learning settings, particularly in challenging scenarios involving long task sequences. The substantial performance gains highlight CoSO's s strong potential for real-world continual learning.
In summary, our contributions are as follows:
We propose CoSO, novel continual learning framework that fine-tunes pre- pre-trained models via continuous cradient-deriver subspaces, enabling efficient adaptation to sequential tasks. We introduce a lightweight mechanism to maintain the historical task subspace, enabling CoSO to keep current updates orthogonal to the historical subspace and thereby mitigate task interference.
We conduct extensive experiments, demonstrating CoSO's s superior performance over prior PEFT-based continual learning methods across various datasets and settings.
2
Related Work
In this section, review related work on continual learning and low-rank optimization in offline learning.
2.1 Continual Learning
Continual learning [De Lange et all 2022 Van de etal 2022, Masana et all 2022 Wang et all 2024, Zhou all 2024h] aims to enable neural networks to incrementally learn from a sequence of tasks while retaining previously learned knowledge. These approaches broadly fall into five categories [Wang et all, 2024]]: regularization-basee methods [Zenke et all 2017, Kirkpatrick et all,
2017
Li and Hoiem, 2017]. replay-based methods [Lopez-Paz and Ranzato, 2017, Rebuth all
methods methods Madaan methods [LYonn et all, 2018, ct 2019 Sokar et all, 2021. Liang and Li 2023). Regularization-base methods introduce additional loss terms to constrain parameter updates, preventing drastic changes in parameters that are important for early tasks. Replay- based methods store a small subset of training samples from previous tasks in limited buffer and periodically replay these samples alongside new data, allowing the model to rehearse earlier knowledge. Optimization-bant
201
Chaudhry et al, 2 L et all 2020. Sun all 2022), optimization-
Paraatabar all 2020. Sahactaal 2021, Wang all 202IT] representation-
atal
2022
Pham el all M124]] and architecture-
methods manipulate the update directions of each task according to preserved information of previous tasks. Representation-ban -based methods utilize statistical information of features to calibrate classifiers. Architecture- based methods dynamically modify network architectures, dedicating specific model capacity for new tasks.

---

Early continual learning approaches typically initialize their models with random weights. The strong generalization capabilities of foundation models, especially vision transformers [Dosovilskay atl 202 have made pre-trained architectures an increasingly attractive solution for continual learning iZhouettal 20225). Recent developments parameter-efficien fine-tuning (PEFT) based continual learning methods [Gaa all 2025. Liang and L1, 2022! Lu et an, 2022 eu 202S have facilitated efficient adaptation of foundation models through selective parameter optimization,
substantially lowering computational requirements. Existing PEFT-based methods can be broadly categorized into two groups: (1) prompt-based techniques that focus on optimizing learnable tokens
[ester et all, 2021, Wang et all, 2022a Smith et all 2023, Lu all, 20222 and (2) LoRA-based
methods that adjust parameters within constrained low-dimensional subspaces [Gia ct all 2025
Liang and Li 2022) all R025].
Among prompt-hased approaches, I.2P [Wang et all 2022B) introduces task-specific prompt tokens to modulate the pre-trained model's behavior, but struggles with knowledge transfer between tasks. DualPrompt [Wang et all, 2022a] addresses this limitation by maintaining both task-specific and task- invariant prompts, enabling better knowledge sharing. CODA-Prompi [Smuth et all 20X3 further enhances adaptation flexibility through dynamic prompt composition from shared pool. VPT-NSP" Lu all 2022 learns each task by tuning learnable prompts in the null space of previous tasks' features. However, these methods influence model behavior indirectly through learnable tokens, which restrict the model's ability to capture complex task-specific features.
Complementary to prompt-based methods, LoRA-based approaches directly update model param- eters in a parameter-efficient manner. InfLoRA [Liang and Li, e0zel] constrains the parameter up- dates within a predetermined subspace to reduce the interference between tasks. SD-LoRA [Wii atall 20255] decouples the learning of the magnitude and direction of LoRA components. However, both methods confine weight updates to a specific low-rank subspace, which inherently limits the model's 's learning capacity. Unlike these methods, CoSO updates the parameters across series of subspaces, enabling the learning of full-rank weights and thereby enhancing the model's flexibility,
2.2 Low-rank Optimization in Offline Learning
Low-rank adaptation (LoRA) [Hu et anl 2022] has gained significant attention for its ability to reduce computational and memory requirements when fine-tuning pre-trained models [Mao et all 2022 Zhang al. 2023]. Specifically, LoRA reparameterizes the update of linear layer's 's weights AW = BA E R where B E R A E Rrxn are low-rank matrices. By freezing the original weights and only updating the low-rank components, LoRA enables parameter-efficient fine- tuning while preserving performance in many downstream tasks. However, it has been demonstrated all 2024] that low-rank weight updates limit the performance compared to full-rank fine- tuning. Recent works [[ossonn et al 2023, Zhao et all [I have shown that neural network gradients often exhibit low-rank structure. Instead of approximating the weight matrix as low rank, GaLore [Zhao et all 2 directly leverages the low-rank gradients to optimize the model. This methodology enables memory-efficient optimization through effective dimensionality reduction in gradient spaces.
To be concrete, GaLore utilizes the singular value decomposition (SVD) of G2 E Fm xn to compute a low-rank projection matrix P. E Rr where < is the target rank. Leveraging Pt, Ga- Lore transforms the gradient G, into a compact form P G, to achieve memory-efficient parameter updates. At each training step f, the gradient update can be decomposed into three operations:
R. P. P.TG. G.
(forward projection)
N. = Adam(R)) RR))
(adam optimizer update)
G. P.N.
(backward projection)
The projection matrix P: is periodically updated through SVD to follow the evolving gradient sub- space. Utilizing the final gradient G,, GaLore updates the model parameters with learning rate ry:
W. Wt-1 G
Compared to LoRA, Gal not only reduces memory storage from (mn + 3mr + 3nr) to (mn + mr + 2nr), but also achieves higher model capacity by directly optimizing in the most relevant gradient subspaces rather than constraining updates to predefined low-rank structure.
3

---

Stage 1: Update parameters
Project
W
Orthogonal projection
Update +
Project back
W
Orthggnal to M
Gradient Grpt
P
Nr
6.
Stage 2: Update historicul subspuce
Optimization space
Qr.1
Frequent
Directions
Residual subspace
Low-rank approximation of gradients
Task speciffi subspace
Task
Figure 1: CoSO optimizes the parameters in continual low-rank subspaces, enhancing the learning capacity of models. To mitigate forgetting, the optimization subspaces of the current task are set to be orthogonal to the historical task subspace. While learning a task, CoSO consolidates the low-rank approximation matrices {Qr,+ Y into a task-specific component Sr,7 through Frequent Directions. The dedicated component then used update the historical task subspace spanned by M
3 Methodology
In this section, first introduce the necessary preliminaries, then present the details of ap- proach.
3.1 Preliminaries
In continual learning, model needs to learn sequence of tasks while retaining knowledge of previ- ous tasks. We consider the class- incremental learning setting, where task identities are unavailable inference time and access historical data prohibited during learning new tasks [[Wng et all 2022a Smith at 2023, Liang and L. R02 [ al 2024, Wi et al 2025]. We denote the task sequence as D (Di, ..., ) where each task dataset D, = {(xi )) contains n, input- label pairs. Following recent work [[Wang et all 2022a, Gao et all 2023]. we adopt pre-trained Vision Transformer (ViT) [Dosovitskay et all 20221] as the backbone network, denoted as fe(:) with parameters θ and classifier h-)(-) with parameters 4, thus the model is h4 (fe(-)). Formally, the hidden state Y of feature X at the linear layer é, can be calculated as Y W where We is the weight matrix of the linear layer. Let G denote the gradient at the I-th training step of the
linear layer& in task For simplicity, we omit the symbol using W to refer W and Gy to refer Gf in the following sections.
3.2 Continuous Subspaces Optimization
Inspired by GaLore İZhao all R022| we propose CoSO to address the rigidity of single sub- space adaptation methods through multiple subspaces optimizing. However, directly using GaLore causes severe interference between different tasks in continual learning. To minimize the interfer- ence, CoSO enforces orthogonal constraints between current and historical subspace during training. Motivated by memory consolidation in cognitive neuroscience [Dudai, 2004], CoSO estimates task- ssscispecffic subspace to consolidate knowledge upon learning each task. This subspace preserves critical learning directions of the task based on gradients at all training steps, and is incrementally integrated into the historical task subspace, enabling efficient knowledge accumulation. The whole process of CoSO is illustrated in Figure II We first introduce how to optimize the model in con- tinuous subspaces in this section. Then we present how to update the historical task subspace in Section a

---

In continual learning, the key challenge is to prevent new task updates from interfering with pre- viously learned knowledge. Building on the insight that gradient updates in neural networks typ- ically lie in the span of input features [Saha et all, 202TIJ, we develop an approach that leverages this gradient-input feature relationship to minimize task interference through orthogonal projection. Specifically, we maintain an orthogonal basis matrix M,-1 that spans the gradient subspace ac- cumulated from all previous tasks prior to the current task 7. Since gradients inherently encode information about the input features they were computed from, this historical subspace captures the principal directions that were important for learning previous tasks. We recognize that gradient steps along these historical directions would cause maximal interference with past learning, while gradi- ent steps orthogonal to this space result in minimal interference. For each gradient GT., computed during training on task T, we project it onto the orthogonal complement of historical subspace:
Gr,, Gr.t - -M -M--MM,.
This projection removes the gradient component aligned with the learning directions of previous tasks, leaving only the orthogonal component G²,, for updating the parameters. When we update the
weight matrix using these orthogonal gradients. i.e., AW = Enn G²,,, the parameter changes occur in directions that have minimal overlap with the optimization trajectories of previous tasks. This approach effectively partitions the parameter space, preserving directions important for past tasks while utilizing orthogonal directions for new learning. The orthogonal projection thus provides a principled way to balance plasticity for new tasks with stability for old tasks, enabling the model to expand its capabilities while mitigating interference.
However, updating the model with the full orthogonal gradient G incurs substantial memory head and high computational cost, particularly in vision transformers. To achieve memory-efficient fine-tuning, follow GaLore [Zhao et all 2022) and decompose G² E Rmxa using singular value decomposition (SVD) to get the projection matrix Pr.:
UVV = SVD,, (G²,,)
(2)
Pr.+ = U|:
where and are the dimensions of the original weight matrix, rı < n. is the target projection rank, and SVD,, (-) denotes a truncated SVD that retains the top-r1 singular values. Subsequently, we project the orthogonalized gradient G²,, into the low-rank subspace spanned by P.,,, effectively reducing the memory footprint of parameter updates:
Ry, G.,.G..
(3)
Then R, is updated by Adam Kingma
2014 as follows:
Mr. = (3) + R..)) /(1 - -1)
Vr, = (B2 + 32) .R²,) /(1- -3))
(4)
Nr.1 Mr,/ (VVre+e).
where B1, f2 are decay rates, My, is the first-order momentum, and V-, is the second-order mo- mentum. The low-rank normalized gradient N,, is then projected back to update the parameters with learning rate ry:
Gr, Pr.NNr. Wr,1 = Wr.-1 - 17 Gr.--
(5)
Because Pr. is computed from the projected gradient G²,,, which is orthogonal to the historical subspace spanned by My-1, any parameter updates derived from Pr. remain in the null space of previous tasks' feature spaces. Consequently, the linear layer's output for every earlier task remains unchanged, preventing interference at the representation level. Since Pr. is dynamically changed to capture the most important directions of G²,,, we are optimizing the model in continuous subspaces rather than fixed one, thereby expanding the model's representational adaptability. To balance computational efficiency, we update the projection matrix Pr. every K steps. By updating Rr,t in lower dimension space, the memory requirement is reduced from (mm + 3mr, + --3rr) to (mi + mri + 2nrr) compared to LoRA-based methods, such as Infl.RA [Liang and Lij, 2022 and SD- LoRA [W all 20%
5

---

3.3 Historical Task Space Update
Task-Specific Subspace Estimation. To update the orthogonal basis matrix M of the historical task space, we need to efficiently estimate a task-specific subspace, which retains the critical gradient directions of the current task. Specifically, for task т, the model undergoes T training steps, produc- ing sequence of gradients [G',,,.... Gr,r), where C²,, E R To identify the primary direc-
tions of these gradients, we consider the accumulated covariance matrix Σ G² G,G,G. E Rmx" which integrates information from all training steps and characterizes the subspace where most up- dates occur. However, directly maintaining such accumulated covariance matrix would be com- putationally expensive, requiring O(m²nl") time complexity. This particularly challenging for transformer-based models where the parameter dimension m, are typically in the order of thou- sands.
To ensure computational efficiency, we use Frequent Directions (FD) [Ghashami et all. 2016, Wan and Zhang, 2018, [022], deterministic matrix sketching algorithm, maintain a low-rank ap- proximation of streaming gradients. The FD algorithm processes the gradients sequentially while providing guarantee on approximation quality [Wan and Zhang, 2021, Yang al, 2025]. Specif- ically, we first compute a low-rank matrix QT, E R2xx22 with 12 & n. through singular value decomposition (SVD):
UEV = SVD,,, (G),,)
(6)
--==U
Here, SVD,,, denotes a truncated SVD that retains the top-ry singular values. The resulting low- rank matrix Qr, enables to efficiently approximate the gradient covariance matrix:
Qr, ~ G.,
Based on this approximation, we further compute a sketch matrix Sr., E IR that incrementally consolidates the gradient covariance information from all training steps up to step This consoli- dation is achieved by combining the previous sketch matrix Sr.-1 with the current approximation Qr,-- The update of S,,, is as follows:
U'I'V' = ((S,,. Qr,1)
(8)
Sr, = U' U/V22222 - oebrg a x// "2.222
We initialize S,,1 Qr,1, 1. and after iterations, we obtain the final task task-specific sketch matrix Sy, which satisfies:
ST E G GTT
By analyzing the dominant singular vectors of Sr,T, we can effectively estimate the principal sub- space of the current task. Note that update Sr, every K steps match the update frequency of projection matrix Pr.,, ensuring consistency in our approximation process. The effectiveness of CoSO relies the accuracy of low-rank approximation, which is formalized through the following Proposition M
Proposition 1. Given a sequence of projected gradients (Gr,)T= and low-rank matrix {Qr,IT= where G²_, E RR and Qr,t. E Ruxrr, The final sketch matrix is Sr E Rmxrr, Let A =
Di GT Ā Ā=ET QT, For any k < 12 the approximation error is bounded by:
1| [A]x
| ST. S Do
of
(10)
r2 -
where at is the (r2 + 1) singular value of G¹ and [Ā]x h is the minimizer of Í| [A]s l overall rank k matrices.

---

Remark. Because the gradients often exhibit low-rank structure [Cosson et at 2023 Zhao et al
2024J, their singular values decay rapidly. Consequently, the error Et= of would be negligibly small when 12 exceeds the intrinsic rank of the gradients. By maintaining low-rank sketch matrix Sr,T, we reduce the cost of compuling -i= G G. from Omm"nT) to Ommnr?)) where 12 & m. Proposition Q ensure that our low-rank approximation captures the most significant directions in the gradient space. The error bound provides practical guidance for choosing the rank r2: larger values lead to better approximation at the expense of additional computation and memory. To better preserve the task information, we set 12 to be slightly larger than T1, where T1 is the projection rank introduced in Section B.A The proof is provided in Appendix B1
Update Orthogonal Basis Matrix. Once the final task- sketch matrix Sr,7 is computed,
use it to update the orthogonal basis matrix M,-1 to incorporate the optimization subspace of the task T. First, we extract the principal directions of the current task by performing SVD on its sketch matrix:
U,E,V, SVD(S+;r).
(1I)
Then, we determine the number of directions to retain based on the sum of squared singular values. Following the principle of matrix approximation with SVD, we select is as the biggest value that satisfies:
6thy
(12)
where Esn E (0, 1] is a threshold hyperparameter controlling the ratio to preserve, and o, is the i-th singular values in descending order. This criterion ensures that the selected le directions capture at least 6th fraction of the total variance in the gradient space. Finally, we expand the orthogonal basis matrix Mr-1 by incorporating these new directions:
M, = MM--,, U+(:; k]]-
(13)
The above selection and update process ensure that we capture the most important learning directions for each task while maintaining orthogonality between different tasks' subspaces.
Due to space constraints, the complete CoSO algorithm is presented in Appendix B
4 Experiments
We conduct comprehensive experiments with varying numbers of sequential tasks to evaluate CoSO's effectiveness across multiple datasets. We first outline our experimental settings, then present detailed results and analyses.
4.1 Experimental Settings
Datasets and Evaluation Metrics. Following previous works [Wang et all 20226, Liang and Li 20244], we evaluate CoSO on three widely-used continual learning benchmarks: ImageNet- [Hendrycks et al] 2021],, CIFAR 100 (Krihhvskyy 2009), and DomainNet (Peng et all 201)..
ImageNet-R contains 200 classes from ImageNet with artistic style variations. Similar to existing
works [Smith all 20225 Liang and Li R024, Wu et anl 202S], we create three different splits of
ImageNet-R: 5 tasks with 40 classes per task, 10 tasks with 20 classes per task, and 20 tasks with 10 classes per task. For CIFARI00, we divide into 10 tasks, each containing 10 classes. NomainNet consists of 345 classes across six distinct domains and is split into 5 tasks, with 69 classes per task.
We evaluate our method using two complementary metrics that are widely adopted in existing continual learning methods [Wang et all 2022R, Liang and Li, 20244, Wi all 2025). The first metric is the final accuracy ACCr, which evaluates the model's overall performance across all tasks after the complete training process. The second metric the average accuracy ACCT, which measures the model's learning stability throughout the training sequence and is calculated
as ACCT = + ACC; where T denotes the total number of tasks. These two metrics cap- ture both the model's ability to learn new tasks and retain knowledge of previously learned tasks, providing comprehensive assessment of continual learning performance.
7

---

Table 1:
Results (%) on ImageNet-R with varying numbers of tasks (5, and 20). All reported results with mean and standard deviation are computed over 3 independent runs.
ImageNet-R (5 Tasks)
ImageNet-R (10 Tasks)
ImageNet-R R (20 Tasks)
Method
ACC₂
ACC₂
ACC10
1CC
ACC20
2CC
L2P
65.03=0.03
69.97
60.77
68.90=0.55
58.6440.31
+0.35
DualPrompt
68.24410.2
71.8210.39
65.30
69.62 10.29
60.47
65.91 10.52
CODA-P
73.65.10.15
77.88 1..30
72.10
76.90
67.16.0.11
12.34
InfLoRA
17.53
12.14
74.43
80.50
70.30
77. 77.0440.06
SD-LoRA
79.15
23.01
77.34
82. 82.04_0.24
75.26+0.37
80.22+0.72
VPT-NSP²
79.72=0.19
84.33
77.87 10.10
83 83.09=0.20 26226
81.32++021 75.42+0.27
CoSO
82.10+0.13
86.38
81.10
85.56
78.19+0.28
83.69+0.12
90
90-
Accuracy (%)
Accuracy (96)
Accuracy (96
880
80
75
70-
70
70
C350
Caso
CODAMTIEE:
65
SOLIRA
LIP
SOLARA
127
60
Isfaa
60
3 Number of Tasks
10
10
20
Number of Tasks
Number Tasks
(a) 5 tasks
(b) 10 tasks
(c) 20 tasks
Figure 2: The detailed performance during the learning of ImageNet-R on (a) 5 tasks, (b) 10 tasks,
and (c) 20 tasks.
Baselines and Implementation Details. We compare CoSO with several state-of-the-art PEFT- based methods: L2P [aagg et 20226), DualPrompt [aang et al, 20222 CODA-Prompl (CODA- P) [Smith et all 2023). InfLoRA [Liang and Li PIZA VPT-NSP- et اا and SD-LoRA
eW all (025). Comparing against both prompt- based and LoRA-bases methods allows us to comprehensively evaluate the effectiveness of CoSO. In addition to the ViT-B/16 [Dosovitskiy et all 2021] pretrained on ImageNet-1 K, we also evaluate a self-supervised ViT-B/16 obtained with DINO
Caron 20ZT]. Details of the experimental setup provided in Appendix Q
4.2
Experimental Results
We evaluate CoSO against state-of-the-arr continual learning methods across different experimental settings. Table shows the performance comparison on ImageNet-R under various task partitions (5, 10, and 20 tasks). Across all partitions, CoSO delivers the highest final accuracy (AC') and average accuracy (ACC), confirming its robustness to mitigate forgetting. For the most challenging setting (20 tasks), CoSO attains 78.19% final accuracy and 83.69% average accuracy. while the best baseline method achieves 75.42% and 81.32%, respectively. For the ImageNet-R 10 tasks scenario, CoSO improves the final accuracy by 3.23% and the average accuracy by 2.47% compared to best baseline method. Likewise, in the 5 tasks setting, CoSO still leads by 2.38% in final accuracy and 2.05% in average accuracy. This margin highlights CoSO's exceptional resistance to forgetting and its strong capacity to integrate new knowledge without eroding prior learning.
Figure illustrates the evolution of accuracy throughout the continual learning process for various methods evaluated on ImageNet-R, RR It is evident that CoSO consistently maintains superior perfor- mance relative other approaches, both during the intermediate phases and at the end of training. This ongoing superiority underscores CoSO's effectiveness in reducing interference from newly in- troduced tasks, resulting in a significantly slower decline in accuracy compared to competing meth- ods. Complementary results in Table 2 reveal the same trend on CIFAR 100 and DomainNet. On the DomainNet benchmark, CoSO outperforms the best baseline method by 1.75% in final accuracy and 1.37% in average accuracy, confirming its ability to generalize across heterogencous visual domains.

---

Table Results (%) on CIFARIO0 (10 Tasks) and DomainNet (5 Tasks). All reported results with mean and standard deviation are computed over independent runs.
CIFARIOO (10 Tasks)
DomainNet (5 Tasks)
Method
ACC10
1C
ACC3
ACCs
L2P
82.6440.26
00.90
70.03=0.09
75. 75.65+0.06
DualPrompt
84.68
90.1210.05
72.25
77.84 17.84
CODA-P
86.6010.37
91.4661.20
73. 73.16
78.7510.04
InfLoRA
86.85.0.08
91.4510.16
73.09
79.21
SD-LoRA
87.30
21.81
73.20=0.12
79.03+0.04
VPT-NSP²
88. 2..09
92.48
72.52=0.13
78. 40.68
CoSO
88.77+0.16
92.99
74.27
80.05+0.04
Table 3: Ablation study results (%) on ImageNet-R with varying numbers of tasks (5, 10 and 20).
ImageNet-R (5 Tasks)
ImageNet-R (10 Tasks)
ImageNet-R (20 Tasks)
Method
ACCg
ACC₁
ACC10
ACC10
ACC20
ACC²0
w/o Orth
79.35
85.22
75.90
83.43
69.75
78.88
w/o FD
80.72
85.44
78.83
84.45
76.68
82.41
CoSO
82.37
86.46
80.72
85.67
78.27
83.62
A detailed analysis of computational and memory costs are presented in Appendix D The additional results with DINO [Caron et at 2021] are provided in Appendix El
Ablation Study. We conduct comprehensive ablation studies on ImageNet-R benchmark to validate the individual contributions of the orthogonal projection mechanism and the Frequent Directions (FD) based subspace consolidation. Specifically, we compare CoSO with two variants. The first variant (w/o Orth) removes the orthogonal projection, which directly uses the original gradients Gr for optimization instead of the orthogonally projected gradients G',,- This variant optimizes parameters in continuous subspaces without any orthogonality constraint, thereby ignoring task in- terference. The second variant (w/o FD) retains orthogonality but, instead of employing FD to consolidate all intermediate gradients from the current task, constructs the task-specific subspace using only the final subspace obtained at the end of that task.
The results are summarized in Table EL Eliminating orthogonal projection (w/o Orth) leads sharp performance drop (8.52% in final accuracy) on 20 Tasks setting, highlighting the importance of excluding new gradients from the historical subspace prevent interference. Replacing FD with the simplified strategy that builds each task-specific subspace from only the final gradient subspace (w/o FD) also degrades performance, lowering final aceuracy by 1.65%, 1.89% and 1.59% for 5, 10 and 20 Tasks settings, respectively. This drop confirms that aggregating all intermediate gradi- ents through incremental FD updates captures richer task information than using single terminal subspace. Across the table, the full method delivers the highest final and average accuracies, indi- cating that both orthogonal projection and FD consolidation are indispensable for robust continual learning.
5 Conclusion
In this paper, we propose Continuous Subspace Optimization for Continual Learning (CoSO). CoSO optimizes the pre-trained models within continuous subspaces. By maintaining orthogonality be- tween the current task's 's optimization subspace and that of historical tasks, CoSO effectively miti- gates the interference. CoSO maintains compact task-specific component while learning a task. After completing the current task, the task-specific component is used to update the historical task subspace. Extensive experiments on standard benchmarks demonstrate that CoSO consistently out- performs -hee--rr-t baselines in both final accuracy and average accuracy over time, confirming its effectiveness and robustness across diverse data streams. In the future, challenging open prob- lem is to extend CoSO to multimodal task settings.

---

Acknowledgments and Disclosure of Funding
This work was partially supported by National Science and Technology Major Project (0222D01148011 and NSFC (U23A20382).
References
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision 2021.
Arslan Chaudhry, Mare' Aurelio Ranzato, Mareus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In International Conference on Learning Representations, 2019a.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv: 10486, 2019b.
Romain Cosson, Ali Jadbabaie, Anuran Makur, Amirhossein Reisizadeh, and Devavrat Shah. Low- rank gradient descent. IEEE Open Journal of Control Systems, 2:380-395, 2023.
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, (4(7))3366-3385 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
Yadin Dudai. The neurobiology of consolidations, or, how stable is the engram? Annual review of psychology, 55:51-86, 2004.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for contin- ual learning. In Proceedings of the 23rd International Conference on Artificial Intelligence and 2020.
Statistics, volume 108, pages
Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang. A unified continual learning framework with general parameter-efficient tuning. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1 11483-11499 2023.
Mina Ghashami, Edo Liberty, Jeff M. Phillips, and David P. Woodruff. Frequent directions: Simple and deterministic matrix sketching. SIAM Journal Computing, 45(5): 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The Many Faces of Robustness: A Critical Analysis of Out-of -fittr--tiiiiitionn Generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8320 8329, 2021.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022.
Diederik Kingma, Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.690. 2014.
10

---

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barosska-BBwisska, Demis Hass- abis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 11 114(13):551-33526 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Pro- cessing, pages 3045-3059, 2021.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: continual structure learning framework for overcoming catastrophic forgetting. In Proceedings of the 36th International Conference on Machine Learning, pages 3925-3934, 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40 (111:22233529999 2017.
San-Shuolln Liang and Wu-Jun Li. Adaptive plasticity improvement for continual learning. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7816-7825, 2023.
Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference-free low-rank adaptation for continual learn- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 23638 -2687,,,377,, 2024.
Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. Mnemonics training: Multi- class incremental learning without forgetting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
David Lopez-Paz. and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems 30, pages 6470-6479, 2017.
Yue Lu, Shizhou Zhang, De Cheng, Yinghui Xing, Nannan Wang, Peng Wang, and Yanning Zhang.
Visual prompt tuning in null space for continual learning. In Advances in Neural Information Processing Systems 37, pages 7878-7901, 2024.
Divyam Madaan, Jachong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang. Representational continuity for unsupervised continual learning. In International Conference on Learning Repre- sentations, 2022.
Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma. Jiawei Han, Wen-tau Yih, and Madian Khabsa. Unipelt: unified framework for parameter-efficient language model tuning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022.
Mare Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D. Bagdanov, and Joost van de Weijer. Class-incremental learning: Survey and performance evaluation on image clas- sification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5513-5533, 2022.
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 24:109-165, 1989.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang. Kate Saenko, and Bo Wang. Moment matching for multi-souree domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1406-1415. 2019.
Quang Pham, Chenghao Liu, and Steven C. H. Hoi. Continual learning, fast and slow. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(1):134-149) 2024.
AivestreeAAvisee Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. iCaRL: Incremental Classifier and Representation Learning. Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 5533-5542, 2017.

---

Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning, In International Conference on Learning Representations, 2021.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante- Boniila, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual de- composed attention-based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11909-11919, 2023,
Ghada Sokar, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Spacenet: Make free space for continual learning. Neurocomputting 439:1-11, 2021.
Qing Sun, Fan Lyu, Fanhua Shang, Wei Feng, and Liang Wan. Exploring example influence in continual learning. In Advances in Neural Information Processing Systems 35, pages 27075- 27086, 2022.
Gido M Van de Ven, Tinne Tuytelaars, and Andreas S Tolias. Three types of incremental learning. Nature Machine Intelligence, 4(12): (225:1188-11) 1185-1197, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems 30, pages 6000-6010, 2017.
Yuanyu Wan and Lijun Zhang. Efficient adaptive online learning via frequent directions. In Pro- ceedings of the 27th International Joint Conference on Artificial Intelligence, pages 2748-2754, 2018.
Yuanyu Wan and Lijun Zhang, Approximate multiplication of sparse matrices with limited space. In Proceedings of the 35th AAAI Conference on Artificial Intelligence, pages 10058-10066, 2021.
Yuanyu Wan and Lijun Zhang. Efficient adaptive online learning via frequent directions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):691-6622, 2022.
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A Comprehensive Survey of Continual Learning: Theory, Method and Application. IEEE Transactions Pattern Analysis and Machine Intelligence, 46(08))5362-538) 2024.
Shipeng Wang, Xiaorong Li, Jian Sun, and Zongben Xu. Training networks in null space of feature covariance for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 184-193, 2021.
Zifeng Wang. Zizhao Zhang. Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Ya Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. European Conference on Computer Vision, pages 631-648, 2022a.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149, 2022b.
Yichen Wu. Hongming Piao, Long-Kai Huang, Renzhen Wang, Wanhua Li, Hanspeter Pfister, Deyu Meng, Kede Ma, and Ying Wei. SD-IoRA: Scalable decoupled low-rank adaptation for class incremental learning. In International Conference Learning Representations, 2025.
Wenhan Xia, Chengwei Qin, and Elad Hazan. Chain of lora: Efficient fine-tuning of language models via residual learning. arXiv preprint arXiv:2401.00 4..40.00111,,11 2024.
Sifan Yang, Yuanyu Wan, Peijia Li, Yibo Wang, Xiao Zhang, Zhewei Wei, and Lijun Zhang. Dimension-free adaptive subgradient methods with frequent directions. In Proceedings of the 42nd International Conference on Machine Learning, pages 71249-71277 2025.
Jaehong Yoon, Funho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference Learning Representations, 2018.
12

---

Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning, pages 3987-3995, 2017.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-effiiiet fine-tuning. In International Con- ference Learning Representations, 2023.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient LLM training by gradient low-rank projection. In Proceedings of the 41st International Conference on Machine Learning, pages 61 6112--61143, 2024.
Da-We Wei Zhou, Hai-Long Sun, Jingyi Ning, Han-Jia Ye, and De-Chuan Zhan. Continual learning with pre-trained models: A survey. In Proceedings of the 33rd International Joint Conference on Artificial Intelligence, pages 8363-8371, 2024a.
Da- Wei Zhou, Qi-Wei Wang. Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Class- incremental learning: A survey, IEEE Transactions Pattern Analysis and Machine Intelligence, 46(12):9851-9873 2024b.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Section & and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Section X
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
Please refer to Section C for the assumptions. Please refer to Appendixes for the complete proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear the main paper or supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer to Section ! and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data also be provided via
often one good way accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Please refer to the supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:)
wwabbbmisssoooooiicy.. for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
Please refer to Section ! and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question:
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: We report the main results with mean and standard deviation, which puted over 3 independent runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Section @ and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The authors have read the NeurIPS Code of Ethics and ensured that our re- search conforms to it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper is about continual learning and does not involve societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g,, deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper is about continual learning and does not have risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards
allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have properly cited all data, code, and models used in this paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, snperswithcodd www.cooccaaaooo datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
If assets are released,
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Question:
Answer:
[NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This research does not involve LLMs as any important, original, or non- standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.M policy Cettps:/nneura http://aaaaaaacecaeeerrrrae///// / for what should or should be described.
20

---

A Proof of Proposition 1
Recall that (G C 1R m is the sequence of projected gradients for task т, A, G.. )T=
A = Ei= C Gr A Qr, Ā -i= Qr, and sketch matrix ST, 1Rmx.2
Because Qr, is the rank 72 approximation of Gr,,, for every step t, we have
|| Acll2 = oi
(14)
where or is the (r2 1) singular value G Using the triangle inequality together with Eq. (LE),
||A Āll
Σ (A. A.)
Σ
A
A.
12
Σ
oz.
(15)
Since use FD to compute Sr, based on {Qr, 10 from Theorem 1.1 of Ghashami al we have
[0]6]]
||A - [A] 7.2
|| S
T
(16)
1'2 k
where [A]a is the minimizer of A [A]xll overall rank k matrices. Applying the triangle inequality to |A - A.,,, S. ST. and substituting Eq. (USS) and (If) gives
||A Sr,r 2|
||A
Álla 1| ST
T
[A] 113B |
(17)
o
- k
1=1
which is exactly (IC)).
B CoSO Algorithm
We present the the detailed procedure in Algorithm M.
C Experimental Setups and Implementation Details
Following existing works [Smith et all 2022% Wu et all 2025|, we adopt VIT-B/16 [Dosovitskly atall [02 tre-trained ImageNet-2IK: and fine-tuned on ImageNet- 1 our backbone model, which consists of 12 transformer blocks. For fair comparison, all methods use the same ViT-B/16 backbone and optimizer. Additionally, we also evaluate self-supervised ViT-B/16 obtained with DINO [Caron et all 2021]. The optimization is performed using Adam [Kingma, 2014] optimizer with 31 0.9 and 3g 0.999. The training epochs vary across datasets: 40 epochs for ImageNet- R. 20 epochs for CIFARI00, and 5 epochs for DomainNet. We maintain consistent batch size of 128 across all experiments. Results are averaged over independent runs, and report the corresponding standard deviation. Notably, CoSO only optimize the output projection layers in multi-head attention module rather than QKV transformations.
We present the detailed hyperparameter settings of CoSO in Table ! These hyperparameters are carefully tuned to balance memory efficiency and performance, reflecting the varying complexity of the datasets. The hyperparameter settings of baseline methods are following existing work [[ang atall 2022a, Smith et all 2023, Liang and Li,, 2024 Lu Luet all, 2024, all 2025]. For all datasets, we employ minimal data augmentation, consisting of random resized cropping to 224 x 224 pixels and random horizontal flipping during training, without any additional augmentation techniques. To prevent overfitting, we followed VPT-NSP² [Lu 20224], setting the temperature parameter in the cross-entropy loss to for all datasets. All experiments were conducted on NVIDIA A6000 GPUs with 48GB memory using PyTorrc 2.
The projection rank (r1) determines the dimensionality of the low-rank subspace for gradient pro- jection. For simpler datasets like CIFAR100, a lower value of rı 15 is sufficient, while more
21

---

Algorithm CoSO for Continual Learning
1: Input: A layer weight matrix W R= step size 7), decay rates B1, 82, projection rank 71, FD rank r2. threshold and update gap K.
2: Initialize first-order moment Mo Rmxr
3: Initialize second-order moment Vo E R +- 0
4: Initialize sketch matrix S.,0 E mx
5: Initialize orthogonal projection matrix Mo t 0
6: for Task ....N do
7: for step / E...T
8:
G f- V Vw..L(W..
Compute mini-batch gradient for task b Orthogonal projection
9:
Gr < G, M,-1 -M--MMM-GGGG
10:
ift mod K = 0 then
11:
UEV = =VD((G,,))
12:
Pr. = U(;;:n] Update Sr, through Eq. (6) and (
Compute projection matrix Pr. Use FD to consolidate gradient information
13:
14:
else
15:
Prt +- Pr,t-
16:
S + Sr,-1
17:
end if
18:
RT, t- PTT.G"
Project orthogonal gradient into low rank space
19:
Use RI,, to compute Nr. through Eq. (
Update R,, by Adam
G.D t PT.
Project gradient back original space
21:
Wr, + 7,,-- - Gr,6
22:
end for
23:
Update the historical subspaces basis matrix My-1 through Eq.
and Table 4: Hyperparameter settings for different datasets.
24:
end for
Hyperparameter
CIFAR100
ImageNet-R
DomainNet
Projection rank (r1)
15
50
70
Frequent directions rank (r2)
100
120
160
Update gap (K)
20
Threshold (Exr)
0.98
0.98
0.98
complex datasets such as ImageNet-R and DomainNet require higher values (r1 50 and rı = 70, respectively) to capture a richer set of gradient directions. The Frequent Directions rank (r2) is con- sistently set higher than Ti across all datasets. This design choice ensures that CoSO can capture broader range of directions, reducing information loss during continual learning. As the dataset complexity increases, 72 is adjusted upward to retain more task information.
The update gap K is adjusted based on the characteristics of each dataset. For DomainNet, we use larger update gap (K 20) due to its larger and more diverse task structure, where frequent updates may become redundant. In contrast, CIFAR100 and ImageNet-R exhibit rapid gradient changes, necessitating smaller K. Finally, the threshold (c) uniformly set 0.98 across all datasets. This value is selected to maintain high retention rate of gradient information within the subspace.
D Analysis of Computational and Memory Costs
We conducted a comparative analysis of CoSO and baseline methods with respect to computational cost (reported as estimated GFLOPs) and memory usage, as summarized in Table EL CoSO requires half the computational cost of prompt- baspt methods (such as L2P, DralPrompt, and CODA-P), as avoids the need for twice forward passes through the network. In terms of memory usage, CoSO is on par with other low-rank adaptation techniques such as InfLoRA (13.44). Its slightly higher memory footprint (13.61) stems from using larger rank for gradient subspace approximation, which enables better capture of task-specific patterns and leads to superior performance. Notably,
22

---

Table 5: Comparison on ImageNet-R (10 Tasks) in terms of computation (GFLOPs) and memory
usage.
Method
GFLOPs
Memory Usage (G)
L2P
70.24
12.90
DualPrompt
70.24
12.96
CODA-P
70.24
12.97
InfL.oRA
35.12
13.44
SD-LoRA
35.12
15.62
VPT-NSP²
35,83
11.54
CoSO
35.12
13.61
Table 6: Results (%) on ImageNet-R R (10 Tasks). All reported results with mean and standard devia- tion are computed over independent runs.
Method (DINO) ImageNet-R (10 Tasks)
ACC10
ACC10
L2P
61.9440.45
68. 18..770.22
DualPrompt
60.4040.18
67.6540.0r
CODA-P
64.63
72.20+00.00
InfLoRA
67.21
76.40+t0.0
SD-LoRA
69.7840.63
65. 63.7340.35
VPT-NSP"
69.68+0.20
77. 77.2440.16
CosO
71.60
79.28
simply increasing the rank for InfLoRA would not yield similar improvements, as its performance limited by the constraint of fixed subspaces. Compared with SD-LoRA (15.62), which ineurs the greatest memory overhead, CoSO offers more efficient alternative while delivering competitive performance. Overall, these results highlight CoSO's ability to strike favorable balance between computational efficiency and memory usage, making it scalable solution for continual learning across diverse tasks.
Additional Experiment Results on ImageNet-R
To further verify CoSO's generality, test self-supervised ViT-B/16 backbone trained with DINO [Caron et all 022 on ImageNet-R (10 Tasks). The results are presented in Table R CoSO outperforms the best baseline method with considerable margin, confirming its ability to generalize across various vision transformers.
23


Reviews:
[{"review_id": "ZYPdJvdnJi", "reviewer": "Reviewer_fJcc", "summary": "This paper proposes a method called CoSO to address the catastrophic forgetting issue of pre-trained models in continual learning. Unlike existing approaches that typically constrain parameter updates within a fixed low-rank subspace, CoSO optimizes the model across a series of continuous, dynamically generated low-rank subspaces derived from gradient singular value decomposition (SVD), thereby enhancing the model's learning capability.", "questions": "1. Appendix D states that due to the non-linearity of Softmax, CoSO is not applied to the QKV transformation layers within the attention mechanism. Could you clarify what proportion of the total fine-tuned parameters are excluded as a result? Furthermore, how would you assess the performance loss incurred by not being able to enforce orthogonality on these critical layers?\n2. Regarding the baseline PEFT methods in the experiments, was data replay employed during fine-tuning? Since data replay is a common technique for mitigating catastrophic forgetting, have the authors considered including a fine-tuning baseline with data replay for a more comprehensive comparison?\n3. Is the current comparison against baseline methods fair? When compared to methods like InfLoRA, how much of CoSO's performance gain can be attributed to its continuous subspace optimization strategy versus its larger number of trainable parameters (given that r2 > r1)? Would you consider conducting additional experiments to compare performance under a similar total parameter budget, for instance, by increasing the rank of InfLoRA to match the effective rank of CoSO?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths:\n1. The paper is exceptionally well-written, logically structured, and easy to comprehend.\n2. Inspired by GaLore, CoSO employs continual learning with continuous subspace optimization to address two key pain points of existing methods. It avoids the model capacity limitations of fixed subspaces and actively prevents forgetting through orthogonal projection, presenting a novel approach.\n\n\n## Weaknesses:\n1. CoSO introduces several key hyperparameters, including projection rank `r1`, FD rank `r2`, update interval `K`, and threshold `ε_th`. As shown in Table 4, these hyperparameters vary significantly across different datasets. Does this suggest that the method requires problem-specific tuning to achieve optimal performance?\n2. Tables 1 and 2 report the method's performance in terms of final task ACC and AVG ACC. These metrics are indirect for assessing knowledge retention. Continual learning typically employs more direct measures like Backward Transfer (BWT). The absence of such metrics makes it difficult to precisely evaluate the effectiveness of mitigating catastrophic forgetting.\n3. To maintain theoretical orthogonality, CoSO restricts its optimization scope to the output projection layer, excluding the QKV layers. Compared to PEFT methods that can update QKV layers, this may limit the model's adaptability and sacrifice potential performance improvements.\n4. The experiments exclusively use datasets from the visual domain, such as ImageNet, the CIFAR series, and COCO for object detection. The study does not include tasks from other modalities, like natural language processing. This limits the method's generalizability within a universal fine-tuning paradigm.", "was_revised": true, "final_justification": "My initial review raised several concerns about the completeness and fairness of the experimental evaluation. The authors have addressed most of these concerns by providing new experimental results.\n\nSpecifically, the addition of the BWT metric, comparisons against replay-based methods, and fair-budget comparisons have strengthened the paper's empirical claims. The remaining limitation—the focus on vision tasks—is minor in comparison and has been properly acknowledged by the authors.\n\nGiven that the key experimental concerns have been addressed, the paper's technical contributions are clearer. I will therefore maintain my positive score.", "rebuttal": "We sincerely appreciate your valuable feedback. We have carefully considered your concerns and provided a detailed explanation below.\n\n---\n**W1.** CoSO introduces several key hyperparameters, including projection rank $r_1$, FD rank $r_2$, update interval $K$, and threshold $\\epsilon_{th}$. These hyperparameters vary significantly across different datasets. Does this suggest that the method requires problem-specific tuning to achieve optimal performance?\n\n**R1.** We appreciate the reviewer's concern about hyperparameter sensitivity. In our experiments,  we fixed $\\epsilon_{\\text{th}}$ = 0.98 across all datasets. This setting is found to be robust in practice. The update interval K determines how frequently the projection and sketch matrices are computed. On larger datasets like DomainNet, we use larger $K=20$ to reduce overhead, while for smaller datasets with more volatile gradients, we use $K=1$ for better performance. Many existing PEFT-based continual learning methods also involve dataset-specific hyperparameters such as prompt length or LoRA rank. We followed the standard practice by tuning all hyperparameters on validation splits, ensuring fairness in comparison.\n\nTo assess the impact of $r_1$ and $r_2$, we conducted sensitivity experiments varying their values on ImageNet-R. The results (final acc and average acc) below indicate that CoSO remains robust across a wide range of rank settings, with only minor performance fluctuations.\n\n| Method          | IMNR-5      | IMNR-10     | IMNR-20     |\n| --------------- | ----------- | ----------- | ----------- |\n| CoSO ($r_1=30$) | 82.17/86.06 | 80.52/85.40 | 77.90/83.03 |\n| CoSO ($r_1=40$) | 82.15/86.13 | 80.23/85.35 | 78.03/83.25 |\n| CoSO ($r_1=50$) | 82.10/86.38 | 81.10/85.56 | 78.19/83.69 |\n| CoSO ($r_1=60$) | 81.93/86.25 | 80.60/85.43 | 78.18/83.75 |\n\n| Method           | IMNR-5      | IMNR-10     | IMNR-20     |\n| ---------------- | ----------- | ----------- | ----------- |\n| CoSO ($r_2=80$)  | 81.63/86.01 | 80.50/85.44 | 78.22/83.62 |\n| CoSO ($r_2=100$) | 81.95/86.10 | 80.50/85.58 | 78.35/83.54 |\n| CoSO ($r_2=120$) | 82.10/86.38 | 81.10/85.56 | 78.19/83.69 |\n| CoSO ($r_2=140$) | 82.38/86.14 | 81.93/85.75 | 77.60/83.15 |\n\n---\n**W2.** Continual learning typically employs more direct measures like Backward Transfer (BWT). The absence of such metrics makes it difficult to precisely evaluate the effectiveness of mitigating catastrophic forgetting.\n\n**R2.** We appreciate the reviewer's suggestion to include more direct metrics such as BWT. Our primary evaluation focuses on final accuracy and average accuracy, which are widely adopted in existing PEFT-based continual learning works. These metrics jointly reflect adaptation and retention, enabling fair comparisons with state-of-the-art methods.\n\nFollowing the reviewer's suggestion, we compute BWT for all methods on ImageNet-R 20 tasks to evaluate the effectiveness of mitigating forgetting. The results below show that CoSO achieves the highest BWT, indicating minimal forgetting.\n\n| Method     | BWT (IMNR-20) |\n| ---------- | ------------- |\n| L2P        | -6.89         |\n| DualPrompt | -4.59         |\n| CODA-P     | -4.50         |\n| InfLoRA    | -7.46         |\n| VPT-NSP    | -7.32         |\n| SD-LoRA    | -6.39         |\n| CoSO       | -3.97         |\n\n---\n**W3.** To maintain theoretical orthogonality, CoSO restricts its optimization scope to the output projection layer, excluding the QKV layers. Compared to PEFT methods that can update QKV layers, this may limit the model's adaptability and sacrifice potential performance improvements.\n\n**R3.** We thank the reviewer for raising this point. Our goal is to achieve parameter-efficient fine-tuning by updating as few parameters as possible while maintaining strong performance. Following this principle, CoSO restricts optimization to the output projection layers and keeps all other parameters (including the QKV layers) frozen. To assess the effectiveness of this selective update strategy, we compare CoSO against baselines such as InfLoRA and SD-LoRA, which fine-tune the QKV layers. CoSO consistently outperforms these methods across multiple benchmarks. This suggests that tuning the projection layers alone is sufficient for effective adaptation, validating our design choice.\n\n---\n**W4.** The experiments exclusively use datasets from the visual domain, such as ImageNet, the CIFAR series, and COCO for object detection. The study does not include tasks from other modalities, like natural language processing. This limits the method's generalizability within a universal fine-tuning paradigm.\n\n**R4.** We thank the reviewer for pointing out the scope of our experimental setup. While our current experiments focus on the visual domain, this choice is intentional to ensure a fair comparison with strong and well-established baselines in vision-based continual learning. Notably, the core mechanisms of CoSO, namely gradient-based subspace projection and orthogonality constraints, are not limited to the visual modality. These mechanisms are inherently general and can be extended to transformers used in NLP. As part of our ongoing work, we are actively exploring extensions to vision-language settings, including the adaptation of CoSO to CLIP-based models. However, such extensions involve non-trivial re-implementations and are beyond the scope of this initial study. We intend to explore this extension in our future research.\n\n---\n\n**Q1.** Appendix D states that due to the non-linearity of Softmax, CoSO is not applied to the QKV transformation layers within the attention mechanism. Could you clarify what proportion of the total fine-tuned parameters are excluded as a result? Furthermore, how would you assess the performance loss incurred by not being able to enforce orthogonality on these critical layers?\n\n**A1.** We appreciate the reviewer's question regarding the exclusion of QKV layers. In ViT-B/16, the QKV layers account for approximately 25% of the total parameters. Theoretically, CoSO can be applied to the remaining 75% of parameters, including projection layers and FFN layers, which still comprise the majority of the transformer's components.\n\nAs discussed in **R3**, updating more parameters does not always yield better results. By freezing most of the model parameters (including the QKV layers), we are able to better preserve and leverage the generalization capabilities of the pre-trained backbone, allowing CoSO to achieve strong performance with relatively few trainable parameters.\n\n---\n**Q2.** Regarding the baseline PEFT methods in the experiments, was data replay employed during fine-tuning? Have the authors considered including a fine-tuning baseline with data replay for a more comprehensive comparison?\n\n**A2.** We thank the reviewer for raising the question regarding data replay. As discussed in Section 3.1, we focus on a rehearsal-free setting in which the model is not allowed to access or store data from previous tasks. This setting is widely considered more challenging, as it forgoes the benefits of revisiting past data. Our setup aligns with recent state-of-the-art PEFT-based continual learning methods. To ensure fair comparisons, all baselines included in our experiments also follow this rehearsal-free protocol.\n\nTo address your concern, we conduct additional experiments on ImageNet-R of MEMO[1], a typical replay-based method. For a fair comparison, we use 40 images per class for replay. In addition, following reviewer ndbX's suggestion, we include RanPAC[2] and EASE[3], two representative methods that adopt prototype-based replay. These additional results further validate the superior performance of CoSO.\n\n| Methods | IMNR-5       | IMNR-10     | IMNR-20     |\n| ------- | ------------ | ----------- | ----------- |\n| MEMO    | 72.18/80.02  | 70.12/79.04 | 67.60/77.98 |\n| EASE    | 77.53/82.51  | 76.95/82.29 | 75.75/81.96 |\n| RanPAC  | 79.54/83.85  | 78.04/83.35 | 75.03/80.98 |\n| CoSO    | 82.10/86.38  | 81.10/85.56 | 78.19/83.69 |\n\n---\n**Q3.** Is the current comparison against baseline methods fair? When compared to methods like InfLoRA, how much of CoSO's performance gain can be attributed to its continuous subspace optimization strategy versus its larger number of trainable parameters (given that r2 > r1)? Would you consider conducting additional experiments to compare performance under a similar total parameter budget, for instance, by increasing the rank of InfLoRA to match the effective rank of CoSO?\n\n**A3.** We appreciate the reviewer's insightful concern regarding fairness in baseline comparisons. We would like to clarify that the projection rank $r_1$ defines the rank of the gradient subspace, and does not add any extra trainable parameters to the model. The FD rank $r_2$ is not used to directly update model weights, but helps construct the task-specific subspace via the FD. The improved performance is stems from CoSO's ability to (1) optimize in multiple evolving subspaces rather than a fixed one, and (2) enforce orthogonality to mitigate task interference.\n\nTo further address your concern, we conduct additional experiments where we increase the rank in InfLoRA to match the rank of CoSO. As discussed in Appendix E,  increasing the rank of InfLoRA would not yield improvements.\n\n| Method            | IMNR-5      | IMNR-10     | IMNR-20     |\n| ----------------- | ----------- | ----------- | ----------- |\n| InfLoRA (rank 50) | 75.45/81.63 | 70.95/79.40 | 64.57/73.89 |\n| InfLoRA (rank 30) | 76.52/82.43 | 72.30/79.87 | 68.98/76.06 |\n| InfLoRA (rank 10) | 77.53/82.24 | 74.43/80.50 | 70.30/77.04 |\n| CoSO              | 82.10/86.38 | 81.10/85.56 | 78.19/83.69 |\n\n---\n**Reference**\n\n[1] A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning. ICLR 2023.\n\n[2] RanPAC: Random Projections and Pre-trained Models for Continual Learning. NeurIPS 2023.\n\n[3] Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning. \nCVPR 2024."}, {"review_id": "j81x07hoA8", "reviewer": "Reviewer_JZv6", "summary": "This method (CoSO) is established on pre-trained models to tackle class-incremental learning problem. To mitigate catastrophic forgetting, Coso updates model parameters in the direction orthogonal to the historical task subspace. To address the limitation of restricted parameter update in a fixed low-rank subspace, Coso fine-tunes the model in a series of low-rank subspaces rather than a single one.", "questions": "See Weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThis work leverages gradient projection to restrict parameters update for class-incremental learning with parameter-efficient finetuning. Experiments validate that CoSO consistently surpasses SOTA methods. \n\nWeaknesses:\n1. Prior works [1][2] leverage gradient orthogonal projection to aid visual prompt tuning, instead of low-rank adaptation, for class-incremental learning. Why does the CoSO outperform them in various continual benchmarks? Is the better performance of CoSO attributed to the utilization of LoRA?\n2. Prior NSP[1] and PGP[2] both are plug-and-play methods, and can be applied to VPT-Seq, L2P, DualPrompt, CLIP, etc. The paper does not discuss whether the CoSO is plug-and-play. \n3. In my opinion, the historical task subspace $M_{t-1}$ should span the representation subspace of all previous tasks, rather than the gradient subspace, in order to make the proposition 1 valid. But in this paper, authors consider $M_{t-1}$ as basis matrix of the gradient subspace.\n\n[1] Visual prompt tuning in null space for continual learning. In NeurIPS, 2024\n\n[2] Prompt Gradient Projection for Continual Learning. In ICLR, 2024.", "was_revised": true, "final_justification": "The core idea of orthogonal gradient projection in this paper is similar to the prior literature PGP, but the specific implementation is distinct. In PGP, the update is performed on prompts since PGP is a prompt-based method. In contrast, this work updates parameters of the linear projection layers in multi-head attention module. Both methods address the class-incremental learning with the help of pre-trained models. I think this paper deserves my final score.", "rebuttal": "We sincerely thank you for your constructive feedback on our paper. We hope our responses adequately address the concerns you raised.\n\n---\n**W1.** Why does the CoSO outperform NSP[1] and PGP[2] in various continual benchmarks? Is the better performance of CoSO attributed to the utilization of LoRA?\n\n**R1.** Thank you for raising this insightful question. While prior works such as NSP[1] and PGP[2] leverage orthogonal gradient projection to enhance prompt-based continual learning, CoSO differs in both its update scope and optimization strategy, leading to consistently stronger performance.\n- Prior methods[1, 2] constrain updates to the additional prompt tokens appended to the transformer, limiting the model's ability for task-specific adaptation. In contrast, CoSO directly updates the projection layers of the pre-trained vision transformer, which significantly expands the learning capacity of the model for continual adaptation.\n- In our baselines, we also include state-of-the-art LoRA-based continual learning methods such as InfLoRA [3] and SD-LoRA [4]. InfLoRA restricts parameter updates to a fixed subspace, which inherently limits flexibility. In contrast, CoSO performs optimization in a series of dynamically evolving gradient subspaces. These subspaces are explicitly enforced to be orthogonal to the historical task subspace, enabling CoSO to continually discover new learning directions while avoiding interference.\n\nTo summarize, CoSO's superior performance stems from the novel design that jointly enables dynamic subspace exploration and orthogonal knowledge preservation, which are absent in prior Prompt-based or LoRA-based approaches.\n\n---\n**W2.** The paper does not discuss whether the CoSO is plug-and-play.\n\n**R2.** Thank you for this valuable suggestion. CoSO is indeed a plug-and-play optimization strategy in continual learning pipelines. Unlike approaches that require model architectural augmentation (e.g., prompts or adapter), CoSO operates solely on the optimization procedure. It only requires access to the gradients of the parameters being optimized and performs low-rank projections and orthogonalization during the update. Therefore, CoSO can be directly integrated with other methods (e.g., prototype-based) that update a subset of backbone parameters without changing the model architecture. We thank the reviewer for encouraging us to make this clearer in the manuscript.\n\n---\n**W3.** The historical task subspace $\\mathcal{M}_{\\tau-1}$ should span the representation subspace of all previous tasks, rather than the gradient subspace.\n\n**R3.** Thank you for this insightful question. We assume that both the feature and gradient subspaces lie in the same space and capture similar task-specific update directions. Working in the gradient subspace allows for a seamless integration with our low-rank projection-based optimization strategy. Specifically, we can directly apply orthogonality constraints to parameter updates without requiring access to intermediate feature representations. This effectively prevents interference by ensuring that current gradient updates are orthogonal to those from previous tasks. As a result, the risk of increasing the loss on prior tasks is minimized, thereby preserving previously learned knowledge.\n\nWe will revise Section 3.2 to make the above rationale clearer. We thank the reviewer again for raising this point.\n\n---\n**Reference**\n\n[1] Visual prompt tuning in null space for continual learning. NeurIPS, 2024\n\n[2] Prompt Gradient Projection for Continual Learning. ICLR, 2024.\n\n[3] InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning. CVPR 2024.\n\n[4] SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning. ICLR 2025"}, {"review_id": "c9GL6KD2WX", "reviewer": "Reviewer_ndbX", "summary": "This paper introduces CoSO, a continual learning method that leverages low-rank gradient subspaces to fine-tune pre-trained models. It dynamically adapts a model by optimizing within a series of low-rank subspaces. These subspaces are derived from gradients via SVD and are kept orthogonal to historical task subspace to reduce interference. CoSO incrementally updates a task-specific subspace using the Frequent Directions. Experiments on benchmark datasets show that CoSO outperforms state-of-the-art methods.", "questions": "- Proposition 1 states that $G^\\prime$ lies in the null space of $X$. However, in practice, $W$ is updated using $\\tilde{G}$ in Eq.(7). Does Eq.(7) still satisfy Proposition 1? To avoid confusion, the authors should provide a clear explanation.\n\n- Why does CoSO restrict updates to the linear projection layers and not to FFN parameters? Could this approach be extended to update the MLP layers in FFN as well?\n\n- Could CoSO be applied to non-vision tasks like NLP? If so, what modifications would be needed for it to work effectively?\n\n- Could CoSO be combined with other continual learning approaches, such as regularization-based or prototype-based?", "limitations": "Please refer to \"Strengths And Weaknesses\" and \"Questions\" above.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- The proposed CoSO extends low-rank adaptation by using dynamically evolving subspaces, allowing updates across multiple orthogonal subspaces. This design increases the model’s learning capacity, and the orthogonality explicitly reduces interference with previously learned knowledge.\n\n- The authors incorporate Frequent Directions to incrementally capture and compress gradient information into a low-rank task-specific subspace. This strategy effectively reduces memory and computational costs while preserving the most significant update directions for each task.\n\n- Comprehensive experiments across multiple datasets demonstrate that CoSO consistently outperforms baseline methods. The well-executed ablation studies effectively isolate the contributions of its key components (orthogonalization and FD).\n\n\n### Weaknesses\n- CoSO seems to involve quite a few hyperparameters. Specifically, $r_1$ and $r_2$ in particular appear very similar. Could the authors explain the specific difference between them, and whether they could potentially be merged into a single parameter?\n\n- The baseline methods are mainly prompt-based and LoRA-based. It would be helpful if the authors include comparisons with a broader range of recent studies, such as RanPAC [1*], EASE [2*], etc.\n\nReference:\n\n[1*] RanPAC: Random Projections and Pre-trained Models for Continual Learning. NeurIPS 2023.\n\n[2*] Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning. CVPR 2024.", "was_revised": true, "final_justification": "The authors have provided a detailed and satisfactory response to my previous comments. Since my primary concerns have been adequately addressed, I confirm my positive rating of this manuscript.", "rebuttal": "We sincerely appreciate your thoughtful and encouraging feedback on our work. Below, we provide a detailed response to your concern.\n\n---\n**W1.** Explain the specific difference between $r_1$ and $r_2$, and whether they could potentially be merged into a single parameter?\n\n**R1.**  Thank you for the insightful question regarding $r_1$ and $r_2$. We fully agree that a model should ideally involve as few hyperparameters as possible to promote simplicity and ease of tuning. While these two parameters may appear similar, they are designed to serve different purposes.\n\nThe projection rank $r_1$ is used during the optimization process to control the dimensionality of the gradient subspace in which parameter updates are performed. A smaller $r_1$ ensures that the projected gradient is memory-efficient and computationally light, which is especially critical during training.\n\nOn the other hand, the Frequent Directions rank $r_2$ is used to construct the task-specific subspace by consolidating gradient directions across training steps. This subspace serves as a compact summary of the current task's critical update directions and is appended to the historical task subspace to preserve prior knowledge. To achieve a more accurate estimation of the task-specific gradient subspace, $r_2$ is set larger than $r_1$, which allows it to capture a broader spectrum of gradient variation over time. \n\nYes, we can merge $r_1$ and $r_2$ into a single parameter. But we find that tuning them separately leads to better performance in practice. A relatively larger $r_2$ becomes important when $r_1$ is small, as it enables better approximation of the task-specific subspace.\n\n---\n**W2.** The baselines are limited to Prompt-based and LoRA-based methods. It would be helpful to include comparisons with recent studies, such as RanPAC and EASE.\n\n**R2.** Thank you for the valuable suggestion. Following your suggestion, we conducted additional experiments on ImageNet-R using the official implementations and matched settings. The results further demonstrate the superior performance of CoSO across diverse continual learning paradigms.\n\n| Methods | IMNR-5 | IMNR-10  | IMNR-20 |\n| ------- | ----------- | ------------ | ------------ |\n| EASE    | 77.53/82.51 | 76.95/82.29  | 75.75/81.96  |\n| RanPAC  | 79.54/83.85 | 78.04/83.35  | 75.03/80.98  |\n| CoSO    | 82.10/86.38 | 81.10/85.56  | 78.19/83.69  |\n\nOur study focuses primarily on continual learning with pre-trained vision transformers under the parameter-efficient fine-tuning (PEFT) paradigm. The baseline methods we originally included are state-of-the-art PEFT-based approaches widely adopted in recent literature, ensuring a fair and relevant comparison in this specific setting. While RanPAC and EASE are valuable prototype-based methods that operate in the feature space via class prototypes, they follow a different paradigm from PEFT-based approaches.\n\n---\n**Q1.** Does Eq.(7) still satisfy Proposition 1?\n\n**A1.** Thank you for the thoughtful question. Yes, Equation (7) still satisfies Proposition 1. In CoSO, we first project the raw gradient $G_{\\tau,t}$ into the null space of the historical task subspace, yielding $G_{\\tau,t}^{\\prime}$. All subsequent steps in Equation (7), including low-rank projection, Adam update, and projection back, operate within the subspace spanned by $G_{\\tau,t}^{\\prime}$. As a result, the final update $\\tilde{G}_{\\tau,t}$ still satisfies Proposition 1.\n\n---\n**Q2.** Why does CoSO restrict updates to the linear projection layers and not to FFN parameters? Could this approach be extended to update the MLP layers in FFN as well?\n\n**A2.** Thank you for this question. Our goal is to achieve parameter-efficient fine-tuning by updating as few parameters as possible while maintaining strong performance. Following this principle, CoSO restricts optimization to the output projection layers and keeps all other parameters (including the FFN layers) frozen. By focusing on the projection layers, we achieve effective continual adaptation while significantly reducing computational and memory overhead. Theoretically, CoSO can be extended to other modules such as FFN layers. \n\n---\n**Q3.** Could CoSO be applied to non-vision tasks like NLP? If so, what modifications would be needed for it to work effectively?\n\n**A3.** Thank you for this interesting question. While our current work focuses on vision tasks using pre-trained vision transformers, the idea underlying CoSO, which involves projecting gradients into continuous subspaces and enforcing orthogonality across tasks, is model-agnostic and applicable to standard transformer architectures. Consequently, the algorithm can be directly adapted to popular NLP models such as LLaMA by targeting the corresponding projection layers, where gradient-based subspace adaptation remains effective. This presents a promising direction for cross-modal continual learning. We intend to explore this extension in our future research.\n\n---\n**Q4.** Could CoSO be combined with other continual learning approaches, such as regularization-based or prototype-based?\n\n**A4.** Thank you for the insightful question. CoSO is designed as a subspace-based optimization framework that operates at the level of parameter gradients and explicitly enforces orthogonality between tasks. Due to this design, it naturally overlaps with the objective of many regularization-based methods (e.g., EWC), which also aim to prevent interference through constrained updates. As a result, combining CoSO with regularization-based approaches may offer limited additional benefit.\n\nIn contrast, prototype-based methods operate in the feature space and are conceptually orthogonal to CoSO. These methods, which store and use class prototypes to facilitate classification and knowledge retention, can be naturally integrated with CoSO by replacing or augmenting the classifier head while retaining CoSO for backbone optimization. This hybridization can potentially improve performance by leveraging both parameter-space and feature-space strategies."}, {"review_id": "phvgZIcRyL", "reviewer": "Reviewer_HjST", "summary": "The authors propose CoSO, a continual learning method for adapting foundational vision models. It fine-tunes pretrained models within continuous subspaces derived from the SVD of gradients. To mitigate interference, the method enforces orthogonality between the optimization subspace of the current task and those of previous tasks. After completing a task, its task-specific component is accumulated to form a historical subspace, which is used in subsequent tasks. Experiments on several continual learning benchmarks demonstrate that CoSO outperforms recent baselines such as SD-LoRA.", "questions": "1. Please clarify why strict orthogonality (i.e., no interference) is necessary. Could shared representations benefit learning in some cases?\n2. Please elaborate on how CoSO differs from SD-LoRA in design, efficiency, and empirical performance.\n3. Please highlight the technical novelty of CoSO beyond combining known techniques.\n4. Please include more detailed evaluation to show how well CoSO mitigates forgetting and addresses stability-plasticity.\n5. SD-LoRA includes insightful empirical analyses, such as demonstrating the importance of previously learned directions for the current task. Could the authors provide similar analyses to show the role of historical subspaces in CoSO?", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. Continual learning on foundational models is an emerging and important research area with practical significance, yet it remains underexplored.\n2. The algorithm is simple yet effective and grounded in solid theoretical analyses (Propositions 1 and 2).\n3. CoSO shows strong empirical performance, outperforming recent baselines such as SD-LoRA on vision tasks.\n\n**Weaknesses:**\n\n1. While enforcing orthogonality between task-specific subspaces reduces interference, it is unclear why complete separation is necessary. Is it always desirable for new task updates to be entirely independent of prior knowledge? Could past knowledge be beneficial for the current task? If strict independence is required, why not simply train a separate model per task?\n2. CoSO and SD-LoRA appear to share several similarities. A more detailed comparison is needed, including algorithmic rationale and computational efficiency. For instance, CoSO (1) assigns a distinct subspace to each task and (2) leverages previously learned subspaces during training—both also present in SD-LoRA through task-specific LoRA adapters. Clarification on how CoSO differs and what drives its empirical improvements would be valuable.\n3. The method builds on existing techniques such as SVD, GaLore, and FD, with orthogonality as the key addition for continual learning. A more thorough explanation of its technical novelty would strengthen the paper.\n4. All evaluation metrics are averaged over current and past tasks. More fine-grained analysis would help assess the method’s ability to mitigate forgetting and maintain the stability-plasticity trade-off.", "was_revised": true, "final_justification": "The authors clarify that the orthogonality constraint in CoSO is not meant to enforce complete independence from past knowledge, but rather to mitigate harmful gradient interference during optimization. They also clarify the distinctions between CoSO and SD-LoRA, both of which are for continual learning. The authors then provide additional experimental results on balancing stability and plasticity by showing Backward Transfer (BWT). Thus, I maintain my positive score.", "rebuttal": "We sincerely appreciate your constructive feedback. We have carefully considered your concerns and our responses are provided as follows.\n\n---\n**W1 & Q1.** It is unclear why complete separation is necessary. Is it always desirable for new task updates to be entirely independent of prior knowledge? Could past knowledge be beneficial for the current task? If strict independence is required, why not simply train a separate model per task? Please clarify why strict orthogonality (i.e., no interference) is necessary. Could shared representations benefit learning in some cases?\n\n**A1.** We thank the reviewer for raising this important point regarding the necessity of orthogonal subspaces. We clarify that the orthogonality constraint in CoSO is not meant to enforce complete independence from past knowledge, but rather to mitigate harmful gradient interference during optimization. By projecting the current task's gradients orthogonally to the subspaces important for previous tasks, CoSO prevents overwriting past knowledge while still allowing parameter updates in complementary directions. \n\nNotably, the orthogonality constraint is applied only during optimization. CoSO maintains a single shared pre-trained model throughout the entire training process. During inference phase, all model parameters jointly contribute, ensuring that useful representations from earlier tasks are retained and leveraged in future predictions. Therefore, past knowledge can benefit the current task.\n\nWe note that our setting follows the task-agnostic inference protocol, where the task label is unavailable at test time. In this setting, even if a separate model were trained per task, it would be infeasible to determine which model to use at inference. In contrast, CoSO maintains a single unified model that achieves strong performance without requiring any task labels at test time, demonstrating its practical effectiveness and scalability under realistic continual learning scenarios.\n\n---\n**W2 & Q2.** CoSO and SD-LoRA appear to share several similarities. A more detailed comparison is needed, including algorithmic rationale and computational efficiency. Please elaborate on how CoSO differs from SD-LoRA in design, efficiency, and empirical performance.\n\n**A2.**  We appreciate the reviewer's insightful observation and agree that both CoSO and SD-LoRA share the common goal of continual learning through parameter-efficient fine-tuning. However, the two approaches differ in both design and implementation, as clarified below:\n\n**(1) Method Design:**\nSD-LoRA assigns a separate LoRA adapter to each task, where each adapter comprises a pair of low-rank matrices. While SD-LoRA decouples magnitude and direction to enable flexible adaptation, it does not explicitly prevent task interference. In contrast, CoSO avoids task-specific modules by optimizing a shared backbone. To mitigate interference, CoSO enforces gradient orthogonality, ensuring updates remain orthogonal to prior task subspaces.\n\n**(2) Memory Efficiency:**\nCoSO leverages gradient-based low-rank projection to optimize a single shared backbone, enabling memory-efficient training without maintaining multiple sets of adapter weights. This optimization strategy ensures that the GPU memory consumption remains constant, regardless of the number of tasks. In contrast, SD-LoRA must retain all previously learned adapters in memory during training to prevent forgetting, resulting in the GPU memory usage that grows linearly with the number of tasks. This becomes a bottleneck when handling long task sequences. As shown in Appendix E (Table 5), CoSO consumes less memory than SD-LoRA, demonstrating better scalability in both theory and practice.\n\n**(3) Empirical Performance:**\nCoSO consistently outperforms SD-LoRA across all benchmarks. For example, on ImageNet-R (20 tasks), CoSO improves final accuracy by 2.93% and average accuracy by 3.47% (Table 1). These results highlight the effectiveness of gradient space orthogonalization and dynamic subspace composition in enhancing knowledge retention and adaptation.\n\nIn summary, CoSO differs from SD-LoRA by (i) using online gradient-derived subspaces rather than static adapters, (ii) explicitly preventing forgetting via orthogonal projection, and (iii) requiring less memory. We will revise the paper to make these distinctions clearer.\n\n---\n**W3 & Q3.** The method builds on existing techniques such as SVD, GaLore, and FD, with orthogonality as the key addition for continual learning. A more thorough explanation of its technical novelty would strengthen the paper. Please highlight the technical novelty of CoSO beyond combining known techniques.\n\n**A3.** We thank the reviewer for raising this important point. We would like to clarify the high-level motivation behind CoSO and its key technical contributions.\n\nRecent continual learning methods with pre-trained models often follow a common paradigm: adapting a fixed subset of parameters (e.g., prompts or low-rank adapters) to mitigate forgetting while preserving generalization ability of foundation models. Among them, LoRA-based approaches (e.g., InfLoRA) constrain updates within a fixed low-rank subspace. While this reduces interference, it also limits the model's learning capacity.\n\nOur work takes a novel perspective: instead of introducing prompts or adapters, we tackle the limitation of learning capacity by rethinking the optimization space. Our motivation is to fully exploit the expressive potential of pre-trained models by approximating full-rank fine-tuning in a parameter-efficient way, without modifying the model architecture. To this end, we propose to optimize in a sequence of evolving subspaces following GaLore, each derived from current gradients via truncated SVD. This dynamic subspace formulation allows CoSO to retain adaptation flexibility while remaining memory efficient.\n\nHowever, unconstrained subspace evolution across tasks leads to severe forgetting. To address this, we introduce a gradient orthogonality constraint that ensures updates for each task do not interfere with learned knowledge. A major technical challenge is that each task comprises multiple subspaces across training steps. To make orthogonalization feasible, we aggregate these local subspaces into a compact component using Frequent Directions, which allows us to track only the essential directions unique to each task.\n\nWe would like to emphasize that CoSO is not a simple combination of SVD, GaLore, and FD. Each component plays a distinct role in addressing specific challenges of continual learning. Together, they form a cohesive framework that improves both learning capacity and knowledge retention. As a result, CoSO consistently outperforms existing Prompt-based and LoRA-based methods across benchmarks.\n\n---\n**W4 & Q4.** More fine-grained analysis would help assess the method's ability to mitigate forgetting and maintain the stability-plasticity trade-off. Please include more detailed evaluation to show how well CoSO mitigates forgetting and addresses stability-plasticity.\n\n**A4.** We appreciate the reviewer's suggestion to include more detailed evaluation. Our primary evaluation focuses on final task accuracy and average accuracy, which are widely adopted in existing PEFT-based continual learning works. These metrics provide a comprehensive view of both adaptation (learning new tasks) and retention (preserving previous knowledge), and allow for direct comparison with prior state-of-the-art methods.\n\nFollowing the suggestion of reviewer fJcc, we have computed Backward Transfer (BWT) for all methods on ImageNet-R 20 tasks to further evaluate the effectiveness of mitigating catastrophic forgetting. The results below show that CoSO not only achieves the highest final and average accuracy, but also obtains the highest BWT (closest to zero), indicating minimal forgetting.\n\n| Method     | IMNR-20     | BWT (IMNR-20) |\n| ---------- | ----------- | ------------- |\n| L2P        | 58.64/65.57 | -6.89         |\n| DualPrompt | 60.47/65.91 | -4.59         |\n| CODA-P     | 67.16/72.34 | -4.50         |\n| InfLoRA    | 70.30/77.04 | -7.46         |\n| VPT-NSP    | 75.42/81.32 | -7.32         |\n| SD-LoRA    | 75.26/80.22 | -6.39         |\n| CoSO       | 78.19/83.69 | -3.97         |\n\n---\n**Q5.** SD-LoRA includes insightful empirical analyses, such as demonstrating the importance of previously learned directions for the current task. Could the authors provide similar analyses to show the role of historical subspaces in CoSO?\n\n**A5.** We thank the reviewer for this insightful point. SD-LoRA retains all task-specific adapters and is therefore able to directly analyze the importance of previously learned adapters by reusing them during training. In contrast, CoSO does not store the actual parameter updates or direction vectors, preserving only the historical subspace constructed via sketching. While this difference prevents us from conducting similar analysis as in SD-LoRA, we can design experiments to assess the contribution of the historical subspace. \n\nSpecifically, our ablation study removes the orthogonality constraint, allowing updates to overlap with past subspaces. The results show a substantial performance drop (e.g., −8.52% final accuracy in the 20-task setting), indicating that the historical subspace plays a crucial role in preserving past knowledge. This result strongly suggests that the accumulated subspace effectively mitigates interference."}]

Author rebuttal:

Meta-review: This paper presents a new continual learning method named Continuous Subspace Optimization (CSO), which maintains and updates a subspace representation for each task using online PCA. The method aims to preserve past knowledge by minimizing interference with previously learned subspaces and by using regularization strategies that guide optimization toward underexplored parameter directions. Empirical results show improved forgetting and stability metrics across standard continual learning benchmarks. The paper is clearly written, the method is simple and modular, and the ablation studies are generally well executed.

Despite the overall positive reception, reviewers raised concerns about novelty and depth: subspace-based methods are not new, and the proposed update mechanism, while efficient, appears incremental. Furthermore, the theoretical motivation is limited, and the analysis of interference and orthogonality is mostly empirical. The rebuttal clarified the conceptual distinctions from prior work, and reviewers acknowledged the method's practicality and performance. Given the competitive results, strong presentation, and ease of implementation, I recommend acceptance, though not at the level of spotlight.
Decision: Accept (poster)