Title: Continuous Subspace Optimization for Continual Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Subspace Optimization

Continual Learning
Abstract: Continual learning aims to learn multiple tasks sequentially while preserving prior knowledge, but faces the challenge of catastrophic forgetting when adapting to new tasks. Recently, approaches leveraging pre-trained models have gained increasing popularity in mitigating this issue, due to the strong generalization ability of foundation models. To adjust pre-trained models for new tasks, existing methods usually employ low-rank adaptation, which restricts parameter updates to a fixed low-rank subspace. However, constraining the optimization space inherently compromises the model's learning capacity, resulting in inferior performance. To address this limitation, we propose Continuous Subspace Optimization for Continual Learning (CoSO) to fine-tune the model in a series of subspaces rather than a single one. These sequential subspaces are dynamically determined through the singular value decomposition of the gradients. CoSO updates the model by projecting gradients onto these subspaces, ensuring memory-efficient optimization. To mitigate forgetting, the optimization subspace of each task is constrained to be orthogonal to the historical task subspace. During task learning, CoSO maintains a task-specific component that captures the critical update directions for the current task. Upon completing a task, this component is used to update the historical task subspace, laying the groundwork for subsequent learning. Extensive experiments on multiple datasets demonstrate that CoSO significantly outperforms state-of-the-art methods, especially in challenging scenarios with long task sequences.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose CoSO, a continual learning method for adapting foundational vision models. It fine-tunes pretrained models within continuous subspaces derived from the SVD of gradients. To mitigate interference, the method enforces orthogonality between the optimization subspace of the current task and those of previous tasks. After completing a task, its task-specific component is accumulated to form a historical subspace, which is used in subsequent tasks. Experiments on several continual learning benchmarks demonstrate that CoSO outperforms recent baselines such as SD-LoRA.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. Continual learning on foundational models is an emerging and important research area with practical significance, yet it remains underexplored.\n2. The algorithm is simple yet effective and grounded in solid theoretical analyses (Propositions 1 and 2).\n3. CoSO shows strong empirical performance, outperforming recent baselines such as SD-LoRA on vision tasks.\n\n**Weaknesses:**\n\n1. While enforcing orthogonality between task-specific subspaces reduces interference, it is unclear why complete separation is necessary. Is it always desirable for new task updates to be entirely independent of prior knowledge? Could past knowledge be beneficial for the current task? If strict independence is required, why not simply train a separate model per task?\n2. CoSO and SD-LoRA appear to share several similarities. A more detailed comparison is needed, including algorithmic rationale and computational efficiency. For instance, CoSO (1) assigns a distinct subspace to each task and (2) leverages previously learned subspaces during training—both also present in SD-LoRA through task-specific LoRA adapters. Clarification on how CoSO differs and what drives its empirical improvements would be valuable.\n3. The method builds on existing techniques such as SVD, GaLore, and FD, with orthogonality as the key addition for continual learning. A more thorough explanation of its technical novelty would strengthen the paper.\n4. All evaluation metrics are averaged over current and past tasks. More fine-grained analysis would help assess the method’s ability to mitigate forgetting and maintain the stability-plasticity trade-off.\n\nQUESTIONS:\n1. Please clarify why strict orthogonality (i.e., no interference) is necessary. Could shared representations benefit learning in some cases?\n2. Please elaborate on how CoSO differs from SD-LoRA in design, efficiency, and empirical performance.\n3. Please highlight the technical novelty of CoSO beyond combining known techniques.\n4. Please include more detailed evaluation to show how well CoSO mitigates forgetting and addresses stability-plasticity.\n5. SD-LoRA includes insightful empirical analyses, such as demonstrating the importance of previously learned directions for the current task. Could the authors provide similar analyses to show the role of historical subspaces in CoSO?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CoSO, a continual learning method that leverages low-rank gradient subspaces to fine-tune pre-trained models. It dynamically adapts a model by optimizing within a series of low-rank subspaces. These subspaces are derived from gradients via SVD and are kept orthogonal to historical task subspace to reduce interference. CoSO incrementally updates a task-specific subspace using the Frequent Directions. Experiments on benchmark datasets show that CoSO outperforms state-of-the-art methods.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The proposed CoSO extends low-rank adaptation by using dynamically evolving subspaces, allowing updates across multiple orthogonal subspaces. This design increases the model’s learning capacity, and the orthogonality explicitly reduces interference with previously learned knowledge.\n\n- The authors incorporate Frequent Directions to incrementally capture and compress gradient information into a low-rank task-specific subspace. This strategy effectively reduces memory and computational costs while preserving the most significant update directions for each task.\n\n- Comprehensive experiments across multiple datasets demonstrate that CoSO consistently outperforms baseline methods. The well-executed ablation studies effectively isolate the contributions of its key components (orthogonalization and FD).\n\n\n### Weaknesses\n- CoSO seems to involve quite a few hyperparameters. Specifically, $r_1$ and $r_2$ in particular appear very similar. Could the authors explain the specific difference between them, and whether they could potentially be merged into a single parameter?\n\n- The baseline methods are mainly prompt-based and LoRA-based. It would be helpful if the authors include comparisons with a broader range of recent studies, such as RanPAC [1*], EASE [2*], etc.\n\nReference:\n\n[1*] RanPAC: Random Projections and Pre-trained Models for Continual Learning. NeurIPS 2023.\n\n[2*] Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning. CVPR 2024.\n\nQUESTIONS:\n- Proposition 1 states that $G^\\prime$ lies in the null space of $X$. However, in practice, $W$ is updated using $\\tilde{G}$ in Eq.(7). Does Eq.(7) still satisfy Proposition 1? To avoid confusion, the authors should provide a clear explanation.\n\n- Why does CoSO restrict updates to the linear projection layers and not to FFN parameters? Could this approach be extended to update the MLP layers in FFN as well?\n\n- Could CoSO be applied to non-vision tasks like NLP? If so, what modifications would be needed for it to work effectively?\n\n- Could CoSO be combined with other continual learning approaches, such as regularization-based or prototype-based?\n\nLIMITATIONS:\nPlease refer to \"Strengths And Weaknesses\" and \"Questions\" above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This method (CoSO) is established on pre-trained models to tackle class-incremental learning problem. To mitigate catastrophic forgetting, Coso updates model parameters in the direction orthogonal to the historical task subspace. To address the limitation of restricted parameter update in a fixed low-rank subspace, Coso fine-tunes the model in a series of low-rank subspaces rather than a single one.

STRENGTHS AND WEAKNESSES:
Strengths:
This work leverages gradient projection to restrict parameters update for class-incremental learning with parameter-efficient finetuning. Experiments validate that CoSO consistently surpasses SOTA methods. 

Weaknesses:
1. Prior works [1][2] leverage gradient orthogonal projection to aid visual prompt tuning, instead of low-rank adaptation, for class-incremental learning. Why does the CoSO outperform them in various continual benchmarks? Is the better performance of CoSO attributed to the utilization of LoRA?
2. Prior NSP[1] and PGP[2] both are plug-and-play methods, and can be applied to VPT-Seq, L2P, DualPrompt, CLIP, etc. The paper does not discuss whether the CoSO is plug-and-play. 
3. In my opinion, the historical task subspace $M_{t-1}$ should span the representation subspace of all previous tasks, rather than the gradient subspace, in order to make the proposition 1 valid. But in this paper, authors consider $M_{t-1}$ as basis matrix of the gradient subspace.

[1] Visual prompt tuning in null space for continual learning. In NeurIPS, 2024

[2] Prompt Gradient Projection for Continual Learning. In ICLR, 2024.

QUESTIONS:
See Weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method called CoSO to address the catastrophic forgetting issue of pre-trained models in continual learning. Unlike existing approaches that typically constrain parameter updates within a fixed low-rank subspace, CoSO optimizes the model across a series of continuous, dynamically generated low-rank subspaces derived from gradient singular value decomposition (SVD), thereby enhancing the model's learning capability.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n1. The paper is exceptionally well-written, logically structured, and easy to comprehend.\n2. Inspired by GaLore, CoSO employs continual learning with continuous subspace optimization to address two key pain points of existing methods. It avoids the model capacity limitations of fixed subspaces and actively prevents forgetting through orthogonal projection, presenting a novel approach.\n\n\n## Weaknesses:\n1. CoSO introduces several key hyperparameters, including projection rank `r1`, FD rank `r2`, update interval `K`, and threshold `ε_th`. As shown in Table 4, these hyperparameters vary significantly across different datasets. Does this suggest that the method requires problem-specific tuning to achieve optimal performance?\n2. Tables 1 and 2 report the method's performance in terms of final task ACC and AVG ACC. These metrics are indirect for assessing knowledge retention. Continual learning typically employs more direct measures like Backward Transfer (BWT). The absence of such metrics makes it difficult to precisely evaluate the effectiveness of mitigating catastrophic forgetting.\n3. To maintain theoretical orthogonality, CoSO restricts its optimization scope to the output projection layer, excluding the QKV layers. Compared to PEFT methods that can update QKV layers, this may limit the model's adaptability and sacrifice potential performance improvements.\n4. The experiments exclusively use datasets from the visual domain, such as ImageNet, the CIFAR series, and COCO for object detection. The study does not include tasks from other modalities, like natural language processing. This limits the method's generalizability within a universal fine-tuning paradigm.\n\nQUESTIONS:\n1. Appendix D states that due to the non-linearity of Softmax, CoSO is not applied to the QKV transformation layers within the attention mechanism. Could you clarify what proportion of the total fine-tuned parameters are excluded as a result? Furthermore, how would you assess the performance loss incurred by not being able to enforce orthogonality on these critical layers?\n2. Regarding the baseline PEFT methods in the experiments, was data replay employed during fine-tuning? Since data replay is a common technique for mitigating catastrophic forgetting, have the authors considered including a fine-tuning baseline with data replay for a more comprehensive comparison?\n3. Is the current comparison against baseline methods fair? When compared to methods like InfLoRA, how much of CoSO's performance gain can be attributed to its continuous subspace optimization strategy versus its larger number of trainable parameters (given that r2 > r1)? Would you consider conducting additional experiments to compare performance under a similar total parameter budget, for instance, by increasing the rank of InfLoRA to match the effective rank of CoSO?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-06"}