Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learning
Authors: Nikhil Shivakumar Nayak
Krishnateja Killamsetty
Ligong Han
Abhishek Bhandwaldar
Prateek Chanda
Kai Xu
Oleg Silkin
Mustafa Eyceoz
Hao Wang
Aldo Pareja
Akash Srivastava
Venue: ICLR 2026 Poster (2026)
Keywords: Continual Learning
Parameter-Efficient Fine-Tuning
Full Fine-Tuning
Catastrophic Forgetting
Singular Value Decomposition
Geometric Constraints
Orthogonal Subspaces
Low-Rank Subspaces
Constrained Optimization
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a constrained fine-tuning method for continual learning in LLMs using SVD and effective rank to guide updates in subspaces spanned by low singular vectors, significantly reducing catastrophic forgetting and outperforming SOTA methods.

Abstract: Continual learning in large language models (LLMs) is prone to catastrophic forgetting, where adapting to new tasks significantly degrades performance on previously learned ones. Existing parameter-efficient methods often limit model expressivity or introduce new parameters per task, creating scalability issues. To address these limitations, we introduce **Orthogonal Subspace Fine-Tuning (OSFT)**, a novel parameter-efficient approach for continual learning. OSFT leverages adaptive singular value decomposition (SVD) to dynamically identify and preserve critical, high-rank parameter subspaces that encode prior knowledge. All updates for new tasks are constrained to be strictly orthogonal to these preserved subspaces, which minimizes interference while maintaining a fixed parameter count and avoiding the need to store task-specific gradients. We extensively evaluate OSFT on standard continual learning benchmarks using both encoder-decoder (T5-Large) and decoder-only (LLaMA-2 7B, Mistral-7B) models across diverse tasks. Empirically, our method achieves a state-of-the-art trade-off between learnability and knowledge retention, dominating the Pareto frontier, with **up to 7\% higher** average accuracy than recent baselines like O-LoRA, and **reduces forgetting to near-negligible levels**. It notably maintains the model's general linguistic capabilities, instruction-following, and safety throughout the learning process. OSFT provides a practical, theoretically grounded, and scalable solution that effectively balances model plasticity and knowledge retention for continual learning in LLMs. Code is available at https://github.com/Red-Hat-AI-Innovation-Team/mini_trainer.

Published as a conference paper at ICLR 2026
SCULPTING SUBSPACES: CONSTRAINED FULL FINE- TUNING IN LLMS FOR CONTINUAL LEARNING
Nikhil Shivakumar Nayak 1,5, Krishnateja Killamsetty'. Ligong Han
Abhishek Bhandwaldar Prateek Chanda', Kai Xu 15, Oleg Silkin Mustafa Eyceoz', Hao Wang Aldo Parejal 1,5, Akash Srivastava 4.5
Red Hat AT Innovation IBM Research
3 IIT Bombay
*Core AI, IBM
MIT-IBM Watson AI Lab
ABSTRACT
Continual learning in large language models (LLMs) is prone to catastrophic for- getting, where adapting to new tasks significantly degrades performance on pre- viously learned ones. Existing parameter-efficient methods often limit model expressivity or introduce new parameters per task, creating scalability issues. To address these limitations, we introduce Orthogonal Subspace Fine-Tuning (OSFT), a novel parameter-efficient approach for continual learning. OSFT lever- ages adaptive singular value decomposition (SVD) to dynamically identify and preserve critical, high-rank parameter subspaces that encode prior knowledge. All updates for new tasks are constrained to be strictly orthogonal to these pre- served subspaces, which minimizes interference while maintaining a fixed param- eter count and avoiding the need to store task- gradients. We extensively evaluate OSFT on standard continual learning benchmarks using both encoder- decoder (T5-Large) and decoder-only (LLaMA-2 7B, Mistral-7B) models across diverse tasks. Empirically, our method achieves a state-of-the-art trade-off be- tween learnability and knowledge retention, dominating the Pareto frontier, with up to 7% higher average accuracy than recent baselines like O-LoRA, and re- duces forgetting to near-negligible levels. It notably maintains the model's general linguistic capabilities, instruction- -rrrucctoon-olloiingg and safety throughout the learning process. OSFT provides practical, theoretically grounded, and seal- able solution that effectively balances model plasticity and knowledge retention for continual learning in LLMSs. Code is available at /tttts::/githii com/ Innovatoovaiiinen Team/ Team/mini. mini trainer
INTRODUCTION
Large language models (LLMs), such as GPT-3 (Brown al. 2020 PaLM (Chowdhery et al. 2023), and LLaMA-2 (Touvron et al., 2023), have achieved remarkable successes across a broad range of natural language tasks. However, deploying these models in dynamic, real-world scenarios presents fundamental challenge: how can we efficiently adapt them to new tasks and evolving data distributions without losing their valuable pre-trained knowledge?
Consider an enterprise LLM that must continuously learn new product information, regulatory up- dates, and domain-specific terminology. Traditional full rameters-nne not only incurs prohibitive computational costs but also leads to catastrophic forgetting (McCloskey & Cohen 1989 Kirkpatrick et al., (017]. where the model's performance on previ- ously learned tasks deteriorates dramatically. This creates an impossible choice: maintain separate models for each task (multiplying infrastructure costs) or accept degraded performance on earlier capabilities.
all billions of pa-
Parameter-Effiiieet Fine-Tuning (PEFT) methods like Adapters (Houlsby et al.) (Hu et al. 2022) reduce computational costs by freezing the pre-trained model and introducing small
2019 and LoRA
"Correspondence Nikhil Shivakumar Nayak <nayak. ni kh 12608 126088gma 1 com>.

---

Published as a conference paper at ICLR 2026
Average Backward Transfer (%)
-10
-30
-40
30
60
Average Learnability on New Task (%)
FFT (100%)
O-LoRA (1%)
PerTaskFT (800%)
Replay (100%)
Our Method (56%)
MTL (100%)
SeqLoRA (3%)
Figure 1: Pareto frontier TRACE Benchmark. Each curve shows the trade-off between average immediate task accuracy and average backward transfer (BT, our forgetting metric; more negative BT means more forgetring) for given method; higher values indicate hetter performance both axes. Our approach dominates the frontier achieving the best overall performance (learnability -forgetting) while being parameter efficient. Legend entries report the average fraction of trainable parameters used (approximate). For this figure, all methods are run on -L-7M----B--- B-Chtt with the same training schedule; we sweep small grid over key hyper-parametesd per method: learning rate {10-3, 10 10-6) } for FFT, replay buffer size E (5%, 10%. 15%} } of previous-task data for Replay, LoRA rank 2 (2%, 3%, 4%} of the matrix dimension for SeqLoRA, LoRA rank R (0.5%. 1.0%, 1.5%] for O-LoRA (all with orthogonality regularization À 0.5) and two additional O-LoRA points at fixed rank 1.0% with À E (0.2, 1.0), and average effective trainable rank Ro (50%, 56%, 62%} across the 8 tasks for Ours. PerTaskFT and MTL serve upper bounds, obtained by training a separate model per task and by joint multi-task training on the full data, respectively.
trainable modules. However, their restricted parameter budget limits adaptation capacity, and they struggle in continual Cearning-eithe accumulating modules per task requiring complex merging strategies. Continual Learning approaches like EWC (Kirkpatrick et al., (017) and O- LoRA (Wang al. (2023)) attempt address forgetting but either provide only soft constraints that slow rather than prevent forgetting, or operate within fixed low-rank subspaces that may not align with the model's natural capacity distribution.
The key insight missing from these approaches is that not all parameter directions are created equal. Recent work (Sharma et al. 2023) reveals that neural network weight matrices contain substantial redundancy- many parameter directions, particularly those with small singular values, contribute minimally model behavior. This suggests we could identify and repurpose these "dormant" direc- tions for new tasks while preserving critical knowledge-encoding directions.
Building on this insight, we propose Orthogonal Subspace Fine-Tuning a novel parameter- efficient method that fundamentally rethinks how models adapt to new tasks. As illustrated in Figure our approach operates through three synergistic mechanisms:
1. Adaptive Subspace Identification: We decompose each layer via SVD to separate criti- cal knowledge-bearce directions (high singular values) from underutilized capacity (low singular values) that can be safely repurposed.
2. importance-Guided Allocation: We measure each layer's importance via input-output cosine similarity. High-similarity layers that primarily preserve features (e.g.. early atten- tion layers) receive more protected singular directions to maintain stability. Low-similarity layers that transform representations (e.g., final ML.s)) allocated more adaptable
2

---

Published as a conference paper at ICLR 2026
Ti-1
Minimum retention ratio
Target retention ratio
Shared
Span
Transformer
,0
mrr
1 + (trr)
Subspace
Subtepuoe
Singular Value Decomposition
Layer o
low
Layer Importance
VOT
Saroed = Now
Ti
Layer 0
cos
Vargen Vila
VTV)
Task
(
60
Subspace wio
Shared Span
Layer W
High Rank Subspace
high
(
() Thigh
V
(D) high
T1
T2
T3
TN Dx
D,
Low Rank Subapace
1 low
(x low
(DT) low
Incoming Tasks
Figure 2: Overview of our Adaptive SVD-based Continual Fine-tuning Method. For each pa- rameter matrix in the network, we perform SVD decomposition to identify high-rank components (associated with larger singular values) that encode crucial knowledge from previous tasks, and low-rank components (associated with smaller singular values) that contribute minimally to model performance. When learning a new task, gradient updates are projected onto the low-rank subspace orthogonal to previous task representations, allowing full parameter updates while minimizing catas- trophic forgetting.
pacity for learning new tasks. This adaptive allocation automatically balances stability and plasticity across the network according to each layer's functional role.
Orthogonal Gradient Projection: We constrain all gradient updates to remain strictly orthogonal to preserved subspaces, creating impenetrable barrier against forgetting.
Unlike existing methods that either waste parameters (full fine-tuning), sacrifice expressivity (fixed adapters), accumulate modules (progressive approaches), our design achieves all desired prop- erties simultancously. Figure [ empirically validates this claim-oo our method dominates the Pareto frontier, achieving superior learning with minimal forgetting while minimizing the parameter usage.
1.1 OUR CONTRIBUTIONS
1. An orthogonal subspace approach to parameter-efficient fine-tuning: We propose a theo- retically grounded method that partitions weight matrices via adaptive SVD identify and reuse low-importance parameter subspaces with minimal interference. This effectively balances the plas- ticity needed for new tasks with stability retain prior knowledge.
2. Adaptive capacity allocation without extra memory: Our method dynamically allocates pa- rameter budgets across layers based on their functional role while maintaining a fixed footprint, avoiding new modules stored gradients for each task and thus scaling gracefully to many tasks.
We demonstrate consistent gains across classifi- cation, generation, math, and reasoning benchmarks using T5-Large, LLaMA-2 7B, and Mistral-7B models. Our approach achieves better accuracy, stronger knowledge retention, and nearly negligible forgetting while preserving general linguistic capabilities, instruction-follooingg and safety.
State-of-the-art performance on diverse tasks:
We provide in-depth analyses verifying the effective repurposability of low-rank subspaces, showing that these directions can be used for new tasks without degrading old ones. Our experiments confirm practical robustness while theoretical analysis proves tighter bounds catastrophic forgetting.
Thorough empirical and theoretical validation:
3

---

Published as a conference paper at ICLR 2026
The remainder of this paper is structured as follows. Section reviews relevant literature. Section presents our algorithm in detail. Section 4 provides experimental validation. Section concludes with key insights and future directions.
2 RELATED WORK
Continual learning in large language models aims to acquire new knowledge without catastrophi- cally forgetting the old. Existing methods typically achieve this by either restricting which parame- ters are updated or by constraining how full-parameter updates are performed.
Parareeer-restrrittedd approaches isolate task knowledge by modifying only a small subset of weights. Parameter- Eaffciete-Efficientt Fine-Tuning (PEFT) methods like Adapters (Houlsby et al., 2019) and LoRA (Wang et al. 2023a| Liang& & Li (024) freeze the base model and train a few new parameters per task. While this isolates updates, can limit expressiveness and adds parameter overhead that scales with the number of tasks. Recent SVD-based variants such as MiLoRA (Wang et al.] (025)
and PiSSA (Meng al., 2024) further factor weight matrices and interpret low- or high-singular- value components as LoRA-style adapters: MiLoRA freezes the high-singular--alue components and updates only the low-singular-vald components and updates the high--ingular-valuus components instead. OSFT is structurally closest to MiLoRA in that it also fine-tunes only low-singular-vall
components, while PiSSA freezes the www-sinullar-ald
components, but differs in two key ways: (i) we constrain updates via an orthogonal subspace projection so that gradients remain in the complement of the preserved subspace, and (ii) we select ranks adaptively on per-layer basis using an input-output cosine similarity importance score rather than fixing a global rank for all layers. Similarly, sparse fine-tuning methods (Panda et al., 2024; Bhardwaj et al., 2025) update a small fraction of the original weights, but their selection often relies on heuristies like gradient magnitude. Our approach differs by using Singular Value Decomposition (SVD) to identify critical subspaces, choice theoretically grounded in the connection between singular values and the loss landscape's curvature, offering a more principled selection method.
Update-constraining approaches modify all parameters but impose constraints protect prior knowledge. Regularization methods such as EWC (Kirkpatrick et al.] (017) penalize changes to important weights but cannot fully prevent interference, leading to gradual performance decay. A closer line of work, gradient projection, constrains updates to be orthogonal to subspaces learned from past tasks. However, leading methods like GPM (Saha et al., 202[] and SGP (Saha & Roy 2023) derive these subspaces from task activations. This creates critical bottleneck, as their mem- ory requirements grow linearly with the number of tasks, rendering them impractical for billion- parameter models. Other full- full-parameter strategies like standard fine-tuning (Luo et al., (2055) and model merging (Jang et al. 2024; Yadav et al. (023) represent extremes of catastrophic forgetting or prohibitive computational cost, respectively.
OSFT's design motivated by the limitations of the following existing strategies.
Fixed-Rank and Regularization Methods. A simpler approach might be fixed-rank projection (e.g., freezing the top k singular vectors all layers), but this ignores layer heterogeneity and can either over-preserve (hurting plasticity) or under-preserve (causing forgetting). Regularization methods like EWC are often insufficient for LLMs, as their diagonal Hessian approximations fail to capture the complex, non-diagonal curvature of the loss landscape, leading to subpar knowledge retention RRtteer et al. 2018 Heckel| 2022 Kruengkrai & Yamagishi (022).
Activation- Projection (GPM/SGP)). A related line of work uses gradient projection but dif- fers fundamentally. Methods like Gradient Projection Memory (GPM) (Saha et al. (021) and Scaled Gradient Projection (SGP) (Saha & Roy| 2023) perform SVD task activations to build a basis for important subspaces. Our approach differs in four key ways:
1. Object of SVD: We perform SVD directly on model weights to approximate high- curvature directions, whereas GPM/SGP operate on activations.
2. Memory Scalability: Our memory overhead is constant, as we only store the singular vectors of the current weights. In contrast, GPM/SGP accumulate activation-derived bases, causing memory to grow linearly with the number of tasks. This makes them impractical for billion-parameter L.LMs.
4

---

Formally, perform Singular Value Decomposition (SVD) each weight matrix W() layer I: Published as a conference paper at ICLR 2026
3. Adaptive Ranks: Our method uses a layer-wise importance score to adaptively allocate rank, offering a more flexible balance stability and plasticity.
4. Target Scale: OSFT is the first weight-SVD projection method validated on billion- parameter LLMs,, whereas prior methods were demonstrated smaller-scale models.
Our work introduces method that combines the expressive capacity of full-model updates with a scalable and theoretically grounded constraint mechanism. Unlike methods that limit updates to small parameter subsets, we leverage the entire model, and unlike prior projection methods, weight-based SVD approach maintains constant memory overhead, making practical for continual learning in ofte-off-thearr language models.
METHODOLOGY
Our approach, Orthogonal Subspace Fine-Tuning (OSFT), addresses continual learning in large lan- guage models by leveraging adaptive low-rank updates guided by Singular Value Decomposition (SVD). We strategically preserve critical knowledge from previous tasks by constraining parame- ter updates away from dominant (high-rank) singular directions, while enabling model adaptation within complementary (low-rank) directions.
3.1 PROBLEM SETUP AND NOTATION
Let the parameters of an LLM be denoted as:
0 (W(), W W(O),
where each ( R x represents the weight matrix of layer I. Practical deployments involve matrices with millions or billions of parameters, underscoring the necessity of efficient continual updates.
Given sequential tasks (D1, D2,... Dr). each defined by data pairs {( = our goal is to sequentially adapt parameters 9 task Dj without significant performance degradation on previ- ously learned tasks D....., D.-1. Training repeatedly from scratch is computationally prohibitive, necessitating efficient incremental updates.
3.2 LOW-RANK AND HIGH-RANK SUBSPACES VIA SVD
Extensive empirical evidence shows neural network parameters possess substantial redun- dancy (Sharma et al. 2023 Hartford al. 2024), where directions associated with small singular values minimally impact critical model knowledge. Conversely, larger singular values typically en- capsulate vital knowledge. Empirical verification of this low-rank assumption is in Appendix A.4 Leveraging this observation, we propose:
Projecting parameter updates away from high singular-value directions, preserv- ing previously acquired knowledge, and utilizing low singular-value directions for adaptation to new tasks.
( U''O((
(1)
where singular values in E are sorted in descending order. We compute this decomposition once per task, adding minimal overhead compared to full model training.
3.3 DETERMINING LAYER IMPORTANCE VIA INPUT-OUTPUT SIMILARITY
Inspired by AdaSVD Li et al. 2025), we quantify layer importance using cosine similarity between a layer's input activations X and its linear outputs (!) = wxxx()) Specifically, when evaluat- ing layer importance for task f + 1, compute the similarity using data samples from the previous
5

---

Published as a conference paper at ICLR 2026
task t as follows:
1 N
10
cosine_similarig (X (1) x!!)
where N denotes the number of data samples from task 1. Higher similarity indicates minimal di- rectional change, signifying that the layer predominantly preserves rather than transforms activation representations. Such layers are essential for retaining features and ensuring stable propagation of information across tasks. Importance scores are also normalized to have an average one across layers: 1 E 7 = 1. While empirically we observe 7()) is consistently positive, for robustness, we clip any negative raw cosine similarity values zero before normalization.
We explored alternative ann-aapprxiimttion strategies, including LASER (Sharma et al. 2023 SPECTRUM's Marchenko-Past
thresholding (Hartford et al.) 2024p, and entropy-based effec- tive rank (Roy & Vetterli (007). In a continual learning setting, these approaches either fail to capture layer-wise variability under sequential tasks do not yield stable thresholds across hetero- geneous datasets. An alternative notion of representational similarity is centered kernel alignment (CKA) (Kornblith et al. 2019). which compares activations across models or layers. CKA empha- sizes different aspect of representation geometry focusing cross-representation similarity, while cosine similarity is designed to capture how tightly layer's outputs are aligned with its inputs. A CKA-based variant of our importance measure natural extension.
3.4 ADAPTIVE RANK SELECTION
Given the importance of the layer 7()), we introduce two hyperparameters controlling the retention of singular vectors:
Minimum Retention Ratio (mrr), ensuring minimal essential retention even for the least critical layers.
Target Retention Ratío (trr), defining the upper retention bound for highly critical layers.
The fraction of singular vectors preserved at each layer is computed as:
frac mrr + (00) (trr mrr).
The number of singular vectors retain 1:01) I min(d d.
The singular vectors (t) v/O. high) and low-rank subspaces. In practice, we found
then partitioned into high-rank (U high
that setting mur 0.1 and trr 0.8 yields robust performance across our benchmarks. Our method is not overly sensitive to these values; ablation studies show that while performance degrades significantly if retention is too aggressive, mild perturbations (#0.05) result in minimal (<1%) accuracy changes. For new applications, we recommend starting with these defaults and performing small grid search on the first task. See Appendix A.9 for the complete ablation study.
3.5 ORTHOGONAL GRADIENT UPDATES LOW-RANK SUBSPACE
To minimize catastrophic forgetting, we enforce updates to lie within the low-rank subspace orthog- onal to the high-rani directions. This is achieved by projecting the gradients:
(!))) proj = vw - - high ((U)) high) (W(O high
(v()) high
(4)
( Here, high TRdoxAl and ( high E dd are the dense matrices containing the top (9()) singu- lar vectors. This operation computes the component of the gradient that lies within the high-rank subspace and subtracts it, ensuring the final update is strictly orthogonal to the preserved directions.
3.6 ORTHOGONAL UPDATES VIA REPARAMETERIZATION AND GRADIENT HOOKS
Our goal is to confine all updates to the low-rank subspace, making them orthogonal to the frozen high-rank directions. While one achieve this by projecting the full weight gradient, more computationally efficient and elegant solution implemented by reparameterizing the weights and using gradient hooks. This process involves two main steps:

---

Published as a conference paper at ICLR 2026
1. Reparameterization and Freezing: After performing SVD weight matrix WO) replace it with its underlying SVD components.
The high-rank components (U
X)U)) hight highht
V high ) are registered as frozen buffers in the
model (i.e., non-trainable).
The low-rank components (U ters.
Vill are registered as new trainable parame-
low Zlow
During the forward pass, the full weight is reconstructed on-the-fly (W W high W Iow During backpropagation, gradients are only computed for the trainable low-ramk SVD com- ponents.
2. Maintaining Orthogonality with Gradient Hooks:
Simply training the low-rank com- ponents could cause their basis vectors (the columns of U and Vlow) to "drift" and lose their perfect orthogonality with the frozen high-rank basis vectors. To prevent this, attach gradient hook to the trainable parameters. After the gradients (e.g., VU are computed, this hook projects them to be orthogonal to the high-rank basis vectors (e.g., Unig)) This acts as a maintenance step, guaranteeing the mathematical integrity of the subspaces throughout training.
Our OSFT procedure is summarized in Algorithm
3.7 COMPUTATIONAL AND MEMORY ANALYSIS
Our method is designed to be computationally efficient and scalable. The primary additional cost is the SVD, which performed once per layer per task before training begins. Refer Appendix A.IT for detailed computational cost and memory efficiency analysis.
3.8 THEORETICAL JUSTIFICATION OF ADAPTIVE RANK SELECTION
We rigorously justify our adaptive rank selection method through formal theoretical analysis using a second-order Taylor expansion the task-specific loss landscape, detailed in Appendix [A.3] This analysis explicitly demonstrates that preserving parameter directions associated with the highest Hessian peanvalless-rrresentti directions greatest curvature-fffecte trophic forgetting.
minimizes catas-
However, explicitly computing and decomposing the Hessian is computationally prohibitive for large-scale language models. Therefore, employ an efficient approximation inspired by em- pirical evidence from Haink (2023), who show a robust correlation between the Hessian's 's largest eigenvalues and the largest singular values of the model's weight matrices. By retaining the top singular vectors-correesonndee corresponding to critical knowledge from previous tasks we effectively approx- imate freezing the high-curvature Hessian directions while allowing updates within the subspace defined by lower singular values.
Further supporting our approach, empirical findings (Sharma al., 2023 Li al. 2025) highlight
that layers with higher input-output similarity exhibit significantly greater Hessian curvature. Our adaptive layer-wise rank allocation strategically exploits this property: layers identified as crucial (high input-output similarity) receive greater singular vector retention, thereby preserving essential knowledge.
4 EXPERIMENTAL RESULTS
We comprehensively evaluate our adaptive SVD>based continual learning method on established continual learning benchmarks, comparing it extensively with recent state-of-the-art (SOTA) base- lines, notably O-LoRA Wang et al. (2023a). Our experiments aim to demonstrate the effectiveness, scalability, and practicality of our approach in realistic continual learning scenarios. For all bench- marks, fixed task sequences from prior work and report results averaged across multiple orders. To provide deeper insight into learning and forgetting, we include per-task accuracies (both immediately after training and at the end of the sequence) in Appendix A. We compare against
7

---

Published as a conference paper at ICLR 2026
recent SVD baselines MiLoRA and PISSA; results reported in Appendix A..6 We also compare against the SOTA sparse fine-tuning method LoTA (Panda et al. 2024). which uses sparsity masks to preserve task-specific information; results and analysis are provided in Appendix A.7
4.1 BENCHMARKS AND EVALUATION PROTOCOL
We adopt two widely-used benchmarks reflecting varying levels of complexity and task diversity:
Standard Continual Learning Benchmark (5 Tasks) introduced by Zhang et al. 2015), consisting of classification tasks: AG News, Amazon Reviews, Yelp Reviews, DBpedia, and Yahoo Answers.
Extended Continual Learning Benchmark (15 Tasks), introduced by Razdaibiedina et al. 20033)
combining tasks from multiple sources, including GLUE (Wang et al. 2019) (MNLI, QQP, RTE, SST-2), SuperGLUE (Wang et al. 2020) (WiC, CB, COPA, MultiRC, BoolQ), and IMDB, along
with the original 5-task benchmark.
TRACE Benchmark (8 Tasks). In addition, we evaluate on TRACE (Wang et al. 2023b an 8-task instruction-tume tunnnng continual learning benchmark covering domain-specific tasks, multilingual capa- bilities, code generation, and mathematical reasoning. Following TRACE, we report two metrics: Average Accuracy (AA) and Average Backward Transfer (BT). Let As,j denote the accuracy on task j after training on task i in sequence of T tasks. We define:
1 T-
AA
AT.
BT
(Ar. - Ajs)
(5)
T
We interpret forgetting -TT: more negative BT corresponds to more forgetting, while values closer to zero indicate better retention of earlier tasks.
We evaluate two popular large language model architectures, T5-Large (encoder-decoder) and LLaMA-2 7B (decoder-only), using the widely-adopted metric of Average Accuracy (AA), com- puted across all tasks after training on the final task. To ensure robustness, we follow standard protocols, averaging results over three independent runs with randomly permuted task sequences. Implementation details, hardware configurations, and training hyperparameters for both T5-Large and LLaMA-2 7B models are provided in Appendix A.10
4.2 BASELINE METHODS
We position our method clearly against representative continual learning paradigms:
Sequential full-model fine-tuning (SeqFT); serves lower-bound baseline, prone to catastrophic forgetting.
Parameter--ffficient LoRA variants including SeqLoRA, IncLoRA, and the recent SOTA, O-LoRA Wang et al. (2023a), which utilize low-rank adapters.
Replay-based approaches, such as standard replay buffers.
Regularization methods, including Elastic Weight Consolidation (EWC) (2017) and Learning without Forgetting (LwF) Li& Hoiem (0117).
(EWC)) Kirkpatrick al.
Prompt-based techniques, including I.2P Wang et (2022 and ProgPrompl Razdaibied ina et al. (2023).
Model-merging methods: we include SLERP (Jang et al. 2024) and TIES Yadav et al.) 2023 Both methods operate by combining separate per-task models rather than maintain- ing a single continually updated model.
PerTaskFT: trains separate model per task, offering strong performance but requiring extensive computational resources and storage.
Multi-task Learning (MTL): trains a single model simultancously on all tasks, represent- ing ideal upper bound by relaxing continual learning constraints.
8

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of Average Accuracy (%) across standard continual learning benchmarks using the T5-Larg model.
5-Task CL Benchmark
15-Task CL Benchmark
Method
Order-1
Order-2
Order-3
avg
Order-4
Order-5
Order-6
avg
SeqFT
18.9
24.9
41.7
28.5
7.4
7.4
7.5
7.4
SeqLoRA
44.6
32.7
53.7
43.7
2.3
0.6
1.9
1.6
IncLoRA
66.0
64.9
68.3
66.4
63.3
58.5
61.7
61.2
Replay
55.2
56.9
61.3
57.8
55.0
54.6
53.1
54.2
EWC
48.7
47.7
54.5
50.3
45.3
44.5
45.6
45.1
LwF
54.4
53.1
49.6
52.3
50.1
43.1
47.4
46.9
1,2P
60.3
61.7
61.1
60.7
57.5
53.8
56.9
56.1
LFPTS
67.6
72.6
77.9
72.7
70.4
68.2
69.1
69.2
O-LoRA
75.4
75.7
76.3
75.8
72.3
64.8
71.6
69.6
OSFT (ours)
75.3
74.0
78.4
75.9
71.6
69.6
72.7
71.3
SLERP
40.5
43.0
45,8
43.1
2.4
1.5
2.7
2.2
TIES
35.0
38.5
37.8
37.1
7.8
7.1
5.8
6.9
ProgPrompt
75.0
75.1
75.1
78.0
77.7
77.9
77.9
PerTaskFT
70.0
70.0
70.0
70.0
78.1
78.1
78.1
78.1
MTL (Upper Bound)
80.0
80.0
80.0
80.0
76.5
76.5
76.5
76.5
Table 2: TRACE benchmark performance using LLaMA-2-7B-OO Average Accuracy (AA) and Backward Transfer (BT) percentages are reported.
Method
AA (%)
BT (%)
SeqFT
23.0
-8.3
LoraSeqFT
9.2
-24.6
O-LoRA
41.3
-6.2
OSFT (ours)
48.4
-7.1
PerTaskFT
57.6
NA
MTL
52.3
NA
4.3
MAIN RESULTS
Table shows that OSFT outperforms or matches all baselines on both 5-task and 15-task bench- marks. Importantly, compared O-LoRA-the the current SOTA parameter-efficient baseline method achieves superior accuracy, particularly in the more challenging 15-task scenario (71.3% vs. 69.6%). highlighting its effectiveness maintaining task knowledge Notably, while PerTaskFT achieves high performance, it requires training separate models per task, making it computationally impractical. MTL represents an idcalized scenario, training on all tasks simultancously, thus serving as upper-bound performance indicator. A comparison with model merging methods, SLERP and TIES, is provided in Appendix A.8 with corresponding results cluded in T Ablation results for rank selection and gradient projection are in Appendix A.9
extended task sequences.
4.4 PERFORMANCE ON THE TRACE BENCHMARK
To further illustrate our method's capability in more realistic continual learning environments, we evaluate it on TRACE Wang et al. (2023b), which includes diverse and challenging instruction- tuning tasks.
Results in Table empbasiee our method's ability to effectively retain and transfer knowledge across tasks. Our approach achieves notably higher average accuracy, with slightly lower backward trans- fer compared to O-LoRA, demonstrating a strong balance between robustness to forgetting and adaptability to new tasks, both critical for practical deployments. In settings where tasks lated, we also observe positive transfer effects. For example, in TRACE when fine-tuning on the NumGLUp GU---mm and HumGLUF-dd tasks, the accuracy NumGLUE increases after training

---

Published as a conference paper at ICLR 2026
Table 3: Comparison of general ability scores across six diverse evaluation tasks between the base LLaMA-2-7B chat model and our adaptive SVD-based continual learner.
Model
MMLU
GSM
BBH
TydiQA
BoolQA
PIQA
Base Instruct Model OSFT (ours)
46.6
26.1
40.2
23.5
70.5
76.2
47.7
7.7
34.2
35.8
76.6
77.6
Table Win Tie Lose (%) for cnerructionfolllwwinn and safety evaluations against the LLaMA- 2-7B-Chat base model.
Instruction (Helpfulness)
Safety Tie
Method
Win
Tie
Lose
Win
Lose
Replay
10
72
0
88
12
LoRASeqFT
3
4
94
0
86
14
SeqFT
14
34
53
0
98
2
OSFT (ours)
24
56
20
18
78
4
on NumGLUE-ds, indicating that adaptation on the later task can improve performance on the ear- lier, related task rather than degrade it.
Retention of General Capabilities and Safety. We explicitly evaluate the preservation of general and safety after continual learning using benchmarks proposed by TRACE. Table 3 illustrates our method's effectiveness in preserving or enhancing core language capabilities including factual knowledge, commonsense reasoning, and multilinguality compared to the original instruction-tunee model. Reasoning tasks like GSM8K suffer post- cost-training degradation, a known issue across methods and reported in TRACE Wang et al. (2023b). Prior work highlights that continual learning without explicit reasoning supervision (e.g., chain-of-though augmentation) insufficient to preserve these capabilities; however, our method can augmented with such tech- niques to mitigate this degradation. Table demonstrates that our method also retains instruction- following ability and safety performance compared to basclines.
abilities, instruction
CONCLUSION
As large language models (LLMs) become increasingly central to real-world applications, contin- ually adapting them without erasing prior knowledge is essential. We presented novel continual learning framework that uses adaptive singular value decomposition (SVD) to isolate low-rank sub- spaces for new tasks while preserving eritical directions for previously acquired knowledge. Un- like parameter-efficienn techniques that freeze most weights or add modules per task, our method operates all model parameters with fixed memory, preventing catastrophic forgetting through thogonal subspace updates. Extensive empirical evaluations demonstrate our method's 's effectiveness across diverse benchmarks: (1) On the 5-task benchmark with LLaMA-2 7B, we achieved 79.6 % accuracy, surpassing the current SOTA by over 3 percentage points; (2) or the challenging /5- task sequence with La-Ler, we reached 71.3% accuracy, outperforming all parameter-effice competitors; (3) On the realistic TRACE benchmark with LLaMA-2 7B- 7B-Chat, our method attained 48.4% average accuracy without requiring simultaneous multi-task access or multiple specialized models. Crucially, our approach preserved general capabilities, instruction-follwinn behavior, and safety throughout continual learning essential properties for deployment in production environ- ments. Our method OSFT provides mathematically principled solution the fundamental tension between stability and plasticity in neural networks, offering a scalable path toward continuously evolving language models that efficiently accumulate knowledge without forgetting. Our work tablishes a practical approach for real-world deployment of continually adapting language models. Limitations and future directions discussed Appendix A.2
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We provide all details needed to reproduce our results. The algorithm is specified in Section 3 with the training procedure summarized in Algorithm [ Benchmarks, metrics, and evaluation protocol are described in Section 4 Theoretical assumptions and full proofs are provided in Appendix A.3
Computational cost and memory analysis are in Appendix II.II with ablations in Appendix A.9 Per-tast results and task orders are reported in Appendix Implementation and hyperparameters
are given in Appendix (A.10 A code repository with scripts and configs is linked in Appendix A.13 Our disclosure of L.M usage is in Appendix A.14

---

Published as a conference paper at ICLR 2026
REFERENCES
Kartikeya Bhardwaj, Nilesh Prasad Pandey, Sweta Priyadarshi, Viswanath Ganapathy, Shreya Kadambi, Rafael Esteves, Shubhankar Borse, Paul Whatmough, Risheek Garrepalli, Mart Van Baalen, Harris Teague, and Markus Nagel. Sparse high rank adapters, 2025. URL https:
13175
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Hesbert-Voss, Gretehen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlísh, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle. M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL bttps::/procceddnng.. wtwp:eddiinnnnuuuuuuuee nwwrips wtwpurrppsccccppee... cc/paper haapseeerr_lleeeeeeeeeeeeeeeeeee fww.es/ppper///222222 F11e:/55.00006000000677016122662
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 15846. Curran Associates, Inc., http::eeeeeddnnnss..neeee n co/pper files/paper/2020. pdf
2020. URL /atps:://proceeddigg
File F111//8a222711500022229903333333
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, Parker Schuh, Kensen Shi, Sashank Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Bar- ret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Mcier-Hellstem Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: scaling language modeling with pathways. J. Mach. Learn. Res., 24(1), January 2023. ISSN 1532-4435.
David Haink. Hessian eigenvectors and principal component analysis of neural network weight http:/ppp...aaatttccccccooollaa.
matrices. ArXiv, abs/2311 1.0452,, 2023. URL CorpusID:26483333
Eric Hartford, Lucas Atkins, Fernando Fernandes Neto, and David Golchinfar. Spectrum: Targeted training on signal to noise ratio, 2024. URL
org. 06623
Reinhard Heckel. Provable continual learning via sketched jacobian approximations. In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 10448-1047)) PMLR, 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. sirameter-efffcient transfer learning for nlp, 2019. URL RtRLsttss///axxivooom
902. 1902.0051
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation large language models. In International Con- ference on Learning Representattons, 2022. URL http::://ppeereviee. net forum?
Id=nZevKeeFYES
Young Kyun Jang, Dat Huynh, Ashish Shah, Kan Chen, and Ser-Nam Lim. Spherical linear interpolation and next-anchoring for zero-shot composed image retrieval. In European Conference Computer Vision pp. 254. Springer, 2024.
12

---

Published as a conference paper at ICLR 2026
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barnska-arrinska, Demis Has- sabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic for- getting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521- 3526, 2017. doi: 10. 10.10733pnas.1611 1aaas..611115511 18351441 10. 10.1073/pnas. 1611835111113111
URL htt.:ttts:///ww.paeeet http:///wwwwwpp..orgggoo//ee org//////////////////////
Simon Korblithh, Mohammad Norouzi, Honglak Lee. and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519- 3529. PMIR, 2019.
Canasai Kruengkrai and Junichi Yamagishi. Mitigating the diminishing effect of elastic weight consolidation. Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli. Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Scokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na (eds.). Proceedings of the 29th International Conference on Computational Linguistics, pp. 4568 3568-4574, Gyeongju, Re- public of Korea, October 2022. International Committee Computational Linguistics. URL
1.033/
Zhiteng Li, Mingyuan Xia, Jingyuan Zhang, Zheng Hui, Linghe Kong, Yulun Zhang, and Xiaokang Yang. Adasvd: Adaptive singular value decomposition for large language models, 2025. URL
attps://arrxil org/ abs atwpg///b///2.....///// 2502 01403
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions pattern analysis and machine intelligence, 40(12))2935-290) 2017.
Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference-free low-rank adaptation for continual learn- ing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. (6638-23647, 2024.
Yun Luo, Zhen Yang. Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting large language models during continual fine-tuning, 2025. URL https://arxx org/ abs 2308 08747
James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML' 15, pp. 14088-4411 JMLR.org, 2015.
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 24:109-165, 1989. URL
https: eeseanttcsccee
Ptrrg/CorpusID... :1:
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa:
Principal singular values and singular vectors adaptation of large language models. Advances in Neural Information Processing Systems, 37:121038-122 121072, 2024.
Ashwinee Panda, Berivan Isik, Xiangyu Qi, Sanmi Koyejo, Tsachy Weissman, and Prateek Mittal. Lottery ticket adaptation: Mitigating destructive interference in LLMs, 2024. URL https:
/ ipppen/ereee..ettttoooaaaddddddd
Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Alma- hairi. Progressive prompts: Continual learning for language models. In The Eleventh Interna- tional Conference on Learning Representations, 2023. URL htttps://epenrevl/ net/
Forum??d=00JT@BCC..
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 7742-7752, Red Hook, NY, USA, 2018. Curran Associates Inc
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pp. 606-610. IEFF, 2007.
13

---

Published as a conference paper at ICLR 2026
Gobinda Saha and Kaushik Roy. Continual learning with scaled gradient projection, 2023, URL fitps:://arxiz wwtp://g///s/ooooooeoooooo..esst
http://arxiv.oor//////22222222 httpa/////...... 01386
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning,
2021. URL URRp:hts:///arattt org abs htt.///.////.././/////// 01766.976722
Pratyusha Sharma, Jordan T Ash, and Dipendra Misra. The truth in there: Improving reasoning in language models with layer-selective rank reduction. arXiv preprint arXiv:2312. 13558, 2023.
Sidak Pal Singh, Gregor Bachmann, and Thomas Hofmann. Analytic insights into structure and rank of neural network hessian maps, 2021. URL https: http://axvvvvoo httgg/ab//22000 16225
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL https: /arpiii///xvvv../eaa www.iiii.g/iioooooooooooooo abs 2302. 13971
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. 2019.
URL htt.:tttp://aaaiii http:///aaaiii...aaaaa////////// abt/ 07461
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems, 2020. /tthttss:://aa arxiv. owtpvvs.rga/bb//../e. /195 00537
Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. Milora: Harnessing minor singular components for parameter-efficien Ilm finetuning. Proceedings of the 2025 Confer- ence of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pupers), pp. 4823-4836, 2025.
Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023a. htthtts::
//openrevi ew. net. forum? d= d=1.778p77788a
Xiao Wang, Yuansen Zhang, Tianze Chen, Songyang Gao, Senjie Jin, Xianjun Yang. Zhiheng Xi, Rui Zheng, Yicheng Zou, Tao Gui, et al. Trace: A comprehensive benchmark for continual learning in large language models, arXiv preprint arXiv:2310.06762, 2023b.
Zifeng Wang, Zizhao Zhang, Chen- Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy. and Tomas Pfister. Learning to prompt for continual learning. In Pro- ceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 139-149, 2022.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-merging: Re- solving interference when merging models. In Thirty-seventhh Conference on Neural Information Processing Systems, 2023. URL htttts:://openzevee..co
aX3WCC
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Neural Information Processing Systems, 2015. URL httpps:/appi.
semanticschon www.aarorrr/oooos::0333
Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhi-Quan Luo. Why trans- formers need adam: A hessian perspective. In The Thirty-iighth Annual Conference Neu- ral Information Processing Systems, 2024. URL http::t/p//........... http://pppneeeeeeeeeeeeee www.:eenttooorteeeeeee X6rgEpbnj3
/
14

---

subsequent use of Published as a conference paper at ICLR 2026
A APPENDIX
A.1 ALGORITHM SUMMARY
Algorithm Orthogonal Subspace Fine- Tuning (OSFT) Practical Implementation
1: Require: Initial parameters 0 = [WO)_ tasks {D:} hyperparameters mrr, trr.
2: Ensure: Parameters are updated continually while preserving high-rank subspaces.
3: for task = .,...,7 , do
4: Compute layer importance 7() and determine retention count k(0)
5: for layer 1 = 1, ..... L do
6: Decompose w via SVD.
7: Reparameterize Layer:
8:
Store high-rank components (U
high ...) ) as frozen buffers.
9: Store low-rank components (U low!... as new trainable parameters.
Register gradient hook on the trainable SVD components enforce orthogonality.
11: end for
while not converged on task DD do Sample mini-batch.
13:
In the forward pass, reconstruct effective weight w
w w low
high
Backward pass computes gradients for trainable SVD components (e.g., V low
16:
The gradient hook automatically projects these gradients.
17:
The optimizer updates only the (projected) trainable SVD components.
18:
end while
19:
end for
A.2 LIMITATIONS AND FUTURE WORK
While our approach achieves strong performance across a range of benchmarks, few directions re- main open for further refinement. (1) Rank Estimation Heuristies: Although current rank selection method performs robustly in practice, future work could explore more principled, data- driven heuristics to fine-tune retention ratios with even greater precision. (2) Layer-Speeific Opti- mization: Our method currently applies SVD to all weight matrices; selectively targeting specific layer types (e.g., attention matrices) may offer further efficiency gains with minimal trade-offs. (3) Long-Horizon Adaptation: In scenarios with very large numbers of tasks, more adaptive capacity management or online adjustment of subspace budgets may further enhance scalability. (4) Samples from Previous Task: When we use the cosine-similarity based rank selection to set the effective rank, OSFT requires small fixed-size buffer of task (1-1) data when starting task &; this is fixed memory requirement and does not increase with the number of tasks in the sequence. The alter- native weneetterninnd-theehseeeee also performs well in practice. These natural extensions to core framework, which remains effective and practical in current continual learning settings.
schedule for effective rank does not require any old-task data and
A.3 THEORETICAL ANALYSIS: TIGHTER FORGETTING BOUNDS VIA ADAPTIVE SVD
We now formally derive hierarchy of catastrophic forgetting bounds that rigorously demonstrate the advantage of our adaptive rank selection approach compared to both naive full fine-tuning and uniform low-rank projection methods. In essence, this section shows how protecting high-curvature directions (i.e., large Hessian eigenvalues) minimizes sonelting-motivatinng weight-matrix SVD as tractable approximation.
Lemma 1 (Second-Order Approximation of Catastrophic Forgetting). Consider a model with pa- rameters g(k) after training task k. and subsequent parameters 0(++1) -(()) + 20 after learning task + 1. Assuming V 1 (8()) ze 0 (i.e. task k's loss is near-optimal at g(k) the catastrophic forgetting task k can be approximated by:
1 AL & L (a(())) = A0 HRA0,
(6)
15

---

Published as a conference paper at ICLR 2026
where H₃ (x((()) is the Hessian of the loss function ((%)
Proof. Step 1: Taylor Expansion. Expanding L3 ((++1) B(() + A0 via Taylor's theorem:
(x((((+)) = (x(((()) + ( AAT H& + 0(1400))).
==0
Step 2: First-Order Term Vanishes. Since ((k) represents a (local) optimum for task we have VL (0(k) O, thereby eliminating the first-order term.
Step 3: Dominant Quadratic Term, The remaining quadratic term 1/2 AO 2₂ dominates for- getting.
Lemma 2 (Block-Diagonal Approximation of the Hessian). Consider Transformer model with parameters partitioned into layers such that:
(e)( vec(W(())) ..... v...,..
The Hessian matrix Hg at the optimum g(k) can be approximated as block-diagonal with respect to layers:
H k
0
H
2
0
H₂ R0
(8)
H
where each ( represents the intra-layer Hessian for layer l. Under this approximation, the quadratic form decomposes as:
TO HRA R
vec(AW((0) H (e)(AW((1)
Proof. The block-diagonal approximation is theoretically justified by analyses showing the Hes- sian of neural networks, especially Transformers, is dominated by intra-layer terms with negligible cross-layer interactions (Singh et al., 2021 Martens & Grosse 2015). Empirical evidence from Transformer models further supports this structure: Hessian spectrum analyses reveal minimal mag- nitude in off-diagonal inter-layer Hessian blocks compared to the intra-layer blocks (Zhang et al.) 2024).
Empirical Validation: As shown in Zhang et al. (2024), inter-layer Hessian blocks in Transformers exhibit 10x smaller Frobenius norms than intra-layed layer blocks, with cross-layer correlations below 0.1 pretrained models. This justifies treating layers independently for curvature analysis.
Norm Equivalence:
Note that (ec(AW(()) H vee'vec(WWW is equivalent to (AW(), !!! AW(")) where (r.) is the Frobenius inner product. Thus, the quadratic form directly ties to layer-wise Frobenius norms.
In practice, optimization and continual learning algorithms that assume a block block-diagonal Hessian, such Kronecker- Facttrrd Approximate Curvature (K- (K-FAC) (Martens & Grosse (2015) and struc- tured Laplace approximations (Ritter et al.] 2018), consistently demonstrate effectiveness in lever- aging layer-wise curvature without significant loss of accuracy. Thus, the approximation is both theoretically sound and empirically validated.
Lemma (Relationship Between Layer Importance and Curvature). The layer importance measure 1(0), defined as:
70
1
cosine similarity x!!) y(())
(10)
N
where x.()) are layer inputs and ( spectral properties of the layer-wise Hessian H(())
w(0) x are layer outputs, positively correlates with the
16

---

Published as a conference paper at ICLR 2026
Proof. Layers with high importance scores (high similarity between inputs and outputs) tend to preserve activation patterns rather than significantly transform them. These layers typically serve as information conduits in the network, maintaining critical features learned for task k.
Empirically, these high-importance layers exhibit higher sensitivity to parameter perturbations. When a layer primarily passes information forward with minimal transformation (high ) per- turbations to its parameters directly interfere with this information flow, causing large changes in the loss function. Mathematically, this translates to larger eigenvalues in H(() indicating steeper curvature.
Conversely, layers with lower ((t) values significantly transform their inputs, suggesting these layers are more adaptable. Perturbations to these layers' parameters cause smaller changes in the loss landscape, resulting in smaller eigenvalues in H
This relationship has been verified empirically in multiple studies (Sharma et al. 2023 Li et al., 2025 consistently showing a positive correlation between measures of layer importance and the magnitude of Hessian eigenvalues.
Intuition: Consider a layer that merely passes input features (high π()). Perturbing its weights W(( directly distorts critical task-k features, causing large loss changes (high curvature). In con- trast, layers transforming inputs (low 7(0) allow parameter changes without catastrophic feature distortion, corresponding to flatter curvature.
Preserving Large Hessian Eigenvalues Minimizes Forgetting, Combining these lemmas, we see that directions with large Hessian eigenvalues impose the greatest risk for catastrophic forgetting: even small updates along those directions yield substantial loss increases for old tasks.
Theorem 1 (Hierarchy of Forgetting Bounds). Assuming equal parameter update magnitudes || across different fine-tuning strategies, the forgetting bounds satisfy:
Aduptive SVD <. Fixed-Rank <,
Full Fine-tuning
(11)
Specifically:
Full Fine-tuning:
A S Amax(HH) C, 2
(12)
Fixed-rank:
AL S max (x p
(13)
Adaptive (Ours):
AL S m
(E) (1
(14)
where r(E) mPr + 10 (trr - mrr) is our adaptive rank allocation based on layer importance.
Moreover, under the condition that layer importance 7(8) positively correlates with Hessian curva- ture (Lemma)) 3), we have:
max L (o+1 f
max )max((H))
(15)
ensuring our adaptive approach provides strictly tighter forgetting bounds.
Proof. We establish the hierarchy of bounds by proving each inequality separately.
Part max; (X,2) < Amax Hx). By the diagonaa approximation (Lemma 20. Amax H+) =
maxe(> (9)) From Lemma high- 7(0) layers have larger \ Since x S A for all by the
ordering of eigenvalues, we have:
max (1 (6) (741) S (ax (0) Amax Amax(HR).
For the full fine-tuning case, we need to bound AO H,A0. By the Rayleigh quotient property, for any symmetric matrix Hę and non-zero vector A0:
Rayleigh Quotient Proof for Full Fine-tuning Bound:
T H₂A0
I Amax (H+),
17

---

Published as a conference paper at ICLR 2026
where Amax (H+) is the largest eigenvalue of Hg. This holds because the maximum value of the Rayleigh quotient equals the largest eigenvalue.
Rearranging, get.
AOT 4g
S Amax((H ||00||2 Amax (Hx) c.
Hence the forgetting bound for full fine-tuning is:
L ~ A0' Hx A0 I (max)())) ||0012
Part 2: max, m (00)+1 < max, ( 7411
Let 8" arg max, X be the layer with the largest post-projection eigenvalue in the fixed-rank approach. By Lemma 3 this layer typically has high curvature and thus high importance f((") Under adaptive allocation strategy, that high-importance layer obtains larger rank allocation (r(E*) r), ensuring:
((-)+11 X max
For any other layer + E,
X 7(0+1 x max (141))
either because r(t) (for other high- ieghimmprttance layers) or because 1 importance layers). Hence maxe (0)
X (for low-
((0)++1 maxe(2/41) implying a strictly tighter bound than
fixed-rank.
Combining Parts and 2 completes the proof of the bound hierarchy.
(16)
Adaptive (Ours)
Fixed Rank
Full Tuning
where 8 = arg maxt A+1 the highest-curvature layer.
(9)
On the Equal-Norm Assumption The assumption || across different fine-tuning strate- gies isolates the impact of update directions but does not imply optimality. In practice:
Adaptive SVD may achieve lower forgetting even with smaller norms by avoiding high- curvature directions.
Full tne-tuning could offset poor directional alignment with larger updates, but this risks catastrophic forgetting.
Future work should analyze the Pareto frontier of the accuracy accuracy-forgeting trade-off under variable norms.
This assumption purely theoretical device, not a claim about how hyperparameters are tuned in practice.
Key Theoretical Insights
Under equal parameter update budgets:
Full fine-tuning suffers worst-case forgetting bounded by Amax
Fixed-rank projection improves this by capping directions via uniform low- rank selection, but misallocates rank to some layers.
Adaptive SVD aligns per-layer rank r(e) with curvature (via (()), giving strictly tighter forgetting bounds.
18

---

Published as a conference paper at ICLR 2026
Corollary (Forgetting Reduction with Adaptive SVD). Under the equal parameter update mag- nitude assumption, our adaptive SVD achieves strictly less forgetting than fixed-rank rank or naive full fine-tuning. This gap widens when:
Layer importance [(0 varies significantly across layers.
The Hessian spectrum shows heavy tails few large eigenvalues dominate).
Proof. Follows directly from Theorem
and the established bound hierarchy:
Adaptive
A A
Fixed-rank
< A Full
Practical Approximation via Weight-Matrix SVD. While the above results show that retaining large prssiinn-eggevalle directions is essential to minimize forgetting, computing Hessian eigen- vectors is intractable for large language models. Recent empirical findings (Haink) (023) indicate that these high-curvature directions often overlap significantly with top singular vectors of the weight preservine large singular values- as
matrices. Hence, our method uses SVD-based rank pragmatic surrogate for preserving large Hessian eigenvalues. By focusing lower singular- value directions for new-task updates, we effectively contain catastrophic forgetting without the prohibitive overhead of Hessian decomposition. This aligns with the theoretical ideal of limiting updates where curvature is highest, but in computationally feasible manner.
This theoretical framework underpins higher curvature) get more singular directions retained, while less critical layers can be more aggres- sively pruned. As shown Section 4 this approach consistently outperforms naive full fine-tuning and uniform low-rank baselines in mitigating forgetting and stabilizing knowledge across tasks.
adaptive SVD strategy: high-importance layers (with
A.4 EMPIRICAL VALIDATION Low RANK APPROXIMATION
We conducted in-depth analysis of the Granite 8B model architecture validate findings from prior literature suggesting that the weight matrices in transformer layers are effectively low- rank (Sharma et al., 2023 Hartford et al. 2024). This implies that these matrices can be accurately approximated using low-rank Singular Value Decomposition (SVD), revealing unused capacity that can potentially be leveraged to learn additional tasks or improve performance existing ones. Since Granite shares a similar architecture with LLaMA, our findings are directly applicable to LLaMA and offer broader insights into decoder-only transformer architectures and large language models in general.
Table 5: Leaderboard average results for attn ww..roj..wiigtttt across varying low-rank reduc- tion levels. Middle layers showed slightly better robustness than early layers. The baseline here refers to the original Granite 8B model without any low-rank approximation.
Reduction %
Above Baseline
Below Baseline
10%
3
50%
4
90%
2
10
99%
2
10
99. 75%
0
11
We examined all attention and feedforward projection matrices across all layers of Granite 8B, and report results for four key matrices: the attention value and key projections, and the two feedforward projection matrices that follow attention. Based prior observations from LASER Sharma et al. (003)) suggesting that later layers benefit most from rank reduction often leading to improved downstream performance when gigh-frequency components layers 28, 29, 34, and 39 out the model's 40 layers. We performed SVD-based low-rank approx- imations varying reduction levels (e.g., retaining only 1%, 50%, 90% of the original singular
ooteet removed- report findings from
19

---

Published as a conference paper at ICLR 2026
Layer: V_PRO) - Metrics Across Reduction Percentages
IFEval
BBH
MATH Lvi 5
0.495
IFEval
0.55
MATH Lv1
0.090
B8H
Reduction
Reduction
0.085
Reduction
0.54
50%
10%
0.485
10%
50%
0.060
50%
10%
0.53
Easclinc
0.075
Basalinc
30
32
34
36
38
30
32
34
36
38
30
32
34
36
36
Layer Number
Layer Number
Layer Number
GPQA
MUSR
MMLU-PRO
0.292
0.48
0.47
GPQA
0.290
0.46
MMLL-PPO
0.326
0.288
Reduction
0.45
Reduction
0.324
Reduction
10%
50%
0.44
50%
0.322
50%
0.286
99%
99%
99%
Boseine
0.43
Baselire
Bavelime
36
38
30
32
32
34
36
38
Layer Number
Layer Number
Layer Number
Figure 3: Effect of low-rank approximation on the attn. v-proj. weight (value projection ma- trix) across selected layers Granite 8B, evaluated on the Leaderboard benchmark.
Layer: FC_IN Metrics Across Reduction Percentages
IFEval
BBH
MATH Lvi 5
0.10
0.55
0.495
0.08
0.490
IFEval
0.45
10%
88H
0.485
0.480
Reduction
10%
MATH
0.04
Reduction
10%
50%
50%
50%
0.35
0.475
0.00
30
32
34
36
38
30
32
34
38
30
32
34
36
38
Layer Number
Layer Number
Layer Number
GPQA
MUSR
MMLU-PRO
0.50
Reduction %
Reduction %
0.290
0.334
0.49
50%
MMLU-PRO
GPQA
0.285
90%
0.332
99%
0.48
Baseline
Baseline
Reduction
0.280
0.330
50%
0.275
99%
Baseline
0.46
36
38
30
32
34
36
Layer Number
Layer Number
Layer Number
Figure 4: Effect of low-rank approximation on the www.maat.ppppp.mmmmmmmmmmmmmmmma weroj.weeghte weight (first feedforward projection) across selected layers in Granite 8B, evaluated the Leaderboard benchmark.
20

---

Published as a conference paper at ICLR 2026
Layer: FC_OUT-
Metrics Across Reduction Percentages
IFEval
BBH
MATH Lvi 5
0.55
0.495
0.50
IFEval
MATH Lv15
B8H
Reduction 10%
Redurtion
0.08
Reduction
0.40
0.485
30%
10%
50%
50%
50%
0.35
99%
99%
0.07
95%
Baseline
Bosctine
Boseline
0.460
34
36
38
30
32
32
34
36
38
Layer Number
Layer Number
Layer Number
GPOA
MUSA
MMLU-PRO
0.295
0,290
0.475
0.330
GPOA
0.285
MUSR
0.470
0.290
Reduction %
0.465
Reduction %
0.328
Reduction %
0.270
0.275
39%
50%
0.460
95%
50%
10%
0.326
0.327
95%
Baseline
Baseline
Basaline
0.455
34
36
38
30
32
34
36
38
Layer Number
Layer Number
Layer Number
Figure 5: Effect of low-rank approximation on the wwtp:/owwwwoowwww.oooooo......o. weight (third feedforward projection) across selected layers in Granite 8B, evaluated on the Leaderboard benchmark.
vectors), and evaluated the impact of each intervention on performance the Open LLM Leader- board v2 benchmark"] consisting of six tasks MMLU-Pro, GPQA, MuSR, MATH, IFEval, and BBH. Consistent with prior work, we observed that some low-rank approximations maintained or even improved performance, highlighting the redundancy and compressibility of these matrices (see Figures 3 4 and [ and Table 55). Each experiment involved a single intervention defined by tuple specifying the layer number, matrix type, and reduction percentage.
Our approach assumes that lower singular vectors can safely accommodate new knowledge without significant forgetting. Specifically, our method relies on the premise that fine-tuming in the directions of low singular vectors will not interfere with previously learned tasks. This assumption holds only if the data from earlier tasks lie predominantly in the subspace spanned by the high singular vectors. If task-specific information from earlier tasks resides in the span of the low singular vectors, mod- ifying these directions could lead to interference-est expecially if the associated singular values were previously small (effectively suppressing higher- frequerfryyyency components or noise), but are increased during learning new tasks, thereby reactivating those suppressed directions. Formally, we expand the weight matrix via SVD as:
W Σ OF U.!
(17)
To empirically verify this, we investigate whether the output components of previous tasks in the hidden layer, when projected onto the low singular vector subspace, are negligible. In particular, we compute the 1.2 norm of the matrix-vector product between the outer product of each singular vector pair u,v, and the input vector (from previously learned task) without scaling by the corresponding singular value. This helps determine whether the old task input lies in the null space of the low singular vectors merely yield small outputs due to low singular values. If the L2 norms of the matrix-vector products corresponding to low singular vectors are near zero, we can safely update these directions for new tasks without affecting the prior task.
We perform this analysis on the mip. down proj. . weight matrix in layer 34 of Granite 8B using data from previously learned task. The results are presented in Figure 6 As expected, the output norm steadily decreases from left to right, where the x-axis corresponds to singular vector indices sorted in descending order of singular values. The three highest singular directions yield
m1 leaderboarreoooo hwwp//oardd/bboo.
21

---

Published as a conference paper at ICLR 2026
L2 Norms of Matrix-Vector Product Across Singular Vectors
Smoothed Norms of Matrix-Vector Product
Average Norm of Outputs
40
30
20
0
1000 Singular Vector Index (i) in Descending Order of Singular Values
0
2000
3000
4000
Figure 6: L2 norms of matrix-vector products for each singular vector component in the mlp. down_proj. weight matrix (layer 34, Granite 8B), using inputs from a previously learned task. The clear downward trend confirms that low singular directions have minimal activation for the learned task.
norms of 55.5, 18.1 and 1.8, respectively, indicating sharp drop in signal strength after the top components. This supports the theoretical redundancy hypotheses (Chen et al. 2020 Sharmaa et al, 2023), validating adaptive low-rank continual learning strategy. In particular, this layer retained performance even after 99% rank reduction, matching the performance of the unmodified Granite 8B model the Leaderboard benchmark.
These diagnostic experiments laid the groundwork for our final approach, which leverages projected gradient descent restricted to low-rank subspaces. Importantly, these subspaces are adaptively se- lected to minimize interference with previously Icarned tasks while preserving expressive capacity for learning new ones. Detailed analysis of singular value statistics across all layers and matrix types is provided in Appendix A.5
A.5 SINGULAR VALUE STATISTICS AND RANK ANALYSIS OF THE GRANITE 8B MODEL
To better understand how select which singular vectors to fine-tune within model weight ma- trices, we analyzed the singular value statistics of each matrix using tools from Random Matrix Theory (RMT). Specifically, fostributionn-olllovvngg the approach SPECTRUM (Hartford et al. 2024) to distinguish signal from noise. Singular values that fell below this bound were treated as noise, allowing us estimate the effective rank of each matrix. However, we observed that, under this criterion, all weight matri- the Granite 8B model appear be full-rank. This outcome is attributed to the violation of the core assumptions of the Marchenko- Pastur nawely that matrix entries are independently and identically distributed whichh which clearly does hold in trained language models where parameters are highly structured and correlated. Consequently, we adopted a scaled thresholding approach, in- formed by descriptive statistics such as the minimum, mean, median, and maximum singular values within each layer.
examined the of the lower bound of the Marchenko-Pastur
To support the adaptive rank selection strategy introduced in the main paper, we performed a comprehensive analysis of the singular value spectra across all weight matrices in the Granite 8B model. For each matrix type (e.g., j.proj, k-proj, v-proj, o-proj, gat p.roj. up-pro down.projj, compute and visualize the distribution of minimum, maximum, mean, and median singular values across all transformer layers (Figures/7 We also construct heatmap illustrating the variation of mean singular values throughout the network (Figure 14). These statistics provide
22

---

Published as a conference paper at ICLR 2026
useful insights into which low singular vectors and corresponding subspaces suitable for fine- tuning during continual learning.
Singular Value Statistics for q_proj
Min Singular Value
Max Singular Value
Mean Singular Value
8
Median Singular Value
Singular Value
2-
0
10
15
20
25
30
35
40
Layer Number
Figure 7: Singular value statistics for the attn.
attn.t.pro........ q.proj. wei ght matrix across Granite 8B layers.
Singular Value Statistics for k_proj
Min Singular Value
Max Singular Value
Mean Singular Value
Median Singular Value
Singular Value
10
15
20
25
30
35
40
Layer Number
Figure 8: Singular value statistics for the attn. k.proj. we whi matrix across layers.
A.6 COMPARISON WITH SVD BASELINES: MILORA AND PISSA
MiL.RA PiSSA are closely related to our setting, as they also use SVD to decompose weight matrices and then update only subset of singular directions using LoRA-style adapters. MiLoRA performs SVD the weight matrix, freezes the high-singular-ve value components (interpreted as the "pretrained base"), and updates the now-sinuular-valu components trainable adapters. PiSSA ap- plies the opposite split: it freezes the low-singular-var value components and updates the high-singular- value components adapters.
For a direct comparison, we follow the math reasoning setup from the MiLoRA paper and fine-tune LLaMA-2-7B on the MetaMathQA dataset (395K samples combining GSM8K and MATH), We
23

---

Published as a conference paper at ICLR 2026
Singular Value Statistics for proj
Min Singular Value
3.0
Max Singular Value
Mean Singular Value
Median Singular Value
2.5
Singular Value
2.0
0.5
0.0
10
15
30
35
40
Layer Number
Figure Singular value statistics for the attn. v-proj weight matrix across layers.
Singular Value Statistics for o_proj
Min Singular Value
Max Singular Value
Mean Singular Value
Median Singular Value
Singular Value
10
15
20
25
30
35
40
Layer Number
Figure 10: Singular value statistics for the attn.c o proj. weight matrix across layers.
evaluate on the GSM8K and MATH test sets and report Exact Match (EM) on the final checkpoint. Table summarizes the results.
Table 6: Comparison with MiLoRA and PiSSA on the MetaMathQA setup using LLaMA-2-7B, We report Exact Match (EM) on GSM8K and MATH, along with the average.
Method
GSMSK
MATH
Avg.
Full FT
66.5
19.8
43.2
LoRA
60.6
16.9
38.7
PiSSA
58.2
15.8
37.0
MiLoRA
63.5
17.8
40.7
OSFT (ours)
69.7
18.2
43.95
24

---

Published as a conference paper at ICLR 2026
Singular Value Statistics for gate proj
Min Singular Value
Max Singular Value
Mean Singular Value
Singular Value
8-
Singular Value
6-
0-
0
25
30
35
40
Layer Number
Figure 11: Singular value statistics for the mlp. gate proj. wei matrix across layers.
Singular Value Statistics for up_proj
Min Singular Value
Max Singular Value
Mean Singular Value
Median Singular Value
Singular Value
4
2-
10
15
20
25
30
35
40
Layer Number
Figure 12: Singular value statistics for the mlp. up proj. wei weight matrix across layers.
OSFT slightly improves on the full fine-tuning average and outperforms both MiLoRA and PiSSA on this math reasoning setup, indicating that constraining updates orthogonal low-singular- value subspace can match or exceed strong SVD-based PEFT baselines while maintaining the con- tinual learning benefits studied in the main paper.
A.7 COMPARISON WITH SPARSE FINE- TUNING
Sparse fine-tuning approaches aim to mitigate forgetting by selectively freezing or reactivating im- portant parameters. We compare our method against Lottery Ticket Adaptation (LoTA) (Panda et al.) 2024), which uses sparsity masks to preserve critical weights from source task while adapting to new task. We reproduce the task transfer setup from Table 5 of their paper using the Mistral-7B- model. Each experiment begins with an instruction follooiig task (Task A), followed by transfer to a downstream task (Task B). The goal is to preserve performance on Task A while adapting effec- tively to Task B.
25

---

Published as a conference paper at ICLR 2026
Singular Value Statistics for down proj
Min Singular Value
Max Singulaar Value
Mean Singular Value
Median Singular Value
Singular Value
4
2-
10
Layer Number
Figure 13: Singular value statistics for the mlp. dwwn. weight matrix across layers.
Heatmap of Mean Singular Values Across Layers
1.07
0.72
0.48
0.50
1.38
1.03
1.43
0,51
0.65
0.58
1.32
1.15
1.34
0.91
0.51
0.65
0.58
1.28
1.20
1.30
0.89
0.53
0.66
0.59
1.27
1.21
1.30
0.87
0.53
0.66
0.61
1.28
1.19
1.31
0.85
0.48
0.62
0.58
1.27
1.18
1.30
0.54
0.63
1.28
0.52
0.65
1.28
1.19
1.32
0.53
0.66
0.56
1.30
1.18
1.34
0.89
0.51
0.67
0.54
1.29
1.18
1.33
0.92
0.51
0.66
1.28
1.17
1.33
0.87
0.50
1.28
1.16
1.33
0.80
0.46
228
37 353122 313022282220022 202222222222 2423222100 2223222221111111111 201112201111111111
1.29
1.16
1.33
0.77
Mean Singular Value
1.31
1.16
1.34
0.80
0.55
0.62
Number
1.32
1.15
1.36
0.79
0.52
0.61
0.58
1.31
1.16
1.35
0.82
0.54
0.64
0.62
1.31
1.17
1.34
0.77
0.55
0.61
0.65
1:18
0.56
0.64
1.29
1.20
1.31
0.82
0.59
0.66
0.68
1.29
1.21
1.30
01000
0.14
0.00
0.29
0x
1.29
1.21
1.30
0.00
0.14
0.00
0.30
1.29
1.22
1.30
0.32
0.61
0167
0.73
1.28
1.22
1.31
0.15
0100
1.28
1.22
0.16
0.00
0.35
1.22
0.74
1.28
1.22
1.32
0.2
0.00
0.58
1.29
1.21
1.33
0.75
0.61
0.63
0.77
1.31
1.20
1.34
0.73
0.59
0.60
0.75
1.31
1.19
1.34
0.67
0.60
0.57
0.79
1.33
1.18
1.35
0.69
0.61
0.58
0.80
1.34
1666
1.35
0.67
0.90
1.35
1.34
0.72
0.49
1.00
1.35
1.34
0.49
0.75
0.48
1.07
1.33
16
1.33
0.38
0.79
0.42
1.26
1.27
1.23
1.35
0.38
0.80
0.42
1.21
mlp.down_pme praj mlp.gate_proj
mip.up_prm)
selt self_attn.k wttpttttkkk_mmm __rojjjjjjjj_ooo self, att_atttnaaaaooooa_o_aaa self, wwwpjattn.___p_ __oro self_ saff_aattnvv_pp_ Layer Name
Figure 14: Heatmap of mean singular values across matrices and transformer layers Granite 8B.
As shown in Table our method OSFT consistently matches outperforms LoTA in mitigating for- getting (Task A) while achieving comparable or higher downstream task performance (Task B). For example, GSM8K and Inthuntt method achieves stronger retention and higher accuracy
26

---

Published as a conference paper at ICLR 2026
Table 7: Comparison between LoTA and method sequence of two tasks. Each row corre- sponds to a setup where the model is first trained on the instruction-folloiing task (Task A) and then fine-tuned on downstream task (Task B). We report the utility (accuracy task-specific metric) on both Task A and Task B after training on the second task. Values in parentheses indicate the drop in Task task alone. Lower drop in Task A utility indicates better forgetting mitigation; higher Task B utility indicates better learning and adaptation.
Task performance compared to the Baseline, where the model is trained on that
Task B
Task A Method
Task B Method
Task A Utility (L)
Task B Utility (J)
Instruction Follow
Baseline
19.0
Baseline
59.8 (-)
GSM8K
FFT LoTA
OSFT (ours)
18.6 (0.4)
60.1 (-)
LoTTO
17.8 (1.2)
59.1 (0.7)
Baseline
56.7 (-)
MathInstruct
FFT
OSFT (ours)
17.8 (1.2)
60.1 (-)
LoTA
LoTA
16.0 (3.0)
55.5 (1.2)
Baseline
83.5 (-)
Reasoning
FFT
OSFT (ours)
18.3 (0.7)
82.1 (1.4)
LoTA
LoTTO
16.5 (2.5)
83.7 (+)
Baseline
77)0
GSMSK+ARc+SLL
FFT LoTA
OSFT (ours)
14.7 (4.3) 15.9 (3.1)
76.2 (0.8)
LoTTO
73.8 (3.2)
Bascline
93.1 (-)
Safety
FFT
OSFT (ours)
71.8 (21.3)
LoTA
LoTTO
63.4 (29.7)
on the new task. On Reasoning, we maintain near-zere zero drop in Task A utility, while LoTA suffers larger decline. These results demonstrate the effectiveness of the full-parameter constrained update approach relative to sparse masking.
A.8 COMPARISON WITH MODEL MERGING TECHNIQUES
We compare against two model merging techniques-SLERR (Spherical Linear Interpolation) and TIES (Task-Informed Ensemble Synthesis)-to assess their applicability the continual learning setting. SLERP was applied by merging full model weights sequentially: after each task, the model was interpolated with the next task's model on the unit hypersphere. TIES was applied to linearly combine task-specific LoRA adapters using weights tuned a held-out validation set. Our method OSFT significantly outperforms both (see Table T. In continual learning benchmarks involving many tasks, such the 5-task and 1.5ttakk settings examined here, finding effective merge strategies becomes increasingly challenging. Moreover, even after identifying an optimal strategy, extensive hyperparameter tuning, experimentation, and expert knowledge are typically required to merge mod- els effectively without compromising task performance over long task sequences. This complexity makes such merging approaches less practical compared to our proposed method.
A.9 ABLATION STUDIES
To better understand the contribution of key components in our method, we conduct two ablation studies using the L-LMM-22 7B model on the standard continual learning benchmark comprising classification tasks (AG News, Amazon, Yelp, DBpedia, Yahoo). These ablations designed to evaluate: (I) the importance of accurate effective rank estimation for singular vector selection, and (2) the necessity of constraining updates remain within the low-rank subspace via projection.
(1) Impact of Inaccurate Effectivee Rank Estimation: Our method relies on computing an ef- fective rank per matrix based input-output activation similarity, which informs the threshold for partitioning singular vectors into high- and low-rank subspaces. To test the importance of this esti- mation, reduce both the minimum and target retention ratios (mrr and trr) half their original
27

---

Published as a conference paper at ICLR 2026
values. This results in more aggressive fine-tuning by retaining fewer high singular vectors, thus allocating more of the matrix capacity to learning new tasks. However, this also increases the risk of overwriting components important for previous tasks. As shown in Table 9 this ablation leads to substantial performance drop of just over 28 percentage points (from 79.6% to 51.5%) emphasizing the importance of accurately estimating the effective rank ensure that task-relevant subspaces preserved.
Ablation results over mrr and trr are shown in Table 8 The default setting (0.10, 0.80) gives the best average accuracy. Nearby values such as (0.05, 0.70) and (0.20, 0.90) perform similarly (within ~4-8 points), showing that OSFT is reasonably robust moderate changes. Very low retention (0.05, 0.40) and "flat" schedules where mir = tı (e.g., (0.50, 0.50j) hurt performance, and the extreme retention (0.70, 1.00) case performs worst, confirming that overly weak retention or overly high retention both degrade results.
Table 8: Sensitivity of OSFT to mrr and trr on the 5-task standard continual learning benchmark with LLaMA-2 7B. Average accuracy (%) over all tasks is reported.
mrr
trr
Avg. accuracy (%)
0.10
0.80
79.6
0,05
0,40
51.5
0.05
0.70
75.9
0.20
0.90
71.1
0.50
0.50
55.8
0.70
1.00
48.0
(2) Unconstrained Fine- Tine-Tuning of Low Singular Vectors: our method, gradient updates are projected back into the low-rank subspace to prevent interference with high-rank directions. This ablation removes that constraint: we freeze the high singular vectors but allow unconstrained updates the low singular vectors, meaning that during optimization, updates are not restricted stay within the initially identified low-rank subspace. This allows the low singular vectors to drift into the space previously occupied by high singular vectors, leading to potential interference and loss of previously acquired knowledge. As expected, this results in catastrophic forgetting, with accuracy dropping from 79.6% 31.2%. addition, since only the low singular vectors are updated while the high ones are frozen, each new task is forced to be learned in restricted subspace, limiting the model's overall expressiveness. Together, these factors result highlighting the necessity of maintaining orthogonality between new task updates and previously learned subspaces.
50-point accuracy drop,
Table 9: Ablation results on the LLaMA-2 7B model using the standard 5-task continual learning benchmark.
Method
Average Accuracy (%)
OSFT (ours)
79.6
(1) Halved mir/ur (aggressive effective rank approximation)
51.5
(2) No projection (unconstrained low-rank updates)
31.2
A. IMPLEMENTATION DETAILS
We detail the implementation of all experiments presented in this work. Our study utilizes both encoder- and decoder-only language models. For all continual learning experi- ments including the 5-task and 15-task benchmarks, as well as the TRACE benchmark repli- cate the task sequences. prompts, and dataset configurations as established in O-LoRA Wang et al. 2023a) and TRACE Wang al. (2023b).
T5-Large. Experiments with the T5- model were conducted single NVIDIA H100 GPU using standard PyTorch training in full precision. We used a constant learning rate of 10-5 with the AdamW optimizer and total batch size of training for one epoch per task. For each
28

---

Published as a conference paper at ICLR 2026
classification dataset, sampled 1,000 examples per class (where available) to construct balanced training sets, following the protocol established in Wang et al. (2023a). All runs were performed with fixed random seed, and checkpoints were saved after each task for evaluation and reproducibility.
LLaMA-2 7B and Mistral-7B. All experiments with the LLaMA-2 7B and Mistral-7B models were conducted on a server equipped with 8 NVIDIA H100 GPUs, using the DeepSpeed library with Stage 2 optimization. Gradient checkpointing was enabled, and training was performed with per-GPU batch size of 1 (resulting in an effective batch size of 8). We used the Adam optimizer with a learning rate of 1 x 10-5, weight decay of 0.01, 31 = 0.9, 32 = 0.999, and e = 1 x 10 All continual learning runs were trained for one epoch per task. After backpropagation, projection steps were applied to the gradients to constrain updates within the designated low-rank subspaces.
Our SVD configuration was automatically generated by analyzing specific matrices in each transformer block block-namely, q-proj, k_proj, v.proj, o.proi, gate-proj. up-proj. and down proj. Among the various strategies explored for determining which singular vectors to retain, we found empirically that two approaches consistently performed best. The first allocates
fraction of singular vectors for task in n-task i-l
fixed budget by freezing the top Y
quence. The second uses adaptive rank selection based on layer importance scores, as described in Section 3.4 where the number of retained singular vectors per layer is computed using the normal- ized importance (()) from Section 3.3 The remaining components were fine-tuned using projected gradient descent within the low-rank subspace.
Clarification mrr, Irr choice and 7(0) handling. We introduced two key hyperparameters. minimum retention ratio (mrr) and target retention ratio (trr) in Eq. equation 3 Empirically, we set these values as mrr = 0.1 and trr 0.8, which consistently yielded strong results across bench- marks with Ta-Large, LLaMA-2 7B, and Mistral-7B. As shown in Appendix|A.9] (PPeendiiA_9(TTaleeee9 halving these parameters (more aggressive fine-tuning on the new task) significantly reduced performance (from 79.6% down to 51.5%). demonstrating the importance of appropriately balancing stability and plasticity. Conversely, mild perturbations around the defaults (e.g.. #0.02 or #0.05) produced less than 1 1% change in accuracy, confirming robustness. Thus, starting from our recommended default (0.1, (.8) and performing a modest grid search around these values on early tasks is simple and effective strategy for practical tuning. Additionally, if prior knowledge about task difficulty is available, the retention rates can be adjusted accordingly.
Regarding negative layer importance scores (()); By definition, we compute 7()) the cosine sim- ilarity between the input activations ( and output activations y()) Empirically, we observed these values were consistently positive (ranging roughly between 0.5-0.8) across all layers and tasks. Nevertheless, for robustness, if any raw cosine similarity were negative, we explicitly clip it to zero before normalization, ensuring:
Σ10=1
This guarantees that the retention ratio
mrr X (full rank).
A.11 RUNTIME AND RESOURCE ANALYSIS
To assess the deployment practicality of our method, we analyze both theoretical and empirical resource overheads compared full fine-tuning (FFT).
Memory Efficiency during Training. The key efficiency gain comes during training. By freezing the high-rank components, we avoid storing their gradients and optimizer states. For an x matrix, the memory for parameters, gradients, and Adam optimizer states is roughly 4n2 for full fine-tuning, whereas method requires approximately 2n² + 6nr (for storing U/V factors and low-rank optimizer states, where r is the trainable rank). OSFT is therefore more memory-efficienn whenever < n/3, which typically holds in continual learning. In practice, < n/3 is sufficient
29

---

Published as a conference paper at ICLR 2026
because starting from instruction-tun tuned model, most prior capabilities must be retained, and one- third of the subspace capacity is generally adequate to learn new tasks. This constant memory footprint is significant advantage methods that store past gradients activation bases.
SVD Time Complexity. The time complexity of computing the SVD of an weight matrix is O(n"). We perform SVD once per matrix at the start of each task. For LLaMA-2 7B, where typical weight matrices have dimensions 4096 x 4096 and the input sequence length is L 512, this cost is on par with ~4 forward passes through a transformer block, since the cost of a single forward pass is OTT T.) Empirically, performing SVD all relevant matrices of LLaMA-2 7B takes approximately minutes on a single H100 GPU. This is less than 4% the total training time for typical 7k-sample dataset, becoming negligible for larger datasets.
Empirical Runtime. We profiled the wall-clock runtime of performing SVD all relevant ma- trices in LLaMA-2 7B. Using a single H100 GPU:
Time for SVD (all matrices): ~2 minutes
Time for fine-tuning on 7k-sample dataset for epochs: 25 minutes
Thus, the SVD overhead <10% of total training time on small datasets, and becomes negligible on larger datasets, Moreover, the SVD cost scales linearly with the number of tasks and matrices, making it practical even in hong-horizon continual learning.
Rank Estimation Cost. Rank selection based on capacity heuristics (e.g., dividing total budget by number of tasks) adds no computational cost. When using adaptive rank selection via input-output similarity, only require a single forward pass over ~50 samples from the previous task-an overhead of under 30 seconds on modern GPUs.
Comparison with Gradient Storage. Many existing continual learning methods store task- specific gradients or importance weights. For each task, this requires storing additional n² tensor per matrix, leading to Okk)²) total memory for k tasks. Our method avoids this entirely, making it significantly more scalable.
Our method incurs only a small one-time per-task cost for SVD computation and requires no addi- tional memory for gradient storage. Theoretical and empirical results confirm that it both scalable and suitable for deployment in resource-constrained environments.
A.12 TASK SEQUENCES AND PER-TASI PERFORMANCE
Across all three experimental settings the 5-task standard CL benchmark, the 15-task longer se- quence benchmark, and the 8-task TRACE benchmark-wv we strictly adhered the original config- urations of O-LoRA Wang et al.] (2223a) and TRACE Wang et al. (2023b). This included using the same datasets, task instructions for prompting models during classification and generation, and identical training and validation sample counts and label distributions per task. Task sequences were replicated exactly to ensure consistency across evaluations and facilitate fair comparisons.
To better assess forgetting and learning dynamics, we report per-task performance for one repre- sentative task sequence from the 5-task (Table and 15-task (Table T2p continual learning bench- marks, and the task sequence in the TRACE benchmark (Table 13 Each task reports two metries: the accuracy immediately after the task is learned (reflecting plasticity), and the final accuracy after all tasks trained (reflecting stability and forgetting).
These breakdowns provide more granular view of both adaptation (learning) and forgetting across tasks. The final results in the main paper (e.g.. Table [I] are averaged across multiple such task orders. The same orders were used for both T5-Large and LLAMA-2 7B experiments.
A.13 CODE AVAILABILITY
The implementation of the OSFT training method, including dataset processing and training pipelines is available at:
30

---

Published as a conference paper at ICLR 2026
Task orders for all six sequences used in the Standard Continual Learning benchmark experiments.
Table 10:
Order
Task Sequence
dbpedia + amazon + yahoo - ag
2
dhpedia amazon ag yahoo
3
yahoo + amazon - ag - dbpedia
4
mnli + cb + wic + copa + qqp - boolqa + rtc - imdb
+ yelp ~ amazon - sst - dbpedia ag + multirc - yahoo
5
multire- + boolqa + wic + mnli + cb + copa + qqp - rtc - imdb
sst 2 - dbpedia - ag + yelp amazon + yahoo
6
yelp - amazon mnli cb copa qqp - rte imdb
+ sst > dbpedia + ag + yahoo > multirc - boolqa + wic
Table 11: Per-task performance on the 5-task benchmark (Order 1, T5-Laage Model). Task order: dbpedia + amazon + yahoo + ag. Average Accuracy: 75.3, Backward Transfer: -3.7.
Task
Accuracy (After Task)
Accuracy (Final)
dbpedia
98.9
97.2
amazon
53.5
46.1
yahoo
74.3
68.7
ag
89.1
89.1
https:://ithuu..com/eee An--oootvttt wwwoonnTTTnnnniiiiaeeaaoaiainiee ceaiiiittrrinnn
A.14 USE OF LARGE LANGUAGE MODELS
Consistent with the ICLR 2026 disclosure policy, used large language models (LLMs) only to aid or polish writing (grammar and English fluency) and for minor assistance in editing experimental code (e-g., debugging syntax corrections). LLMs analysis, methodology design, data analysis, or result generation. All scientific contributions, theo- retical derivations, core code development, and experiments were implemented and validated solely by the authors.
not used for research ideation, theoretical
31

---

Published as a conference paper at ICLR 2026
Table 12: Per-task performance on the 15-task benchmark (Order 4, T5-Large Model). Task order:
mnli - cb + wic + copa + पчР boolqa + rte - imdb - yelp + amazon + sst-2 - dbpedia + ag + multire + yahoo. Average Accuracy: 71.6, Backward Transfer: -5.5.
Task Accuracy (After Task)
Accuracy (Final)
mnli cb
74.9
62.3
83.6
75.0
wic
57.2
51.2
copa
54.4
47.0
qqp
85.8
82.4
boolqa
83.2
77.3
rte
83.4
80.5
imdb
96.1
94.5
yelp
59.7
48.6
amazon
54.2
49.1
sst-2
92.3
93.3
dbpedia
98.5
94.6
ag
84.8
72.1
multirc
78.1
75.8
yahoo
69.8
69.8
Table 13: Per-task performance on the TRACE benchmark (LLaMA-2-7B-Chaal Task order: C-
STANCE - FOMC MeetingBank - Py150- - ScienceQA - NumGLUE-cm + NumGLUE- ds + 20Minuten. Average Accuracy: 48.4, Backward Transfer: -7.1.
Task
Accuracy (After Task)
Accuracy (Final)
C-STANCE
0.48
0.42
FOMC
0.65
0.58
MeetingBank
0.56
0.49
Py150
0,60
0.48
ScienceQA
0.73
0.67
NumGL.EF.
0.37
0.28
NumGLUE-ds
0.54
0.51
20Minuten
0.44
0.44
32


Reviews:
[{"review_id": "TRcLc6K8Wr", "reviewer": "Reviewer_5uQ5", "summary": "This paper proposed the orthogonal subspace fine-tuning (OSFT) method in continual learning. It performs SVD on each layer and utilizes the cosine similarity between input activations and its linear outputs to determine high-rank subspace and low-rank subspace of each layer weight. To preserve the important previous knowledge, OSFT orthogonally updates in low-rank subspaces by projecting gradients onto low-rank subspaces. Experimental results on continual learning benchmarks show the effectiveness of OSFT.", "questions": "1. In this paper, OSFT updates on the subspaces consisting of smaller singular values of the weights, but in CorDA [1] and SVD-LLM [2], both works utilize the input information along with the weights to compute SVD for obtaining the subspaces of smaller/larger singular values. Especially, SVD-LLM states that “truncating smaller singular values in SVD could lead to significant compression loss”, which means there is no linear relationship between performance and singular values obtained only by weight itself. Can authors explain the OSFT’s differences from the conclusions of these two papers?\n\n[1] CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning, NeurIPS 2024.\n\n[2] SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression, ICLR 2025.\n\n\n2. Can authors compare the experimental performance of OSFT with SAPT, InfLoRA, or CorDA(Knowledge-preserved adaptation version)?\n\n\n3. In line 15 in Algorithm 1, OSFT “computes gradients for trainable SVD components”. What’s the interval step of computing SVD components? Does OSFT compute SVD at each training step? If so, the computation and training time of OSFT is significantly large, which is not efficient and not practical. Can authors compare the training time and memory cost of OSFT with O-LoRA, SAPT, or InfLoRA?\n\n4. Compared to LoRA-based methods, OSFT needs to compute the gradients of full-dimensional weights, which is very expensive for LLMs. Although the paper title is about “constrained full fine-tuning in LLMs”, FFT is costly for LLMs in downstream tasks. Also, if OSFT belongs to FFT in LLMs, then experiments may need to be redesigned since the current used benchmarks, like GLUE and SuperGLUE, are sampled with 1000 training samples, which are too small for FFT.\n\n5. What’s the choice of effective rank of OSFT in Figure 1? Can authors clarify the experimental settings of OSFT in Figure 1? It’s confusing about how to obtain the three different dots of OSFT in Figure 1.\n\n6. The paper does not discuss in detail why to choose cosine similarity to compute each layer’s importance. Since there are other useful measurements, for example, centered kernel alignment (CKA) [1] measures the similarity between the intermediate activations of two models at each layer. Can authors discuss the difference between these similarities?\n\n[1]  Similarity of neural network representations revisited, ICML 2019.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed OSFT considers the difference between high-rank and low-rank subspaces to achieve orthogonal gradient updates in directions which are orthogonal to previous knowledge.\n\n2. The proposed OSFT adaptively allocates parameter budgets across layers, rather than treating them equally, which balances stability and plasticity across the network based on the role of each layer.", "weaknesses": "1. The computation cost of OSFT would be huge since it needs to compute full-dimensional gradients during training, and then projects full-dimensional gradients to low-rank parameter subspaces obtained by computing SVD on each layer weight.\n\n2. The motivation of the design for OSFT is not well constructed and discussed. For example, authors only cited one paper without any discussion to support the choice of cosine similarity. Since this similarity choice should be one contribution of this paper, which connects with the determination of low-rank subspace, it should be discussed and evaluated carefully.\n\n\n3. Experiments lack recent parameter-efficient continual learning baselines, SAPT [1], InfLoRA [2], and CorDA [3]. Also, Table 1 does not show the performance on each task order, which is important to show the robustness of the proposed method, and Table 8 in the appendix only shows order 1. Besides, it’s not clear about the task order in Table 3.\n\n[1] SAPT: A shared attention framework for parameter-efficient continual learning of large language models, ACL2024.\n\n[2] InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning, CVPR2024.\n\n[3] CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning, NeurIPS 2024.\n\n\n4. The ablation study disappears in the experiment section. In Eq (3), mrr and trr should be evaluated with different values to show the performance or robustness of OSFT. The authors just mentioned “ablation studies show that while performance degrades significantly if retention is too aggressive”, but authors do not provide these results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q1H9rTPnKF", "reviewer": "Reviewer_oGvj", "summary": "The paper proposes OSFT for continual learning in large language models. OSFT first performs SVD on each weight matrix to identify high‑rank directions that encode prior knowledge and low‑rank directions that can be safely reused. It computes a layer‑importance score and retains more singular vectors in crucial layers, allocating more adaptable capacity to less crucial ones. During training, OSFT projects gradients orthogonally to the preserved subspaces. The authors support this design with a theoretical analysis linking forgetting to Hessian curvature and with the use of top singular vectors as a practical proxy for high‑curvature directions.", "questions": "1. Is there a reason why authors did not measure forgetting per task and forward transfer with comparison methods (A.11 partly helps, but no aggregate forgetting stats are given)?\n2. Existing LoRA-based methods do not appear to suffer from performance degradation as severe as that observed with the proposed method on the GSM8K benchmark. In this regard, is there a particular reason why experiments with LoRA-based approaches were not conducted?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-structured and easy to read.\n2. The paper includes experiments on traditional benchmarks.", "weaknesses": "1. Gradient projection onto an orthogonal complement is a known CL idea. OSFT’s twist is: project in weight space using SVD‑identified directions, not activation space; and make the preserved rank per layer adaptive. The layer‑importance measure is explicitly inspired by AdaSVD, and the theoretical appeal of preserving top singular directions is attributed to an external empirical correlation. This strengthens the sense of a careful integration of existing ideas more than a conceptual break‑through.\n2. The proxy that top singular vectors ≈ high‑curvature/Hessian directions is justified by citing external robust correlation, not by measurements on the authors’ own T5/LLaMA runs. The paper does not report a Hessian–SVD alignment check on its trained models. \n3. The proposed method activates more parameters than O-LoRA but fewer than full fine-tuning. By doing so, it improves performance compared to O-LoRA while mitigating forgetting relative to full fine-tuning. This suggests that the method effectively identifies an appropriate number of parameters to utilize. If there exist experiments that adjust the number of active parameters without applying the proposed method, it would help clarify the unique contribution of the proposed approach.\n4. The paper cites Inflora (Liang & Li, 2024) and LoTA (Panda et al., 2024). LoTA is only compared in an appendix two‑task transfer, not in the main CL sequences; Inflora is not benchmarked at all, despite being specifically about interference‑free low‑rank continual learning. This weakens the empirical positioning.\n5. Figure 1 notes “crucial hyper‑parameters varied for each method,” but the ranges, tuning protocol, and compute budget per method aren’t given. \n6. It’s unclear whether baselines received equivalent tuning or comparable trainable‑parameter fractions—particularly important because OSFT is shown at 56% while LoRA baselines are 1–3%. Claims of frontier domination are therefore budget‑confounded.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lKCL3YiayO", "reviewer": "Reviewer_uSxf", "summary": "The authors propose **Orthogonal Subspace Fine-Tuning (OSFT)** for continual learning. They utilize adaptive singular value decomposition (SVD) to dynamically identify and preserve critical, high-rank parameter subspaces. They first introduce *importance-guided allocation*, i.e., measuring each layer’s importance via input–output cosine similarity (where low-similarity layers are allocated more adaptive capacity). This determines importance scores, which, together with retention ratios, define the fraction of singular vectors to preserve per layer. The authors perform *adaptive subspace identification*, i.e., decomposing each layer via SVD to find critical components with a selected rank. Lastly, they apply *orthogonal gradient projection*, i.e., projecting gradients to be orthogonal to the critical subspaces. Concretely, they freeze high-rank components, and project the low-rank component's gradients to be orthogonal to the high-rank basis vectors.\n\nThey evaluate the method across two continual-learning benchmarks and more realistic TRACE benchmark, using encoder–decoder models (T5-Large) and decoder-only models (LLaMA-2-7B, Mistral-7B). They report average accuracy, forgetting measures, and retention of general capabilities and safety.", "questions": "1. O-LoRA appears to use significantly fewer parameters, and its trade-off curve leads to a wider curve which seems desirable. Could you comment on that? In Figure 1 - why change the rank in O-LoRA, and not the strength of regularization? Similarly, for replay-based baselines, why the learning rate is changed and the amount of data?\n2. The related work section does not discuss MiLoRA [1] and PiSSA [2], which seems like a significant oversight - can you include them, also in the experimental section? Especially MiLoRA seems like a crucial method.\n3. Can the authors explain what happens when the number of tasks grows large? How does that affect required SVD rank, is there a threshold before capacity becomes an issue? Can the authors relate the SVD subspace of task 0 and task 5, for example? How is interference with early tasks prevented over time?\n4. Can the authors further motivate the choice of a *weight-space* approach versus a *function-space* (activation-based) approach?\n5. The cited empirical findings (Sharma et al., 2023; Li et al., 2025) show that layers with higher input–output similarity exhibit greater Hessian curvature. Could the authors expand on this relationship?\n6. Why do PerTaskFT and MTL report “N/A” for the BT metric?\n7. Page 6 L297 - what is the reference for that?\n\n[1] MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning. (2025) Hanqing Wang and Yixia Li and Shuo Wang and Guanhua Chen and Yun Chen\n\n[2] PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models. (2024) Fanxu Meng and Zhaohui Wang and Muhan Zhang", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a thoughtful and well-motivated approach to preserving critical subspaces in continual learning.\n\n---\n\n- The paper includes computational analysis as well as theoretical justification.\n- A strong set of comparison methods is included.\n- A large number of datasets are used in the evaluation.\n- The related work section is generally good.\n- The hyperparameters (e.g., mrr, trr) appear to have sensible default values, and the authors provide practical guidance for tuning.\n- The evaluation follows standard protocols, averaging results over three independent runs with randomly permuted task sequences.", "weaknesses": "### **1. Methodological Clarity & Design**\n\n- Data from task (t) is used when evaluating layer importance for task (t+1).\n- The methods section lacks a clear, cohesive narrative.\n- The method does not appear particularly parameter-efficient relative to other PEFT approaches.\n- The claim that the method “retains and transfers knowledge across tasks” may be too strong; empirical evidence for *transfer* should be clarified.\n\n---\n\n### **2. Missing Definitions & Introductions**\n\n- SLERP, TIES, and TRACE are not properly introduced before being used in comparisons.\n- Figure 1 uses terms (TRACE, average backward transfer, forgetting) without definition.\n- Cosine similarity is referenced conceptually but never explicitly visualized or explained in the results.\n\n---\n\n### **3. Comparisons & Baselines**\n\n- Prior work [1] shows that full fine-tuning forgets more; therefore, full fine-tuned baselines may not be appropriate, and stronger PEFT baselines should be included.\n\n---\n\n### **4. Presentation & Figure Issues**\n\n- Several plots require cleanup and clarifications; Figures 1 and 2 are presented out of order.\n    - Figure 2 is difficult to interpret (e.g., unclear connection between “layer 2 = layer ℓ”, too many elements, cosine similarity not referenced).\n\n---\n\n### **5. Organization & Structure**\n\n- Section 3.7 duplicates material that would be more appropriately placed in the related work section.\n\n[1] LoRA Learns Less and Forgets Less. (2024) Dan Biderman and Jacob Portes and Jose Javier Gonzalez Ortiz and Mansheej Paul and Philip Greengard and Connor Jennings and Daniel King and Sam Havens and Vitaliy Chiley and Jonathan Frankle and Cody Blakeney and John P. Cunningham", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cPuuI07ncx", "reviewer": "Reviewer_zn3T", "summary": "The paper proposes OSFT (Orthogonal Subspace Fine-Tuning), a continual learning method for LLMs that (i) performs per-layer SVD of weight matrices, (ii) treats the top singular directions as a high-rank subspace presumed to carry previously acquired knowledge, and freezes them, and (iii) constrains all parameter updates for new tasks to the orthogonal complement (low-rank subspace). To avoid a one-size-fits-all rank, the method uses an input–output cosine similarity proxy to adaptively determine, per layer, how many top singular directions to preserve. Implementation can be realized either by explicit gradient projection or by a reparameterization that freezes the high-rank factors and optimizes only the low-rank factors. The authors provide a second-order analysis suggesting a tighter bound on worst-case forgetting than both unconstrained full fine-tuning and fixed-rank projection, and report empirical improvements over strong orthogonal/low-rank baselines across multiple sequential learning benchmarks.", "questions": "- Stability of general capabilities: how can the evaluation scope and statistical rigor be strengthened?\nCould the authors expand to mathematics (e.g., GSM8K/MathBench), code (HumanEval/MBPP/Codeforces-style reasoning) and tool use (ReAct/ToolBench), and report multi-seed mean±std under compute-matched settings? Additionally, plotting capability drift curves across task order (before/after sequential updates) would offer a more granular view of the stability–plasticity dynamics.\n\n- SVD–curvature relationship: can the link be verified quantitatively and leveraged adaptively?\nThe approach relies on the empirical premise that top singular directions approximate high-curvature/importance directions. Could the authors, for representative layers, extract the leading Hessian/Fisher eigenvectors and measure principal angles to the top singular vectors? Further, could they compare “protect SVD-top vs protect Fisher-top vs a joint criterion,” and explore Fisher-weighted singular-value ranking or multi-criteria scoring to define the preserved subspace? Such analyses would more directly substantiate the curvature surrogate assumption and might yield an even more robust adaptive preservation rule.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Conceptually transparent and practically implementable “stability–plasticity” mechanism.\nThe paper operationalizes “protecting prior knowledge” as a geometric constraint on update directions: disallow updates along top singular modes and permit updates only in their orthogonal complement. This aligns with the intuition that model behavior is most sensitive along dominant spectral directions while leaving substantial slack in the complementary subspace for learning. Importantly, the mechanism does not require rehearsal buffers or per-task parameter growth. Both explicit gradient projection and a reparameterized module (freezing high-rank, training low-rank) are described, making integration into standard PyTorch/PEFT pipelines straightforward.\n\n- Layer-wise adaptive capacity allocation that mitigates the brittleness of fixed-rank approaches.\nBy estimating a layer’s “knowledge-carrying importance” via input–output similarity and then allocating more preserved high-rank directions to important layers, the method avoids uniform rank settings that can either under-protect critical layers or overconstrain unimportant ones. This per-layer adaptivity is well-motivated for Transformers—whose layers differ in function—and empirically reduces sensitivity to rank choice while improving overall stability–plasticity trade-offs.\n\n- A coherent theory–practice loop.\nThe paper offers a second-order analysis yielding a hierarchy of forgetting bounds (adaptive projection < fixed-rank projection < unconstrained full fine-tuning) and leverages the widely observed empirical link between large singular directions and high-curvature/importance directions as a computationally feasible surrogate for direct curvature estimation. The result is a self-consistent narrative—curvature → dominant spectral directions → orthogonal update constraint—that ties mathematical justification to implementable proxies.\n\n- Reasonably broad empirical coverage with diagnostic ablations.\nEvaluations span both encoder–decoder and decoder-only architectures, short and long task sequences, and more comprehensive sequential scenarios, with comparisons to state-of-the-art orthogonal/low-rank baselines (e.g., O-LoRA–style methods). Ablations (e.g., removing orthogonal projection; reducing preserved rank) produce large and consistent degradations, highlighting that both the orthogonality constraint and the adaptive preservation rule are functionally essential. The fact that model size remains constant across tasks is also practically appealing for deployment.", "weaknesses": "- Computational budget and fairness (stronger budget-controlled comparisons are needed).\nAlthough OSFT maintains constant model size across tasks, its per-task training still operates in the full parameter space under constraints, which can entail higher effective degrees of freedom and greater compute/time than small-parameter PEFT schemes (e.g., LoRA/adapters). The paper lacks strictly budget-matched comparisons (e.g., equal GPU-hours, equal training steps, equal hyperparameter search budgets) and energy/latency reporting. Without such controls, reported gains may partially reflect additional compute rather than methodological superiority. A detailed performance–cost table/curve would materially strengthen the claims.\n\n- Breadth of general capability and safety evaluations (evidence remains narrow).\nThe paper asserts that general abilities and alignment/safety are preserved after continual updates, yet the evaluation scope is limited relative to the breadth of LLM competencies (mathematical reasoning, code generation, tool use, long-context robustness, multilingual performance, jailbreak resistance, hallucination propensity, etc.). Stronger conclusions would require wider, multi-dimensional assessments with multi-seed mean±std and budget-controlled settings to rule out noise and tuning bias.\n\n- Limited coverage of recent open-source model families (e.g., Qwen3).\nExperiments primarily focus on T5-Large and LLaMA-2/7B-scale models. Given the rapid adoption of newer families—Qwen3, Llama-3, etc.—the method’s external validity would be clearer with systematic evaluations on these up-to-date backbones, ideally alongside the model-specific best practices for PEFT/CL provided by those communities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)