Merge before Forget: A Single LoRA Continual Learning via Continual Merging
Authors: Fuli Qiao
Mehrdad Mahdavi
Venue: ICLR 2026 Poster (2026)
Keywords: Continual learning
Low-rank adaptation
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: A single LoRA continual learning via continual merging

Abstract: Parameter-efficient continual learning has emerged as a promising approach for large language models (LLMs) to mitigate catastrophic forgetting while enabling adaptation to new tasks. Current Low-Rank Adaptation (LoRA) continual learning techniques often retain and freeze previously learned LoRAs or generate data representations to overcome forgetting, typically utilizing these to support new LoRAs learn new tasks. However, these methods not only ignore growing computational memory with tasks and limited storage space but also suffer from potential task interference due to the lack of effective LoRA merging mechanisms. In this paper, we propose a novel continual learning method that orthogonally initializes and sequentially merges LoRAs updates into a single unified LoRA. Our method leverages orthogonal basis extraction from previously learned LoRA to initialize the learning of new tasks, further exploits the intrinsic asymmetry property of LoRA components by using a time-aware scaling mechanism to balance new and old knowledge during continual merging. Our approach maintains constant memory complexity with respect to the number of tasks, minimizes interference between past and new tasks via orthogonal basis initialization, and improves performance over asymmetric LoRA merging via adaptive scaling. We provide theoretical analysis to justify our design and conduct extensive experiments across diverse continual learning benchmarks using various LLMs, demonstrating the effectiveness and efficiency of our method.

Published as a conference paper at ICLR 2026
MERGE BEFORE FORGET: A SINGLE LORA CONTIN- UAL LEARNING VIA CONTINUAL MERGING
Fuli Qiao
Mehrdad Mahdavi Pennsylvanie State University mzm61 60psu. edu
Pennsylvania State University fvg501 5 ppsu. edu
ABSTRACT
Parameter-efficient continual learning has emerged as promising approach for large language models (LLMs) to mitigate catastrophic forgetting while enabling adaptation to new tasks. Current Low-Rank Adaptation (LoRA) continual learning techniques often retain and freeze previously learned LoRAs or generate data repre- sentations to overcome forgetting, typically utilizing these to support new LoRAs learn new tasks. However, these methods not only ignore growing computational memory with tasks and limited storage space but also suffer from potential task interference due to the lack of effective LoRA merging mechanisms. In this paper, we propose a novel continual learning method that orthogonally initializes and sequentially merges LoRAs updates into a single unified LoRA. Our method lever- ages orthogonal basis extraction from previously learned LoRA to initialize the learning of new tasks, further exploits the intrinsic asymmetry property of LoRA components by using a time-aware scaling mechanism to balance new and old knowledge during continual merging. Our approach maintains constant memory complexity with respect to the number of tasks, minimizes interference between past and new tasks via orthogonal basis initialization, and improves performance over asymmetric LoRA merging via adaptive scaling. We provide theoretical analy- sis to justify our design and conduct extensive experiments across diverse continual learning benchmarks using various LLMs, demonstrating the effectiveness and efficiency of our method.
INTRODUCTION
Large Language Models (LLMs) (Raffel et al., 2020 Achiam al. [003 Touvron et al. 2023) have been growing the cornerstone of modern machine learning, achieving remarkable performance across wide range of downstream tasks. However, despite their impressive capabilities, LLMS still suffer from catastrophic forgetting (McCloskey & Cohen) 1989 7hu et al. [2024a] Yang et al.) 2024b) when fine-tuning sequential tasks, and their huge model capacity makes full fine-tuning computationally expensive and memory-intensive (Zhao et al. 2024a). These challenges have led to increasing attention in parameter-ee marameter-efficien continual learning, particularly via techniques such as LoRA (Low-Rank Adaptation) (Hu et al. 2022). that injects trainable low-rank matrices A and B into pre-trained models, enabling task adaptation with minimal additional parameters. Existing methods have shown progress in mitigating forgetting in LLMs through LoRA-basee continual learning. For example, O-LoRA (Wang et al. 2023) freezes previously learned LoRAs and incrementally learns new tasks their orthogonal subspace; InfLoRA (Liang & Li 2024) preserves prior LoRAs and uses dese-deenndent input matrices to define orthogonal subspaces for initializing new ones; SAPT-LoRA (Zhao et al.) (2024b) retains earlier LoRAs and leverages generated previous tasks' data to align new LoRA learning with shared modules; SD-LoRA (Wu et al. 2025) incrementally decouples the learning of magnitude and direction LoRA components while preserving directions learned from previous tasks. However, these methods either keep and freeze previously learned LoRAs, resulting in parameter growth of the form [BA B, or generate and maintain task-specific data representations, leading (i) linear growth in memory usage with the number of tasks, (ii) limited scalability due to constrained storage space, and (iii) potential task interference in the absence of principled LoRA merging mechanisms, These limitations motivate the question:

---

Published as a conference paper at ICLR 2026
(a) LoRA A (q)
(b) LORA B (q)
(c) LoRA A (v)
(d) LoRA B (v)
Figure I: Cosine similarity between 15 tasks from the large number of tasks benchmark for fine-tuned
q and v attention LoRA A and B in the last layer (32nd) of Alama-2-7B-chin
Can we enable continual learning only using single shared LoRA, without learning or storing
lask-specific LoRAs or data representations?
To address this question, we take inspiration from model merging (aaripov et al. 2018 Draxler et al. 2018: Wortsman et al. (2022), emerging paradigm that aims to combine multiple sseciseec models into a single unitied model without retraining (Stoica et al., 2024 Ilaarco et al. 2023 Yaday et al. 2023; Ortiz-Jimenez al. 2023). By extending this idea, frame continual learning
sequential model merging problem, where the objective shifts from keeping all task-specific LoRAs or data to continually integrating their updates into single shared LoRA as new tasks arrive. Recently, model merging has successfully been extended to the LoRA regime: KnOTS (Stoica et al. 2025) leverages singular value decomposition to project LoRA updates into shared latent space, where existing merging methods can be applied; LORA-LEGO (Zhao et al. (025) decomposes LoRAs into minimal semantic units via grouping and clustering, enabling a reconstruction of multiple LoRAs into one. However, these LoRA merging methods generally assume concurrent access to all task-specific LoRAs fine-tuned from the same pre-trained model, which limits their applicability to the continual merging scenarios (Dziadzio et al. 2025), where tasks arrive sequentially. In such settings, the order of merging becomes critical and may degrade the performance of the final model. Moreover, continual LoRA merging remains underexplored in existing literature. While in the
setting, continual merging has received more attention, e.g., OPCM (Tang et al. 2025) sequentially projects new model updates onto subspaces orthogonal to the previously merged model and uses adaptive scaling to mitigate interference. However, these methods are not designed for LoRA, and the objective
of merging differs from that of continual learning (Ortiz-Jimenez et al. 2023). These challenges lead to the following question we aim to answer:
How can we enable continual learning through -ased continual merging?
We answer this question by maintaining single pair of low-rank matrices (A, B), shared across tasks. Achieving this necessitates addressing key challenges, including how to initialize and continually update the shared LoRA to effectively balance the trade-off between forgetting and generalization. Moreover, in contrast to full-model continual merging, A and play different roles in continual merging with LoRA. For instance, prior works (Zhu et al. 2024b Sun al. 2024 Zhang al.) 2023b Kopiczko et al.] 2024) have shown that in LoRA tunettunng. training B (initialized to zero) is critical for the performance, even randomly initialized A often suffices, but reversing the roles of A and B substantially decreases performance. To further investigate the asymmetry of LoRA components, we separately fine-tune 15 tasks from a standard large number of tasks benchmark (Wang al.
2023) in continual learning using 15 independent LoR As on Llama-2-7B-chat (Touvron et al. 2023). and compute cosine similarity of A and B across 15 tasks using their last layer LoRA. Figure I shows that A exhibits significantly higher similarity across tasks compared to B, suggesting that LoRA components follow inherently different learning dynamics. This motivates us treat and B differently in continual merging.
To address the above questions, we propose a novel parameter-eficeeeeee efficieett continual learning method via continual merging into a single LoRA, which initializes new task learning in an orthogonal subspace and sequentially merges LoRA updates. We name this method SLAO (Single LoRA continual learning with Orthogonal initialization via continual merging). Specifically, SLAO initializes each new task learning LoRA using orthogonal basis extracted from previously learned LoRA components, and exploits the asymmetric roles of A and B by applying time-aware scaling mechanism that balances knowledge retention and plasticity during continual merging. As shown in Figure 2 our approach ensures constant memory overhead regardless of the number of tasks. Additionally, it
2

---

Published as a conference paper at ICLR 2026
reduces interference between past and new tasks via orthogonal basis initialization and enhances
performance through adaptive continual merging that considers LoRA asymmetry.
Summary of contributions. This paper makes the follow- ing key contributions: (1) A novel parameter-efficient con-
LuRk Momury Usage (M4)
S1A0
220
O-LORA
tinual learning method for LLMs that continually merges new task LoRAs into a single LoRA via orthogonal basis
initialization and time-aware scaling mechanism, reduc- ing catastrophic forgetting and improving generalization. (2) A theoretical analysis of how our design mitigates for- getting and improves intransigence. (3) Comprchensive experiments on various continual learning benchmarks using Llama models and Qwen models of varying sizes, demonstrating effectiveness and efficiency of our proposed method.
Number of Tooes
Figure 2: Comparison of SLAO and O- LoRA memory usage of large number of tasks benchmark via Llama-2-7B-che
RELATED WORKS
Continual Learning. Continual learning aims to retain knowledge of previously learned tasks while adapting new data. It faces two main challenges: (I) catastrophic forgetting (McCloskey & Cohen 1989), where the performance of the model on earlier tasks significantly degrades as it learns new ones; and (2) knowledge transfer, where the model leverages previously acquired knowledge to improve learning on new tasks. Existing approaches are divided into three categories:
(i) Rehearsal--are based methods employ memory buffer to store samples from previous tasks, enabling joint training with new tasks. Dark Experience Replay (Buzzega et al., 2020) encourages consis- tency with past knowledge by aligning the model's current logits with those sampled earlier in the optimization trajectory. CLEAR (Rolnick et al. 2019), an experience replay method, effectively mitigates catastrophic forgetting in multi-task reinforcement learning. Gradient episodic mem- ory (Lopez-Paz & Ranzato 2017) stores task-specific gradients and projects new gradients to avoid interference with previous knowledge.
(ii) Regularization- based methods utilize constraints on the parameters of the model to prevent model updates of new tasks from interfering with knowledge acquired on previous tasks. Elastic weight consolidation, EWC (Kirkpatrick et al. 2017), uses Fisher Information Matrix to identify and protect parameters critical for previous tasks. Orthogonal Gradient Descent, OGD (Farajtabar et al., 2020), projects gradients of new tasks onto subspace that preserves model outputs previous tasks, while ensuring the direction remains effective for learning new tasks.
(iii) Architecture-basee methods dynamically adjust the structure of the model to isolate task-specific weights or expand model capacity (Mallya & Lazebnik 20188] Wang et al. (2022)). Packnet (Mallya & Lazebnik (0188) performs iterative pruning and network re- re-training. Progressive Prompts (Raz daibiedina et al.] (223)) mitigate forgetting by maintaining a growing sequence of soft prompts, where
each new task contributes an additional prompt.
Parameter-efficient continual learning. LoRA-based continual learning has emerged as a practical and parameter-efficient technique for adapting LLMs to sequential tasks. O-LoRA (Wang et al. 2023) freezes previously learned LoRAs and incrementally learns new tasks in their orthogonal subspace; InfLoRA (Liang & Li 20244) preserves prior LoRAs and uses task-dependent input matrices define orthogonal subspaces for initializing new ones; SAPT-LoRA (Zhao et al.) 2024b) retains earlier LoRAs and leverages generated previous tasks' data align new LoRA learning with shared modules; SD-LoRA (Wu et al. 2025) incrementally decouples the learning of magnitude and direction in LoRA components while preserving directions learned from previous tasks.
Merging and Continual Merging. Model merging (Garipov et al.) 2018 Draxler et al. 2018
Wortsman et al., 2022) has emerged as an efficient paradigm that combines multiple task aask-specffic models into a single unified model without retraining (Stoica et al. 2024 Ilharco et al. 2023 Yaday
et al. 2023 Ortiz-Jimente et al. 2023). This idea has recently extended to LoRA-based adaptation:
KnoTs S (Stoica et al. 2025) leverages singular value decomposition to project LoRA updates into shared latent space, where existing merging methods can be applicd: LoRA-LEGO (Zhao et al. 2025)
decomposes LoRAs into minimal semantic units via grouping and clustering, enabling a reconstruc-
3

---

Published as a conference paper at ICLR 2026
tion of multiple LoRAs into However, both LoRA merging and full model merging generally assume simultaneous access to all task-specific LoRA or model fine-tuned from the same initial pre-trained model, which limits their applicability the continual merging scenarios (Dziadzio et al., 2025), where tasks arrive sequentially. Moreover, continual LoRA merging remains underexplored in existing literature. While in full-model settings, continual merging has received more attention, i.e., OPCM (Tang et al. 2025) mitigates interference by sequentially projecting new model updates onto subspaces orthogonal to the previously merged model, combined with adaptive scaling.
BACKGROUND AND MOTIVATION
Problem setup. Let fw, : - Y denote pre-trained model parameterized by Wo E R=x0, which remains frozen throughout the continual learning (CL) process. Here, X' and ) represent the input and output spaces, respectively. We consider a sequence of T tasks. For each task L E {1, 21 ..., T), the model is continually fine-tuned using the LoRA algorithm based on its associated training dataset D, = {(Xe,i, H= and evaluated on a separate test dataset D/ {{X,, Y(,)) where N and N' denote the number of training and testing samples, respectively. The goal is to continually learn a single set of LoRA parameters, specifically, matrices B Rm*xr and A R= with r < min(m, n2) such that the resulting merged LoRA model remains competitive with models optimized for expected risk (multi-task objective):
Eix. L(( Yi),
min B.A
where L, denotes the empirical risk (e.g., cross-entropy or mean squared error) for task
In the single shared LoRA setting for CL, we restricted to maintaining only pair of LoRA parameters, denoted by A'erge and Bieerge across all tasks. When task arrives, we fine-tune on its training data, possibly initialized with current merged models {A B to obtain task- specific LoRA parameters Bn E Rmxr and An, E R="x. The fine-tuned model for task is represented as parameters Bi A 1--1
After fine-tuning, we merge the previously accumulated LoRA merge with the new task-specific parameters An, resulting in an updated merged representation B A marge- Due to the inherent asymmetry in LoRA components, the merging of the LoRA components performed separately for and as formalized below:
BC t = MorrinuallMerge B B(B Bn,t), A'eerge Continual A(A mergoi Aa.t); 1
where Bn, 0 and An,o is initialized using Gaussian distribution, following the standard LoRA initialization (Hu et al. 2022). B Blherge BR, 1 and A An, are initialized as the first task
fine-tuned LoRA, and B At
merge merge is optimize Equation |
3.1 OPPORTUNITIES AND CHALLENGES IN CONTINUAL LEARNING VIA CONTINUAL MERGING
Storage and memory efficiency is one the core advantages of continual merging for CL. Unlike existing continual learning methods that benefit from freezing and retaining previously fine-tuned LoRAs, using continual merging after fine-tuning task only requires storing fixed number of
LoRAs: (1) the current merged LoRA and (2) the fine-tuned LoRA to be merged. This strategy results constant memory complexity of O( + [Al) O((m + n)r), where |B| + |A| denote the parameter sizes of single LoRA. Critically, this memory requirement remains independent of the number of sequential tasks T. contrast, as shown in Figure 2 existing freezing- continual learning methods require storing all LoRAs, incurring linear memory complexity of
O B + A)). which is ( n)r), growing linearly with the number of tasks.
Training efficiency is an evident advantage of continual merging for CL. Prior works do not simply keep previous tasks' LoRAs without any operations. Instead, when training new tasks, prior works use these multiple LoRAs during training through constraints, i.c., making new task LoRA parameters orthogonal to all previous LoRAs, heavily increasing computational cost during training. However, continual merging in CL would only use the parameters of single previously fine-tuned LoRA to initialize new task LoRA parameters before training, avoiding extra computation during training.
Difference between continual learning and merging mainly in the objective. In the context of
4

---

Published as a conference paper at ICLR 2026
multi-task model merging, the task arithmetic property, defined by Ortiz-Jimenez et al. (2023), refers to the ability to add task-specifie vectors without interfering with performance on other tasks. However, in CL, the objective extends beyond retention: the model must both preserve previously acquired knowledge and generalize effectively to unseen data. Hence, while merging can support CL, its underlying objectives are entirely equivalent those of CL., leading to fundamental differences in both theoretical analysis and algorithmic design.
3.2 ORTHOGONAL INITIALIZATION MOTIVATED BY LORA NTK ANALYSIS
To inform our algorithmic design, we evaluate the performance CL using LoRA by two key metrics, forgetting error (Lin al. 2023) and intransigence error (Li et al. 2023), defined as below:
(1) Forgetting error: It measures how much knowledge of old tasks has been forgotten after learning the current task. Specifically, after learning task t [ TT, the average forgetting over all old tasks is defined as:
i E [1,
F Σ (c.(Wo + B,Ad) (W + B,A.))
(2)
In Equation 2 C.W + B,A.) (Wo + B,A,) denotes the performance difference between B,A; (result after training task i) and B,A, (result after training task t) on test data of task i.
(2) Intransigence error: It evaluates the ability of the algorithm to adapt to task after having already adapted to a sequence of old tasks.
I I. = E=En (C, + -,A,) CW + B,A)))
In Equation 3 (W + B,A,) - L, + A;) denotes the performance difference between B? A; (optimal result of training task i) and B;A, (result after training task i) on test data of task i.
To examine these errors, we draw on the empirical observation (Malladi et al. 2023) that, when prompt-based fine-tuning employed (Schick Schütze 2021 Gao et al. 2021), ) the fine-tuning of pre-trrined language model tends to remain within the Neural Tangent Kernel (NTK) regime. Specifically, under the NTK regime, assuming D; = (X,, Y))eee(,.....)) the empirical risk for task I using LoRA can be approximated as (Jang et al. 2024)
N C, Σ
((fw, + (Vw fww.wwoo (X.) B,A)) Y.)
(4)
As detailed in Appendix B (Lemmal D by extending the analysis in Jang et al. (2024); Maurer (2016). show that, under the NTK regime, the term in forgetting error can be bounded as follows:
C.WWo + B,Ar) C. + B,A.) s (BA B,A, V fw. (X.))l
K
K
S
Σ j=1
B,A, - B.A (x.II S
Σ j=1
B.A. - BAA.
(5)
where K is output dimension and V make |, B:A,lle should be minimized. Thus, the term in forgetting intransigence decomposition can be written as:
w Ww l| as small as possible. Similarly, minimize Equation 3 |B.A, A, B; A
(X,) R. Thus, to minimize Equation 2 should
B.A. B.A. + B: B A |
B, (A. IF + || B. B.) A, || + [B( -;) |I B;)A;l
(6) error in CL
Algorithmic motivation. From above bound, we observe that prrgetting-intransppe with LoRA depends asymmetrically on the choice frozen and trainable components. For example. freezing A and fine-tuning B is at least as effective, if not better, than the reverse (Zhu et al. 2024b). However, if we apply freezing A CL, then || A. A = 0 but ||A, A [| may be unintentionally increased due to random A,. Instead, if we propose to fine-tune A, and extract I,. to initialize A via Q:-1, then we have A (0) (A(() (O))T I, where [1,..., f). This orthogonal structure not only keeps geometric consistency across tasks but also allows A, (t 2 j > f). to remain well-aligned with previous A,, ie. AT] x I,. thereby minimizing both A., - A, l| and ||A A [|p. This motivates our
orthogonal basis Qi-1 from A,-1, where QT-
design of orthogonal initialization. The complete derivation is in Appendix B.1
5

---

Published as a conference paper at ICLR 2026
3.3 CONTINUAL MERGING MOTIVATED BY LORA ASYMMETRY ANALYSIS
To provide the analysis of merging B. we consider a scenario where single LoRA is continually fine-tuned for sequential tasks, which means each task starts from the previous task's fine-tuned
LoRA. We initialize task 1 as B = 0 and Ao ~ NN(0, o²) (Hu et al. 2022). After fine-tuning with
T steps, we obtain parameters:
Wo + (Bo + AB ( AAI) Wo + AB + AABAAA
(7)
Since Bg 0, we have that
((A))) Bulle 0, || 1|F #
(8)
Based on theorem in Hao et al. (2024) we write task 1 fine-tuned LoRA:
B (T)(T)) A Ao + (T)
(9)
Using recursion, for task i where 2 2, the orthogonality measures become:
|AB, B ||r |A,-1AA | (A + n 2 A,, + - A.-2f
= (n) 1|r
(B(T)
(10)
AT
(11)
Our insight is that the second term in Equation I has a smaller magnitude when learning rate is not large. since when 1) < 1/L. lim 17 (t)l < 1 (Hao et al. 2024). thus the second term is significantly smaller than the first term, and |A BF < A A ||r- Hence, the update of B is more orthogonal to its initialization than the update of A is its initialization. Based on the findings in Wei et al. (025), task vectors in model merging are inherently close orthogonal to minimize interference, which indicates that in our case, merging rather than merging A provides better task isolation and reduced interference, motivating our choice to perform merging B. Then, for the operation of merging B, we build on garameter-efficiee module linear arithmetic composition, including addition and negation. Linear connectivity implies that model parameters fine-tuned from the same pretrained checkpoint can be added improve generalization (Wortsman) et al. 2022), property that extends to PEFT adapters, whose small updates likewise allow linear composition Zhang et al. 2023a). Hence, we can write merging operations on B using task vectors
B Bmerge Bmerge A (Buew Bmerge)
(12)
This makes the foundation of the operation of merging B continual merging.
METHODOLOGY
4.1
SLAO: SINGLE LORA CONTINUAL LEARNING
Based on the above analysis, we propose our method, SLAO, utilize continual merging into single LoRA to minimize task interference and improve generalization for CL. SLAO is motivated by four key insights: (1) Orthogonal Retention: To minimize forgetting error and intransigence error, it is crucial to maintain orthogonality in the LoRA components across tasks; (2) Continual Merging: To reduce memory usage in LoRA- CL. continually merging new task fine-tuned LoRA updates into a single merged LoRA A is highly efficient strategy; (3) Asymmetry of LoRA: Given the distinct learning roles of LoRA components A and B, they should be handled separately; and (4) Time-aware Scaling: To retain prior knowledge while adapting to new tasks, the merging process for new LoRA updates should be scaled a time-aware manner that reflects its training trajectory.
The SLAO consists of two main operations: (1) Initialize each new task learning LoRA by extracted orthogonal basis from the previous task's 's fine-tuned LoRA; (2) After fine-tuning on the new task, employ adaptive time-varying scaling for new LoRA updates to merge into the merged LoRA. The complete procedure is outlined in Algorithm [ and illustrated in Figure Starting with the first task's 's fine-tuned LoRA BR, An,1 by standard fine- tuning, our method iterauvely integrates LoRA updates of subsequent tasks in a continual fashion.
utilize the asymmetry of LoRA components
Orthogonal basis extraction for initialization. For new task i, first extract orthogonal basis of

---

Published as a conference paper at ICLR 2026
Algorithm 1 SLAO: Single LoRA Continual Learning
Initialize B Bmergee B Amerge An,1, scaling factor (11 number of tasks T.
2: for = 2 to T do
3:
Q: = QR(((AAA)))))) (A ".). Q, = signn(dia (R.)) // Extract orthogonal basis of An
(0) Q, B B // Initialize An, and BR, for task i
4:
5:
Bn, t fine-tune
Cune(Wo, (Wo. BR. (1.) // Fine-tune An, and BR,s for task
6:
A merge An.i
7:
B morge = B merge + A(1) B B
8:
Use merged LoRA B merge A merge for inference until new task comes
9: end for
10: return B merge
T
previous fine-tuned An,i- use that to initialize A making (4 decomposition to extract orthogonal matrix from i which is:
(0)
I,.. We utilize QR
Q,R: = - - Q: Q; sign(diag( R2))
ft.
Q
(13)
As result, the initialization An. has orthogonal rows. For B, we directly initialize B by Ba.i- which is the fine-tuned B of previous task
Asymmetrically merging LoRA via time-aware scaling, After fine-tuning task i, we merge its A 1i=1 merge)-
LoRA updates into { (Bmerge>
). Due to the intrinsic asymmetry of B and A in LoRA,
update A merge An,i and we merge by time-aware coefficient X(i) for new task updates:
B Binerge
Bi-l X(i) (BB,
B Binerge)
(14)
where X(1) is introduced to maintain a consistent magnitude of the merged B's deviation from previous tasks throughout the merging process. In method, the scaling, factor be set to A(1) +. which follows the continual merging method proposed in Tang et al. (2025). The findings in Ihaaroo et al. (2023); Tang et al. (2024) indicate that task vectors from different tasks tend to be approximately orthogonal, and since Bs across tasks are approximately orthogonal to each other, as shown in Figure L which indicates that B task vectors
approximately orthogonal. This orthogonality makes (()) = vs a natural choice for the scaling factor, since helps maintain the
magnitude of parameter changes across merging steps Tang et al. 2025)
4.2 DYNAMICS OF SLAO
To better understand the effectiveness of SLAO, inspired by the analysis of Hao et al. (2024), in the following theorem we analyze the dynamics of task-specific parameters' update in CL scenario.
Theorem 1. Let the parameters and B be updated using SGD at each step s for task as follows: (V
A = = A 1 (B;)) (VwC:) B B n) w (A)))
(15)
where is the learning rate. We assume A = ! A SA(S) and B B, +n (O))
(0)
holds with such functions f and I' for for1, I,...,,, and || ES= W. L for every S during training task i, which implies that the model stays within a finite Euclidean ball. If we assume A (0) (O))
(L=(1-(F(L 1-
I,, in this case, the dynamics of A, satisfies [[fa(s))l2 S the dynamics of B satisfies fB(s) = - WW (i) + I). When small, we have fB(s) R - -Ez (V C.). Thus B = (4)),, and total update for B, AB, - -n (V)
and
The proof is deferred to Appendix This analysis, under the orthogonal initialization of A, suggests that B may update across different initialization subspaces, effectively increasing the rank of and thereby aiding generalization. We note that the key difference between ours and Hao et al. 2024 lies in the initialization of LoRA: while they use standard initialization with Bi")
(0) O, we initialize B
using the previously fine-tuned LoRA parameters, resulting in ((()) A 0, complicating the analysis.
7

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Models and datasets. We evaluate our approach across three Llama models: Llama-2-7B-chee Blama-2-13B-char and Llama-3-2-3B, and two Qwen models: Qwen2.5-3B and Qwen2.5-7B. All experiments conducted on NVIDIA A100 GPUs utilizing DeepSpeed repository. We consider three continual learning benchmarks: (1) Standard CL benchmark: AG News, Amazon, Reviews, Yelp Reviews, DBpedia, and Yahoo Answers. (2) Large number of tasks: five standard CL bench- mark tasks, four GLUE tasks (MNLI, QQP, RTE, SST-2), five SuperGLUE tasks (WiC, CB, COPA, MultiRC, BoolQ). and IMDB movie reviews. Following O-LoRA (Wang et al. (023). each task uses 1000 randomly sampled training samples and 500 validation samples per class. (3) SuperNI Benchmark (Wang et al., 2022a): diverse collection of NLP tasks with expert- covering dialogue generation, information extraction, question answering, summarization, and sen- timent analysis. We follow task selection and ordering in SAPT ZZhao et al. 2024b), using 1,000 training instances and 100 for validation/testing per task.
instructions,
Baselines. We compare our method SLAO with the following baselines: (1) Continual learn- ing baselines: SeqLoRA: sequentially fine-tunes single LoRA multiple tasks without con- straints; IncLoRA: incrementally adds a new LoRA per task while freezing previous LoRAs; O-
LoRA (Wang et al. 2023); InfLoRA (Liang & Li 2024): SAPT-LoRA (Zhao et al. 20T4b).MTL:
single model is trained jointly on all tasks; LoRM (Salami et al.
2025); CorDA
adaptation) YYngg al. 2024bb Magmax (Marezak al. 2024).
(2) LoRA merging baselines:
LoRA-LEGO (Zhao et al. 2025); KnOTS (Stoica et al. 2025). (3) Continual merging baseline:
OPCM (Tan et al., 2025). To fairly evaluate existing merging methods LoRA-based continual
learning, we extend full-model merging methods to LoRA and equally treat components of LoRA,
and all merging methods are achieved sequentially.
Evaluation metrics. To evaluate our proposed approach, we employ three key metrics: (1) av- erage accuracy (AA), calculated as the mean accuracy across all tasks after training on the last task: + Σ/=1 ai,T, where ds is accuracy for classification tasks and Rouge-L for other tasks; (2) backward transfer (BWT) (Lin et al. 2022), defined -_1 E
ag,;). and experimental results are shown in Appendix D.4 (3) maximum order-normalized performance disparity (MOPD) and average order-normalize performance disparity (AOPD) (Yoon et al. 2020). which evaluate order-robustness, and experimental results shown in Appendix D.5
5.2 OVERALL RESULTS
Continual learning performance results analysis. As shown in Table [ our method consistently outperforms all data-free baselines across three benchmarks using Llama-2-7B-chat. GoRA-Basel continual learning: SeqLoRA performs worst among LoRA-based methods, as unconstrained continual fine-tuning on a single LoRA causes severe forgetting. InCLoRA improves by freezing prior learned LoRAs to isolate subspaces, though its subspace separation is simple. InfLoRA outperforms O-LoRA in standard CL benchmark due to orthogonal input-based subspaces, but drops large number of tasks and SuperNI benchmark, due to sensitivity to manually tuned DualGPM threshold. SAPT-LoRA achieves the highest average performance among LoRA-based methods, but relies on generated previous task pseudo samples, unrealistic in many LLM scenarios, and is more order- sensitive than ours. LORM-BA (begin with freezing B) and LORM-AB yield nearly identical results, suggesting that the freezing order of LoRA components in CL matters little. CorDA performs well on standard CL benchmark and large number of tasks, but drops significantly on SuperNI benchmark, likely due to relying on nullspace selection from pretrained models and lacking time- aware merging. MagMax performs comparably to ours on standard CL. benchmark, slightly worse on large number of tasks, but underperforms on SuperNI benchmark, where task similarity is lower, thus only keeping weights which have the largest absolute value would cause forgetting. LoRA merging baselines: KnOTS and LORA-LEGO perform similarly in the standard CL benchmark, but KnOTS outperforms in the large number of tasks and SuperNI. KnOTS may benefit from flexible SVD-merging mechanism that we apply time-aware scaling on merging, while LORA-LEGO treats tasks equally, lacks prioritization, and is ineffective in complex CL contexts. Continual merging approaches: since OPCM is designed for full model, directly applying it to LoRA by treating its
8

---

Published as a conference paper at ICLR 2026
Table 1: Testing performance (%) on three CL benchmarks using Llama-2-7B-chal across different task orders, where each result is run three random times, where Oi denotes ith task order.
Standard CL Benchmark
Large Number of Tasks
SuperNI Benchmark
Method
01
02
03
avg
04
05
06
avg
01
02
avg
SeqLoRA
73.3
76.2
78.4
76.0
69.1
66.0
71.1
68.7
18.4
26.8
22.6
IncLoRA
75.3
77.3
78.3
77.0
72.2
71.6
73.8
72.5
22.0
25.6
23.8
O-LoRA
76.1
76.3
79.2
77.2
74.0
72.0
74.6
73.5
23.3
28.4
25.9
InfLoRA
78.4
80.4
79.9
79.6
69.4
67.4
72.5
69.8
16.5
22.1
19.3
SPAT-LORA
82.9
81.8
78.7
81.1
84.7
78.9
82.2
81.9
53.2
48.5
50.9
LoRM-BA
76.0
76.8
78.3
77.0
71.4
69.0
70.3
70.2
25.6
18.7
22.2
LoRM-AB
77.5
74.7
75.9
76.0
71.0
69.5
70.2
70.2
25.6
23.7
24.7
CorDA
78.4
79.3
80.0
79.2
73.4
72.7
74.0
73.4
20.9
16.0
18.5
MagMax
80.1
80.6
80.3
80.3
72.3
73.5
74.5
73.4
15.3
7.0
11.2
KnOTS
67.9
65.9
70.8
68.2
61.5
60.1
58.0
59.9
34.6
30.1
32.4
LORA-LEGO
68.3
66.0
70.9
68.4
58.8
58.7
53.2
56.9
32.8
26.7
29.8
OPCM
61.9
62.0
56.7
60.2
51.9
52.8
46.9
50.5
11.6
12.3
12.0
SLAO (ours)
80.1
80.8
80.4
80.4
75.0
74.4
75.1
74.8
38.7
35.7
37.2
Multi-Task
80.9
78.1 1
45.2
Table 2: Comparison of initialization strategies on testing performance across three standard CL
benchmarks using 1-ama-2-B--aal under different task orders, where Oi denotes ith task order.
Standard CL Benchmark
Large Number of Tasks
SuperNI Benchmark
Initialization
01
02
03
avg
04
05
06
avg
01
02
avg
Random (zero)
66.4
62.4
68.4
65.7
61.4
60.3
57.2
59.6
33.3
28.9
31.1
Last-Merge
80.1
80.8
80.1
80.3
74.7
72.8
75.0
74.2
37.4
30.5
34.0
Last-FT (ours)
80.1
80.8
80.4
80.4
75.0
74.4
75.1
74.8
38.7
35.7
37.2
two components identically leads suboptimal performance. The results under L1a-a-2-13B-chat, Qwen2.5-3B, and Qwen2.5-7B (Yang et al., 2024a) are shown in Appendix D
Impact initialization strategies. We compare three different initialization strategies for learning new tasks: (1) random (zero) initialization, (2) initialization from last merging point, (3) initialization from last fine-tuning point (ours). shown in Table 2 initializing from last fine-tuning point consistently outperforms other two strategies across all three benchmarks. Using last merging point performs slightly worse, while random (zero) initialization performs the worst. The performance gap due to how initialization affects LoRA's learning trajectory and merging way. Random initialization places far away from optimal task-specific A', making intransigence worse. Initialization from last merging point fixes time coefficients after merging back to single LoRA, limiting its flexibility, while initialization from last fine-tuning point allows the merged LoRA to implicitly reweight previous tasks' updates when merging. This adaptive adjustment yields better CL performance.
Asymmetry in LoRA merging. To investigate asymmetry in LoRA merging, we compare different continual merging strategies for LoRA: (1) Freeze Merge B (FREA-MB), (2) Freeze B Merge A (FREB-MA), (3) Fine-tune BA Merge A (FTBA-MA), (4) Fine-tune BA Merge BA (FTBA-MBA), (5) Fine-tune BA Merge (FTBA-MB). As shown Table only FTBA-MB consistently outperforms other strategies, except ours. FTBA-MBA has comparable performance compared to FTBA-MB, but FTBA-MA yields the poorest performance among fine-tuning LoRA methods, When freezing one component of LoRA, FREA-MB is better than FREB-MA, consistent with Zhuu et al. (2024b) that freezing and fine-tuning B at least better than the reverse. It highlights asymmetry in LoRA components and importance of asymmetric merging based on their fundamental roles in adaptation.
Effect of model variants and sizes. We evaluate our method on three LLM variants: Llama-2-78 chat, 1amma-2-13B-hhe Llama-3-3B (Grattafiori et al. 2024). The results in Table indicate that

---

Published as a conference paper at ICLR 2026
Table Comparison of merging strategies on testing performance on three standard CL benchmarks using Llama-2-785 2-----hathh across different task orders, where Oi denotes ith task order.
Standard CL Benchmark
Large Number of Tasks
SuperNI Benchmark
Merging
01
02
03
avg
04
05
06
avg
01
02
avg
FREB-MA
77.7
78.0
76.2
77.3
70.8
66.1
72.1
69.7
13,9
25,4
19.7
FREA-MB
78.7
79.3
78.0
78.7
72.3
73.0
73.5
72.9
23.7
29.2
26.5
FTBA-MA
76.2
79.0
79.9
78.4
71.6
69.5
74.1
71.7
21.1
30.7
25.9
FTBA-MBA
79.7
80.4
80.2
80.1
73.2
73.9
74.5
73.9
33.8
32.7
33.3
FTBA-MB
79.3
80.8
80.1
80.1
74.1
74.0
74.8
74.3
32.4
35.2
33.8
SLAO (ours)
80.1
80.8
80.4
80.4
75.0
74.4
75.1
74.8
38.7
35.7
37.2
Table 4: Comparison of model variants and sizes testing performance across three standard CL benchmarks using Llama-2-7B-chh in different task orders, where Oi denotes ith task order.
Standard CL Benchmark
Large Number of Tasks
SuperNI
Benchmark
Model
01
02
03
avg
04
05
06
avg
01
02
avg
Llama-3-2-3B
74.3
75.8
75.3
75.1
73.3
72.5
74.9
73.6
32.7
34.6
33.7
Alama-2-7B-cha
80.1
80.8
80.4
80.4
75.0
74.4
75.1
74.8
38.7
35.7
37.2
Llama-2-13B-chat
80.8
81.1
81.1
81.0
76.5
75.9
76.0
76.1
42.3
42.2
42.3
model variants and model size play crucial roles in average performance across different task orders and benchmarks. Llama-3-3B performs the worst, while in same generation of Llama, larger models consistently achieve better accuracy: Llama-2-13B-che to Llama-2-7B-- catastrophic forgetting and improving generalization continual learning. Moreover, we observe that larger models exhibit greater robustness task order variations compared smaller models.
relatively has higher accuracy compared This trend suggests that increased model capacity enhances both reducing
76
74
Accuracy
Accuracy
77.5
72
75.0
70
72.5
66
70.0
0.1
0.5
0.9
Aldi (Ours)
0.1
0.5
0.9
Aldi (Duns)
15
0.5
0.9 AIt (Durs)
Time scaling factor
Time scaling factor
Time scaling factor
(a) Standard CL. BBnnchmar-775
(b) Large Number of tasks-7B
(c) SuperNI Benchmark-3B
Figure 3: Comparison of model performance across time coefficients.
Time-varying coefficient analysis. To evaluate impact of adaptive vireinarynng scaling in continual merging for CL, we compare against fixed factors {0.1, 0.5, 0.9} on three benchmarks via Llama- 2-7B-chat and 3-2-3BB As shown in Figure 3 adaptive strategy consistently achieves highest average accuracy with lower variance across task orders and models. For simpler standard CL benchmark, larger fixed value 0.9 outperforms smaller one 0.1, while for more complex long benchmarks, smaller values perform better; 0.5 is relatively stable but consistently suboptimal.
CONCLUSION
In this work, we proposed a novel parameter-efficien continual learning based on continual merging of LoRA, enabling no additional training access to any data representations. Our approach leverages the orthogonal basis from previous fine-tuned LoRA to initialize for new task learning and constructs single shared merged LoRA via time-aware scaling, thus ensuring constant memory usage regardless task number. Through comprehensive experiments, we demonstrated the effectiveness and efficiency of our method across multiple benchmarks and model scales.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This work was partially supported by NSF EFMA Award #2318101 and NSF CAREER Award #2239374.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1ige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprini arXiv:2303.0877 2023.
Pictro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33:15920-155 15930, 2020.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference machine learning, pp. 1309-1318.
PMLR, 2018.
Sebastian Dziadzio, Vishaal Udandarao, Karsten Roth, Ameya Prabhu, Zeynep Akata, Samuel Albanic, and Matthias Bethge. How to merge multimodal models over time? In ICLR 2025 Workshop on Modularity for Collaborative, Decentralized, and Continual Deep Learning, 2025.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International conference artificial intelligence and statistics, pp.
PMLR, 2020.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3816-3830, Association for Computational Linguistics, August 2021.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. Xrviv preprint arXiv:2407.2178. 2024.
Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters are secretly gradient compressors. In Forty-first International Conference Machine Learning, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICIR, 1(2):3, 2022.
Wei Hu, Lechao Xiao, and Jeffrey Pennington. Provable benefit of orthogonal initialization in optimizing deep linear networks. In International Conference on Learning Representations, 2020.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023.
Uijeong Jang, Jason D. Lee, and Ernest Ryu. LoRA training in the NTK regime has no spurious local minima. In Forty-first International Conference on Machine Learning, 2024.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska. et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In The Twelfth International Conference Learning Representations, 2024.
11

---

Published as a conference paper at ICLR 2026
Haoran Li, Jingfeng Wu, and Vladimir Braverman. Fixed design analysis of ialtarriiation-bbaeee continual learning. In Conference on Lifelong Learning Agents, pp. 513-533. PMLR, 2023.
Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference- low-rank adaptation for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2024.
Sen Lin, Li Yang. Deliang Fan, and Junshan Zhang. Beyond not-forgetiing: backward knowledge transfer. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Continual learning with
Sen Lin, Peizhong Ju, Yingbin Liang, and Ness Shroff. Theory on forgetting and generalization of continual learning. In International Conference on Machine Learning, pp. 21078-21 20000. PMLR, 2023.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. kernel-based view of language model fine-tuning. In International Conference on Machine Learning, pp. 23610-23641. PMLR, 2023.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference Computer Vision and Puttern Recognition, pp. 7765-7773, 2018.
Daniel Marezak, Bartlomiej Twardowski, Tomasz Trzciński, and Sehastian Cygert. Magmax: Lever- aging model merging for seamless continual learning. In European Conference on Computer Vision, pp. 379-395. Springer, 2024.
Andreas Maurer. A vector-contraction inequality for rademacher complexities. In Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27, pp. 3-17. Springer, 2016.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 36:66727-66754, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. Progressive prompts: Continual learning for language models. arXiv preprint 12311:2301.12314,
2023.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019.
Riccardo Salami, Pietro Buzzega, Matteo Mosconi, Jacopo Bonato, Luigi Sabetta, and Simone Calderara. Closed-form merging of parameter-efficien modules for federated continual learning. In The Thirteenth International Conference on Learning Representations, 2025.
Timo Schick and Hinrich Schütze. Exploiting cloze- quuze-quessions for fho-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 255 269. Association for Computational Linguistics, April 2021.
12

---

Published as a conference paper at ICLR 2026
George Stoica, Daniel Bolya, Jakob Brandt Bjorner, Pratik Ramesh, Taylor Hearn, and Judy Hoff- man. Zipit! merging models from different tasks without training. In The Twelfth International Conference Learning Representations, 2024.
George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman. Model merging with SVD to tie the knots. In The Thirteenth International Conference Learning Representations, 2025.
Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving loRA in privacy-preservingg federated learning. The Twelfth International Conference on Learning Reprexentations, 2024.
Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. Parameter-efficient multi-task model fusion with partial linearization, In The Twelfth International Conference on Learning Representations, 2024.
Anke Tang, Enneng Yang, Li Shen, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. Merging models on the fly without retraining: A sequential approach to scalable continual model merging, arXiv preprint 2000:2250000022,, 2025.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09928: 2023.
Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap. al. Supper-nturallisstuutions:: Generalization via declarative instructions on 1600+ nlp tasks. arXiv preprint arXiv:2204.00770 07700, 2022a.
Zifeng Wang, Zizhao Zhang, Chen-Yu Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Plister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 139-149, 2022b.
Yongxian Wei, Anke Tang, Li Shen, Zixuan Hu, Chun Yuan, and Xiaochun Cao. Modeling multi-task model merging as adaptive projective gradient descent. In Forty-second International Conference on Machine Learning, 2025.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pp. 23965-23998, PMLR, 2022.
Yichen WU, Hongming Piao, Long-Kai Huang, Renzhen Wang, Wanhua Li, Hanspeter Pfister, Deyu Meng, Kede Ma, and Ying Wei. SD-IoRA: Scalable decoupled low-rank adaptation for class incremental learning. In The Thirteenth International Conference on Learning Representations, 2025.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES- merging: Resolving interference when merging models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1515 2024a.
Yibo Yang, Xiaojie Li, Zhongzhu Zhou, Shuaiwen Leon Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem. CorDA: Context-oriented decomposition adaptation of large language models for task-aware parameter-efffcienn fine-tuning. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024b.
13

---

Published as a conference paper at ICLR 2026
Jachong Yoon, Sachoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and order-robust contin- ual learning with additive parameter decomposition. International Conference on Learning Representations, 2020.
Jinghan Zhang, Junteng Liu, Junxian He, al. Composing parameter-cfficient modules with arithmetic operation. Advances in Neural Information Processing Systems, 36:12589-12610, 2023a.
Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv preprint arXiv:2308.03303, 2023b.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-effcciet LLM training by gradient low-rank projection. In Forty-first International Conference on Machine Learning, 2024a.
Weixiang Zhao, Shilong Wang, Yulin Hu, Yanyan Zhao, Bing Qin, Xuanyu Zhang, Qing Yang, Dongliang Xu, and Wanxiang Che. Sapt: A shared attention framework for parameter-efficien continual learning of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pp. 11641-11661, 2024b.
Ziyu Zhao, Tao Shen, Didi Zhu, Zexi Li, Jing Su, Xuwu Wang, and Fei Wu. Merging loRAs like playing LEGO: Pushing the modularity of loRA to extremes through rank-wise clustering. In The Thirteenth International Conference on Learning Representations, 2025.
Didi Zhu, Zhongyisun Sun, Zexi Li, Tao Shen, Ke Yan, Shouhong Ding, Chao Wu, and Kun Kuang. Model tailor: Mitigating catastrophic forgetting multi-modal large language models. In Forty-first International Conference on Machine Learning, 2024a.
Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz. Sáez de Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry in low-rank adapters of foundation models. In Forty-first International Conference Machine Learning, 2024b.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
The appendix is organized as follows:
Appendix A
is the of L.Mss.
Appendix
provides theoretical analysis.
Appendix
provides the overview of SLAO.
Appendix
provides more experimental setup details and additional experimental results.
Overall Results on Llama-2-13B-chatt D.I
Results Qwen2.5 models D.2
Impact of initialization strategies for existing LoRA merging methods D.3
Continual learning performance on backward transfer [D.4
Continual learning performance on Order-normalized Performance Disparity (MOPD and )OPD)
Choice of orthogonal decomposition strategy D.6
Asymmetry of LoRA D.7
Impact of learning rate D.8
Impact of the rank of LoRA
D.9
Comparison training cost D.10
Comparison of orthogonality among LoRA-based CL methods D.II
Descriptions of task sequence orders D.12
A THE USE OF LARGE LANGUAGE MODELS (LLMs)
We utilize LLMs to polish the paper writing.
B THEORETICAL ANALYSIS
B. THE FORGETING-INTRANSGEENN ERROR DECOMPOSITION UNDER NTK
Lemma 1. (Jang et al. 2024 Maurer 2016) Assume D is i.i.d N random samples sampled from
probubility distribution P. Lei An = [X; + Sw, (X.) (Vw Jw, (X;), 8) (,)8)))
RR || S
D E Rmxny is class of affine predictors with bounded nuclear norm D. For 1 S K, suppose
V WW [X)]F S R almost surely with respect to the random data X; ~ P. For S i i N. suppose L = E(., Y.) is G-Lipschitz continuous on A on the first argument (with respect to the Euclidean norm) for almost surely with respect to the random data X, C P. That is
((x)) (,(a' (X2)) S Gila(X,, a' 22(22)ll2 for any a, a' E A. X,, X2 DDx (16)
Proof. First, let g X + R be function satisfying the following property with 0: 1g(X1,.... X,-1, X;, X.+1..... XN) - g(X,,..... X,,1, X X,+1...., X Se
(17)
for all XI, XI,..., XN, X E X. Then, for all e > 0,
2c²
P(gg( X Xx) - Elg( X
XN)]I < exp
(18)
Ne²
Take to be g sup | (((6)) (()) L(dn) + L())), which is a function of X1, Xi,..., X N.
Since |/. <. D implies ||0|| 5 D) and by the Lipschitz continuity of {(,, Y)). we have the following
for any (X, 1, Y.) D:
x,) + 1w (X.) + (G(X,), 56), Y.) (G( YII < G 6, G(X))))
K
S G
Σ
100 5 (X)))
15

---

Published as a conference paper at ICLR 2026
K
G
Σ
100
8
G
1=1
K
G
Σ
4D2.
R
2GRDVK.
(19)
Thus, from this lemma, we apply it to LoRA-based continual learning and have
WW + B,A.) (W + B,A,) s G|| B, - B,A,, V fwo (x,))ll2
G
Σ
B.A BAA V fw,,)) (x,)
G
B.A: - B
(20)
where K is the output dimension. From this, we can see that to minimize forgetting error, we should make B,A, B,A,Ii || as small as possible. Similarly, to minimize intransigence error, B,A, B A should be also small. Besides, it's evident that La W + B, - LWW + B, 0. Thus, we makes |B B.A. D and B.A B A || S D, then we have
F + +I
Σ (C.(W, + B.A.) LW + B:A.)) + Σ i=1 (C. + B,A,) ( + B7 A7))))
Σ (G.WW + B,A.) L.W. + B.A,)) + + i=1 (L, B,4,) L.(W + B; AAA))
E (C.WW) + +BA,) L. B.A.)) + (L.(Wo + B,A,) L.W.o + BT
Σ 1= G Σ
|.A, BA +G
[BA: BA
GR E
B.A. BA +
-B
4DGR Σ VK
(21)
To make both B.A, B,A, || and B.A, B;A; minimized, the forgetting intransigence decomposition can be written as:
I.A, B,A. + B,A, A;
[A, A.) +I B,) BIB I
(22)
Algorithmic motivation. From the above bound, observe that generalization error in CL with LoRA depends asymmetrically on the choice of frozen and trainable components. Interestingly, this insight contrasts with standard fine-tuning practices. For example, concluded in (Zhu et al. 2024b), freezing A and fine-tuning B is at least as effective, if not better, than the reverse. However, apply freezing A CL. implies A - A, |F
which may unintentionally increase A - A/ due to limited task-specific expressiveness. In Hu et al. (2022). A is initialized to random Gaussian matrix satisfying E[A A [] = I,.. Instead, if we propose to fine-tune A; while extracting orthogonal basis Q from A;-1, where QT I, to initialize A via
Q Q then will have (1)) (A ( I, where i E [1,...,4] .]. This orthonormal structure
16

---

Published as a conference paper at ICLR 2026
only keeps geometric consistency across tasks but also allows A, (t 2 > i), to remain well-aligned with previous A, i.c. A[ R I,, thereby minimizing both ||A, A.lle and ||A, A; l| This motivates our design of continual merging with orthogonal initialization to reduce forgetting and maintain adaptability.
B.2 DYNAMICS OF LOW-RANK ADAPTERS UPDATED BY SLAO
We analyze the dynamics of A and B in continual learning setting. This analysis, under the orthogonal initialization of A, suggests that B may update across different initialization subspaces, effectively increasing the rank of B and thereby aiding generalization.
Theorem Let the parameters and B be updated using SGD at each step s for task as follows:
A = A (B)) (V L.), B B - ( w (A))
(23) )
where is the learning rate. We assume A = A +n (A(5) and B (0) (
holds with such functions fA and fo..or,,.... I,...,& 8, and || Σ> Vw training task i, which implies that the model stays within a finite Euclidean ball. If we assume A (A. ) I,. this case, the dynamics of A, satisfies [[fa(s) S the dynamics of B satisfies fB(s) (V (nf () + I). When ni small, we have fB(s) = -E C.). Thus B = (AA and total update for B, AB, (VwL)) (A.
(()) |F S L every S during
(1 and 1-
SB(0) = 0. For 8 > 0, assume Aj = Ao + (s) and B (s) Since the first task training the same as the
Proof. We start by noting the fact that for Task 1, when & = 0, [a))
LoRA fine-tuning in Hao et al. (2024) for the dynamics of A and B we have:
+
Ai rBi) (i)
Ao + As) - 1/² (V)
= Ao (1
(24)
and
B -B
=nfB(s+1)Ar
(25)
Thus, by rearranging the terms, have:
1-1 fa(s) = -n Σ f0())
(26)
1=0
fB(8) EVV.CGMM
(27)
Since Wo + BA R Wo + ABAs when learning rate 77 is small, the change in B dominates the final weight update. Thus, if freezing A, obtain
AB ~ 17
Σ
Ci
A
(28)
and
fn(8) ~
W
(29)
Task (>> 1): when fa
For s > 0: Assume A = A + n, 00) fA(s) and B = B(" (A) hold with such
functions SA and In for 1, 1. Then, for 8 1, have
A - (B,) (Vw
17

---

Published as a conference paper at ICLR 2026
(0) - + n (0) (B " (Ww,fi)
= A
(0) + 74, (0) [a(() - -(((B()) + ( W: Li)
= 4.0 + nA, (fa(s) - (w)
(
(Vw
(30)
We would like to express A as
A A
(0) + +1A fA($+1)
(31)
So compare both sides:
n
( (0) fa(s 1) - (fA(s) (f -(B (V...C
(32)
Divide both sides by 7,, rearrange:
A fa(s +) (fA(s) -
- )B. (Vw, (2)
(33)
Since by our initialization A, (A = L,, then we have
A (0) fa(s
1) 1)
A
- (s) (( (Vw,CC))
(34)
fA(s +1) - - -00_0)00000///ww0_44444""" ( 1) = /A()) (3) (B (Vw.f.)
(35)
(1(
(36)
f Σ (416)- (VV,,) CW)
(37)
1=
For B, we have:
B = B (V
B (4 (VV, (0) + +nA fa(S))
( (0) (A( (T) - (VA (0) , (0) fA()))
B (n (0) - (VV (0) - - (VV
B + 1 (fB(s) - (Vw. F.) - VVW, (4)
(38)
Thus,
fB(s + fB(() (Ww.C)) w.. (
= fB((s) (1) +I)
Σ (w.C))(5T
(39)
1=
Then, have:
[f) =
11
(mfa(m) + CT) + TTB (Vwol
1=0
8-1
Σ (W,
Σ (Vw,Ct)
j=m+1
+
V (Ww (W..CT)TTWWW
0= m=
Σ (A (0)) (BB W: C2)
+ 1)
18

---

Published as a conference paper at ICLR 2026
2 Σ vA.
+ +n + 11 ( )
(40)
m=0
(B
[I T
Σ (B (0)
1 S vr
(41)
21.21-- then 1-101""
If [f]](())]| Sa,-
[[a(()]l S L
- (2)))
+ + +22
1 1/2
_
1 L2
T2?
- + 12
(42)
1 1-1212
have
[|a()))l2 I[
(43)
If 1 < 1/L,
vllfa(8)]lr S
1- 122
S 1)12(1 - (181399)
1 122
na,
(44)
Thus, have r|]ff(>)|| n0,,- The dynamics
fA(8) - - Σ [O()).) --(s)=--) s-1 Vw. ())
(45)
In our algorithm, we fine-tune A in our algorithm and we have n)lfa()) < I, then
V wo (
(Vwa.?), SB(8) = -=---v
2 (v.. C2)
Sa($) =
(46)
1=0
m=U
1=0
Therefore, we have
8 AB ~ -1
Σ Wi, (T
(47)
0=0
C OVERVIEW OF SLAO
We show the detailed overview of SLAO in Figure 4 It presents framework where fine-tuned LORA and merged LoRA are processed over time, and specializes key components: (1) orthogonal initialization, and (2) time-aware continual merging.
19

---

Published as a conference paper at ICLR 2026
Time
Orthogonal
Berrge
Time-aware Continual Merging
B
Initialization
Aqa
Apu
An
A(1)
B B
B
Ap
Bosss
Barre
Anu+3
Aguust
A
An2+1
Figure 4: Overview of SLAO. Left area is framework where fine-tuned LoRA (orange) and merged LoRA (gray) are processed over time. Right area (2 blue boxes) highlights key components: (1) Orthogonal initialization for new task
learning LoRA, where orthogonal basis is extracted from (0)
AR, to initialize An.i+1 such that A +1++1 1 (A (A) 1 = I, and BR 1 initialized by previous
BR.; (2) Time-aware continual merging for B Bn,++ and B merger and update A 4+1
merge via An,tt1.
D EXPERIMENTS DETAILS
Our experiments conducted on 4 NVIDIA A100 GPUs using the DeepSpeed repository. We evalu- ate five LLMs: Llama-2-7B-chan Llama-2-13B-char Llama-3-2-3B, Qwen2.5-3B, and @wen2.55-BB Each individual experiment (e.g., running single task order from the large number of tasks bench- mark on Llama-2-1 1-B-hhat) can be executed on single A A100 GPU. We apply LoRA to the query and value projection matrices in the attention modules of each model, with fixed rank of Each task order is evaluated over 3 random seeds.
For Standard CI. benchmark and the large number of tasks benchmark on 1lama-2-B--ael and 1lama-2- 1amma-2-1300 1-3-hhhht,, we follow the setting in Wang et al. (2023), and train the models with one epoch, a constant learning rate of le-4, the training batch size 1, and the gradient accumulation step And for Standard CL benchmark and the large number of tasks benchmark on Llama-3-2-3B we set the learning rate as le-4. For SuperNI benchmark using Jlamm-2-7B-chh and Llama-2-133 1amma-2-13B-hattt we follow the learning rate, training batch size, and gradient accumulation steps in Zhao et al. (2024b), and we use 5e-5 and train the models with five epochs with a training batch size of 2, and gradient accumulation steps of And for SuperNI benchmark Llama-3-2-3B. we set the learning as 5e-5.
D.1 OVERALL RESULTS ON LLAMA-2-13B-CHE
Continual learning performance analysis. As shown in Table 5 our method consistently outper- forms all data-free baselines across two benchmarks using 1lama-2-13B-oo- model.
LoRA-Based continual learning: IncLoRA improves upon SeqLoRA by freezing previously learned LoRAs to isolate subspaces, though its subspace separation is simplistic. SeqLoRA performs a little better than O-LoRA the standard CL benchmark, since we use the hyperparameter À = 0.5 the orthogonal loss in O-LoRA that may be adjusted along with the different models and datasets, while O-LORA outperforms SeqLoRA and IncLoRA in large number of tasks benchmark. SAPT-LORA achieves the highest average performance among -RRA-based methods, but it relies generated previous task pseudo samples, unrealistic in many LLM scenarios, and is more sensitive to task order- ing than our method. LoRM-BA (from second task, begin with freezing B) and LoRM-AB (from second task, begin with freezing A) yield nearly identical results in large number of tasks benchmark, suggesting that the order of alternating LoRA components in learning sequential tasks does not significantly affect outcomes, but LoRM-BA outperforms LoRM-AB in standard CL benchmark. CorDA performs well standard CL benchmark and large number of tasks, but both of them are lower than our method. MagMax performs comparably to our approach on standard CL benchmark, and slightly worse on large number of tasks, thus only keeping the weights which have the largest absolute value would cause catastrophic forgetting.
LoRA merging baselines: KnOTS and LoRA-LEGO perform similarly in the standard CL bench- mark, but KnOTS outperforms in the large number of tasks. KnOTS may benefit from flexible SVD-merging mechanism so that we apply time-aware scaling on merging, while LoRA-LEGO treats tasks equally, lacks prioritization, and is ineffective in complex CL contexts.
20

---

Published as a conference paper at ICLR 2026
Table Testing performance (%) on three CL benchmarks using Llama-2-13B-cher across different task orders, where each result is run three random times, where Oi denotes ith task order.
Standard CL Benchmark
Large Number of Tasks
Method
01
02
03
avg
04
05
06
avg
SeqLoRA
77.1
77.4
78.6
77.7
74.1
74.2
74.5
74.3
IncLoRA
78.3
79.6
79.5
79.1
74.2
76.1
75.1
75.1
O-LoRA
76.1
77.1
78.5
77.2
75.5
75.5
74.8
75.3
SPAT-LORA
83.2
82.4
80.1
81.9
83.9
80.2
82.3
82.1
LORM-BA
78.4
80.2
80.4
79.7
74.9
71.9
70.5
72.4
LoRM-AB
76.2
74.1
75.3
75.2
74.3
70.1
72.3
72.2
CorDA
79.1
80.4
80.6
80.0
75.9
75.8
72.9
74.9
MagMax
80.9
80.6
80.7
80.7
73.7
73.4
76.0
74.4
KnOTS (zero init)
71.9
73.1
73.9
73.0
66.8
65.5
66.0
66.1
LoRA-LEGO
73.0
72.3
72.9
72.7
64.3
63.3
64.0
63.9
OPCM
68.8
63.6
61.6
64.7
57.6
60.2
58.8
58.9
SLAO (ours)
80.8
81.1
81.1
81.0
76.5
75.9
76.1
76.2
Multi-Task
81.4
79.2
Table 6: Comparison of merging strategies testing performance on two CL benchmarks using Lama-2-13B--hee across different task orders, where Oi denotes ith task order.
Standard CL Benchmark
Large Number of Tasks
Method
01
02
03
avg
04
05
06
avg
FREB-MA
77.7
77.5
76.4
77.2
69.1
68.2
69.3
68.9
FREA-MB
79.4
79.2
78.9
79.2
73.2
73.8
73.2
73.4
FTBA-MA
79.2
80.1
80.8
80.0
75.1
75.0
75.8
75.3
FTBA-MBA
80.7
80.9
80.8
80.8
75.4
75.4
76.0
75.6
FTBA-MB
80.6
80.9
80.9
80.8
75.8
75.7
76.0
75.8
SLAO (ours)
80.8
81.1
81.1
81.0
76.5
75.9
76.1
76.2
Continual merging approaches: since OPCM is designed for full model, directly applying it to LoRA by treating its two components identically leads to suboptimal performance.
Asymmetry in LoRA merging. To investigate the asymmetry in LoRA merging, we compare different continual merging strategies for LoRA: (1) Freeze A, merge B (FREA-MB), (2) Freeze B. merge A (FREB-MA) (3) Fine-tune BA, merge A (FTBA-MA), (4) Fine-tune BA, merge BA (FTBA-MBA), (5) Fine-tune BA, merge (FTBA-MB), As shown Table 6 only fine-tuning LoRA and merging B consistently outperforms other strategies, except ours. Fine-tuning LoRA and merging BA has comparable performance compared to fine-tuning LoRA and merging B. but fine-tuning LoRA and merging A yields the poorest performance among fine-tuning LoRA methods. When freezing one component of LoRA during training, freezing A and merging B is much better than freezing B and merging A, consistent with conclusion in Zhu et al. (2024b) that freezing A and fine-tuning B is at least better than freezing B and fine-tuning A. This highlights the asymmetry in LoRA components and the importance of asymmetric merging based their fundamental roles in adaptation.
D.2 RESULTS IN QWEN2.5 MODELS
To evaluate the performance of our SLAO using state-of-the-ard LLMs, Qwen2.5-3B and Qwen2.5-7B to compare with the other two LoRA-based continual learning baselines on the SuperNI benchmark.
21

---

Published as a conference paper at ICLR 2026
Table 7: Comparison of testing performance on SuperNI benchmark using Qwen2.5-3B and Qwen2.5-
7B models across two different task orders.
Qwen2.5-3B
Qwen2.5-7B
Method
01
02
avg
01
02
avg
O-LoRA
31.1
29.8
30.5
34.3
32.6
33.4
InfLoRA
35.6
25.6
30.6
43.5
31.0
37.3
SLAO
37.8
32.4
35.1
41.0
35.5
38.3
Table 8: Comparison of initialization strategies for existing LoRA merging methods on testing performance on two CL benchmarks using Llama-2-7B-chat and Llama-2-13B-chh across different task orders, where Oi denotes ith task order.
Standard CL Benchmark
Large Number of Tasks
Init
Model
Method
01
02
03
avg
04
05
06
avg
Random (Zero)
7B
KnOTS
67.9
65.9
70.8
68.2
61.5
60.1
58.0
59.9
Random (Zero)
7B
LoRA-LEGO
68.3
66.0
70.9
68.4
58.8
58.7
53.2
56.9
Last-FT
7B
KnOTS
79.7
80.7
79.9
80.1
74.0
72.5
75.0
73.8
Last-FT
7B
LoRA-LEGO
78.8
79.8
79.5
79.4
72.2
70.4
74.2
72.3
Random (Zero)
13B
KnOTS
71.9
73.1
73.9
73.0
66.8
65.5
66.0
66.1
Random (Zero)
13B
LoRA-LEGO
73.0
72.3
72.9
72.7
64.3
63.3
64.0
63.9
Last-FT
13B
KnOTS
80.1
79.6
80.3
80.0
75.4
74.9
75.8
75.4
Last-FT
13B
LORA-LEGO
79.8
80.0
80.2
80.0
75.3
73.8
76.0
75.0
When evaluating SuperNI benchmarks in Table 7 SLAO consistently better than other two baselines and achieves the best average performance on both Qwen2.5-3B and Qwen2.5-7B models, demonstrating its robustness on Qwen2.5 model sizes. Also, the performance of SLAO on Qwen2.5- 3B is better than that Qwen2.5-7B.
D.3 IMPACT OF INITIALIZATION STRATEGIES FOR EXISTING LORA MERGING METHODS
We compare the testing performance of different initialization strategies for existing LoRA merging methods in Table where we compare two strategies: random (zero) initialization and the last fine-tuning point of previous tasks. For zero initialization for new tasks, when using Llama-2-7B-che and 1lama-2-13B-char KnOTS and LoRA-LEGO perform similarly in the standard CL benchmark. but KnOTS outperforms in the large number of tasks. For last fine-tuning point initialization, KnOTS and LoRA-LEGO perform similarly in the standard CL benchmark and the large number of tasks, while KnOTS is slightly over LoRA-LEGO. All results in the last fine-tuning point initialization are significantly better than the zero initialization. These results show that our last fine- ane-tuning point initialization has better performance than zero initialization in continual merging scenarios.
D.4 CONTINUAL LEARNING PERFORMANCE ON BACKWARD TRANSFER
We evaluate the performance of backward transfer (BWT) using -lama-2-7B-aa on the large number of tasks benchmark. As shown Table 9 SLAO demonstrates strong backward transfer ability. Among all methods, SeqLoRA performs the worst due to its lack of mechanisms to prevent forgetting. KnOTS and LoRA-LEGO also underperform, as they are primarily designed for model merging rather than continual learning. IncLoRA exhibits limited BWT performance, while O- LoRA achieves better results by enforcing orthogonality during learning. InfLoRA slightly trails O-LoRA, and CorDA performs worse, possibly due to its reliance nullspace projection without tine-aware updates. SAPT-LoRA achieves the best BWT overall, though it benefits from synthetic data from previous tasks, which may not be feasible in realistic settings. Between the two variants of LoRM, LoRM-BA slightly outperforms LoRM-AB. Finally, MagMax and OPCM show comparable performance, both designed balance update integration during continual merging.
22

---

Published as a conference paper at ICLR 2026
Table Testing performance (%) of the average backward transfer (BWT) on large number of tasks using Llama-2-78 2-mhh-h-- across different task orders.
Method
BWT
Method
BWT
SeqLoRA
-17.2
LoRM-BA
-6.7
IncLoRA
-9.6
LoRM-AB
-4.1
O-LoRA
-4.0
MagMax
-3,8
InfLoRA
-4.9
OPCM
-3.9
SAPT-LoRA
-2.9
KnOTS (zero init)
-14.1
CorDA
-4.5
LoRA-LEGO
-15.6
SLAO (ours)
-3.5
Table 10: Comparison of MOPD and AOPD on testing performance across three standard CL
benchmarks using Llama-2-7B-char in different task orders.
Standard CL Benchmark
Large Number of Tasks
SuperNI Benchmark
Method
MOPD
AOPD
MOPD
AOPD
MOPD
AOPD
O-LoRA
9.84%
5.79%
17.87%
8.53%
22.16%
11.63%
SAPT-LORA
8.75%
5.69%
18.94%
9.65%
25.28%
12.38%
InfLoRA
8.23%
2.54%
19.58%
10.01%
23.42%
11.46%
SLAO (ours)
1.72%
1.30%
15.17%
7.16%
18.94%
10.76%
D.5 CONTINUAL LEARNING PERFORMANCE ORDER-NORMALLZED PERFORMANCE DISPARITY
We evaluate the performance of Order-normalized Performance Disparity Yoon et al. 2020) using Alama-2--BB-hhe on three benchmarks. Order-normalized Performance Disparity is used to evaluate order-sensitivity for each task t, defined as the disparity between its performance on R random task sequences:
OPD; = max P P......PP'!
...)) min P
P")
(48)
where P, denotes the performance of task t to the task sequence r. The Maximum OPD de- fined MOPD max(OPD,,.... OPD;) and the Average OPD is defined as AOPD EL OPD,, evaluate robesttobustness on the whole task set. Lower scores of both metrics indicate higher robustness.
Table shows the performance of MOPD and AOPD on three benchmarks with their task sequences. Our SLAA shows the most stable performance across different task orders, indicating that it handles order sensitivities better compared to other baselines. While SAPT-LoRA achieves higher scores in Table I it heavily depends on past tasks' data information, so that SAPT-LoRA suffers from greater variation different task orders, mostly due to its past pseudo-sample generation.
D.6 CHOICE OF ORTHOGONAL DECOMPOSITION STRATEGY
In our algorithm, SLAO, we use QR decomposition to extract the orthogonal basis from previous LoRA A. To evaluate the effectiveness of QR decomposition, we compare it against orthogonal bases derived from (I) singular value decomposition (SVD), where the product UV! forms an orthogonal approximation, and (2) randomized SVD, where the product QU forms an orthogonal approximation. As shown Table II QR initialization performs similarly to the SVD approach on standard CL benchmark and large number of tasks, but the performance of QR SuperNI benchmark is better than that of SVD. The performance of randomized SVD is not better than SVD and QR across these three benchmarks.
23

---

Published as a conference paper at ICLR 2026
Table 11: Comparison of orthogonal decomposition strategies on testing performance on two CL
benchmarks using Alama-22-7--oe across different task orders, where Oi denotes ith task order.
Standard CL Benchmark
Large Number of Tasks
SuperNI Benchmark
Method
01
02
03
avg
04
05
06
avg
01
02
Randomized SVD
72.6 69.1
73.3
71.7
52.3
62.5
57.8
57.5
11.8
22.8
17.3
SVD
79.9
80.8
80.2
80.3
75.3
74.4
75.1
74.9
36.9
33.7
35.3
QR
80.1 80.8 80.4
80.4
75.0
74.4
75.1
74.8
38.7
35.7
37.2
D.7 ASYMMETRY OF LORA
We separately fine-tune tasks from the SuperNI benchmark using LoRAs on Llama-2-7B- chat (Touvron et al. 2023), and compute cosine similarity of A and B across 15 tasks using the last layer LoRA. Figure shows that exhibits significantly higher similarity across tasks compared to B, suggesting that LoRA components follow inherently different learning dynamics.
(a) LoRA (q)
(b) LoRA B (g)
(c) LoRA (V)
(d) LoRA B (V)
Figure 5: Cosine similarity between 15 tasks from SuperNI benchmark for fine-tuned and v attention
LoRA A and B in the last layer (32nd) Llama-2-7BB Mamma-2-B-chat
D.8 IMPACT OF LEARNING RATE
To assess the effect of different learning rates, we evaluate SeqLoRA on the standard CL benchmark using L-lama-2-7B-e rate of le-4 achieves the best performance, while 1e-3 performs the worst, significantly degrading the overall results. This highlights the importance of careful learning rate selection LoRA-base continual learning.
with learning rates of 1e-3, le-4, and le-5. As shown in Table [2] learning
Table 12: Impact of learning rate testing performance of SeqLoRA using Llama-2-7B- 2-7B-chhtt across three task orders in Standard CL Benchmark, where Oi denotes ith task order.
Standard CL Benchmark
learning rate
01
02
03
avg
1e 3
6.0
0.0
19.7
8.6
1e 4
73.3
76.2
78.4
76.0
le 5
73.6
71.8
76.0
73.8
D.9 IMPACT OF THE RANK OF LORA
assess the effect of different LoRA rank values our algorithm by comparing three rank settings on both the standard CL benchmark and the large number of tasks benchmark using Llama-2-133 chat. As shown in Table 13] a rank of yields the best performance on the standard CL benchmark, while rank of performs best the large number of tasks benchmark. Overall, the performance differences across the three ranks are relatively small, suggesting that our method is robust to the choice of rank.
24

---

Published as a conference paper at ICLR 2026
Table 13: Impact of the rank LoRA on testing performance of SLAO using Llama-2-13B-aa
across Standard CL Benchmark and large number of tasks, where Oi denotes ith task order.
Standard CL Benchmark
Large Number of Tasks
rank
01
02
03
avg
04
05
06
avg
80.6
81.0
81.1
80.9
77.2
76.3
77.1
76.9
80.8
81.1
81.1
81.0
76.5
75.9
76.1
76.2
80.3
80.9
80.9
80.7
76.9
75.5
75.9
76.1
Table 14: Comparison of training cost across three standard CL benchmarks using Llama-2-7B-chat
(average cost across different task orders, FTBA-MBAOI: FTBA-MBA with orthogonal initialize A).
Standard CL Benchmark
Large Number of Tasks
SuperNI Benehmark
Method
Peak GPU
GPU
Peak GPU
GPU
Peak GPU
GPU
Memory
Walltime
Memory
Walltime
Memory
Walltime
O-LoRA
35.71GB
01:21:49
37.55GB
02:47:06
38.06GB
02:48:34
InfLoRA
45.21GB
01:47:48
57.66GB
04:57:22
58.99GB
05:01:01
FTBA-MBA
35.29GB
00:51:43
35.64GB
01:59:51
36.01GB
02:02:07
FTBA-MBAOI
35.43GB
00:51:27
36.23GB
01:59:12
37.59GB
02:01:35
FTBA-MB
35.17GB
00:51:36
35.47GB
01:59:26
35.91GB
02:01:49
SLAO (ours)
35,24GB
00:50:58
35.61GB
01:59:00
35,94GB
02:00:08
D.10 COMPARISON OF TRAINING COST
We compare the training cost among several baselines in Table 14 We use single NVIDIA A100 GPU to fine-tune Llama-2-7B-chhe It shows that our SLAO is both memory usage efficient and training efficient, since only compute one-time QR matrix factorization at initialization, avoiding additional computational cost during training. Besides, we observe that the walltime under orthogonal initialization is often smaller than the walltime without orthogonal initialization. There is similar conclusion in Hu et al. (2020), which proves that drawing the initial weights from the orthogonal group can speed up convergence. Therefore, SLAO provides an ideal balance between performance, memory usage, and training speed.
D.11 COMPARISON OF ORTHOGONALITY AMONG LORA-BASED CL METHODS
Initialization:
(a) O-LoRA: New task's LoRA B,A, is randomly initialized and not orthogonal to previous tasks' LoRAs (BiA1,... B:...,,,
(b) InfLoRA:
Use all of new task data compute its input matrix H, on current full model parameters, and use all previous i - tasks' gradient spaces which denote as M, to make new A, Rxx lie N n M,' where N; is the subspace spanned by the columns of H, B; RRdx is initialized as zero
(c) SLAO: Extract orthogonal basis from previous fine-tuned A,-1 E Rr as new task's
A, which makes A.A, I,. B, gdxr is initialized as previous fine-tuned B;
Training:
(a) O-LoRA: Compute orthogonal loss to make new task's A, E gd orthogonal to all previous tasks' A, and update A, and B;.
(b) InRLoRA:
Compute standard cross entropy loss and update B, E RRdxr
(c) SLAO: Compute standard cross entropy loss and update B, E d and A, E Brxd
Post-Training:
(a) O-LoRA: Store all tasks' LoRA [B..,.... .,A.).
(b) InfLoRA: Use new task i data to compute new task input matrix R; on new learned
B; then compute new gradient orthogonal bases memory M, through DuaIGPM,
25

---

Published as a conference paper at ICLR 2026
where M; represents the gradient space of all tasks. Then, integrate B,A; to W and store the updated gradient space M, of all tasks. (c) SLAO: Merge B, to previously merged B and keep fine-tuned B;, merged B, and fine-tuned A,,
Overall, InfLoRA and SLAO both focus on the orthogonality of initialization and post-training. while O-LoRA focuses on the orthogonality of the updating process during training. Moreover, for initialization, InfLoRA makes new A; lie at the intersection of input matrix and previous gradient spaces M,, while SLAO extracts orthogonal basis from previous fine-tuned A; new A; for post-training, InfLoRA computes and stores all previous tasks' orthogonal gradient spaces, while SLAO uses the asymmetry of LoRA to obtain merged B.
D.12 DESCRIPTIONS OF TASK SEQUENCE ORDERS
We report task descriptions and their metrics used for our CL experiments across Llama models in Table 13 and Table T6] And we show eight task orders in Table I7
Table 15: Descriptions of datasets in Large Number of Tasks benchmark and first datasets from standard CL benchmark.
Dataset name
Category
Task
Domain
Metric
1. Yelp
CL Benchmark
Sentiment analysis
Yelp reviews
Accuracy
2. Amazon
CL Benchmark
Sentiment analysis
Amazon reviews
Accuracy
3. DBpedia
CL Benchmark
Topic classification
Wikipedia
Accuracy
4. Yahoo
CL Benchmark
Topic classification
Yahoo Q&A
Accuracy
5. AG News
CL Benchmark
Topic classification
News
Accuracy
6. MNLI
GLUE
Natural language inference
Various
Accuracy
7. QQP
GLUE
Paragraph detection
Quora
Accuracy
8. RTE
GLUE
Natural language inference
News, Wikipedia
Accuracy
9. SST-2
GLUE
Sentiment analysis
Movie reviews
Accuracy
10. WIC
SuperGLUE
Word sense disambiguation
Lexical databases
Accuracy
11. CB
SuperGLUE
Natural language inference
Various
Accuracy
12. COPA
SuperGLUE
Question and answering
engss,encyclopedaa
Accuracy
13. BooIQA
SuperGLUE
Boolean question and answering
Wikipedia
Accuracy
14. MultiRC
SuperGLUE
Question and answering
Various
Accuracy
15. IMDB
SuperGLUUE
Sentiment Analysis
Movie reviews
Accuracy
Table 16: Descriptions of 15 datasets in SuperNI benchmark.
Dataset number
Dataset name
Task
Metric
1. task639
multi-vwo--user-utttrann--eee generation
dialogue generation
Rouge-I.
2. task 1590
seglommay-ttx--gemeerre
dialogue generation
Rouge-L
3. task 1729
personaccht--eeneratt----
dialogue generation
Rouge-L
4. task 181
outcome extraction
information extraction
Rouge-L.
5. 748
information extraction
Rouge-L
6. task 15 10
ovaluaation-ellationeeetttaaaaa
information extraction
Rouge-L
7. task002
quoref-answer--eeneraiionn
question answering
Rouge-L
8. task073
commonseensq--aansveeeeennnnnii
question answering
Rouge-L
9. 1
sciq-
answer-gennerationn
question answering
Rouge-L
10. task511
ceddir-fff-long--etts-smmmmmm
summarization
Rouge-L
11. task
ksum-summarization
summarization
Rouge-L.
12. 152
samsum-summary
summarization
Rouge-L.
13. task363
sst2-polarity
sentiment analysis
Accuracy
14. task875
emotion-classifcaatn
sentiment analysis
Accuracy
15. task 687
sentimentl 40-classification
sentiment analysis
Accuracy
26

---

Published as a conference paper at ICLR 2026
Table 17: Eight different task orders.
Order
Model
Task Sequence
Llama-2-7B-char
1
Llama-2-13B-chatt
dbpedia-> amazon - yahoo - ag
Llama-3-2-3B
Llama-2-7B-chat,
2
Llama-2-13B-chart
dbpedia-> amazon - ag yahoo
Llama-3-2-3B
3
Blama-2-13B-chatt
yahoo + amazon ag dbpedia
Llama-3-2-3B
Llama-2-7B-chat
mnli + cb wic copa + qqp + boolqa iiddb
4
Llama-2-13B-Bn 1-B-hthht,
yelp + amazon + sst-2 ~ dbpeddaa - ag -multirc - yahoo
Llama-3-2-3B
Llama-2-7B-che
multire boolqa + wic + mnli + cb - copa q qop - + imdb + sst-2 + dbpedia + + yelp - amazon +yahoo
5
Hlama-2-13B-hhal
Llama-3-2-3B
Alama-2-7B-chan
yelp amazon mnli + cb - copa + qqp ~ imdd sst-2 + dbpedia - ag - yahoo - multire -boolqa + wie
6
Llama-2-13B- 13B-chat,
Llama-3-2-3B
Llama-2-78 2-B-chat,,
task + task363 > t2s - task181 - task002
(SuperNI)
Llama-2-13B-as
~ttsk15100 task639 1729 task073 1590
Llama-3-2-3B
task748 - task5 - task5591 - task 1687 - task875
Llama-2-7B-chat
task748 - task073 - task1590 task639 -> task
2
(SuperNI)
B-ahh-,,
task1687 task591 task363
task1510-
task1729
Llama-3-2-3B
task181 - task511 - task002 - task1299 - task875
27


Reviews:
[{"review_id": "czYgZDlcL5", "reviewer": "Reviewer_tRRS", "summary": "This paper proposes SLAO, an algorithm that can avoid forgetting while maintaining a single shared LoRA across all tasks, avoiding the linear memory growth of storing multiple LoRAs. The authors provide NTK-based theoretical analysis to motivate orthogonality and asymmetry handling, and show consistent improvements over LoRA-based CL methods.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ The motivation of the intrinsic asymmetry property of LoRA is clearly presented in Figure 1.\n+ The paper provides theoretical analyses grounded in NTK theory.\n+ The writing is generally clear and well-organized.", "weaknesses": "+ Unclear mechanism for avoiding forgetting. 1)While I can understand how InfLoRA prevents forgetting by projecting updates into the null space of old task features or directly using old task samples, I find it difficult to see how the proposed orthogonal initialization in this paper achieves the same effect. For the merging step, many prior works in multi-task learning compute task vectors as ΔW = B·A. In your formulation, however, A is replaced, and only B is merged. It remains unclear why merging B alone can effectively prevent forgetting. 2) Another concern is, the definition of task vector here is different from that in MTL, where the fine-tuned parameters subtract the same original weights instead of the merged ones. 3) It would be helpful to explore the effect of the scaling factor $\\lambda(i)$.\n+ The paper claims that the proposed method avoids linear memory growth by maintaining only the merged LoRA and the current task’s LoRA, without storing all task-specific adapters. However, in methods such as O-LoRA and InfLoRA, it is also possible to directly merge different task-specific LoRAs while achieving comparable performance. I would appreciate clarification on how Figure 2 was computed in this regard, and a more detailed explanation of how the proposed approach fundamentally differs from these existing merging strategies in terms of scalability.\n+ While analytically clean, this setting does not fully capture nonlinear behavior in LLM fine-tuning. The link between the NTK bounds and empirical performance could be better validated with empirical NTK measurements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nDeu5c5qde", "reviewer": "Reviewer_TKgi", "summary": "This paper proposes SLAO (Single LoRA Continual Learning with Orthogonal Initialization via Continual Merging), a framework that maintains a single shared LoRA across all tasks by sequentially merging new task updates, eliminating the need for task-specific LoRA storage.  Experiments on Llama-2-7B-chat, Llama-2-13B-chat, and Llama-3-3B demonstrate SLAO’s effectiveness.", "questions": "1. How does SLAO perform when tasks have extreme similarity (e.g., multiple sentiment analysis tasks) or dissimilarity (e.g., sentiment vs. QA)? Does orthogonal initialization overly restrict knowledge transfer for similar tasks, and can the orthogonality constraint be relaxed dynamically?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. SLAO is the first to enable CL with a single shared LoRA via sequential merging.  \n\n2. SLAO is robust to hyperparameters (e.g., LoRA rank, learning rate) and model scales. In particular, performance improves with larger models.", "weaknesses": "1. While SLAO’s training overhead is low, QR decomposition for orthogonal basis extraction adds a one-time cost per task. The paper does not quantify this cost for long sequences (e.g., 50+ tasks) or analyze whether approximate orthogonal methods (e.g., randomized SVD) could reduce the cost without performance loss.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3XjU6JX7rg", "reviewer": "Reviewer_ucvk", "summary": "This paper introduces SLAO, a novel continual learning (CL) method for large language models (LLMs) that leverages Low-Rank Adaptation (LoRA) and continual merging to maintain a single shared LoRA across tasks. Unlike existing LoRA-based CL methods that retain multiple task-specific LoRAs or generate pseudo-data, SLAO sequentially merges new task updates into a unified LoRA via orthogonal initialization and time-aware scaling. The approach ensures constant memory usage, reduces task interference, and improves generalization. Theoretical analysis and extensive experiments on multiple benchmarks and model scales (e.g., Llama-2, Llama-3) demonstrate its effectiveness and efficiency.", "questions": "See the weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ Provides a formal analysis of forgetting and intransigence in the NTK regime and motivates the design via LoRA asymmetry.\n+ The concept of continual merging into a single LoRA is novel and addresses key limitations of existing LoRA-based CL methods.", "weaknesses": "+ The method is designed for llm but evaluated on models that are not state-of-the-art, as well as tasks that are easy for current llms. Models such as qwen2.5/3 series and tasks such as aime, livecodebench or at the same difficulty level are needed. At lease, the reviewer think the tasks should be more diverse.\n+ Could the orthogonal initialization strategy be combined with other PEFT methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ApoQyiE7KH", "reviewer": "Reviewer_YyiC", "summary": "This manuscript addresses the linear growth in memory and computational cost in LoRA-based continual learning (CL), where prior methods store a separate adapter per task. The authors propose SLAO, a novel method that maintains only a single LoRA adapter, achieving constant memory complexity. SLAO frames CL as a sequential merging problem. For each new task, it uses an orthogonal basis from the prior adapter for initialization, fine-tunes, and then performs an asymmetric merge: it replaces the task-invariant A matrix and applies a time-aware weighted average to the task-specific B matrix. Experiments on several benchmarks and Llama models show SLAO outperforms other data-free, constant-memory CL baselines.", "questions": "Refer to Weaknesses for related questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The core insight to treat LoRA's A and B matrices asymmetrically is clever, well-supported by empirical analysis (Fig. 1), and validated in ablations (Table 3), proving its critical role in the method's success.\n\n2. The method is comprehensively evaluated against the relevant baselines (adapter-isolation, merging, etc.) and is clearly shown to be the superior data-free approach. The claims are further supported by thorough ablations for initialization, merging strategy.", "weaknesses": "1. The merge for the A matrix is a full replacement ($A_{merge}^{i} = A_{ft,i}$) by the newest task, which risks forgetting the optimal basis for older tasks. This \"winner-takes-all\" approach needs more discussion.\n\n2. A significant performance gap remains between SLAO and data-rehearsal methods, especially on complex benchmarks. A brief discussion on hybridizing SLAO's constant-memory model with a constant-sized buffer would add context.\n\n3. The theoretical analysis provides good motivation for the design, but the final algorithm isn't a strict derivation. For instance, the scaling factor $\\lambda(i) = 1/\\sqrt{i}$ is adopted from prior work, not derived from the paper's own analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)