TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree
Authors: Yu-Yang Qian
Yuan-Ze Xu
Zhen-Yu Zhang
Peng Zhao
Zhi-Hua Zhou
Venue: ICML 2025 poster (2025)
Keywords: Continual Learning
Supervised Fine-Tuning
Large Pre-trained Models
Low-Rank Adaptation
Hierarchical Task Structure
Primary area: general_machine_learning->sequential_network_and_time_series_modeling

Abstract: Many real-world applications collect data in a streaming environment, where learning tasks are encountered sequentially. This necessitates *continual learning* (CL) to update models online, enabling adaptation to new tasks while preserving past knowledge to prevent catastrophic forgetting. Nowadays, with the flourish of *large pre-trained models* (LPMs), *efficiency* has become increasingly critical for CL, due to their substantial computational demands and growing parameter sizes. In this paper, we introduce TreeLoRA (K-D Tree of Low-Rank Adapters), a novel approach that constructs *layer-wise* adapters by leveraging hierarchical gradient similarity to enable efficient CL, particularly for LPMs. To reduce the computational burden of task similarity estimation, we employ *bandit* techniques to develop an algorithm based on lower confidence bounds to efficiently explore the task structure. Furthermore, we use sparse gradient updates to facilitate parameter optimization, making the approach better suited for LPMs. Theoretical analysis is provided to justify the rationale behind our approach, and experiments on both *vision transformers* (ViTs) and *large language models* (LLMs) demonstrate the effectiveness and efficiency of our approach across various domains, including vision and natural language processing tasks.

TreeLoRA: Efficient Continual Learning via Layer- LoRAs Guided by a Hierarchical Gradient-Similarity Tree
Yu- Yang Qian 12
Yuan-Ze Xu 12 Zhen- Yu Zhang?
Peng Zhao 12 Zhi-Hua Zhou 12
Abstract
algorithms (Bengio et al., 2021; Zhou, 2022; Sutton et al., 2022), where new tasks with different data distributions are encountered sequentially, leading to a performance degra- dation over time (Vela et al., 2022). This sequential na- ture necessitates a learning method capable of continuous adaptation and evolution, named continual learning (CL), in which models must adapt new tasks while retaining knowledge from previous ones to alleviate catastrophic for- getting (Thrun & Pratt, 1997).
Many real-world applications collect data in streaming environment, where learning tasks are encountered sequentially. This necessitates con- tinual learning (CL) to update models online, en- abling adaptation to new tasks while preserving past knowledge to prevent catastrophic forgetting. Nowadays. with the flourish of large pre-trained models (LPMss), efficiency has become increas- ingly critical for CL, due to their substantial com- putational demands and growing parameter sizes. In this paper, we introduce LoR (K-D Tree of Low-Rank Adapters), a novel approach that constructs layer-wise adapters by leveraging hier- archical gradient similarity to enable efficient CL, particularly for LPMs. To reduce the computa- tional burden of task similarity estimation, we employ bandit techniques to develop algo- rithm based on lower confidence bounds to ef- ficiently explore the task structure. Furthermore, we use sparse gradient updates to facilitate pa- rameter optimization, making the approach better suited for LPMs. Theoretical analysis is provided to justify the rationale behind our approach, and experiments on both vision transformers (ViTs) and large language models (LLMs) demonstrate the effectiveness and efficiency of our approach across various domains, including vision and nat- ural language processing tasks.
Existing CL. approaches can be generally classified into three main categories (Wang et al., 2024b): regularization- based methods (Kirkpatrick et al., 2017), which introduce specially designed regularizers prevent the model from forgetting previously learned knowledge; rehearsal- based methods (Shin et al., 2017), which store data from previ- ous tasks in a buffer and reuse it during the ongoing task stream; and rrchitecture-basee methods (Mallya & Lazeb- nik, 2018; Serrà et al., 2018), which design specialized model architectures to dynamically adapt to new tasks and mitigate forgetting. These approaches have demonstrated strong empirical performance for continual learning tasks with conventional machine learning models.
Nowadays, with the rise of the pre-training paradigm, large pre-trained models (LPMs) have excelled in numerous tasks such as natural language processing (Brown et al., 2020) and computer vision (Radford et al., 2021). The necessity of efficient CL, especially the approaches that are tailored to LPMs, has become increasingly demanding, due the substantial computational demands and growing parame- ter sizes. Therefore, it is essential to explore the issue of efficient continual learning, particularly the Continual Fine- Tuming (CFT) of LPMs. Although there exist some continual learning methods developed for LPMs (Scialom et al., 2022: Wang al., 2023b; Dou et al., 2024), they often do not focus the efficiency in CL. typically exhibiting compu- tational complexity that scales linearly with the number of tasks. Consequently, this raises a critical new challenge: How to develop an efficient continual learning approach, particularly fitted for large pre-trained models?
1. Introduction
Machine learning algorithms have achieved significant suc- cess across a wide range of applications. However, dis- tribution shift still poses a challenge to machine learning
"National Key Laboratory for Novel Software Technology. Nan- jing University, China 'School of Artificial Intelligence, Nanjing University, China 'RIKEN AIP, Tokyo, Japan. Correspondence to: Peng Zhao <zhaop@ lamda.nju.cdu.cn..
In this paper, we propose an efficient continual learning ap- proach that facilitates adaptation to new tasks by leveraging task-shared knowledge, while avoiding the linear computa-
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by Hierarchical Gradient-Sinin Similarity Tree
tional complexity that scales with the number of tasks. Our
approach, TreeLoRA (short for "K-D Tree of Low-Rank Adapters"), introduces layer-wise adapters to construct hierarchical free based on the criteria of gradient similar- ity. This design organizes and manages historical tasks efficiently, thereby eliminating linear task-dependence. To ensure overall efficiency, two key components are devel- oped. First, is necessary to estimate the gradient similarity on the fly during training, and we employ bandit techniques and develop an algorithm based on the lower confidence bound (Garivier& & Moulines, 201 1) to efficiently search on the tree structure. Second, to better suit the LPMs, we apply sparse gradient updates to optimize model parameters more efficiently for new task adaptation.
Test Accuracy
60
40
20
TreeLoRA
LoRA
4000
5000 8000
10000
14000 16600
Samples
Figure 1: Test accuracy vs. the number of arrived samples for our TreeLoRA and the vanilla LoRA (Hu et al., 2022) on the Split CIFAR-100 dataset when encountering new task, demonstrating our efficiency adapt to new tasks, owing to mining the hicrarchi- cal suddient-simllriig structure to explore task-shared knowledge.
Our approach shows advantages both in theory and prac- tice. Theoretically, owing to our designed tree structure, our approach achieves tighter regret bounds compared to conventional bandit search methods that do not utilize such a structure, justifying the rationale behind our approach. Empirically, we conduct experiments on vision transformers and large language models, across both the computer vision benchmarks and natural language processing tasks, vali- date the performance and efficiency of our approach. The experimental results demonstrate that, with similar or even improved performance, our method achieves up to a 3.2x speedup for ViTs and 2.4x x speedup for LLMs in training time, compared to previous statc-of-the-an methods, thereby validating the effectiveness and efficiency of our approach.
The goal is to learn a good model that performs well not only on the current task but also on previous ones. Specifically, we denote fi (wy) as the prediction performance (such as the accuracy, SART score, and ROUGE-L score, etc.) evaluated task using the parameters updated up task j. then the overall performance (OP) for the n-th model is
A OP, n
f((w))
Another important goal is to minimize the forgetting rate, which is defined buckward transfer (BWT) that measures catastrophic forgetting by comparing model's performance old tasks before and after learning new ones, that is
Organization. Section 2 introduces the problem formula- tion. Section presents our TreeLoRA approach. Section 4 provides theoretical guarantees. Section 5 reports exper- iments, and Section 6 concludes the paper. Due page limits, defer more empirical studies to Appendix A and related works to Appendix B. All proofs are in Appendix C.
1 BWT,
72 Σ ( f(ww))
which measures the performance degradation on previous tasks. Therefore, the goal is to learn a model w,, that can achieve maximal OP and minimal BWT. Note that we focus challenging task where the identity of the task is not available during testing.
2. Problem Formulation
In this section, we introduce the problem formulation. Our setting is typical continual learning problem (Lange al., 2022), where there are two stages: (i) the offline initializa- tion stage and (ii) the online adaptation stage.
3. Our Approach
This section presents our approach. First, we introduce our novel continual update framework for large pre-trained mod- els, leveraging LoRAs with tree structure. Next, we detail the construction of the tree using bandit algorithm. Finally, we explain how the model is updated efficiently through sparse updates facilitated by the TreeLoRA structure.
(i) In the offline initialization stage, the learner has access to a lot of labeled data sampled from initial distribution Dy defined over the feature space X' and label space y, and can use these data to learn a good pre-trained model parameter Wo Rd
(ii) In the online adaptation stage, sequence of tasks (Ti, (Ti,T,,... To,..., TN) arrive sequentially, where N is the total number. For the n-th task, the learner receives a few labeled data S,, = {x,, v h sampled from distribution Dr.- The learner needs to use these data to update the current model parameter W,, E Rdd
3.1. Constructing Task-Similarity Tree Structure
In this part, we will detail our novel continual update frame- work for LPMs, leveraging LoRAs with a tree structure.
Our key technical insight is that some tasks exhibit similar characteristics and can be grouped together, allowing for
2

---

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by Hierarchical Gradient-Similarig Tree
Tasks CL A
TreeLoRA Structure
Efficient Bandit Search for TreeLoRA
Root
O
Intermediate Node
Intermediate Node
Shallow Layers
O
Deep Layers
inputs
outputs
Leaf nodes
1
task3
task2
task N
Self- Attantion: Layer
Selk- Attention Layer
Self- Attention Layer
Self-
Attantion
Tokenizer
Embedding
Layer
MLP
(a) Our ToR Structure
(b) Assembling ToRALoRA into Transformer based Models
Figure 2: Figure illustrating our approach: (a) We develop the TreeLoRA approach for efficient continual learning, where each node represents task group, and each leaf node corresponds to an individual task. The children of each node represent subgroups that belong to the parent node. (b) Employing TreeLoRA for transformer based models: when encountering new task, we first perform the bandit algorithm to identify the most relevant task group, then assemble the corresponding LoRA adapter to efficiently adapt to the task.
the use of task-shared knowledge to improve performance and accelerate adaptation to new tasks. Conversely, tasks that differ significantly or even conflict with others are han- dled separately to alleviate catastrophic forgetting. With this insight, we propose novel approach that explicitly leverages the hierarchical task-similarity structure to en- hance the efficiency of continual learning. Specifically, we introduce TreeLoRA, designed to explore and manage task relationships, whose effectiveness is shown in Figure 1.
and the TreeLoRA structure is constructed based on the similarity of gradient directions. Specifically, the node Nr (representing set of tasks) is defined as:
ENJ. V₁ = max S [M] llg. gr S &.,, (1)
where 5 is the similarity threshold. Note that we choose I.1- norm as the similarity measure, as Ll-norm is more robust in high- nigh-dimensional spaces (Aggarwal et al., 2001). This hierarchical task structure facilitates efficient adaptation to new tasks while preserving knowledge from previous tasks, By grouping tasks with similar characteristics, shared knowl- edge can be leveraged to accelerate learning and enhance continual learning performance.
Tree Structure Design. Consider a deep neural network as the learning model, which can be divided into multiple layers of parameters and connected by activation functions. Shallow layers (closer to the input) tend to capture task- shared common patterns, while the deeper layers (closer to the output) capture task-specific semantics. Our TreeLoRA designed mirror this structure, as illustrated in Figure 2: the root node represents all tasks, and nodes in shallow layers represent adapters that capture shared patterns across several task groups. Nodes in deeper layers represent task- specific semantics, and the leaf nodes correspond to adapters for individual tasks.
summarize, our TreeLoRA structure begins with a root node and incrementally builds hierarchical tree structure new tasks arrive. For each incoming task, it utilizes bandit algorithm to identify the most suitable branch that shares similar gradient directions. Sparse gradient updates are then performed to efficiently adapt the model parame- ters while retaining previously acquired knowledge. At the end of each task stream, the key parameters of that task are recorded, and the TreeLoRA structure updated accord- ingly. The complete algorithm is outlined in Algorithm 1. In the following sections, we provide detailed explanation of how the tree is adaptively constructed using the bandit algorithm, and our sparse updates mechanism.
Task Similarity Measurement. We organize tasks based on the similarity of the gradient directions tasks with similar gradient directions are grouped together, facilitat- ing shared knowledge extraction and streamlined continual learning. Formally, consider data stream of N tasks. At round n [N]. we receive new task associated with a labeled dataset S,, = (xl, yf) sampled from the task distribution D,, The expected gradient direction for task n of model i (w.r.t. the task Tx)) is defined as:
3.2. Adaptive Tree Construction via Bandit Algorithm
In this section, we demonstrate how the tree structure is con- structed. As illustrated in the previous section, TreeLoRA captures task similarity through gradient direction. A signif- icant challenge, however, lies in the fact that the gradient di- rection can only be determined after calculating gradients of
gk E (x.x))),, V((Wwxx,y))
3

---

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similariy Tree
all previous tasks with linear complexity. This requirement makes it computationally infeasible, it would require per- forming gradient descent (NN) times, for all previous tasks. To overcome this limitation, we introduce a novel bandit algorithm that efficiently constructs the tree structure on the fly with only one gradient query each round, therefore greatly improving the computational efficiency.
Algorithm 1 Tre for Efficient Continual Learning
Input: learning rate a, regularization coefficient À Initialize: task similarity estimation fās = 0, Ak E [M]
Initialize the TreeLoRA structure with a root node;
for n = 1,..., N do
Receive n-th task with labeled data (,=((x, y |=
for = I..... Ma, do
Use LCB to select most promising branch as Eq. (2);
Converting to Multi-Armed Bandits Problem. The key dilemma lies in the costly computation complexity of cal- culating all previous gradients to determine the gradient similarity between the current task and previous tasks. To tackle this challenge, instead of calculating all previous gra- dients, we only select one most promising previous task to query the gradient and estimate the similarity. This scenario naturally aligns with an apporratoonexploiineee Specifically, we convert the challenge multi-armer ban- dits (MAB) problem, fundamental problem in the online learning field, where the learner does not have access to all previous gradients but can only selectively obtain some (or even just onc) gradient from the set of previous tasks.
Employ sparse gradient updates as Eq. (3);
Update the estimated task similarity jik.
end
Record the task-specific adapter for the n-th task; Update the TreeLoRA structure as in Section 3.3.
end
trade-off.
where مoa = [Selectx] Ere (Select)) 57 is the estimated task similarity between the current task and the k-th task group (i.e., the nodes in the branch of the selected leaf node at round t), C is the set of all leaf nodes, and C is the child nodes of the k-th node. The term 2 l/log// t/ng, in the LCB accounts for the uncertainty in the similarity estimation, encouraging exploration of less frequently selected nodes.
Formally, we treat each previous task is (k E [n - 1]) as a bandit arm. The gradient of the current sample is denoted as 8% e V((wn, xw, y ), E {1..... m.), J. and the previous gradient (w.r.t. that tree branch) is g A Ve((w; xx, v.)). At each iteration & E f1...., m. ). the learner can only pull one arm, i.e., choose only one previous task h E [n - 1]. and obtain the corresponding empirical residual gradient Ek which is the sum of the empirical gradients' differences
The LCB is computed for each layer of the model. By calculating the LCB layer by layer, TreeLoRA captures the similarity between tasks at various levels throughout the model hierarchy as illustrated in Figure 2, allowing us to better capture hierarchical task similarities, which is especially advantageous in transformer based models. Our LCB algorithm determines the most promising task group for the current task by selecting the node k with the minimum LCB value. This approach balances the trade-off between exploring potential task similarity relationships and exploiting known, promising task groups to optimize the tree construction process.
Ek A
s(Ve(W,. vih) - Ve(We; vi)]
where S1:] is the low-rank sparsify function that updates only the most relevant parameters, i.e., the task-specific low-rank adapter. Here, d' represents reduced dimensionality such that d' < d. E represents the estimated difference between the gradient of the current sample and previous task-shared gradient. The true target g*", however, is nearest task's expected residual gradient, defined as
Bandit Algorithm on Smooth Tree. By leveraging the hierarchical structure of TreeLoRA, our bandit algo- rithm on a smooth tree (Coquelin & Munos, 2007) signifi- cantly enhances efficiency compared to conventional ban- dit algorithms that do not utilize this structure. As stated in Theorem 1, our algorithm reduces the regret from the conventional O(VN) O(Vlog N) w.r.t. the number of tasks N. Consequently, we achieve faster identification of the most similar task group, which substantially reduces computational complexity. This improvement demonstrates the advantage of TreeLoRA to efficiently search for relevant tasks and mine the hierarchical task structure.
ch- 4 (x,))D, [V((w. x, - Ve(Wi-;x, e) 1=1
where h* is the most relevant task for the current task i. The goal is to design a bandit algorithm to minimize cumulative loss between the selected arm and the optimal arm
Inspired by previous bandit algorithms (Coquelin & Munos, 2007), we introduce Lower Confidence Bound (LCBB) algo- rithm adaptively construct the tree structure. Specifically, the LCB of the k-th node in the TreeLoRA is defined as
2/
Juge
if TEEC C
3.3. Efficient Sparse Updates
LCB₁-
max min {ig- -2 (JEC
log
-80. if f ny
(2)
In this part, we will detail how to efficiently update the model by leveraging the hierarchical task structure. Specif-
4

---

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similariyy Tree
ically, we only sparsely update a portion of the model pa- rameters based on the hierarchical task structure. As shown
Initial model We Root Node
Newly learned sparse params of task t
Previous learned params
in Figure 3, we first identify the most relevant task group for the current task based on our bandit algorithm. We then perform a sparse update on the model parameters, where only the parameters that are relevant are updated.
Intermediate Node 1
Specifically, for the n-th task, we choose the most rele- vant parameters by an adaptive regularization term as the regularizer, which is formally defined as
task
Figure 3: Illustration of the sparse update mechanism, where the blue part is the sparse update part we obtained for the t-th task.
gx
reg
This regularizer term enables adaptively selective parameter updates for the current task group by identifying and mod- ifying only the most relevant parameters while preserving previously learned knowledge. The final loss combines the task loss and the regularizer term, leading to the formal model update definition as follows
within the corresponding task group, ensuring balanced tree growth and adaptive partitioning. After each task, we store the task-specific LoRA adapter and update the tree by inserting adapter into the leaf node of nearest branch by a depth-first search, thus adding new nodes and expanding the tree. If nodes exceed storage budget, we choose the closest adapters and reduce them to a single one. After the training, we select the most recently learned adapter for inference.
wx+ = -/- - .SS [VE(W' 8t) - A V reg
(3)
Remark (Discussion of Efficiency). The hierarchical tree structure and sparse update strategy significantly reduce the computational complexity and storage of the algorithm. Previous methods typically require computing gradients of all previous tasks, resulting in a complexity of (NN) per round. In contrast, our tree-based bandit algorithm avoids this linear computational complexity which scales with the number of tasks, but only explores the relevant tree branches at each round and achieves a complexity of (1). The storage complexity is also minimal, as we store only the low-rank adapter for each task, which is much smaller than the full model parameters. Furthermore, our method requires only constant GPU memory during training, thanks to the efficiency of the bandit search strategy.
where a is the learning rate, S is the low-rank sparsify function, and x denotes the regularization coefficient. Fi- nally, the updated model is obtained based on the previously learned model parameters and assembling the newly learned sparse update part, as illustrated in Figure 3.
This sparse update strategy enables us to efficiently adapt the model to new tasks while preserving knowledge of pre- viously learned ones. At the end of each task stream, we record the key parameters for the n-th task, which is the set of relevant parameters that are updated for the current task, and we add these newly-obtained sparse parameters into the T structure and update the tree structure accordingly, as demonstrated in Algorithm 1.
Practical Implementation. In practice, we draw inspi- ration from LoRA (Hu et al., 2022) to perform the sparse update, a widely used method for efficiently tuning LPMMs. Specifically, we utilize LoRA adapters to approximate the gradient of the current task and then apply sparse updates to model parameters. For the ViTs, to 5; and for LLMs, set it to 64. We clarify that tree depth is not directly determined by the number of tasks, as a single node can contain multiple tasks, allowing it to scale to a large number of tasks. However, the depth should not exceed the number of transformer layers, and is treated as a tunable hyperparameter. Further empirical analysis of tree depth provided in Section 5.2.
Remark 2 (Generality of TreeLoRA). Our TreeLoRA struc- ture is broadly applicable to various machine learning mod- els, including deep neural networks, random forests, and other architectures. While applicable to a wide range of model architectures, it provides particular benefits for transformer-bused models due to their large parameter count and inherent hierarchical structure. The layered organization of transformers naturally aligns with TreeLoRA's hierarchi- cal task structure, enabling efficient knowledge sharing and task adaptation. This makes TreeLoRA especially effective for continual learning with transformer models, though its benefits extend to other architectures as well.
set the tree depth
Inspired by the K-D tree data structure (Bentley, 1990), the task-similarity threshold 8 does not require manual tuning, as it is automatically determined when we construct the tree after each task. Specifically, at each split, the threshold is computed by taking the median of the similarity, i.c., L1- norm, between each task gradient and the mean gradient
4. Theoretical Analysis
In this section, we provide theoretical guarantees for our TreeLoRA. We first introduce the measure of pseudo-regret, then provide regret bound analysis of our algorithm.
As demonstrated in Section 3.2, we convert the task structure
5

---

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by Hierarchical -railait-Similaaity Tree
searching to a bandit problem. In this section, we provide the theoretical guarantee of regret bound for our algorithm. The measure is the pseudo-regrer (Bubeck & Gesa-Bianchi, 2012), which is the difference of cumulative loss between the selected arm and the optimal arm, formally,
Note that J is the set of leaf nodes with a suboptimality gap less than 7, which implies that [ S N. Consequently, Theorem I demonstrates that the regret's 's dependency on the number of tasks improves from the conventional order of VN to problem-dependent order of v If á = +00 and 1j = +oo, the smooth tree structure reduces to the trivial case of non-smoothness, resulting in | = N. In this scenario, we recover the conventional regret bound of the LCB algorithm without structural assumptions, i.e., O(VTN log T*.. On the other hand, in benign case where 8, 1) and JT, can be treated as constants, the regret bound simplifies to O(yTlog (NT)), (NTT)) reducing the dependency on the number of tasks from OVNN) to (log (N)). Theo- rem validates the rationality and effectiveness of our pro- posed bandit search strategy with the hierarchical tree struc- ture. The proof of Theorem I is deferred to Appendix C.I.
T min E-Enn11
Reg - E
where T is the total number of iterations. To establish theoretical guarantees for our bandit algorithm on the tree structure, note that our TreeLoRA naturally groups similar tasks into the same branch, enabling efficient mining and managing of task similarity structure. We formalize this observation of smooth tree as Assumption 1.
Assumption (Assumption An of Coquelin & Munos (2007)). For any node: i In in TreeLoRA with depth d <. D, there exists da > 0, such that for all j E I(N.).
5. Experiments
This section provides the experimental results of our ap- proach. To comprehensively evaluate the effectiveness and efficiency of our proposed TreeLoRA, we conduct empirical experiments across two modern model architec- tures: vision transformers (ViT) and large language mod- els (LLMs), across both the computer vision and the natural language processing tasks.' Our experimental evaluation aims to answer the following three research questions:
e - S.,
where 8 is the smooth coefficient, In & {node A, S $n} is the set of no-optimal nodes, in which A; 2 - gl is the suboptimality gap. L(N,) denotes the set of leaf nodes in the branch of node No, and e represents the residual gradient of task i, formally defined as g²
Bxx,))D, [V((w.; x, - V[((;;;,
Q1: How does TreeLoRA perform compared to existing continual learning methods?
This assumption characterizes the nature smoothness prop-
erty of our TreeLoRA structure, where tasks grouped within
Q2: How efficient is our proposed TreeLoRA in terms of
the same branch are inherently similar by constructing tree as Eq. (1) such that the leaves of a branch have sim- ilar values. The smoother the tree, the smaller da will be. Further, if we assume that the empirical residual gradient t has some unbiased noise that can be eliminated by the sparsify operation, then indicates that E[ = c*, which means the learner can obtain an unbiased estimation of the true residual gradient at each iteration.
the computational efficiency?
Q3: How effective is the TreeLoRA to capture the inher- ent task similarity structure?
5.1. Evaluation on Vision Transformers
We first evaluate on vision transformer models using three widely-adopted computer vision benchmarks, i.e., Split CIFAR-100, Split ImageNet-R, and Split CUB-200.
We are now ready provide the following theoretical guar- antee for our proposed algorithm.
Contenders. To validate the effectiveness of our ap- proach, we compared our TreeLoRA with several recent contenders. The following seven methods can be catego- rized into five groups: (i) The baseline method SeqLoRA, which is a standard LoRA method that naively learns a new methods LoRA adapter for each task; (ii) Rehearsal-
Theorem (Regret Bound). Under Assumption 1, as- sume A, with an exponential sequence 8a 5n. Then, with probability least 1 1/T, the cumulative regret of TreeLoRA after T rounds satisfies:
represented by GEM (Lopez-Paz & Ranzato, 2017), which utilize episodic memory to prevent catastrophic forgetting; (iii) Regularization-ben based methods exemplified by EWC from (Kirkpatrick et al., 2017), which constrain parameter updates to prevent forgetting, (iv) Prompt-based methods
(
(Tg(T)) 0 (VTUU))) log
NT [J,]
8 n,+c
NT n
log
< t is the set of leaf nodes with a suboptimality gap less than 1,, T is the number of rounds, N is the number of tasks, and e a log(2)/ log(1 1/7)) is a constant.
where Jn fleaf node i, i - |
Our code and datasets are available at tetps://githhie com/ www.ii//rree......
6

---

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by Hierarchical Gradient-Similarity Tree
Table Comparison of different continual learning methods on VITs with three benchmark datasets. For each method, we report the average accuracy (Acc (%) 1), backward transfer (B WT (%) L) and training time (TIME (s) () metrics. Results are averaged over three runs with standard deviations. The best results are highlighted in bold.
SeqLoRA
GEM
EWC
L2P
DualPrompt
HiDePrompt
HiDeLoRA
TreeLoRA
Split CIFAR-100
Acc (%)
10.35 1: 0.35
76,46 +: 0.35
84.79 +: 0.09
79.66 0.60
78.83 + 0.53
83.71 + 0.03
88.46 1: 0.04
88.54 : 0.05
Bwi (%)
97.62 0.11
4.37 0.30
5.22 0.04
4.40
4.89 1.67
5.34 0.26
4.33 1: 0.41
4.37 0.15
TIME (8)
694 8
22456 = 31
4091
1240 = 12
1173 = 14
1205 11
692 +
214
Split ImageNet-R
AcC (%) f
8.65 + 0.15
54.45 0.76
55.43 0.54
56.75 0.61
54.81
0.40
62.94 0.91
72.28 0.15
71.94 0.47
Bwr (%t 4
82.55 + 0,93
2.16 0.40
9.10 0.97
3.66 0.16
4.39 + 0.11
3.18 + 0.56
4.16 + 0.05
4.06 0.40
TIME (S)
797 # 11
17244 = 29
4727 + 21
1431 = 10
1357 = 15
1354 = 13
796 +
260 + 5
Split CUB-200
ACC (%) f
9.10 + 0.41
61.28 = 0.45
36.34 + 0.34
39.72 + 0.34
37.52 + 0.37
63.86 = 0.11
73.48 + 1.35
73.66 1: 0.22
BWT (%)) }
83.45 + 0.23
2.64 + 0,30
6.65 + 0.53
7.98 + 0.70
11.91 + 0.04
5.57 + 0.23 315 =
5.38 + 0.21
4.87 + 0.30
TIME (s)
194 + 11
7233 = 18
1702 +
333 + 5
307 =
194 +
86 +
G2V
95
LO
08-
HIDELORA
TreelaRA
Aucuracy (96
Training Time 59
Accurac (N0
86
HiDePrompt
90
80
DuaPrompt
73
Dastfens: lCcerrrst
76
50
100
1.50
200
300
Method
Efficiency (samplers)
(a) eтror curve
(b) efficiency comparison
(c) speed-accuracy comparison
Figure 4: (a) Comparison of performance the Split CIFAR- 100. (b) Comparison of the efficiency of different methods. (c) Comparison of accuracy efficiency (with mean and standard deviation over three runs). The closer the top-right corner, the better the algorithm.
including L2P (Wang et al., 2022b) and DualPrompt (Wang et al., 2022a), which learn prompts while keeping the model parameters frozen; and (v) the hierarchical decomposition methods including HiDePrompt (Wang et al., 2023a) and HiDeLoRA (Wang al., 2024a), which divide the contin- ual learning into sub-components. Further comparative re- sults, including those of SAPT (Zhao al., 2024b) and TASL (Feng et al., 2024), are provided in Appendix A.
Table 2:
Performance comparison of TreeLoRA with different training epochs Split CIFAR-100. Results show that TreeLoRA achieves good performance even with few training epochs.
Ace (%)
Bwt (%)
Time (s) |
TreeLoRA (2 epochs)
87.73 + 0.12
4.13 0,25
23 4 0.27
TreeLoRA (10 epochs)
88.23 1 0.16
4.29 1 0.11
108 1: 1.54
TreeLoRA (20 epochs)
88.54 + 0.05
4.37 = 0.15
214 + 4.12
Results Analysis of VITs Experiments. To answer ques- tion Q1, we compare TreeLoRA with state-of oof-thh-art meth- ods across three benchmark datasets on ViTs. As shown in Table 1, the naive SeqLoRA does not perform well, as it learns a new adapter for each task, leading to catastrophic forgetting. Conventional CL methods, such as GEM and FWC, are not specifically designed for LPMs, and do not achieve the same level of performance as more recent meth- ods. TreeLoRA outperforms previous state-of-the-ar meth- ods, HiDeLoRA and HiDePrompt, achieving the best accu- racy on two out of three datasets. This superior performance can be attributed to TreeLoRA's ability to exploit task simi- larity structures and leverage shared knowledge across tasks.
the training time of different methods. As shown in Table I and Figure 4(b), our Treel A achieves comparable per- formance compared to previous state-of-the-art methods, while requiring significantly less training time, resulting in 3.2x speedup. Figure 4(c) illustrates the speed-accuracy trade-off of different methods, where methods closer the top-right corner are better, as they achieve higher accuracy with less training time. Our TreeLoRA ranks among the top performers and processes nearly three times more samples per second than previous state-of-the-art methods.
We also compare TreeLoRA's performance with different training epochs. As shown Table 2, TreeLoRA achieves a good performance even with just two epochs of train- ing, reaching 87.73% accuracy on the Split CIFAR-100.
Efficiency Evaluation. To answer question Q2, we present

---

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by Hierarchical Gradient-Simllarty Tree
Table 3: Comparison of different continual learning methods various foundation models the TRACE dataset (Wang al., 2023c). For each method, we report the overall performance (OP (%) t), backward transfer (BWT (%) D. and training time (TIME (s) 4) metrics. Results are averaged over three runs with standard deviations. The best results are highlighted in bold.
FIX (ICL)
SeqLoRA
OGD
GEM
EWC
L2P
DualPrompt
HiDeLoRA
O-LoRA
TreeLoRA
mistralai / Mistral 7B- In0.
OF (%)
45.04
46,94.1122
45,44116
52.3211.2
52.45=13
49.32:0.8
51.1411.2
5181109
52.0210.8
54.77=
BWT (%)
11.41#06
12.75=0.7
6.01=0.3
5,98=088
5.34=0.6
6.13+0.5
6,25+0,3
8.13+00.
3,77=044
TIME (8)
1:25
195
7937-4167
199
975:223
9834224
1288
13024-29
510-20
meto-Ilama/ / LLLMM-2-7B--CA
OP (%)
38.9440.3
34.30++1.2
4.
10.08-t-66
42.36=1.2
36.23-0.8
37.69 1.2
41.60:0.8
42.78+0.88
43.52- 1.0
Bwr (%) 4
18.50:10.8
8.061112
6.77:11.2
5,97=088
8.25=0.8
8.03+0.8
7.12:0.4
7.1610.4
3,46=0.4
TIME (8) 4
1132+226
6416+145
7385=158
50283=195
899=22
912+23
1286+:27
1293++28
485-18
google / Gemma-2B-it
OP (5)
32.30+0.2
31.89:0088
32,85+1.4
1..48+115
28.35=116
31.14=1.2
32.42+1.0
33.25=0,9
33.73+0.88
33,41=09
Bwt (%)
10.2
12.27
18.25.10.9
16.96-12
15.77
14.25
13.66.0.5
12.36.10.4
8.50
TIME (8) 4
510:155
25
2892:992
17791=185
413=13
415#13
598:16
592:16
271=15
meta Ilaaa/ /LaMMA
3.2-1B I--ntrru
(%)
31.16104
29.73:11.6
30.12:2.0
32.19:2.0
31,96=1.6
29.38=12
30.76:1.2
33.73::1.2
32.941:0.8
36.14=007
BWT (%)
17.03+1.2
15.20+166
10.74+
11.62=122
13.57=0.8
11.34408
12.36+088
12.89+1.2
7.36-088
TIME (8) 4
482.114
1345145
1547.52
8893
270-11
281111
450114
448.144
179 13
1: Financial policy classification
Table 4: Impact of the LoRA depth (aka tree depth TreeLoRA) on the performance for O-LoRA and TreeLoRA, where we use meta Iaama LLaMA-2-78 2B 7B-Chat the foundation model.
Science reasoning
6
Complex reasoning
Root Node
Arithmetic reasoning
Depth
O-LoRA
TreeL.RRA
3,5
3: Code completion
8
21.43 - 0.6
21.49 + 0.6
0,2,3,5,7
7: German text simplification
16
22.05
0.8
22.62 + 0.9
0,
Meeting summarization
32
37.51
0.7
38.62 + 0.8
O: Chinese stance detection
64
42.78
0.8
43.52 + 1.0
Figure 5: Visualization of the tree structure learned by our ap- proach, using Mistral-7B Instruct-v0.3 as the foundation model.
Instruct. These models represent a diverse range of architec- tures and parameter scales, allowing to assess efficiency and effectiveness across different model sizes and architectures.
This is only marginally lower than the full 20-epoch train- ing (88.54) (88.54%) while requiring only 11% of the training time. The rapid convergence can be attributed to our bandit-based search algorithm, which efficiently explores the tree struc- ture, as well to ToeeLLRRAss ability to facilitate knowledge transfer through its hierarchical design, allowing new tasks to leverage shared knowledge from previously learned ones.
Results Analysis of LLMs Experiments. As shown in Table 3, ToA consistently achieves the best per- formance, outperforming other baselines and surpassing previous state-of-the-ard methods, HiieeLRRA and O-LoRA, This superior performance can be attributed to TreeLoRA's hierarchical task structure, which effectively captures and leverages shared knowledge between similar tasks, while separating conficting tasks into different branches to al- leviate catastrophic forgetting. Notably, TreeLoRA also demonstrates superior computational efficiency, reducing training time by approximately 2.4x compared to other advanced methods across all model sizes.
5.2. Evaluation on Large Language Models
In this part, we evaluate the performance of our TreeLoRA on the large language models (LLMs).
Dataset and Contenders. In this part, we employ the TRACE dataset (Wang et al., 2023c), benchmark designed especially for continual learning with LLMs. It consists of 8 distinct sub-datasets with 200, 000 samples, containing challenging tasks including multilingual capabilities, code generation, and mathematical reasoning.
Visualization of the Learned Tree Structure. To answer question Q3, we visualize the hierarchical task structure learned by our approach. Figure 5 illustrates the hierar- chical task structure learned by TreeL.RA, using Mistral- 7B- 1B-Instruct-vO.3 as the foundation model. The tree struc- ture effectively captures the semantic relationships between different tasks. For example, tasks with similar linguistic characteristics are grouped together: C-STANCE (Chinese
Model Architectures We evaluate all methods on four diffeernnt foundation LLMs: 7B-aral---Insstructt -aaMAA---BB-hhe Bemma-2B-it, and L.LMMA-322-18
8

---

TreeLoRA:
Efficient Continual Learning via Layer- LoRAs Guided by a Hierarchical Cradient-Similarity Tree
Table 5: Ablation study of our approach. Penalty indicates the penalty term in Eq. (3). SCB-Search indicates our bandit search strategy based on lower confidence bound (LCB) for selecting the hest task group as in Section 3.2.
Table Hyperparameter sensitivity analysis on Split CIFAR- -100 We report accuracy (%) with a standard deviation over three runs.
Reg coeff X
Learning rate O
ID
Penalty Search
OP (%)
0.1
88.54 + 0.05
0.001
87.97 + 0.05
( 11)
51.92 T 1.3
0.3
88.58 1 0.04
0.003
88.56 + 0.15
0.5
88.51 + 0.03
0.005
88.54 + 0.05
54.03 + 1.2
0.7 88.49 + 0.05
0.007
88.56 + 0.03
TreeLoRA
54.77 + 1.1
1.0
88.52 + 0.02
0.010
88.34 + 0.17
stance detection) and MeeringBank (meeting summariza- tion) are placed under the same parent node, as they both involve understanding and analyzing natural language dis- course. Similarly, tasks requiring mathematical and logical reasoning capabilities, such as ScienceQA (science reason- ing) and NumGLUE-ds (complex math problems), are clus- tered nearby. This answers the research question Q3 that the proposed TreeLoRA is able to capture the inherent task similarity structure effectively on the fly.
analysis, we fix the learning rate at default value a 0.005 but vary À from 0.1 to 1.0; while for learning rate analysis, we fix À 0.1 but vary from 0.001 to 0.010. Table 6 summarizes our findings across different hyperparameter settings. Note that in our proposed Treel LoRA, the hyper- parameter of task-similarity threshold 5 in Eq. (2) does not need to be tuned, as it is automatically determined when we construct the K-D tree at the end of each task.
The regularization coefficient (A) shows stability in model performance, with accuracy consistently around 88.50% across a wide range of values (0.1 to 1.0). This demon- strates Loe'L inherent robustness to this hyperparam- eter. Similarly, the learning rate (a) exhibits stable perfor- mance within the range [0.003, 0.007]. with peak accuracy of 88.56% both a 0.003 and a = 0.007. While extremely small or large learning rates can affect the perfor- mance, TreeLoRA A maintains consistent performance across broad range of settings. Based on these experiments, we recommend default values of X = 0.1 and a = 0.005.
Impact of Tree Depth. To further explore the impact of tree depth, we conducted experiments to validate how vary- ing tree depth affects the model's performance and training efficiency. The results for LLM settings are reported in Ta- ble 4, which show that TreeLoRA A is relatively robust to the choice of LoRA depth. TreeLoRA consistently outperforms O-LoRA across different LoRA depths, and depth of 64 is recommended for the best trade-off between performance and efficiency. It is important to note that LoRA depth is not determined by the number of tasks, as a single node in the tree can contain multiple tasks, allowing it to scale to a large number of tasks. Additionally, the maximum tree depth should not exceed the number of transformer layers.
Due to page limits, we defer more experiments including extended datasets, comparators, and models to Appendix A.
Ablation Study. To validate each component's s contri- bution in TORA we evaluate it on mistralai/ Mistral- 7B- 1B-Instruct-vO.3 with three variants: (i) baseline model without the gradient-similarity penalty term or LCB-based searching, (ii) model with only the penalty term but using a random search strategy, and (iii) our complete T approach with both the LCB-based search strategy. As shown in Table 5, both com- ponents contribute substantially to the overall performance. The penalty term improves performance by 2.1 % (from 51.92% to 54.03%), demonstrating its effectiveness in guid- ing sparse parameter updates. Adding the LCB- (CB-b search strategy further enhances performance, showing the value of bandit search strategy. When combined, these components work together to achieve the best performance, demonstrat- ing that both are crucial for TresLoRA's effectiveness.
6. Conclusion
In this paper, we investigate the problem of efficient con- tinual learning, especially the approaches that are tailored to LPMs. We introduce TreeLoRA, a novel approach that constructs layer-wise adapters by leveraging hierarchical gradient similarity to facilitate efficient knowledge sharing across tasks. To deal with the computational challenge in task similarity estimation, we develop bandit-based algo- rithm utilizing the lower confidence bound for efficient tree structure exploration, along with sparse gradient updates to better suit the LPMs. Theoretically. our approach enjoys tighter regret bounds compared to conventional bandit meth- ods that do not employ the tree structure. Experiments on both vision transformers and large language models validate the effectiveness and efficiency of our approach, with up to 3.2x speedup for ViTs and 2.4x speedup for LLMs, while maintaining or even improving the performance. In the fu- ture, we plan to extend our framework to consider the more challenging problem of evolving task relationships, where the task structure is time-narying
penalty term and
Hy Hyperparameter Sensitivity Analysis. We conduct hy- perparameter sensitivity analysis to evaluate the robustness of TreeLoRA. Our analysis focuses on two key hyperparam- eters: regularization coefficient X, and learning rate a. For X

---

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by Hierarchical -iailait-Similarity Tree
Acknowledgements
Information Processing Systems 33 (NeurIPS), pp. 1877- 1901, 2020.
This research was supported by National Science Founda- tion of China (U23A20382, 62176117) and Postgraduate Re- search & Practice Innovation Program of Jiangsu Province (KYCX25_ 2_55__033.... Zhen- Yu Zhang supported by JSPS KAKENHI Grant Number JP25K21282.
Bubeck, S. and Cesa-Bianchi, N. Regret analysis of stochas- tic and nonstochastic multi-armed bandit problems. Foun- dations and Trends in Machine Learning, 5(1):1-122, 2012.
Coquelin, P. and Munos, R. Bandit algorithms for tree search. In Proceedings of the 23rd Conference Uncer- tainty in Artificial Intelligence (UAI), pp. 67-74, 2007.
Impact Statement
This paper proposes an efficient continual learning method that could significantly reduce computational resources for fine-tuning, which improves efficiency and sustainability in AI development. By reducing computational demands, our method may help to decrease energy consumption and carbon emissions associated with training AI models, con- tributing to environmentally sustainable machine learning.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLoRA: Efficient finetuning of quantized L.LMs. In Pro- ceedings of the 1 1tth International Conference on Learn- ing Representations (ICLR), 2023.
Dohare, S., Hlemandez-Garriaa J. F., Lan, Q., Rahman, P., Mahmood, A. R., and Sutton, R. S. Loss of plasticity in deep continual learning. Nature, 632(8026) 2024.
References
Aggarwal, C. C., Hinneburg, A., and Keim, D. A. On the surprising behavior of distance metrics in high dimen- sional spaces. In Proceedings of the 8th International Conference on Database Theory (ICDT), Lecture Notes in Computer Science, pp. 420 434, 2001.
Dou, S., Zhou, E.,, Liu, Y., Gao, S., Shen, W., Xiong, L.,, Zhou, Y., Wang, X., Xi, Z., Fan, X., Pu, S., Zhu, J., Zheng, R., Gui, T., Zhang, Q. and Huang, X. Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1932 1945, 2024.
Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., and Tuytelaars, T. Memory aware synapses: Learning what (not) to forget. In Proceedings of the 15th European Conference on Computer Vision (ECCV), pp. 139-154, 2018.
Evron, I., Moroshko, E., Ward, R. A., Srebro, N., and Soudry, D. How catastrophic can catastrophic forget- ting be in linear regression? In Proceedings of 35th Conference on Learning Theory (COLT), pp. 4028-4079, 2022.
Bai, Y., Zhang, Y.-J., Zhao, P., Sugiyama, M., and Zhou, Z.- H. Adapting to online label shift with provable guarantees. In Advances in Neural Information Processing Systems 2022.
35 (NeurIPS), pp-
Feng, Y., Chu, X., Xu, Y., Shi, G.,, Liu, B., and Wu, X. TaSL: Continual dialog state tracking via task skill localization and consolidation. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (ACL), pp. 1266-1279, 2024.
Bailey, B. Tree-based data replay for more efficient LLM continual learning. Master's thesis, Massachusetts Insti- tute of Technology, 2024.
Bengio, Y., Lecun, Y., and Hinton, G. Deep learning for Al. Communication of ACM, 64(7))58-65, 2021.
French, R. M. Catastrophic interference in connectionist networks: Can it be predicted, can it be prevented? In Advances in Neural Information Processing Systems 6 (NIPS), pp. 1176-1 177, 1993.
Bentley, L. K-d trees for semidynamic point sets. In Pro- ceedings of the 6th Annual Symposium on Computational Geometry, pp. 187-197. 1990.
Garivier, A. and Moulines, F. On upper-confidence bound policies for switching bandit problems. In Proceedings of the 22nd International Conference on Algorithmic Learn- ing Theory (ALT), pp. 174-188, 2011.
Brown, T. B., Mann. B., Ryder, N., Subbiah, M., Kaplan, J. Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Advances Neural
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In Proceedings of the 10th International Conference on Learning Representations (ICLR), 2022.
10

---

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by Hierarchical Pradient-Similarity Tree
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, I.., and Chen, W. AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning. In Proceed- ings of the 12th International Conference on Learning Representations (ICI.R, 2024.
Proceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (ACL.), pp. 3505-3523, 2022.
Ouyang. L., Wu, Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J.. Kelton, F., Miller. L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35 (NeurIPS), pp. 27730- 27744, 2022.
Huszár, F. Note on the quadratic penalties in elastic weight consolidation. Proceedings of the National Academy of Sciences, 115(11):E2496-E229)) 2018.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Baabska-Barwinsk... A., Hassabis, D., Clopath, C., Kumaran, D., and Hadsell, R. Overcoming catastrophic forgetting in neural networks. Proceedings of the Na- tional Academy of Sciences, 114(13)33521-333 -3526, 2017.
Pham, Q., Liu, C., and Hoi, S. Dualnet: Continual learning, fast and slow. In Advances in Neural Information Pro- cessing Systems 34 (NeurlPS), pp. 16131-16144, 2021.
Lange, M. D., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A.. Slabaugh, G. G., and Tuytelaars, T. A continual learning survey: Defying forgetting in classifi- cation tasks, IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3366-3385, 2022.
Qian, Y.-Y,, Bai, Y., Zhang. Z.-Y., Zhao, P., and Zhou, Z.-H. Handling new class in online label shift. In Proceedings of the 23rd IEEE International Conference on Data Mining (ICDM), pp. 1283-1288, 2023.
Qian, Y.-.,, Zhao, P., Zhang, Y.-J., Sugiyama, M., and Zhou, Z.-H. Efficient non non-stationary online learning by wavelets with applications to online distribution shift adaptation. In Proceedings of the 41st International Conference on Machine Learning (ICML), pp. 41488344411,,, 2024.
Liang, Y. and Li, W. Inflora: Interference-free low-rank adaptation for continual learning. In Proceedings of the 34th IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pp. 23638 22647, 2024.
Lin, J., Li, X., Wang. P.,, Wang, W., and Song, Y. GaLore: Memory-efficient LLM training by gradient low-rank pro- jection. In Proceedings of the 40th Conference Empir- ical Methods in Natural Language Processing (EMNLP), pp. 14297, 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Ma- chine Learning (ICML), pp. 8748-8763 87688 2021.
Lopez-Paz, D. and Ranzato, M. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems 30 (NIPS), pp. 6467-6476 2017.
Rebuffi, S., Kolesnikov, A., Sperl, G., and Lampert, C. H. iCaRL: Incremental classifier and representation learning. In Proceedings of the 30th IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2001-2010, 2017.
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for sci- ence question answering. In Advances in Neural Informa- tion Processing Systems 35 (NeurIPS), pp. 11332-11343, 2022.
Schwarz, J., Czarnecki, W., Luketina, J., Grabska- Barwinska, A., Teh, Y. W., Pascanu, R., and Hadsell, R. Progress & compress: scalable framework for continual learning, In Proceedings of the 35th International Con- ference on Machine Learning (ICMI.), pp. 4535-4544 2018.
Mallya, A. and Lazebnik, S. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the 31st IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7765-7773, 2018.
Scialom, T., Chakrabarty, T., and Muresan, S. Fine-tuned language models are continual learners. In Proceedings of the 39th Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6107-6122, 2022.
McCloskey, M. and Cohen, N. J. Catastrophic interfer- ence in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, vol- ume 24, pp. 109-165.11 1989.
Serrà, J., Suris, D., Miron, M., and Karatzoglou, A. Over- coming catastrophic forgetting with hard attention to the task. In Proceedings of the 35th International Conference on Machine Learning (ICML.), pp. 4548-4557, 2018.
Mishra, S., Mitra, A., Varshney, N., Sachdeva, B. S., Clark, P., Baral, C., and Kalyan, A. Numglue: A suite of funda- mental yet challenging mathematical reasoning tasks. In

---

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Cradient-Similarity Tree
Shin, H., Lee, J. K., Kim, J., and Kim, J. Continual learning with deep generative replay. In Advances in Neural In- formation Processing Systems 30 (NIPS), pp. 2990-2999, 2017.
TRACE: A comprehensive benchmark for continual learn- ing in large language models. CoRR, abs/23 10.06762, 2023c.
Wang. Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C.-Y., Ren, X., Su, G., Perot, V., Dy, J. G., and Pfister, T. Dualprompt: Complementary prompting for rehearsal-free continual learning. In Proceedings of the 17th European Conference on Computer Vision (ECCV), pp. 631-648, 2022a.
Smith, J. S., Karlinsky, L., Gutta, V., Eescante-Bonilla P.,
Kim, D., Arbelle, A., Panda, R., Feris, R., and Kira, Z. Coda-prompt Continual decomposed attention-based prompting for rehearsal-free continual learning. In Pro- ceedings of the 33rd IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 11909- 11919, 2023.
Wang. Z., Zhang, Z., Lee, C.-Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J. G., and Pfister, T. Learning to prompt for continual learning. In Proceedings of the 35th IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 139-149, 2022b.
Smith, J. S., Valkov, L., Halbe, S., Gutta, V., Feris, R.,
Kira, Z., and Karlinsky, L. Adaptive memory replay for continual learning. CVPR Workshop, pp. 3605-3615, 2024.
Yang. S., Ali, M. A., Wang. C.-L., Hu, L., and Wang, D. MoRAL: Moe augmented lora for Ilms' lifelong learning. ArXiv preprint, arXiv:2402.1 1260, 2024.
Sutton, R. S., Bowling, M. H., and Pilarski, P.M. The alberta plan for ai research. ArXiv preprint, arXiv:2208. 11173, 2022.
Yüksel, S. E., Wilson, J. N., and Gader, P. D. Twenty years of mixture of experts. IEEE Transactions on Neural Networks and Learning Systems, 2378):1177-1193, 2012.
Thrun, S. and Pratt, L. Learning Learn. Springer New York, 1997.
Zenke, F., Poole, B., and Ganguli, S. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 3987-3995, 2017.
Vela, D., Sharp, A., Zhang, R., Nguyen, T., Hoang, A., and Pianykh. O. S. Temporal quality degradation in ai models. Scientific Reports, 12(1):1 1664, 2022.
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The Caltech-UCSD birds-200-2011 dataset. Technical Report, 2011.
Zhang, L.,, Lu, S., and Zhou, Z.-H. Adaptive online learning in dynamic environments. In Advances in Neural Informa- tion Processing Systems 31 (NeurIPS), pp. 1330 1340-134, 2018
Wang, L., Xie, J., Zhang, X.. Huang, M., Su, H., and Zhu, J. Hierarchical decomposition of prompt-based contin- ual learning: Rethinking obscured sub-optimality. In Advances in Neural Information Processing Systems 36 (NeurIPS), 2023a.
Zhang, Y.-J., Zhang, Z.- Zhao, P.,. and Sugiyama, M. Adapting to continuous covariate shift via online density ratio estimation. In Advances in Neural Information Pro- cessing Systems 36 (NeurIPS), pp. 29074-291 13, 2023.
Wang, L. Xie, J., Zhang, X., Su, H., and Zhu, J. Hide-pet: Continual learning via hicrarchical decom- position of parameter-effiieet tuning. ArXiv preprint, arXiv:2407.00 052299, 2024a.
Zhao, P., Zhang, Y.J., Zhang, L., and Zhou, Z.-H. Adap- dynamic
tivity and non-stationarity: regret for online convex optimization. Journal of Machine Learning Research, 25(98): 1-52, 2024a.
Wang, L., Zhang, X., Su, H., and Zhu, J. A comprehen- sive survey of continual learning: Theory, method and application. JEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8):5362 53833, 2024b.
Zhao, W., Wang, S., Hu, Y., Zhao, Y., Qin, B., Zhang, X., Yang, Q. Xu, D., and Che, W. SAPT: A shared atten- tion framework for parameter-efficient continual learning of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 11641-11661. 2024b.
Wang, X., Chen, T., Ge, Q. Xia, H., Bao, R., Zheng, R., Zhang, Q., Gui, T., and Huang, X. Orthogonal subspace learning for language model continual learning. Pro- ceedings of the 40th Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 10658- 10671, 2023b.
Zhou, D.-W., Sun, H.-L.,, Ning, J., Ye, H.-J., and Zhan, D.-C. Continual learning with pre-trained models: A survey. In Proceedings of the 33rd International Joint Conference on Artificial Intelligence (IJCAI), pp. 8363-8371, 2024.
Wang, X., Zhang, Y., Chen, T., Gao, S., Jin, S., Yang, X., Xi, Z., Zheng, R., Zou, Y., Gui, T., Zhang, Q., and Huang, X.
Zhou, Z.HH. Open-enviromment machine learning. National Science Review, (8) 123, 2022.
12

---

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by Hierarchical Gradient-Similarie Tree
A. Additional Experiments
This section provides additional experimental details for the main paper.
A.1. Details of Datasets
For the experiments on ViTs, we evaluate on the following three widely-used benchmark datasets:
Split CIFAR-I00 (Zenke et al., 2017): A widely-used continual learning benchmark that divides CIFAR-100 into 10 sequential tasks, each containing 10 disjoint classes. CIFAR-I00 consists of 100 classes, with each class containing 600 numbers of 32 x 32 color images (500 for training and 100 for testing). Each task comprises 5, 000 training samples and 1, 000 test samples.
Split ImageNet-R R (Wang al., 2022a): A challenging benchmark that segments ImageNet- -'s 200 classes into 10 distinct tasks, with each task containing 20 classes. The original ImageNet-R dataset comprises 30, 000 images (24, 000 for training and 6, 000 for testing), consisting of artistic renditions (including cartoons, graffiti, paintings, origami, and other artistic styles) and challenging variants of ImageNet samples.
Split CUB-200: A fine-gras fne-grained benchmark based on -UB-200-2011 (Wah et al., 2011), which contains 200 different bird species divided into 10 sequential tasks (20 classes per task). The original CUB-200-2011 dataset comprises 11, 788 images (5, 994 for training, 5, 794 for testing) across 200 bird species.
For the experiments on LLMs, we evaluate on the following three widely-used benchmark datasets:
TRACE (Wang et al., 2023c): A benchmark designed especially for the continual learning with L.MSs. consists of distinct datasets spanning challenging tasks, including domain-specific tasks, multilingual capabilities, code generation, and mathematical reasoning, with each task contains 5, 000 instances. Specifically, the eight tasks are: C-STANCE, FOMC, MeetingBank, 15150, ScienceQA, NumGI.UE-cm, NumGLUE-dss and, 20Minuten. All datasets are standardized into unified format, allowing for effortless automatic evaluation of LLMs. Therefore, it contains a total of 200, 000 samples, with 40, 000 training examples and 16, 000 testing examples.
Note that TRACE contains wide range of different tasks, including Jomain-specifit tasks, multilingual capabilities, code generation, and mathematical reasoning. The performance measure for each task is different: For C-STANCE and FOMC tasks, we use accuracy as the evaluation metric assess the model's classification performance. MeetingBank task performance is evaluated using the ROUGE-L score, which measures the longest common subsequence between the generated and reference summaries. For code-related task Py150, we employ similarity score to evaluate the quality of generated code. The ScienceQA task is evaluated using accuracy to measure the correctness of scientific question answering. Both NumGLUE-cm and NumGLUE-ds tasks, which focus on mathematical reasoning, use accuracy as their evaluation metries. Lastly, for the multilingual task 20Minuten, we utilize the SART score to assess the quality of text simplification.
A.2. Details of Contenders
In our experiments, we compare TreeLoRA with the following baselines and state-of-the-art continual learning methods:
SeqLoRA baseline method that sequentially learns new LoRA adapters for the data stream when new task arrives.
GEM (Lopez-Paz & Ranzato, 2017): A rehearsal-based CL method that stores a small set of samples from each task to record their gradients, and performs gradient projection when the new task's ' gradient conflicts with previous task ones.
EWC (Kirkpatrick et al., 2017): A regularization-bass continual learning method that uses the Fisher Information Matrix to identify and protect parameters that are important for previous tasks.
L2P (Wang et al., 2022b): A prompt-based based continual learning method that learns a prompt pool shared by all tasks, where each prompt is associated with a key vector, and instructs the last multi-self attention (MSA) layer in a Prompt Tuning fashion.
DualPrompt (Wang et al., 2022a): Extends L2P by introducing both general and task-specific expert prompts to better handle knowledge transfer, and instruct different MSA layers a Prefix- -eefi-gTuning fashion.
HiDePrompt (Wang et al., 2023a): A hierarchical decomposition prompt learning method that decomposes continual learning objective into hierarchical components: within-task prediction, task-identity inference, and task adaptiee
13

---

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarty Tree
prediction. It uses task-specific prompts replace all task-sharing prompts, and then replaces the task-specific keys with task-identity inference output layer.
HiDeLoRA (Wang et al., 2024a): A generalized extension of HiDePrromp that replaces prompt-based parameter-effiient tuning (PET) with unified framework for CL with pre-trained models and PET. hierarchical decomposition approach for LoRA adapters to enable efficient parameter sharing.
FIX (ICL) (Brown et al., 2020): An in-context learning method that incorporates 6-shot task demonstrations into the prompt, serving as a baseline to demonstrate the model's 's performance without continual learning.
O-LoRA (Wang et al., 2023b): A method that learns tasks in different orthogonal low-rank subspaces to minimize interference between tasks.
SAPT (Zhao et al., 2024b): A continual learning method that aligns sarameter-efficiem tuning with task selection through shared attention mechanism.
TASL. (Feng et al., 2024): A continual learning method based on skill localization that identifies a small subset of parameters responsible for task-specific performance.
InfLoRA (Liang& & Li, 2024): A parameter-efficient fine-tuning method that injects a small number of parameters to reparameterize the pre-trained weights within a subspace to diminish interference between new and old tasks.
A.3. Implementation Details
Implementation Details of ViTs. For the ViTs experiments, we follow similar implementation details to prior works (Wang et al., 2023a; Smith et al., 2023). We adopt pre-trained ViT-B/16 model iBOT-2IK, self-supervised model pre-trained on ImageNet-21K K that achieves state-of-the the-art classification performance as the backbone network. The model is optimized using Adam with B1 = 0.9 and B2 = 0.999, utilizing a batch size of 192 and a constant learning rate at 0.005. We train 20 epochs on Split CIFAR-100 and 50 epochs on other benchmarks to ensure convergence. Input images are preprocessed to 224 x 224 dimensions and normalized the range [0, 1J. We conduct the baselines following their original implementations. Specifically, L2P (Wang et al., 2022b) is implemented with prompt pool size set as 30, and prompt length as 5. DualPrompt (Wang et al., 2022a) employs length-5 task-shared prompts in layers 1-2 and length-20 task-specifi prompts in layers 3-5. GEM (Lopez-Paz & Ranzato, 2017) is implemented with a memory size of 100 samples and memory strength of 0.05. EWC (Kirkpatrick et al., 2017) is implemented with a regularization strength of 50. We set hyperparameters for HiDePrompt following Wang et al. (2023a). HiDeLoRA (Wang et al., 2024a) is implemented with a prompt length as 20, and LoRA low-dimension bottleneck as 10. Note that our approach can be seamlessly integrated into the existing CL frameworks. In our implementation, we incorporate HiDeLoRA's task decomposition methodology into TreeLoRA. The reported accuracy is calculated as the average test accuracy across all tasks after training on the final task. All experiments are conducted using eight Nvidia V100 GPUs with two Intel(R) Xeon(R) Gold 6248R CPUs.
Implementation Details of ViTs and LLMs. For the LLMs experiments, we follow the same learning rate and training epochs following the original setting of the TRACE dataset (Wang et al., 2023c), i.e., a = 1 - 5 and epochs=1, 1, 5, 1, 5, 5, 5 for the methods without LoRA adapters (OGD, GEM, and EWC); and a = le - 4 with epochs=5, 3, 7, 5, 3, 5, 5, 7 for the methods with LoRA adapters (SeqLoRA, L2P, DualPrompt, HiDeLoRA, and O-LoRA). Our TreeLoRA employs a smaller number of training epochs 2, 1, 3, 2, 1, 2, 2, owing to its ability capture the task similarity structure, while other methods using the same reduced number of epochs lead to performance drop. For example, O-LoRA suffers a performance drop from 33733% to 30.60% on the TRACE dataset with Gemma-2b-it when using TreeLoRA's epoch configuration.
To further control the storage complexity, TreeLoRA employs merging strategy in which leaf nodes with highly similar gradient directions are combined when total storage exceeds a predefined threshold, integrating their LoRA adapters into a single one. The merging threshold is dynamically adjusted based on available system resources.
The batch size is set to 4 for all methods. We set the À as 0.5 for the LLMs experiments. All experiments in this section were performed using four Nvidia A800 (80 GB) GPUs with two Intel(R) Xeon(R) Gold 6430 CPUs. The learning rate is set to 1 x 10-3 for all methods. We use DeepSpeed with ZeroStage 2 and mixed-precision training (BF16). The max prompt length is set to 1024 tokens.
We evaluate all the methods with four foundation large language models. These models represent a diverse range of architectures and parameter scales, allowing us to assess TreeLoRA's efficiency and effectiveness across different model sizes and architectures. We describe the key characteristics of each model below:
14

---

TreeLoRA: Efficient Continual Learning via Layer-Wiis LoRAs Guided by Hierarchical Sradient-Similarit Tree
CUM Ta BLUAA
1400
725
HiDeLoRA
TreioRA
700
Accurazy (5)
1000
67.5
70
HiDePromot
600
200
550
40
10
523
DullPropp
20
60
80
100
120
140
Tasks
Method
Efficency (smmpe//))
(a) emror
(b) efficiency comparison
speed-accuracy comparison
Figure 6: (a) Comparison of performance on the Split ImageNet-R. (h) Comparison of the efficiency of different methods. (c) Comparison of accuracy and efficiency (with and standard deviation over three runs). The closer the top-right corner, the hetter the algorithm.
96-
CDV
Duafrorpt HEDLLRA
HiDeloRA
TrooLaRA
80-
HiDaPrompl
250
Accurssy (86)
70
Time
200
Accursry (94)
60
60-
55
50-
40
20-
L2P
DualPrompt
30
L2P
Dushherst Holhoaast HorLark
35
Tasks
Method
Effidency (sample/s)
(a) error curve
(b) efficiency comparison
(c) speed accuracy comparison
Figure (a) Comparison performance on the Split CUB-200. (b) Comparison the efficiency of different methods. (c) Comparison of accuracy and efficiency (with mean and standard deviation over three runs). The closer to the top-right corner, the better the algorithm.
Biatrra-----nstucct--- An advanced instruction-tune model utilizing sliding window attention and grouped-query attention mechanisms. The architecture comprises 32 transformer layers with 0096-dimeniional hidden states, supporting 32k token context window through rotary positional embeddings.
LLaMA-2-7B-chat A decoder-only transformer architecture with 7 billion parameters, trained on 2 trillion tokens. The model incorporates supervised fine-tuning (SFT) and constitutional AI alignment techniques, featuring 32 transformer layers with 4096-dimensional hidden states and 32 attention heads.
Gemma-2B-it A lightweight, state-of-the-art open models from Google, containing 2 billion parameters. The model employs multi-query attention mechanisms and features 18 transformer layers. It incorporates responsible Al constraints and demonstrates strong performance on reasoning and instruction -oitrection-oolowinng tasks.
A compact instruction-tuned language model comprising I billion parameters, optimized for efficient deployment. The architecture consists -048-dimensional hidden states and attention heads, trained on a curated subset of foslloiti data.
Disk Storage and GPU Memory Usage. In all experiments, the storage and GPU memory requirements of TreeLoRA remain consistent and minimal, only requiring approximately 8 GB of GPU memory with MB of additional disk storage for ViTs, 40 GB of GPU memory with 10 MB of additional disk storage for 7B LLMs, and 22 GB of GPU memory with 5 MB of additional disk storage for 2B LLMs during the entire training process. These requirements are almost the same as those of the baseline SeqL.RRA method, ensuring that our TreeL.oRA does not impose any additional storage or GPU memory burden compared compared to the vanilla LoRA.
A.4. Additional Results on VITs
Figure 6 and Figure provide additional experimental results on ViTs. On Split ImageNet-R, as shown in Figure 6(a), ToeeLoRA achieves competitive error rates, performing slightly worse than HiDeLLRRA but outperforming other baseline
15

---

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by Hierarchical Gradient-Similarity Tree
Table 7: Comparison of different continual learning methods on various foundation models on the our collected Math-I.LM dataset. For each method, we report the overall performance (Op (%) 1). and the backward transfer (TW (%) L) metrics, as well as training time. Results are averaged over three runs with standard deviations. The best results are highlighted in bold.
FIX (CCL)
SeqLoRA
OGD
GEM
EWC
L2P
DualPrompt
HiDeLoRA
O-LoRA
TreeLoRA
Math LLM Dataset
OP (
36.92.0.5
40.24.1.00
42.09114
40.75.14
37.25
31,67110
37.72.1.0
42.83.10.7
42.00
45,59 11.67.0.66
BWT (
9.12.10.8
8.0610.5
16.25
17.50.0.6
2.54.004
9,54.10.7
11.92.0.5
2.32
methods. The efficiency comparison in Figure 6(b) reveals that TreeLoRA maintains strong computational efficiency while achieving this competitive performance. Similar trends are observed on the Split CUB-200 dataset, where TreeLoRA demonstrates consistently higher accuracy than all other methods. Furthermore, our TreeLoRA faster than the SeqLoRA, primarily due to Ll-norm sparsification Eq. (3), which reduces the number parameters that need to be updated, and the hierarchical tree structure enables more efficient statistical extraction of task relationships.
We also present results on the specially collected Math-LLM dataset, which includes three tasks related to math and reasoning: ScienceQA (Lu et al., 2022), NumGLUE-cm, and NumGLUE-ds (Mishra et al., 2022). As shown in Table 7, using meta-llama/ / I.L.MA---B--cle as the foundation model, ToeeLoRA achieves the best performance among all methods, and shows higher improvement than the performances on the original TRACE dataset. This is attributed to TreeLoRA's ability to capture the inherent task similarity structure, which is particularly advantageous for task streams with similar characteristics.
A.5. Comparison of FLOPs and Parameter Complexity
To further compare the FLOPs of different methods, we include a comparison using LLaMA-2-7B -hhatt as an example for training a single token on the 10th task, as shown Table 8. Here, m and denote the dimensions of the transformer's parameter matrix, is the LoRA rank, and N is the number of tasks. While TreeLoRA introduces a tree structure to capture task relationships, the additional overhead for tree maintenance is minimal compared to the overall training cost.
Table 8: Comparison of FLOPs and parameter complexities of different methods, where we instantiate the FLOPs using meta-llama 1--MAA---78-CA Chat the foundation model, where we need to train single token on the 10th task.
Method
FLOPs
Parameter Complexity
OGD
2.8 x 1040
((m))
LoRA
4.2 x 106
O((m + n)r)
O-LoRA
4.2 107
O((n n)r
TreeLoRA
4.2 x 106
O((m + (o)r+NN)
A.6. Validation on More Datasets and Task Orders
To validate the stability and robustness of TreeLoRA over long task sequences, we conduct additional experiments on sequence of 15 diverse tasks, including C-STANCE, FOMC, MeetingBank, Py150, ScienceQA, NumGLUTEcmm NumGLUE- ds, 20Minuten, dbpedia, amazon, yahoo, agnews, yelp, BooIQA, and QQP, using meta-llama Llama-3. 3.2--13 1------IItctr as the foundation model. The results are summarized in Table 9. The results demonstrate that TreeLoRA maintains stable performance even with a long sequence of 15 diverse tasks, achieving higher average accuracy and lower forgetting compared to other contenders. Moreover, Treel LoRA shows even better efficiency than short-term task sequences, indicating its scalability for long-term continual learning scenarios.
Table 9: Performance comparison long sequence of 15 tasks for LLMs, where meta-llama Llama-3 3.2-IB 2-1B-Innttruct as the foundation model.
FIX
SeqLoRA
OGD
GEM
EWC
L2P
DualPrompt
HiDeLoRA
O-LoRA
TreeLoRA
(%) BWT (%) 1
41.32
40.71
32.52
35.48
31.46
41.05
41.29
42.38
44.02
45.68
15.72
21.32
18.33
22.22
14.92
15.58
11.23
10.99
6.41
TIME (8)
721
1921
2235
13058
403
411
683
679
251
16

---

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by Hierarchical Gradient-Similarity Tree
We conduct additional experiments validate TreeLoRA and other contenders using the same datasets (i.e., MNLI, CB, WiC, COPA, QQP, BooIQA, RTE, IMDB, Yelp, Amazon, SST-2, DBpediaa Agnews, MultiRC, and Yahoo) and the same task order settings in O-LoRA (Wang et al., 2023b). We use meta-llama The results are shown in Table 10. The results indicate that TreeLoRA achieves stable performances across different task orders, and also improves efficiency (about 1.5x speedup compared to O-LoRA).
as the foundation model.
Table 10: Comparison of overall performance (%) BWT (%) and training time (s) for different task orders the datasets and the same orders as O-LoRA (Wang et al., 2023b).
Task Order
FIX
OGD
GRM
LeqLoRA
HiDeLoRA
O-LoRA
TreeLoRA
Task Order
FIX
HideLoRRA
O-LoRA
TreeLoRRA
Order1
48.25
54.16/ 10.82
14/10/M0 10.25
58.71 8.63
56.32/2.75
59 59.50/2.55 2,51
59.73/ 2.22
Order4
52,13/-
59,42/4.33
/4.67
58.45
Order2
48.75/-
46.82/21.50
46.70
20.93
45 15.76
5341/5.58
52.53/5.65
53.78/ 5:74
Orde55
52,13/-
$4,49/7.52
37,45/4422
58.12/331
48.75/-
43.79/27.31
51.12/ 16.24
49.03/
61.25/3.12
63.82/2.03
62.76/
52.13/-
57.26/6.98
58/02/4.75
59.004.11
Time
684
712
43
56
58
45
Time
124
121
83
A.7. Comparison with More Contenders
We conduct extended experiments to directly compare the performance of TreeLoRA and InfLoRA (Liang & Li, 2024)
on the CIFAR-100 100 dataset using VIT models. The results are shown in Table 11. The results demonstrate that TreeLoRA achieves similar or better accuracy compared to InfLoRA, with lower training times.
Table 11: Comparison of TreeLoRA and InfLoRA on CIFAR-100 with VIT models.
InfLoRA
TreeLoRA
Acc (%) f
85.44
88.54
BWT (%6 |
4.82
4.37
TIME (8) 4
695
214
Besides, further add a comparison with two recent advanced continual learning methods, SAPT (Zhao al., 2024b) and TASL (Feng et al., 2024), as well as other baselines, using meta / LLLMM----B-CCA comparison, generative replay is not employed in SAPT as it may introduce additional information beyond the original data stream. The results are summarized in Table 12. As it illustrated in the table, TreeLoRA achieves the best overall performance in terms of accuracy, forgetting, and efficiency compared to baselines, including SAPT and TASL.
the foundation model. For fair
Table 12: Performance comparison with more contenders including SAPT and TASL LLMs.
FIX
SegLoRA
OGD
GEM
EWC
L2P
DualPrompt
HiDeLoRA
O-LoRA
SAPT
TASL
TreeLoRA
OP (5% f
38.94
34,30
42.09
40.08
42.36
36.23
37.69
41.60
42.78
42.93
43.19
43.52
BWT (%6) L
18.50
8.06
6.77
5.97
8.25
8.03
7.12
7.16
5,49
4.58
3.46
TIME (s)
1132
6416
7385
50283
899
912
1286
1293
1205
1185
485
A.8. Experiments on Extreme-Scale LLMs (13B Model)
To further investigate the computational trade-offs for extreme-scale LLMs, we conduct experiments using 13B model (meta/ / Lllma----3--aaee 2--3ht-ha--hf" The results are reported in Table 13. The results show that TreeLoRA achieves better accuracy while using lower training time compared other contenders. Additionally, the memory (storage) overhead of TreeLLoRA is minimal, requiring only 15 MB. These findings demonstrate the effectiveness of our tree-based adaptation strategy in both performance and efficiency aspects, and its scalability to large-scale models.
Table 13: Performance comparison on LLM (meta 2-lee///LL---3---eelee 2-13--chat-tff
FIX
SeqLoRA
OGD
GEM
EWC
HiDeLoRA
O-LoRA
TreeL.oRA
OP (%) f
40.15
39.16
42.32
43.77
41.23
43.27
44.32
47.13
BWT (%)
15.58
9.72
8.42
10.12
11.27
5.19
3,42
TIME
1525
8712
9931
67819
1835
1839
662
17

---

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by Hierarchical Prradient-Smillari Tree
1
Root Node
Root Node
Root Node
Root Node
0, 2,
0
2
(a)
(b)
(C)
(d)
1,
1,4
1, 4,4, 6
1,
Root
Root Node
Root
Root
Node
Node
3,
3,
0,
2,
0, 3,5
0, 3,
0,
0,2
0,
0, 2,
(c)
(1)
(h)
Figure 8: The evolution of the tree structure in our TreeLoRA under a dynamic task flow in our LLM experiment, using mistralai Mistral-7B Instruct-vO.3 3 as the foundation model. Figures (a) to (h) correspond to task I to task 8 (i.e., f = to 8), respectively. These figures demonstrate how the tree structure evolves over time and effectively captures the relationships among different tasks.
A.9. Evolving of the Tree Structure
We also provide the evolution of the tree structure in our TreeLoRA our LLM experiment, using mistralai/ Mistral-7B- Instruct-v0.3 as foundation model. As shown Figure 8, (a) to (h) correspond to task to task (i.e., to = 8), respectively. These demonstrate how the tree structure evolves over time and effectively captures the task relationships.
B. Related Work
This section introduces related works, including continual learning, low-rank approximation, and mixture of experts.
Continual Learning. The goal of continual learning is to continuously learn from new data while retaining knowledge of previously learned tasks (Dohare et al., 2024; Wang et al., 2024b; Zhou et al., 2024). major challenge catastrophic forgetting (French, 1993; McCloskey & Cohen, 1989; Thrun & Pratt, 1997), where the model forgets previously learned tasks when fine-tuned on new data. There are mainly three main approaches deal with this issue:
Regularization seaudarization-based methods (Schwarz al., 2018) place constraints the model parameters ensure tasks learned without forgetting the old ones. For example, elastic weight consolidation (EWC) (Kirkpatrick et al., 2017) leverages the Fisher Information Matrix to identify and protect parameters critical for previous tasks. Huszár (2018) 8) further improves EWC by maintaining penalties only for the immediately preceding task, reducing over-emphasis on early tasks and lowering storage overhead. Memory-aware synapses (Aljundi et al., 2018) improve upon these methods by estimating parameter importance through output function sensitivity, addressing the limitation of near-zero gradients at loss function local minima.
Rehearsal-based methods (Scialom et al., 2022; Bailey, 2024) store data from previous tasks in buffer and use it train the model alongside the current task. Notable approaches include iCaRL (Rebuffi et al., 2017), which maintains exemplar sets of feature vectors for each class, performs neiiestt classification, and selects representative samples through herding while preserving past knowledge via distillation loss. Gradient episodic memory (GEM) (Lopez-Paz & Ranzato, 2017) maintains gradient information from previous tasks and employs gradient projections to resolve conflicts between current and past task gradients. Smith et al. (2024) propose an adaptive memory replay method that treats sampling of past data as a multi-armed bandit problem. While these methods have shown superior performance across various benchmarks, they typically come with high computational and storage cost, due the need to maintain and update the model based on large-sized buffer of past data, which may scale linearly with the number of tasks.
Architecture-basee based methods (Mallya & Lazebnik, 2018; Serrà et al., 2018) dynamically adjust the model architecture to accommodate new tasks. Inspired by the complementary learning systems theory, DualNet (Pham et al., 2021) introduces a learning framework that incorporates fast and slow learning components, but this approach still relies on a rehearsal buffer for optimal performance. To address these limitations, I.2P (Wang et al., 2022b) proposes shared prompt pool strategy
18

---

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarty Tree
that eliminates the need for explicit task identities during inference while reducing parameter and memory overhead. In this approach, each prompt is linked to key vector and modulates the final multi-self attention (MSA) layer using Prompt Tuning techniques. Further advancing this line of research, DualPrompt (Wang et al., 2022a) enhances the L2P framework by incorporating both general-pun purpose and task-specific expert prompts, aimed at improving knowledge transfer across tasks. Our approach further refines the task architecture by leveraging the task similarity structure to build a hierarchical tree structure and bandit-based searching algorithm, which efficiently adapts to new tasks and achieves comparable performance.
Supervised Fine- Fine-Tuning Tuning. The recent advancement in large language models has demonstrated the effectiveness of supervised tine-tuning approaches. InstructGPT (Ouyang et al., 2022) shows that fine-tuning LLMs with human feedback enhances their cntruuctionfollowiinn capabilities and response quality. To address the computational burden of fine-tuning LLMs, LoRA (Hu et al., 2022) proposes a low-rank adaptation method that substantially reduces trainable parameters while preserving model performance. While these methods have become fundamental techniques for adapting pre-trained models to downstream tasks, addressing distribution shifts between training and testing data remains challenging, particularly in continual learning scenarios where new tasks emerge continuously.
Low-Rank Adaptation. The low-rank approximation has emerged as an effective technique for model adaptation and compression. The pioneering work LoRA (Hu et al., 2022) introduces low-rank adaptation matrices to efficiently fine-tune large language models while preserving their capabilities. Following this direction, QLoRA (Dettmers et al., 2023) combines quantization with low-rank adaptation to further reduce memory usage. More recent works like AdA (Hu et al., 2024) propose adaptive budget allocation strategies for parameter-efficient fine-tuning, while GaLore (Lin et al., 2023) focuses on memory-efficient training through gradient low-rank projection. Our TreeLoRA also takes inspiration from these works to be better suited to the LPMs by considering the low-rank approximation.
Mixture of Experts. Mixture of Experts (MoE) (Yüksel et al., 2012) is a powerful architecture that allows for efficient parallel computation by distributing the workload across multiple experts. Recent works have demonstrated the effectiveness of MoE in continual learning scenarios. LoRAMoE (Dou et al., 2024) introduces framework that combines low-rank adapters with router network, functioning as a plugin version of MoE. By freezing the backbone model and directing specific adapters to leverage shared knowledge for downstream tasks, it effectively mitigates catastrophic forgetting. MoRAL (Yang et al., 2024) combines MoE with LoRA for continual learning of LLMs, demonstrating robust knowledge retention and superior performance through question-answer pairs rather than factual triplets. Although these approaches show promising results in continual learning, they still incur significant computational overhead due to their architectures. Instead, we propose efficient CL. approach that leverages task similarity through hierarchical tree structure and bandit-based searching algorithm, enabling fast adaptation and achieving comparable performance.
Non-stationary Online Learning. Non-stationary online learning aims to design algorithms capable of adapting to distribution shifts in non-stationary environments. A theoretically grounded approach is dynamic regret minimization (Zhang et al., 2018), which evaluates performance relative to a sequence of time-varying benchmark models. A principled framework for optimizing dynamic regret in online convex optimization is the online ensemble framework (Zhao et al., 2024a), which maintains a set of diverse base learners and combines them through a meta-algorithm. Recently, various algorithms have emerged from this framework (Bai et al., 2022; Qian et al., 2023; Zhang et al., 2023). For instance, ATLAS (Bai et al., 2022) addresses label shift by employing base learners with varying learning rates, while the wavelet-based method (Qian et al., 2024) detects changes and restarts using an ensemble of wavelet bases. Non-stationary online learning primarily focuses on adapting to the current task and pays less attention to the issue of catastrophic forgetting, which is more concerned with continual learning. Extending theoretical insights from non-stationary online learning could provide new perspectives for advancing continual learning theory (Evron al., 2022).
C. Proofs
This section presents the proofs that were omitted in the main paper.
C.1. Proof of Theorem 1
Proof. We aim to analyze the regret bound for the TreeLoRA algorithm. Recall that the regret is defined as:
T
T
Reg (T)
e
E
Σ
ck
min E[N]
E E|n Ag;
19

---

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by Hierarchical Srrddint-Simillarty Tree
where "5 is the number of times arm j is pulled up time T, and A ę gk is the suboptimality gap for the arm Decomposing the Regret. We start by decomposing the regret into two terms based whether A, is less than greater than threshold 4;:
Reg(T)
Σ
B[n
D
Σ
B[ugT] -
jEMMAA
JEMAA بA
term (a)
term (b)
Bounding term (a). Since A, S A, we bound term (a) as:
term (a) A
Decomposing and Bounding term (b). For term (b), we further divide the summation into two parts:
term (b) = Σ
Singal-A, +
E[ngr] Dj.
jCT2n/3
3 T2
torm
(c)
term (d)
where 12n/3 [node, E [M] A, S 2n/ is the set of nodes with suboptimality gap less than 21//3.
For term (c), we consider the same event as in the proof of Theorem 3 in (Coquelin & Munos, 2007). Let h be the smallest integer such that 5n S n/3. We have h s log(38/1)
(1/ + 1. Let be node of depth h. If E 2 then, thanks to
Assumption I, we have for all E L(N)), My ps 3 2 p 77. thus E Jq. Therefore.
4NT A?
6 J/J
4NT
term (c) - Σ
Elngal A S log
log
A
A?
jEJ,
JEJn
where the second inequality follows from Lemma
For term (d), we use the same argument as Theorem 4 in (Coquelin & Munos, 2007) and obtain that with probability at least 1 - 1/T, we have
54(35)c 1
(4NT 172
term (d) <
log
where ف log())/logL//>). The term O t1/ depends weakly (linearly) on the depth D (through log() Thus. if fixed and we increase the depth D of the tree, term (d) the dominant term.
Combining Bounds.
Adding the bounds for term (a), term (c), and term (d), the overall regret satisfies:
6| A
4NT
54(30)
4NT
NT
Reg T) < T A
log
log
0
T
A +
log
A?
172
A?
By selecting At as D = //TlTT we have:
Reg(T) S. (1/
(1 log
TTJ,
NT
5
log
NT
|
1/2
which finished the proof.
C.2. Useful Lemmas
We present the following useful lemmas.
Lemma 1 (Theorem 2 and Theorem 3 in (Coquelin & Munos, 2007)). With probability at least 1 - 1/T. all j E for where D Ollog N) is the depth of the tree. Beside, with Assumption I, with probability at least 1 1/T, the number of times node i is visited is at most
any leaf node, the number of times leaf is chosen is at most n4 < X7 log
6 4.2
A
6 Glog (ANT/(AABBB,
n, -A)
20


Reviews:
[{"review_id": "Hzzpp481wJ", "reviewer": "Reviewer_GZLm", "summary": "This paper proposes TreeLoRA, a continuous learning method that builds hierarchical adapters based on gradient similarity, which aims to solve the computational efficiency problem in continuous learning of large pre-trained models (LPMs). By organizing tasks into a K-D tree structure and introducing sparse gradient updates, this method achieves better accuracy than baselines (such as HiDeLoRA) on ViT and LLM, while reducing training time by about 2.4 times. However, the core dynamic update mechanism (such as node addition and reduction rules) is not fully explained, and the complexity comparison with mainstream parameter efficient fine-tuning methods (such as LoRA/O-LoRA) is insufficient, which may affect the credibility of the method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your helpful comments! Below, we address your major technical questions and will revise the paper to improve clarity and resolve any potential misunderstandings.\n\n---\n\n**Q1.** Elaborate more on the construction of the tree structure, including update and expansion, threshold design, relationship between tree depth and number/similarity of tasks.\n\n**A1.** Thanks for the question. The construction of the tree structure is explained in detail below:\n\n- **Update of Tree Structure.** After each task, we store the task-specific LoRA adapter (as in Section 3.3) and update the tree by inserting adapter into the leaf node of nearest branch by a depth-first search (DFS), thus adding new nodes and expanding the tree. If nodes exceed the storage budget, we choose the closest adapters and reduce them to a single one (as in Appendix A.3).\n\n- **Threshold Design.** Inspired by the K-D tree structure [Bentley, 1990], the threshold $\\delta$ does not require manual tuning. Specifically, at each split, the threshold is computed by taking the median of the similarity (L1-norm) between each task gradient and the mean gradient within the corresponding task group, ensuring balanced tree growth and adaptive partitioning.\n\n- **Relationship Between Tree Depth and Number/Similarity of Tasks.** We clarify that tree depth is not directly determined by the number of tasks, as a single node can contain multiple tasks, allowing it to scale to a large number of tasks. However, the depth should not exceed the number of transformer layers (as illustrated in Fig. 1), and it is treated as a tunable hyperparameter. Further empirical analysis of tree depth is provided in **A2** in response to Reviewer bzQv.\n\nWe will add these details in the revision for more clarity.\n\n---\n\n**Q2.** No comparison with standard LoRA or O-LoRA.\n\n**A2.** We believe this is a misunderstanding here — we have compared our approach with the stand LoRA (aka SeqLoRA) and O-LoRA in our experiments, as presented in Table 3 of the submission PDF. The results demonstrate our improvements over basic methods in both performance and efficiency. We also add FLOPs comparison with LoRA/O-LoRA, please refer to **A5**.\n\n---\n\n**Q3.** On Theoretical Claims \"does not consider the adjustment cost of the dynamic tree structure... assumes a static task relationship\"\n\n**A3.** Thank you for the insightful comments. Our regret analysis focuses on a simplified scenario to provide foundational justifications for the proposed algorithm. The elements you mentioned can certainly be incorporated into future work by some modern online learning techniques. For instance, incorporating the *switching cost* would allow us to account for the cost of adjusting the tree structure, and adopting *dynamic regret* could help capture time-varying task relationships. Nonetheless, we believe these extensions are non-trivial to achieve. For instance, the minimax rate for MAB is $\\Theta(\\sqrt{T})$, whereas introducing a switching cost increases it to $\\Theta(T^{2/3})$ [Dekel et al., STOC'14]. Our theoretical results serve as a first step toward tackling more complex scenarios, and we will include these points in the discussion of future work.\n\n---\n\n**Q4.** Theoretical relationship.\n\n**A4.** In Theorem 1, the regret bound is dependent on both the number of tasks $N$ and the task complexity $J_n$ (which is controlled by the similarity between tasks). Consequently, as the number of tasks increases or as tasks become less similar, the performance of our method is expected to deteriorate, requiring more rounds to maintain and search the tree structure.\n\n---\n\n**Q5.** Other concerns about experiments, including:\n\n- [5-1] parameter/FLOPs comparison, additional overhead of tree structure, \"is the parameter growth rate of TreeLoRA strictly lower (than O-LoRA)\"\n- [5-2] evolution of tree structure under dynamic task flow\n- [5-3] impact of different tree depths on performance.\n\n**A5.** Thanks for your suggestions.\n\n- [A5-1] We include a comparison using *LLaMA-2-7B-Chat* as an example for training a single token on the 10-th task, as in table below:\n\n||FLOPs|Parameter Complexity\n-|-|:-:\nOGD|28×10⁹|$\\mathcal{O}(mn)$\nLoRA|4.2×10⁶|$\\mathcal{O}((m+n)r)$\nO-LoRA|4.2×10⁷|$\\mathcal{O}((m+n)rN)$\nTreeLoRA|4.2×10⁶|$\\mathcal{O}((m+n)r+Nr)$\n\nHere, $m$ and $n$ denote the dimensions of the transformer's parameter matrix, $r$ is LoRA rank, and $N$ is the number of tasks.\n\n- [A5-2] We also add an analysis of the evolution of the tree structure under dynamic task flow, which provides a clearer illustration of how TreeLoRA captures task structures over time: https://anonymous.4open.science/r/TreeLoRA/scripts/rebuttal.jpg\n\n- [A5-3] Additionally, we conduct further experiments to evaluate the impact of tree depth on performance, please refer to **A2** for Reviewer bzQv.\n\n---\n\nThank you again for the helpful review. We will revise the paper accordingly and include the related references, such as Rusu et al. (2016) and Wang et al. (2023).", "claims_and_evidence": "The tree structure can dynamically capture task similarity and reduce computational complexity.\nIt does not explain how nodes are dynamically increased or decreased (such as the conditions that trigger splitting). The rationality of the structure is only indirectly proved through visualization, and there is a lack of mathematical description of the dynamic process.", "methods_and_evaluation_criteria": "- The construction of tree structure depends on gradient similarity, but it is not clear:\nHow to update the tree level when new tasks are inserted (incremental or global reconstruction)\nThreshold design for node splitting/merging\nAdaptive relationship between tree depth and number/similarity of tasks\n\n- Only CL methods such as HiDeLoRA are compared, and no theoretical/experimental comparison is performed with standard LoRA (independent adapter for each task) or O-LoRA (orthogonal constraint adapter), which cannot prove its advantages over basic methods.", "theoretical_claims": "The regret bound analysis does not consider the adjustment cost of the dynamic tree structure. The theoretical model assumes a static task relationship, which is inconsistent with the actual dynamic scenario.", "experimental_designs_or_analyses": "The evolution of tree structure under dynamic task flow (such as the tree changes when 10 new categories are added in Split CIFAR-100)\nAverage adapter parameters per task vs. standard LoRA.\nThe impact of different tree depth/width on performance.", "relation_to_broader_scientific_literature": "\\", "essential_references_not_discussed": "Rusu et al. (2016) Progressive Neural Networks\nWang et al. (2023) DyLoRA: Dynamic Low-Rank Adaptation", "other_strengths_and_weaknesses": "\\", "other_comments_or_suggestions": "Add parameter/FLOPs comparison experiment with standard LoRA/O-LoRA\nDiscuss the additional overhead of tree structure maintenance (such as gradient similarity calculation complexity)", "questions_for_authors": "Is node splitting based on a fixed threshold? How to avoid over-complication of the tree structure?\nWhen the new task has low similarity with the existing node gradient, how to expand the tree structure? (Add new branches or increase the layer depth?)\nIs the theoretical relationship between the number of tree levels L and the number of tasks T O(log T)? How to set L in actual experiments?\nCompared with the O(d) parameter increment of O-LoRA (d is the adapter dimension), is the parameter growth rate of TreeLoRA strictly lower?\nIs the sparsity rate of sparse gradient updates related to the tree structure? How to balance sparsity and knowledge retention?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZgsPKm8cyx", "reviewer": "Reviewer_bzQv", "summary": "This paper proposes TreeLoRA, a novel and efficient approach for continual learning in large pre-trained models. TreeLoRA constructs a hierarchical tree structure of LoRAs based on gradient similarity, enabling efficient task adaptation and knowledge sharing. The method employs bandit algorithms to explore task-similarity structure and leverages sparse gradient updates to optimize parameters, demonstrating superior efficiency and performance compared to previous state-of-the-art continual learning methods.\n\n## update after rebuttal\n\nThank the authors for the detailed follow-up and additional experimental results. I appreciate the authors' efforts to extend the evaluation to the full 15-task benchmark and to clarify the role of LoRA depth in TreeLoRA's performance.\n\nHowever, I still find some aspects unclear. Specifically, while the explanation about LoRA depth partially clarifies the observed performance drop in LLMs, it remains ambiguous how TreeLoRA itself solves the issue when the model size becomes larger. Furthermore, although the authors state that TreeLoRA's depth is independent of the number of tasks, the rationale for choosing specific depths for different models is still not clearly explained. For instance, in the LLaMA-2-7b-chat experiments, both TreeLoRA and O-LoRA perform poorly at depth 8 but significantly improve at depth 64, raising the question of whether TreeLoRA consistently outperforms O-LoRA or whether its benefits only appear under particular depths. As there is no empirical evidence indicating a linear relationship between performance and LoRA depth, the observed improvements remain difficult to interpret.\n\nThe newly added results are appreciated and add value to the submission. Moreover, the idea behind TreeLoRA represents a novel and promising research direction. However, I believe further clarification is needed regarding TreeLoRA’s consistent advantage on LoRA depths. Therefore, I maintain my original score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's feedback. In the following, we address each of your technical inquiries.\n\n**Q1.** \"impact of task order on the performance ...this paper does not use the same dataset used by O-LoRA.\"\n\n**A1.** Thank you for your comment. First, we would like to clarify that the TRACE dataset used in our paper consists of 8 tasks, which is ***larger*** than the 4 tasks used in the O-LoRA paper. Moreover, the TRACE dataset includes a ***diverse set*** of tasks, such as text generation and code generation, whereas the datasets used in the O-LoRA paper primarily focus on classification tasks.\n\nTo further address your concern, we conduct additional experiments to validate TreeLoRA and other contenders using the _**same datasets**_ (i.e., dbpedia, amazon, yahoo, and agnews) and  _**same task orders**_ as in O-LoRA. We use *Llama-3.2-1B-Instruct* as the foundation model, and convert classification tasks to text generation tasks. Results of overall performance (%)/BWT (%) indicate that TreeLoRA achieves superior performances across different orders, and also improves efficiency (about 1.5x speedup compared to O-LoRA):\n\nTask Order|FIX|OGD|GEM|SeqLoRA|HideLoRA|O-LoRA|TreeLoRA\n:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:\nOrder1|48.75/0.0|54.16/10.82|54.10/10.25|55.71/8.63|56.32/2.75|59.50/2.51|**59.73/2.22**\nOrder2|48.75/0.0|46.82/21.50|46.70/20.93|45.26/7.70|53.41/5.58|52.53/5.65|**53.78/5.74**\nOrder3|48.75/0.0|43.79/27.31|51.12/16.24|49.03/19.05|61.25/3.12|**63.82/2.03**|62.76/2.23\nTime|-|684|712|43|56|58|**45**|\n\nFurther, we include an experiment involving long task sequences, which is similar to the experimental setup used in O-LoRA's paper. For more details, please refer to the **A1** for Reviewer ygph. We will add these results in the revised version. Thanks again for your valuable comments.\n\n---\n\n**Q2.** There is limited analysis of how the performance of TreeLoRA is affected by key hyperparameters. How does TreeLoRA determine the depth of the K-D tree chosen for ViTs (5) and LLMs (64)? Is there any strategy for guidance? Is there any range for the depth? Does it connect with the number of tasks?\n\n**A2.** Thanks for your comment. We detail the hyperparameter analysis of our method below:\n\n- **Hyperparameter Sensitivity.** In our paper, we provided an analysis of the sensitivity of TreeLoRA's performance to key hyperparameters, such as the regularization coefficient $\\lambda$ and the learning rate $\\alpha$. These results are detailed in Appendix A.6.\n\n- **Impact of the Tree Depth.** To further explore the impact of tree depth, we conducted additional experiments to validate how varying tree depth affects the model's performance, with overall performance (%) and training time shown in the table below:\n\nTree Depth|CIFAR-100 (ViT)|Time (s)\n:-:|:-:|:-:\n1|86.52|171.31\n2|88.22|182.42\n5|**88.54**|**212.66**\n7|88.39|233.17\n\nTree Depth|TRACE (LLM)|Time (s)\n:-:|:-:|:-:\n8|21.49|455\n16|22.62|468\n32|38.62|476\n64|**43.52**|**485**\n\nThese results show that TreeLoRA is relatively robust to the choice of tree depth. For ViT models, a tree depth of 5 provides a good balance between performance and efficiency, while for LLMs, a depth of 64 is recommended. These settings bring slightly better trade-offs between performance and efficiency. \n\nWe clarify that tree depth is not determined by the number of tasks, as a single node in the tree can contain multiple tasks, allowing the structure to scale to a large number of tasks. Additionally, the maximum tree depth should not exceed the number of transformer layers (as illustrated in Figure 1)\n\n- **Impact of the Gradient Similarity Threshold.** Regarding the threshold $\\delta$, as mentioned in Section 3.3, we clarify that it is automatically determined and does not need manual adjustment. Specifically, inspired by the K-D tree data structure [Bentley, 1990], at each split, the threshold is computed by taking the median of the similarity (L1-norm) between each task gradient and the mean gradient within the corresponding task group. This approach ensures balanced tree growth and adaptive partitioning of the gradient space, without the need for manual threshold adjustments.\n\n---\n\n**Q3.** One previous work, InfLoRA (CVPR 2024), also utilizes CIFAR-100 and the same model ViT to conduct the experiments. How does TreeLoRA's performance compare to this work?\n\n**A3.** Thank you for your comment. We add experiments to directly compare the performance with InfLoRA on the CIFAR-100 dataset using ViT models:\n\n||InfLoRA|TreeLoRA\n:-:|:-:|:-:\nAcc (%)|85.44|**88.54**\nBWT (%)|4.82|**4.37**\nTime (s)|695|**214**\n\nThe results demonstrate that TreeLoRA achieves similar accuracy compared to InfLoRA, with lower training times. We will add these results and corresponding discussions in the revised version.\n\n---\n\nWe hope these clarifications address your concerns. We sincerely wish that you can re-evaluate our paper and consider updating the score for our paper. Thank you for your time and feedback!", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods make sense for the problem, but for the benchmark datasets, this paper does not use the same dataset used by O-LoRA, since it is a direct and very related baseline for this paper.", "theoretical_claims": "I checked the correctness of the theorem 1.", "experimental_designs_or_analyses": "I checked the experimental designs. Please see the questions.", "relation_to_broader_scientific_literature": "As mentioned in the paper, the proposed method may help to decrease energy consumption and carbon emissions associated with training AI models, contributing to environmentally sustainable machine learning.", "essential_references_not_discussed": "This paper uses the image datasets CIFAR-10 and ViT but it does not compare with the similar work [InfLoRA] published in CVPR 2024, which also uses the same datasets and the same model.\n\n[1] Liang, Yan-Shuo, and Wu-Jun Li. \"Inflora: Interference-free low-rank adaptation for continual learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "other_strengths_and_weaknesses": "Strengths:\n1. TreeLoRA is a novel hierarchical structure that efficiently groups tasks based on gradient similarity, efficient task adaptation, and knowledge sharing.\n2. This paper provides a theoretical analysis to support the proposed method’s efficiency, demonstrating regret bounds compared to conventional methods.\n3. The proposed method achieves speed improvements while having similar or even better performance compared to current existing methods.\n\nWeaknesses: \n1. There is limited analysis of how the performance of TreeLoRA is affected by key hyperparameters such as the tree depth and gradient similarity threshold. The authors mentioned that the tree depth is set to 5 for ViT and 64 for LLMs, but there is insufficient discussion on how the values are chosen and how sensitive the method is to the choices.\n2. The authors compare TreeLoRA with O-LoRA and other baselines, but there is insufficient analysis of how different task orderings affect performance. O-LoRA explored the impact of different task orders, it would strengthen the evaluation if using the same task orders to compare. \n3. The paper does not use the same datasets as O-LoRA, which makes the current comparison less rigorous and kind of unfair.", "other_comments_or_suggestions": "Include a discussion on the impact of task order on the performance of TreeLoRA.", "questions_for_authors": "1. How does TreeLoRA handle situations where task order changes over time? Since the experiments compare with the recent baseline, O-LoRA, which explored the impact of different task orders, how does TreeLoRA perform on the different task orders in O-LoRA.\n2. How does TreeLoRA determine the depth of the K-D tree chosen for ViTs (5) and LLMs (64)? Is there any strategy for guidance? Is there any range for the depth? Does it connect with the number of tasks?\n3. In the experiment, the TreeLoRA uses image datasets CIFAR-10 as one benchmark, and I found one previous work, InfLoRA (CVPR 2024), which also utilizes this dataset and the same model ViT to conduct the experiments. How does TreeLoRA’s performance compare to this work?\n\n[1] Liang, Yan-Shuo, and Wu-Jun Li. \"Inflora: Interference-free low-rank adaptation for continual learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y3CQeAwgxe", "reviewer": "Reviewer_ygph", "summary": "TreeLoRA presents a continual learning method that enhances the efficiency of updating large pre-trained models. By integrating layer-wise LoRA with a hierarchical gradient similarity tree, it improves knowledge retention while reducing computational costs. TreeLoRA mitigates catastrophic forgetting while maintaining efficiency in VITs and LLMs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive and helpful comments. We provide our response to each question as below.\n\n---\n\n**Q1.** \"The paper presents evidence for TreeLoRA's effectiveness but lacks discussion on the stability and robustness of its tree structure over extended task sequences. A deeper analysis of its evolution in long training sequences, especially in non-stationary environments, would add valuable insight.\"\n\n**A1.** Thank you for your comment. Following your suggestion, we conduct additional experiments to validate the stability and robustness of TreeLoRA over ***long task sequences***, which consist of a total of 15 tasks, including C-STANCE, FOMC, MeetingBank, Py150, ScienceQA, NumGLUE-cm, NumGLUE-ds, 20Minuten, dbpedia, amazon, yahoo, agnews, yelp, BoolQA, and QQP, using _meta-llama / Llama-3.2-1B-Instruct_ as the foundation model. The results are summarized in the following table:\n\n| Metric   |  FIX  | SeqLoRA |  OGD  |  GEM  |  EWC  |  L2P  | DualPrompt | HideLoRA | O-LoRA | TreeLoRA  |\n| -------- | :---: | :-----: | :---: | :---: | :---: | :---: | :--------: | :------: | :----: | :-------: |\n| Op (%)   | 41.32 |  40.71  | 32.52 | 35.48 | 31.46 | 41.05 |   41.29    |  42.38   | 44.02  | **45.68** |\n| BWT (%)  |  0.0  |  15.72  | 21.32 | 18.33 | 22.22 | 14.92 |   15.58    |  11.23   | 10.99  | **6.41**  |\n| Time (s) |   -   |   721   | 1921  | 2235  | 13058 |  403  |    411     |   683    |  679   |    **251**    |\n\nThe results demonstrate that TreeLoRA maintains stable performance even with a long sequence of 15 diverse tasks, achieving higher average accuracy and lower forgetting compared to other contenders. Moreover, TreeLoRA shows even better efficiency than short-term task sequences, indicating its scalability for long-term continual learning scenarios.\n\nAdditionally, we include a figure that illustrates the evolution of the tree structure under dynamic task flow in our LLM experiment. This figure helps to better visualize how TreeLoRA adapts to the evolving task structure over time. The figure is available at the following link: [https://anonymous.4open.science/r/TreeLoRA/scripts/rebuttal.jpg](https://anonymous.4open.science/r/TreeLoRA/scripts/rebuttal.jpg)\n\n---\n\n**Q2.** \"the memory and computational trade-offs for extreme-scale LLMs remain unexamined\"\n\n**A2.** Thanks for your comment. In our paper, we validate our method using small ViT models, as well as large-size language models (1B, 2B, and 7B), which contain commonly used models in the research community [Wang et al., 2023a, Wang et al., 2023b, Dou et al., 2024]. To further investigate the computational trade-offs for extreme-scale LLMs, **_we add an experiment using a 13B model_** (meta / Llama-2-13b-chat-hf), as shown in the following table:\n\n| Metric   |  FIX  | SeqLoRA |  OGD  |  GEM  |  EWC  | HideLoRA | O-LoRA | TreeLoRA  |\n| -------- | :---: | :-----: | :---: | :---: | :---: | :------: | :----: | :-------: |\n| Op (%)   | 40.15 |  39.16  | 42.32 | 43.77 | 41.23 |  43.27   | 44.32  | **47.13** |\n| BWT (%)  |  0.0  |  15.58  | 9.72  | 8.42  | 10.12 |  11.27   |  5.19  | **3.42**  |\n| Time (s) |   -   |  1525   | 8712  | 9931  | 67819 |   1835   |  1839  |  **662**  |\n\nThe results show that TreeLoRA achieves better accuracy while using lower training time compared to other contenders. Additionally, the memory (storage) overhead of TreeLoRA is minimal, requiring only 15 MB. These findings demonstrate the effectiveness of our tree-based adaptation strategy in both performance and efficiency aspects, and its scalability to large-scale models.\n\n---\n\n**Q3.** \"While TreeLoRA is compared to other LoRA-based continual learning methods, benchmarking against non-LoRA-based strategies is limited, such as replay-based approaches. A broader comparison would better contextualize TreeLoRA's advantages within the continual learning landscape.\"\n\n**A3.** Thank you for your comment. It appears there may be a misunderstanding due to our insufficient emphasis. Specifically, we have compared TreeLoRA with several non-LoRA-based continual learning strategies, including replay-based (rehearsal-based) methods, GEM, regularization-based methods such as EWC, and the baseline OGD (which represents full update, a non-LoRA method). As shown in Table 3 and Table 4 of the submission PDF, TreeLoRA outperforms these methods both in terms of performance and efficiency. Additionally, TreeLoRA offers particular advantages for transformer-based models due to the large parameter size and inherent hierarchical structure in these models.\n\n---\n\nWe hope these clarifications address your concerns. We will improve the paper writing to better emphasize these points. Thanks again for your constructive comments.", "claims_and_evidence": "Yes, please refer to the Strengths and Weaknesses section for more details.", "methods_and_evaluation_criteria": "Yes, please refer to the Strengths and Weaknesses section for more details.", "theoretical_claims": "Yes, please refer to the Strengths and Weaknesses section for more details.", "experimental_designs_or_analyses": "Yes, please refer to the Strengths and Weaknesses section for more details.", "relation_to_broader_scientific_literature": "Please refer to the Strengths and Weaknesses section for more details.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "***Strengths***\n1. This paper presents a hierarchical gradient similarity tree for task organization, optimizing parameter updates with improved efficiency. A novel bandit-based similarity estimation reduces complexity, enhancing scalability. Sparse gradient updates further adapt TreeLoRA for ViTs and LLMs.\n\n2. A rigorous theoretical analysis derives tighter regret bounds than standard bandit approaches. The hierarchical structure minimizes computational overhead while preserving task knowledge, ensuring efficiency gains.\n\n3. Experiments on vision and language tasks demonstrate that TreeLoRA surpasses state-of-the-art methods, accelerates ViTs and LLMs, and mitigates catastrophic forgetting with reduced backward transfer.\n\n***Weaknesses***\n1. The paper presents evidence for TreeLoRA’s effectiveness but lacks discussion on the stability and robustness of its tree structure over extended task sequences. A deeper analysis of its evolution in long training sequences, especially in non-stationary environments, would add valuable insight.\n\n2. Additionally, the memory and computational trade-offs for extreme-scale LLMs remain unexamined.\n\n3. While TreeLoRA is compared to other LoRA-based continual learning methods, benchmarking against non-LoRA-based strategies is limited, such as replay-based approaches. A broader comparison would better contextualize TreeLoRA’s advantages within the continual learning landscape.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review."}}, {"review_id": "VsnThipQIZ", "reviewer": "Reviewer_qcDX", "summary": "This paper proposes a novel continuous learning approach, TreeLoRA (K-D Tree of Low-Rank Adapters), which exploits hierarchical gradient similarity to build layer-wise adapters for efficient CL.To achieve even greater efficiency, the authors develop a confidence lower bound based bandit techniques to efficiently explore the task structure. In addition, the authors provide theoretical analyses to demonstrate the validity of the proposed approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's constructive feedback. In the following, we respond to each question.\n\n---\n\n**Q1.** \"Regarding the use of LCB to calculate the similarity between tasks especially in transformer-based models, is it computed only at the last layer of the model, or is each layer calculated individually? I noticed that the figures in the paper seem to indicate that only the last layer is calculated, so why not compute each layer separately, given that the features learned at each layer of the model are different?\"\n\n**A1.** Thank you for your question. We would like to clarify that the LCB calculation in TreeLoRA is indeed performed ***layer by layer*** across the entire model. As described in Equation (2) in our paper, the LCB is computed as follows:\n\n$$\n\\mathrm{LCB}\\_k=  \\begin{cases}  \\widehat{\\mu}\\_k-2 \\sqrt{\\frac{\\log t}{n\\_k}}, & \\text { if } k \\in \\mathcal{L} \\\\\\\\ \n\\max \\\\left\\\\{ \\min\\_{j \\in \\mathcal{C}} \\\\left\\\\{\\widehat{\\mu}\\_j-2 \\sqrt{\\frac{\\log t}{n\\_j}}-\\delta \\\\right\\\\} \\\\right\\\\}, & \\text { if } k \\notin \\mathcal{L}  \\end{cases}\n$$\n\nwhere $\\hat{\\mu}\\_k = \\frac{1}{|\\operatorname{Select}\\_k|} \\sum\\_{\\tau \\in \\{\\operatorname{Select}\\_k\\}} \\hat{\\xi}\\_{\\tau}^k$ is the estimated task similarity between the current task and the $k$-th task group (i.e., the nodes in the branch of the selected leaf node at round $t$), $\\mathcal{L}$ is the set of all leaf nodes, $\\delta$ is the automatically determined threshold, and $\\mathcal{C}$ is the child nodes of the $k$-th node. Therefore, the LCB is computed for each layer of the model. By calculating the LCB layer by layer, TreeLoRA captures the similarity between tasks at various levels throughout the model hierarchy as illustrated in Figure 1, allowing us to better capture hierarchical task similarities, which is especially advantageous in transformer-based models. We will add these details in the revised version of the paper to provide further clarity. Thank you again for your valuable question!\n\n---\n\n**Q2.** \"The authors mention that the setting of the threshold $\\delta$ does not need to be done manually and is done dynamically, why and how is this done?\"\n\n**A2.** Thanks for your comment. Inspired by the K-D tree data structure [Bentley, 1990], the threshold $\\delta$ does not require manual tuning. Specifically, during the construction of the K-D tree after each task, the gradient space is partitioned based on the distribution of task gradients. At each split, the threshold is computed by taking the median of the similarity (L1-norm) between each task gradient and the mean gradient within the corresponding task group. This approach ensures balanced tree growth and adaptive partitioning of the gradient space, without the need for manual threshold adjustments.\n\n---\n\n**Q3.** \"definition of $f_i(w_j)$ does not conform to common notation. It is recommended to swap $i$ and $j$ to write it as $f_j(\\mathcal{T}_i)$, or consider using $\\theta_j$ to represent the model parameters\"\n\n**A3.** Thanks for your comment. We will revise our paper accordingly and use clearer notations, which would enhance the clarity of the paper. Thanks again for your feedback.\n\n---\n\n**Q4.** \"it seems that there is a lack of comparison with some recent advanced continual learning methods [1, 2]\"\n\n**A4.** Thank you for pointing out these two references. Following your suggestions, we add a comparison with these two recent advanced continual learning methods, SAPT [Zhao et al., ACL 2024] and TASL [Feng et al., ACL 2024]. For a fair comparison, we do not employ the generative replay in SAPT. The results, using _meta / LLaMA-2-7B-Chat_ as the foundation model, are presented in the table below:\n\n|Metric|FIX|SeqLoRA|OGD|GEM|EWC|L2P|DualPrompt|HiDeLoRA|O-LoRA|SAPT|TASL|TreeLoRA|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Op (%)|38.94|34.30|42.09|40.08|42.36|36.23|37.69|41.60|42.78|42.93|43.19|**43.52**|\n|BWT (%)|-|18.50|8.06|6.77|5.97|8.25|8.03|7.12|7.16|5.49|4.58|**3.46**|\n|Time (s)|-|1132|6416|7385|50283|899|912|1286|1293|1205|1185|**485**|\n\nWe also add comparison with other recent methods, InfLoRA [Liang and Li, CVPR 2024], please refer to **A3** for Reviewer bzQv for more details. We will add these results to the revised version, and will also add discussions with SAPT and TASL methods in the related work section.\n\n---\n\nWe hope these clarifications address your concerns. Thanks again for your valuable comments.", "claims_and_evidence": "The claims made by the authors are well-supported by clear theoretical proofs and experimental results, which effectively validate their assertions.", "methods_and_evaluation_criteria": "The proposed methods are effective in addressing the problem outlined in the paper.", "theoretical_claims": "I have checked the proof of theory provided by the author and there are no obvious problems.", "experimental_designs_or_analyses": "See Weaknesses (2).", "relation_to_broader_scientific_literature": "The research is a study of model base capabilities, with potential implications for the broader scientific literature.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "Weaknesses:\n\n1. The definition of $f_{i}(w_{j})$ does not conform to common notation. It is recommended to swap $i$ and $j$ to write it as $f_{j}(\\mathcal{T}\\_i)$, or consider using $\\theta_{j}$ to represent the model parameters. This would enhance the clarity of the paper.\n\n2. In the methods compared in this paper, it seems that there is a lack of comparison with some recent advanced continual learning methods [1, 2], which might have better performance than the approach proposed in this paper.\n\n[1] Zhao, W., et al. Sapt: A shared attention framework for parameter-efficient continual learning of large language models. In ACL, 2024.\n\n[2] Feng, Y., et al. Tasl: Task skill localization and consolidation for language model continual learning. In ACL, 2024.", "other_comments_or_suggestions": "Typos: I'm not sure if this is a typo, but I noticed that the variable $j$ in equation (1) seems to be unnecessary, as it not be defined.", "questions_for_authors": "Question:\n\n1. Regarding the use of LCB to calculate the similarity between tasks especially in transformer-based models, is it computed only at the last layer of the model, or is each layer calculated individually? I noticed that the figures in the paper seem to indicate that only the last layer is calculated, so why not compute each layer separately, given that the features learned at each layer of the model are different?\n\n2. The authors mention that the setting of the threshold $\\delta$ does not need to be done manually and is done dynamically, why and how is this done?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a hierarchical, gradient-similarity-based adapter structure (TreeLoRA) for continual learning with large pre-trained models. The main contributions include:
- A novel hierarchical adapter structure that organizes task-specific LoRAs via a K-D tree guided by layer-wise gradient similarity, enabling scalable and efficient parameter reuse.  
- A bandit-based task similarity estimation mechanism and sparse gradient updates that improve learning efficiency in large vision and language models.  
- Extensive empirical validation across ViT and LLM benchmarks, showing improved performance and computational efficiency over strong baselines (e.g., O-LoRA, InfLoRA), alongside theoretical regret bounds supporting the algorithm’s design.

During the response period, the authors thoroughly addressed reviewer concerns, including comparisons with InfLoRA, SAPT, and TASL, ablations on tree depth, task order sensitivity, and LoRA depth. They also added experiments on long task sequences and 13B-scale LLMs. The authors also provided experiments in response to Reviewer bzQv's only remaining concern.

Given the paper’s clear motivation, and strong empirical results, I recommend acceptance. For the final version, the authors are encouraged to (1) further clarify how LoRA depth interacts with model architecture in scaling to LLMs, and (2) more explicitly highlight comparisons with non-LoRA-based baselines and plug-in strategies (e.g., replay, generative modules).
Decision: Accept (poster)