Orthogonal Subspace Learning for Language Model Continual Learning
Authors: Xiao Wang
Tianze Chen
Qiming Ge
Han Xia
Rong Bao
Rui Zheng
Qi Zhang
Tao Gui
Xuanjing Huang
Venue: EMNLP 2023 Findings (2023)
Keywords: continual learning
orthogonal subspace
paramemter efficient tuning
TL;DR: In this paper, we propose Orthogonal LoRA, a novel approach for language model continual learning that effectively addresses the issue of catastrophic forgetting.

Abstract: Benefiting from massive corpora and advanced hardware, large language models (LLMs) exhibit remarkable capabilities in language understanding and generation. However, their performance degrades in scenarios where multiple tasks are encountered sequentially, also known as catastrophic forgetting. In this paper, we propose orthogonal low-rank adaptation (O-LoRA), a simple and efficient approach for continual learning in language models, effectively mitigating catastrophic forgetting while learning new tasks. Specifically, O-LoRA learns tasks in different (low-rank) vector subspaces that are kept orthogonal to each other in order to minimize interference. Our method induces only marginal additional parameter costs and requires no user data storage for replay. Experimental results on continual learning benchmarks show that our method outperforms state-of-the-art methods. Furthermore, compared to previous approaches, our method excels in preserving the generalization ability of LLMs on unseen tasks.

Orthogonal Subspace Learning for Language Model Continual Learning
Xiao Wang**, Tianze Chen Qiming Ge* Han Xia*,
Rong Bao* Rui Zheng* Qi Zhang* x Tao Gui Xuanjing Huang**
School of Computer Science, Fudan University, Shanghai, China
Institute of Modern Languages and Linguistics, Fudan University, Shanghai, China International Human Phenome Institutes (Shanghai) qz, tgui j@fudan.edu..
Abstract
Benefiting from massive corpora and advanced hardware, large language models (LLMs) exhibit remarkable capabilities in language understanding and generation. However, their performance degrades in scenarios where mul- tiple tasks encountered sequentially, also known as catastrophic forgetting. In this paper, we propose orthogonal low-rank adaptation (O-LoRA), a simple and efficient approach for continual learning in language models, effectively mitigating catastrophic forgetting while learning new tasks. Specifically, O-LoRA learns tasks different (low-rank) vector subspaces that kept orthogonal each other in order to minimize interference. Our method induces only marginal additional parameter costs and requires user data storage for replay. Experimental results on continual learning benchmarks show that our method outperforms state-of-the-arr methods. Further- more, compared to previous approaches, our method excels in preserving the generalization ability of LLMs on unseen tasks.
Original update direction
Orthogonal update direction
Gradient subspace of previous tasks
Figure 1: Illustration highlighting the intuition of our approach. O-LoRA mitigates catastrophic forgetting of past task knowledge by constraining the gradient updates of the current task to be orthogonal to the gradient subspace of the past tasks.
Existing continual learning works (Ke and Liu, 2022; Wang et al., 2023a) can mainly catego- rized into rcheursal-basee regularization-basee and architecture-basee approaches. Rehearsal- based approaches (Lopez-Paz and Ranzato, 2017; de Masson D' Autume et al., 2019) allow access to memory buffer with examples from prior tasks and train the model jointly with the current task. Unfortunately, storing and replaying data from previous tasks may raise privacy concerns, especially when sensitive or personally identifiable information is involved. Regularization-bant approaches (Kirkpatrick et al., 2017; Li and Hoiem, 2017; Smith et al., 2023) introduce additional terms in the loss function to penalize changes in important weights, aiming protect carlier learned tasks. They often struggle to handle long task sequences. Architecture-based approaches (Wang et al., 2023e; Razdaibiedina al., 2023) dynamically expand the model capacity or isolate existing model weights to reduce interference.
1 Introduction
Learning tasks sequentially is crucial for de- veloping real-world NLP models (Wang et al., 2023b; Xi et al., 2023), as enables continu- ous evolution when encountering new tasks or knowledge, Although pre-trained models (Devlin et al., 2019; Brown et al., 2020; Raffel et al., 2020; OpenAl, 2023) have achieved tremendous success static tasks (Wang al., 2022a, 2023c), learning multiple tasks sequentially, commonly referred to continual learning, remains challenging (Wu et al., 2022: Luo et al., 2023). As model learns new tasks, tends to forget or lose the knowledge it had acquired for earlier tasks, leading to phenomenon known catastrophic forgetting (McCloskey and Cohen, 1989).
Equal contribution Corresponding Author

---

However, such approaches essentially learn differ- ent expert models for different tasks, limiting their generalization to unseen tasks.
models, incrementally learning new tasks in orthogonal subspaces.
Our method significandly outperforms prior SOTA methods on standard continual learning benchmarks.
Existing methods typically update all tasks within shared vector space, directly affecting the model's hidden layer outputs. Recent studies (Farajtabar et al., 2020; Saha et al., 2021) have highlighted a promising approach to address this issue. By taking gradient steps in the orthogonal direction to the gradient subspaces associated with past tasks, we can effectively mitigate catastrophic forgetting as it prevents interference with the past task loss functions. However, previous approaches either require storing historical data (Chaudhry al., 2019), which raises data privacy concerns, historical data gradients (Farajtabar al., 2020), which becomes impractical for large-scale models.
Experimental results show that our method preserves the generalization ability of large language models unseen tasks, which was lacking in previous approaches.
Background
2.1 Continual Learning Setup
Continual learning (Ke and Liu, 2022: Wang et al., 2023b) focuses on developing learning algorithms to accumulate knowledge on non-stationary data. In supervised continual learning, sequence of tasks
In this work, we propose orthogonal low-rank simple and efficient
D arrive in a streaming fashion. Each task D, = {(*,, 3/) J contains separate target dataset, where ay Xr y E V/. A single model needs adapt them sequentially, with only access Dr the t-th task. general, given a prediction model he parameterized by θ, continual learning seeks to optimize for the following objective across all tasks:
adaptation (O-LoRA) approach for continual learning language models. Our key insight is rooted in the nature of LoRA: large pre-trained models primarily fine-tune within a specific low-rank subspace. With this premise. we hypothesize that the gradient subspaces from previous tasks can be effectively captured by the LoRA parameters. In the context of continual learning, we incrementally learn new tasks in orthogonal subspace while fixing the LoRA parameters learned from past tasks. Figure 1 provides visual representation of how O-LoRA minimizes catastrophic forgetting.
max
Σ
po x))
(1)
A= J.
In this study, we tackle more challenging setting. During the training phase, the model is prohibited from accessing any historical data. In the testing phase, the model predicts sample's 's label without knowing which task it belongs to.
Our method offers three advantages: (1) Data privaie-frenndlines: We require no storage of user data for replay, addressing concerns associated with privacy. (2) Model garameter-friendllnnss. By introducing only marginal cost of additional parameters, our approach enables the learning of new tasks without compromising the performance previous tasks. (3) Generalization-friendiin Our method does not rely on task IDs during testing, making it compatible with instruction tuning paradigm (Wang et al., 2022b), thus preserving LLMs' generalization ability unseen tasks.
2.2 LoRA
When pre-trained models (PTMs) adapt to specific tasks, Hu et al. (2021) has demonstrated that weight updates in PTMs exhibit a low "intrin- sic dimension." " For a pre-trained weight matrix Wintt E Rdxk LoRA constrains its update by representing it with a low-rank decomposition Wint + AW = Wini + AB, where A E Rdxr B Re and the rank < min(d, k). Winit remains fixed during training and does not receive gradient updates, while A and B contain trainable parameters. To illustrate the modified forward pass of LoRA, consider the operation h W.n.. With LoRA, the modified forward pass becomes:
Our main contributions are summarized as follows:
We introduce O-LoRA, a simple and efficient approach for continual learning in language
'The dataset, code can he found at http:::///gtubb.... cmnfriend/O-LoRA
W W AW
Wmna ABz (2)

---

Answer: very posilive
Output
Orthogenal regularization
O-LoRA
Transformer-Basee Language Model
Multi-Head Atention
Weights of the pretrained model
K
V
LoRA weights previous
weights tank
Frozen
Trainable
Input
ih the sentiment the following paragraph? very negative, negative, neutral, positive, very positive
Option:
I love Bocchi the Rock!! I have watched all the three seasons in one night!! No episodes missed!!
Sentiment Analysis
Topic Clessification
Question nswering
The framework of O-LoRA for language model continual learning, First, allowing the integration of human expertise and enhancing generalization by instruction tuning, Next, approximate gradient subspaces of each task respectively uning LoRA. For each sequentially incoming task, we incrementally learn a new LoRA while enforcing orthogonality between the current task's 's LoRA and the past ones.
Figure 2:
3 Orthogonal Low-rank Adaptation
to output text. 2) Options are the output label constraints for task, which represent the set of possible outputs that can be generated by the model for a given input. 3) Text is the imput sentence of task instance. This sequence then fed into the pre-trained language model along with the task instruction and options. 4) Answer is the expected output of the given sample.
In this section, we introduce O-LoRA, illustrated in Figure 2. First, we adopt instruction tuning as our training paradigm. Then, incrementally learn tasks within orthogonal subspace while keeping the LoRA parameters fixed for past tasks. Lastly, we conduct comparative analysis of our method compared to existing approaches.
3.2 Continual Learning in Orthogonal Subspaces
3.1 Instruction Schema
Instruction-follooiing capability is essential to LLMs as an interface between humans and AI models (Wang et al., 2022b; Ouyang et al., 2022; Wang et al., 2023d). We choose instruction tuning training paradigm for two reasons: 1) Incorporating human expertise: The models can leverage prior knowledge and benefit from human expertise by providing explicit instructions. leading to more efficient learning. 2) Enhanced generalization: The explicit guidance helps models capture the underlying principles, enabling better generalization to unseen situations.
Previous methods exhibit a common feature: all tasks undergo updates within shared vector space, directly impacting the hidden layer outputs of the model. Catastrophic forgetting happens in neural networks when the gradient updates with respect to a new task are applied to the model without considering previous tasks.
Farajtabar et al. (2020) propose the Orthogonal Gradient Descent (OGD) method for mitigating this problem, which constrains the parameters to move within the orthogonal space to the gradients of previous tasks. With limited access previous task data, OGD approximates the current gradient of previous data with the gradient in the previous convergence parameters. However, OGD needs to store gradients of all previous data. This can
All task instructions follow the same uniform schema, which composed of 1) Task Definition provides a detailed guide on how an input text (c.g., a sentence or a document) is expected to be mapped

---

be especially intractable for large-scale language models with billions of parameters (Raffel al., 2020; Brown et al., 2020), which have become a standard in the NLP field.
RF
PE
TIF
FWC (Kirkparrick et al., 2017)
A GEM (Chaudhry et al., 2018)
Masson D' D'Autume 2019)
IDBR (Huang et al. 2021)
L2P (Wang al., 2022c]
Is it possible to approximate the gradient direction of previous tasks without storing histor- ical gradienis? In this study, we leverage the low-rank subspace of LoRA (Hu et al., 2021) proxy for the gradient subspace of past tasks. Our fundamental insight is rooted in the nature of LoRA: large pre-trained models primarily fine-tune within a specific low-rank subspace. This characteristic behavior suggests that the LoRA parameters are not mere numerical adjustments but encapsulate crucial model update directions. Therefore, we hypothesize that the gradient subspaces of previous tasks succinctly represented by the LoRA parameters. By learning within subspace orthogonal the LoRA subspace associated with previous tasks, we can prevent interference with past task loss functions, thus mitigating catastrophic forgetting.
LwF Hoiem, 2017)
OGD (Farjjaahar al. 2020)
LFPTS (Qin and Joty, 2021)
EIP (Wang et al., 2023e)
(Razduubiedinss 2023)
O-LoRA
5
Table The comparison between O-LoRA and other continual learning methods. Specifically, RF indicates whether method rehearsal-free. PE indicates whether the method is parameter efficient. TIF indicates whether task-id is available during inference. UT indicates whether the method can be applied to solve unseen tasks.
Lorth (Ai, = Σ klob
(8)
where Oz, k-] denotes the element at the j-th row and k-th column of O2,,, and À is the weights of the orthogonality loss. During the training process, to mitigate forgetting of past knowledge, we fix the previous LoRA parameters {A,, B,ji t}. Following Hu al. (2021), only apply LoRA to the attention weights of queries (W,) and values (W,).
We propose O-LoRA, which incrementally learns new tasks in a direction orthogonal to the LoRA subspace of past tasks while fixing the previous parameters. For each task, we introduce set of LoRA parameters denoted (A,, B.). where A RRdxr, E TRr and the rank r < min(d, k). We approximate the parameter update subspace L4, for the t-th task as the subspace spanned by the column vectors of Ac:
While the number of LoRA parameters grows with the number of tasks during training, we can merge the updates corresponding to the LoRA parameters into the initial parameters to avoid GPU memory inflation.
A = [a], 07 .... aj]
(3)
U, = spant a. a d7}
(4)
Let B, = [b], bY,, by) where bū E B3 represents the linear weighting coefficients of the column vectors in A,
W = W init
A,
(9)
3.3 Comparisons Between O-LoRA and Other Methods
To ensure the orthogonality between the sub- space U and the subspace W, we need to satisfy:
In this section, we compare O-LoRA with other existing continual learning methods across several dimensions: rehearsal-free, parameter efficiency, availability of task-id during inference, and appli- cability to unseen tasks. As shown in Table 1, O- LoRA demonstrates three distinct advantages: data privacy- frinnellefrienninnsssss model parameter- frienddiiredssssssss and generalizationn-rienddll
u. 0, Vu U1 C W.
Therefore, achieving orthogonality between the LoRA subspaces of task (2T') and task (24') can be expressed as:
O2t - A A. 0.
(6)
Finally, our training objective is defined as:
Data srivacy-friendlingg methods (de Masson D' Autume et al., 2019; Huang et al., 2021), which rely on storing past task data in a buffer and replaying it during training,
Rehearsal-bass
h-1
log pa (yl x) + A
Σ
Lorch (Ai, A4) (7)
D.
1=1

---

are not suitable for scenarios with data privacy concerns. Additionally, as the number of training tasks increases, the cost of training new tasks using replay-based methods also grows. In contrast, our method does not require storing historical data, alleviating concerns regarding data privacy. Moreover, since we only modify the training loss, there is no additional training cost incurred.
Large number of tasks Our method's perfor- mance on longer task sequences, posing a greater challenge, is evaluated through experiments on a continual learning benchmark of 15 datasets (Razdaibiedina et al., 2023). This includes five tasks from CL benchmark, four from GLUE benchmark (MNLI, QQP, RTE, SST2) (Wang et al.. 2018), five from SuperGLUE benchmark (WiC, CB, COPA, MultiRC, BoolQ) (Wang et al., 2019), and the IMDB movie reviews dataset (Maas et al., 201 1). Following Razdaibiedina et al. (2023), we select 1000 random samples for training each task and hold out 500 samples per class for validation.
Model garameter-friendlines Many previous methods (Kirkpatrick et al., 2017; Farajtabar et al., 2020) train the entire model parameters for each task, while our method only introduces marginal additional parameters for each task. O-LoRA has lower requirements in terms of computational resources and GPU memory during training. Additionally, because the training of LoRA freezes the pre-trained model parameters, is less prone to forgetting the knowledge acquired during pre-training.
Unseen tasks Generation To assess the impact of our approach on LLMs' generalization ability, initially train an LLM on the Alpaca dataset (Taori al., 2023), an open-source multitask instruction tuning dataset. We then use the pre- trained LLM for sequential training on the standard CL benchmark (Zhang et al., 2015). Our zero- shot benchmark, MMLU (Hendrycks et al., 2020), covers 57 subjects across various domains such as STEM, humanities, and social sciences, assessing world knowledge and problem-solving abilities across various difficulty levels.
aameralization-friendiiee Traditional meth- ods (Kirkpatrick et al., 2017; Chaudhry et al.. 2018; Wang al., 2022c), primarily designed for classification tasks, often fall short in generalizing to unseen tasks due to their narrow task-specific focus. In contrast, O-LoRA employs instruction tuning (Wang al., 2022b) as its training paradigm. By incorporating explicit instructions or demon- strations, the model capture the underlying principles or constraints of a task. This explicit guidance helps the model generalize beyond the specific examples in the training data, enabling to handle unseen situations more effectively. The integration of human expertise through instruction tuming enhances the generalization capabilities of O-LoRA.
4.1.2 Metrics
Let dzjj be the testing accuracy on the i-th task after training on j-th task, the metries for evaluating is Average Accuracy (AA), the average accuracy of all tasks after training on the last task, $ > 0g,
4.1.3 Baselines
We evaluate O-LoRA against 10 baseline methods. Importantly, among these baselines, only prompt- based methods are exceptions; all others utilize the LoRA framework. This uniformity in the founda- tion ensures consistent parameter settings between O-LoRA and its comparatives, guaranteeing fair comparison.
4 Experiments
4.1 Experimental Setup
4.1.1 Datasets
SeqFT (de Masson Autume et al., 2019): train all model parameters on a sequence of tasks (without adding any regularization or replaying samples from the previous tasks).
Standard CL Benchmark We evaluate our ap- proach using the CL benchmark for language models, which consists of five text classification datasets introduced by Zhang et al. (2015): AG News, Amazon reviews, Yelp reviews, DBpedia and Yahoo Answers. We adopt the CL setup for the T5 model, following LFPT5 (Qin and Joty, 2021), and explore three different orders of the benchmark. Appendix A.2 provides the task details, and the sequences tasks used in our experiments are provided in Appendix A.3.
SeqLoRA: fixed-size LoRA parameters are trained on sequence of tasks (without adding any regularization or replaying samples from the previous tasks).
IncLoRA: incremental learning new LoRA parameters on sequential series of tasks

---

Standard CL Benchmark
Large Number of Tasks
Order-1
Order-2
Order-3
avg
Order-4
Order-5
Order-6
avg
SeqFT
18.9
24.9
41.7
28.5
7.4
7.4
7.5
7.4
SeqLoRA
44.6
32.7
53.7
43.7
2.3
0.6
1.9
1.6
IncLoRA
66
64.9
68.3
66.4
63.3
58.5
61.7
61.2
Replay
53,2
56.9
61.3
57.8
55
54.6
53.1
54.2
EWC
48.7
47.7
54.5
50.3
45.3
44.5
45.6
45.1
LwF
54.4
53.1
49.6
52.3
50.1
43.1
47.4
46.9
L2P
60.3
61.7
61.1
60.7
57.5
53.8
56.9
56.1
LFPT5
67.6
72.6
77.9
72.7
70.4
68.2
69.1
69.2
O-LoRA
75.4
75.7
76.3
75.8
72.3
64.8
71.6
69.6
ProgPrompt 75.2
75
75.1
75.1
78.0
77.7
77.9
77.9
PerTaskFT 70.0
70.0
70.0
70.0
78.1
78.1
78.1
78.1
MTL
80.0
80.0
80.0
80.0
76.5
76.5
76.5
76.5
Table 2: Summary of the results on two standard CL benchmarks with T5 15-large model. Averaged accuracy after training the last task is reported. All results averaged over runs.
(without adding any regularization or replaying samples from the previous tasks).
MTL: train a model on all tasks as multi-task learning. This method is the upper bound of continual learning.
Replay: finetune the whole model with memory buffer, and replay samples from old tasks when learning new tasks to avoid forgetting.
4.1.4 Implementation Details
O-LoRA a model-agnostic CL method that can be used with any transformer-based model. our experiments, we use two language models adopted by the previous lines of works CL for NLP: encoder-decoder T5 model (Raffel et al., 2020) and decoder-only LLaMA model (Touvron al., 2023), To compare O-LoRA to the recent CL approaches (Wang et al., 2022e; Qin and Joty, 2021). use the pre-trained T5-large model. To validate the impact of our approach on the generalization ability of LLMs for unseen tasks, we use pre-trained LLaMA- 7B model. All experimental results are reported as the average of runs. For more detailed settings, refer to the Appendix A.
EWC (Kirkpatrick al., 2017): finetune the whole model with regularization loss that prevents updating parameters that could interfere with previously learned tasks.
LwF (Li and Hoiem, 2017): constrains the shared representation layer to similar to its original state before learning the new task.
L,2 (Wang et al., 2022c): uses the input to dynamically select and update prompts from the prompt pool in an instance-wise fashion.
LFPT5 (Qin and Joty, 2021): continuously train soft prompt that simultaneously learns solve the tasks and generate training samples, which are subsequently used in experience replay.
4.2 Main Results
Table presents a performance comparison of O- LoRA and baseline continual learning methods on two CL benchmarks. Following LFPT5, we report the results of three independent runs with different task orders the CL benchmark.
ProgPrompt (Razdaibiedina et al., 2023): adopts task-specific soft prompt for each distinct task, sequentially appending it to prior learned prompts. In essence, it trains individual models per task, leveraging the task ID to select the appropriate model during inference.
Results on Standard Continual Learning Benchmarks On all task orders of the standard CL benchmark. O-LoRA consistently outperforms previous methods by significant margin. Overall, O-LoRA achieves a performance improvement over 24% compared to LFPT5, the previous state-
Per TaskFT: train a separate model for each task.

---

MMLU
CL
LLaMA-7B
34.4
/
Aurthr tarchen
Alpaca-LoRA
37.5
AlpAca-LoRA-CL
23.3
46.7
Alpaca-inc-LoRA-CL
28.6
33.1
Alpaca-OLoRA-CI. 33.6
76.8
Changes
Table Performance comparison different continual learning methods applied to the Alpaca-LoRA-LAl model. These methods are evaluated on MMLU(zero- shot) and CL benchmmark(order 1).
Figure 3: Histogram of prediction loss changes after training on new task. The O-LoRA constraint (Aι 0.5) helps reduce the changes in comparison to when it not present (A1 = 0).
of-thc-art method. Our approach demonstrates comparable performance to multi-task learning and significantly outperforms PerTaskFT, indicating that our method not only effectively avoids catas- trophic forgetting but also leverages knowledge from past tasks for efficient learning new tasks. Performance with Large Number of Tasks On more challenging benchmark with large number of tasks, O-LoRA outperforms the state- of-the-art method, LFPT5, in terms of the average performance across three orders of tasks. While ProgPrompt performs better than our method handling long sequence tasks, its inherent constraints cannot be overlooked. ProgPrompt is strictly tied to tasks it's trained on and leans heavily on task IDs during inference, limiting its generalization ability and making it less adaptive for LLMs. is worth noting that almost all existing continual learning methods perform significantly lower than PerTaskFTT and MTL, indicating that continual learning for a large number of tasks remains a challenging problem.
4.3 Discussions
Does O-LoRA preserve the loss of previous tasks while training new tasks? We assessed the efficiency of the low-rank subspace of LoRA in approximating the gradient subspace of previous tasks. In our evaluation, we applied an orthogo- nality constraint with a weight of 0.5 (Aı 0.5). In comparison, without the constraint (Λ₃ = 0), new LoRA parameters are added for new tasks with historical LoRA, and model parameters are kept fixed. As Figure 3 shows, the O-LoRA constraint helps keep the loss of previous samples low, proving that the O-LoRA constraint effectively counteracts catastrophic forgetting.
How does O-LoRA influence the output of each layer in the model? We examine the variation in hidden states for past task samples in models trained with and without O-LoRA constraints, using the T5-base model. Figure 4 demonstrates that the O-LoRA constraint mini- mizes the variations, hence reducing the forgetting of internal knowledge. We found that lower layers encode more generic semantic knowledge that can be shared across tasks. Conversely, higher layers encode task-specific semantic knowledge and change a lot during new task learning. The decoder can capture relevant information from these rich semantic representations, proving the minimal impact of our method on past tasks.
Impact on the Generalization Ability of LLMs We investigate the impact of O-LORA on the generalization of Large Language Models through continual learning experiments. We start with a fine-tuned LLaMA-7B language model on the Alpaca dataset, then test models with and without the O-LoRA constraint on the MMLU benchmark. With MMLU being a four-classification problem, a 25% accuracy equates to random guessing. According to Table 3, models without O-LoRA (Alpaca-LoRA-CL, Alpaca-LoRA-ine--eee achieve accuracies of 23.3% and 28.6% respectively, comparable random guesses. In contrast, models with O-LoRA average at 33.6% accuracy, demonstrate the effectiveness of O-LoRA in maintaining generalization for unseen tasks.
How do different PLMs influence perfor- mances? We evaluate the performance of models across varying parameter sizes (T5-base, T5-lerge, T5-XL) and distinct architectures (T5, LLaMA) using the standard continual learning benchmark. Our findings are as follows: series, O-LoRA's average accuracy improves as the parameter size increases. 2) Larger network
In the T5

---

Order
Model
2
3
avg
MTL
changes hr CIPTE
T5-base
73.9
75.8
74.5
74.7
78.5
TS-large
75.4
75.7
76.3
75.8
80.0
T5-xl
78.9
79.0
77.9
78.6
79.9
LLaMA-7B
76.8
75.7
75.7
76.1
77.1
Table 4: Comparison of different PLMs' performances across three orders in standard continual learning benchmark. Results also include average accuracy ("avg") and multitask learning performance ("MTL").
Laya!
2.5
Order
r-dim
2
avg
Changees
2
74.2
71.1
73.6
73.0
4
73.0
72.7
74.1
73.3
8
75.6
71.7
71.9
73.1
16
74.5
73.4
74.8
74.2
std
0.92
0.89
1.07
0.47
Figure Variation in hidden across different layers of the T5-base model with and without O-LoRA constraints. (a) illustrates the changes in the hidden states of each layer in the T5 encoder after training on new task. (b) demonstrates the changes in the hidden states of each layer in the T5 decoder. Light blue color indicates the utilization of orthogonality loss, while dark blue represents the absence of orthogonal loss in the training objective.
Table 5: Comparisons of different rank of LoRA.
This experiment is conducted based on T5-Base on the standard continual learning benchmark.
Related Work
5.1 Continual Learning
Continual learning (Ke and Liu, 2022; Wang et al., 2023a) aims to develop learning algorithms that can accumulate knowledge on non-stationary data. Existing works be broadly categorized into rehearsal-based, regularization-bas architecture based approaches. For an in-depth discussion on continual learning in the era of large language models, readers may refer to (Wang et al., 2023b).
sizes appear counteract catastrophic forgetting, approaching the proficiency levels of multitask learning. 3) Notably, even with greater parameter count the LLaMA-7B model, the T5-3B model registered a higher average accuracy. This implics that reroddrr-deoder architectures might be more resistant to forgetting.
and
Rehearsal-based approaches (Lopez-Paz and Ranzato, 2017; de Masson D' Autume et al., 2019; Han et al., 2020; Bai al., 2022) leverage memory buffer that stores examples from previous tasks, training the model jointly with the current task. Experience replay (ER) (Rolnick et al., 2019) is a common strategy employed in rehearsal- based approaches and serves as strong baseline. However, the storage and replay of data from previous tasks raise privacy concerns, particularly when dealing with sensitive information. Regularization-
What is the Optimal Rank r for O-LoRA? To investigate the influence of the rank parameter (r) on the performance of O-LoRA, we conduct experiments using T5-Base on standard CL benchmark. Table 5 presents the results of varying values. Increasing the rank improves the average accuracy of the model to certain extent. However, we observe that there not significant difference in performance between r=2 and 1=16, indicating that the gradient space of the model has a relatively low intrinsic dimensionality.
approaches (Kirkpatrick et al., 2017: Li and Hoiem, 2017; Farajtabar et al., 2020; Smith et al., 2023) incorporate additional

---

terms into the loss function penalize changes in crucial weights. For instance, Orthogonal Gradient Descent (OGD) (Farajtabar et al., 2020) constrains the parameters to move within the orthogonal space defined by the gradients of previous tasks. However, OGD requires storing gradients of all historical data, which becomes infeasible for large language models. Another work introduces C- LoRA (Smith et al., 2023) for continual learning of text-conditioned images, which regularizes the similarity of new LoRA parameters with historical versions, limiting their learning plasticity to new tasks.
Conclusion
In this paper, we introduce O-LoRA, a novel ap- proach that leverages orthogonal subspace learning for continual learning in language models. O- LoRA systematically addresses catastrophic forget- ting by adopting an incremental learning strategy within orthogonal subspaces. Distinguished by its data privacy considerations, efficient model parameter utilization, and robust generalization to novel tasks, our method stands out. Empirical eval- uations underscore O-LoRA's efficacy in tackling the intricacies of continual learning.
Architecture-based approaches (Wang al.. 2023e; Razdaibiedina et al., 2023) focus on dynamically expanding model capacity or isolating existing model weights to mitigate interference between new and old tasks. Progressive Prompts (Razdaibiedina et al., 2023) learns separate prompts for each incoming task and sequentially con- catenates them with previously learned prompts. However, such approaches essentially train distinct expert models for different tasks, which restricts their generalization ability to unseen tasks.
Limitations
While method has demonstrated effectiveness in empirical evaluations, there are a few limitations to consider. Firstly, its performance and appli- cability in more complex scenarios with a large number of tasks, such as hundreds of tasks, require further investigation. Additionally, although our method does not rely on task identification during inference, it still requires task identification during training to train different LoRA parameters for each task. Exploring methods for task-agnostic training would be a valuable future direction. By addressing these limitations, we can enhance the scalability and task aask-tgnostic capabilities of our approach, further advancing the field of continual learning for language models.
In contrast to existing methods, our approach offers unique advantages in terms of data privacy, model parameter efficiency, and generalization capability, as discussed in the previous sections.
5.2 Parameter Efficient Tuning
Parameter Efficient Tuning (PET) (He al., 2021) has emerged as a significant research direction aimed at optimizing model performance while minimizing computational resources and annotation efforts. Various approaches have been proposed to achieve parameter efficiency in tuning. including adapters (Houlsby et al.. 2019), prompt learning (Lester al., 2021), LoRA (Hu et al., 2021), and fine-tuning subsets of the model (Zaken et al., 2021). One particularly promising upproach is the use of low-rank adapters, which have demonstrated effectiveness in adapting models to new tasks with minimal additional parameters. Building upon LoRA, we propose an efficient continual learning neural architecture in this work. Our approach involves layering low-rank adapters on the key and value projection matrices of transformer blocks. By leveraging the benefits of low-rank adapters, we aim to strike balance between model performance and computational efficiency in the context of continual learning.
Acknowledgements
The authors express their gratitude the anony- mous reviewers for their insightful comments. We also wish acknowledge Hang Yan. Even though he wasn't directly involved in this study, his foundational guidance in continual learning has been pivotal to our research. This work was partially funded by Shanghai Academic Research Leader Program 22XD1401100.
References
Guirong Bai, Shizhu He, Kang Liu, and Jun Zhao. 2022. Incremental intent detection for medical domain with contrast replay networks. In Findings of the Association for Computational Linguisties: ACL 2022, pages 3549-3556.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot

---

learners. Advances in neural information processing systems, 33:1877-1901.
Zixuan Ke and Bin Liu. 2022. Continual learning of natural language processing tasks: A survey. ArXiv, abs/2211. 1.22011.
Arslan Chaudhry, Mare Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2018. Efficient lifelong learning with a-gem. arXiv preprint
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz. Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka
arXiv:8//2.00420.
Arslan Chaudbry, Marcus Rohrbach, Mohamed EI- hoseiny. Thalaiyasingam Ajanthan, Puneet Kumar Dokania, Philip H. S. Torr, and Marc' Aurelio Ranzato. 2019. Continual learning with tiny episodic memories. ArXiv, abs/1902.10443
et al. 2017. Over- coming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):321-3326
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirieal Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cyprien de Masson D'Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019. Episodic memory in lifelong language learning. Advances in Neural Information Processing Systems, 32.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-48 4-86,, Minneapolis, Minnesota. Association for Computational Linguisties.
Zhizhong Li and Derek Hoiem. 2017. Learning without forgetting. IEEE transactions pattern analysis and machine intelligence, 40 12)22335-229
David Lopez-Paz and Mare' Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30.
Mehrdad Farajtahar, Navid Azizan, Alex Mott, and Ang Li. 2020. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, pages 3762-3773. PMLR.
Yun Luo, Zhen Yang. Xuefeng Bai, Fandong Meng, Jie Zhou, and Yue Zhang. 2023. Investigating forgetting in pre-trained representations through continual learning. arXiv preprint arXiv:2305.05965
Xu Han, Yi Dai, Tianyu Gao, Yankai Lin. Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2020. Continual relation learning via episodic memory activation and reconsolidation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6429-6440.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang. Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Languuge Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Linguistics.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. 2021. Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:21 10.0433.0
Dan Hendrycks. Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika. Dawn Song. and Jacob Steinhardt, 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03.2 03300.
Michael McCloskey and Neal J. Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 24:109-165.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, 2019. Parameter-efficiee transfer learning for nip. In International Conference Machine Learning, pages 2790-2799, PMLR.
OpenAl. 2023. Gpt-4 technical report. ArXiv,
abs/2303.08774.
Long Ouyang, Jeffrey Wo, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730- 27744.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan AZu, Yuanzhi Li. Shean Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.0966
Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang. 2021. Continual learning for text clas- sification with information disentanglement based regularization. arXiv preprint NXXi::210400545
Chengwei Qin and Shafiq Joty. 2021. Lfpt5: A unified framework for lifelong few-shot language learning based on prompt tuning of 15. arXiv preprint 1r..v:2110.07298.

---

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67.
Xiao Wang, Yuansen Zhang, Tianze Chen, Songyang Gao, Senjie Jin, Xianjun Yang. Zhiheng Xi. Rui Zheng, Yicheng Zou, Tao Gui, et al. 2023b. Trace: A comprehensive benchmark for continual learning in large language models. arXiv preprint arXiv:2310.06700
Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. 2023. Progressive prompts: Continual learning for language models. In The Eleventh International Conference on Learning Representations.
Xiao Wang, Weikang Zhou, Qi Zhang, Jie Zhou, Songyang Gao, Junzhe Wang, Menghan Zhang, Xiang Gao, Yunwen Chen, and Tao Gui. 2023c. Farewell to aimless large-scale pretraining: Influ- ential subset selection for language model. arXiv preprint arXiv:2305. 12816.
David Rolnick, Arun Abuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. 2019. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32.
Xiao Wang. Weikang Zhou, Can Zu, Han Xia, Tianze Chen, Yuansen Zhang. Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, et al. 2023d. Instructuie: Multi-task instruction tuning for unified information extraction. arXiv preprint Xrv:v:3304..000 080085.
Gobinda Saha, Isha Garg, and Kaushik Roy. 2021. Gradient projection memory for continual learning. ArXiv, abs/2103.09762.
Yizhong Wang. Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran. Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney. Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022b. Super-NaturalInstrruca Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 5085-51000 1855-519,,4 Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
James Seale Smith, Ten-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. 2023. Continual diffusion: Continual customization of text-to-image diffusion with c-lora. arXiv preprint arXiv:2304.06027.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois. Xuechen Li. Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following Ilama model. https:// tttthu..co//tttee tattttttttssssssaaaeeeeeeeeee aab/stanford_alpac..
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and efficient foundation language models. arXiv preprint 232.
Zhicheng Wang, Yufang Liu, Tao Ji, Xiaoling Wang. Yuanbin Wu, Congcong Jiang, Ye Chao, Zhencong Han. Ling Wang, Xu Shao, and Wenqiu Zeng. 2023e. Rehearsal-free continual language learning via efficient parameter isolation. ArXiv.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32.
Zifeng Wang, Zizhao Zhang, Chen-Ye Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. 2022c. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 139-149.
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: SkYiv:8804.0461..
Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan- Fang Li, Guilin Qi, and Gholamreza Haffari. 2022, Pretrained language model in continual learning: A comparative study. In International Conference on Learning Representations.
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. 2023a. A comprehensive survey of continual learning: Theory, method and application. ArXiv, abs/2302.00487.
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong. Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.0786.
Xiao Wang. Shihan Dou, Limao Xiong, Yicheng Zou, Qi Zhang. Tao Gui, Liang Qiao, Zhanzhan Cheng. and Xuanjing Huang. 2022a. Miner: Improving vutabl-wocaullary named entity recognition from an information theoretic perspective. arXiv preprint arXiv:2204.04331.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. Bitfit: Simple parfieter-efffcient fine-tuning

---

for transformer- bassed masked language-models arXiv preprint vrXiv22106100190
A.4 Task Instructions
Table 6 shows prompts for different tasks, NLI denotes natural language inference, including MNLI, RTE and CB. SC denotes sentiment analysis, including Amazon, Yelp, SST-2 and IMDB. TC denotes topic classification, including AG News, Dbpedia and Yahoo.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28.
A Appendix
A.5 Detailed results of MMLU Zero-shot
A.1
Implementation Details
All of our experiments on i5 models were con- ducted on a machine equipped with NVIDIA GeForce RTX 3090 and were implemented using DeepSpeed repository. For all orders of task streams, We trained the models with one epoch, constant learning rate of le-3, batch size of 64(a batch size of per GPU), dropout rate of 0. and weight decay rate of O. Only the values X and λ2 are different among order to For order 1, order and order 3, we set Å = 0.5, 0.5, 0.5, 0.5, 22 = 0, O, 0, O. For every task in order 4(NLLI, CB, WiC, COPA, QQP, BoolQA, RTE, IMDB, Yelp, Amazon, SST-2, DBpedia, Agnews, MultiRC, Yahoo),
set λ1 = 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 5, 5, 5, 5, and Aa 0, 0.1, 0, 0, 0.3, 0.1, 0.05, 0, 0.1, 0.1, 0.1, 0, 0.1 respectively. For order 5(MultiRC, BoolQA, WIC, MNLI, CB, COPA, QQP, RTE, IMDB, SST-2, DBpedia, Agnews, Yelp, Amazon, Yahoo), we set 5 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, and À2- 0.1.1, 0.1, 0.1, 0, 0.1, 0.3, 0.1, 0.5, O, 0.1, O, 0.1, 0.1 respectively. For order 6(Yelp. Amazon, MNLI, CB, COPA, QQP, RTE, IMDB, SST-2, DBpedia, Agnews, Yahoo, MultiRC, BoolQA, WiC), we set \ 0.5, 0.5, 0.02, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, and A2 = 0, 0, 0, 0.1, 0, 0.3. 0, 0.1, 0.1, 0, 0.1, 0, 0.1, 0.3 respectively.
A.2 Datasets
Table 4 shows details of the 15 datasets used for our CL experiments, along with their evaluation metries. Overall, we used datasets from CL benchmark (Zhang et al., 2015), GLUE (Wang et al., 2018) and SuperGLUE (Wang et al.. 2019) benchmarks, and added IMDB movie reviews dataset, following (Razdaibiedina et al., 2023).
A.3 Task Sequence Orders
We report task orders used for our CL experiments across T5 and LLaMA models Table 5.

---

Dataset name
Category
Task
Domain
Metric
1. Yelp
CL Benchmark
sentiment analysis
Yelp reviews
accuracy
2. Amazon
CL Benchmark
sentiment analysis
Amazon reviews
accuracy
3. DBpedia
CL Benchmark
topic classification
Wikipedia
accuracy
4. Yahoo
CL Benchmark
topic classification
Yahoo Q&A
accuracy
5. AG News
CL Benchmark
topic classification
news
accuracy
6. MNLI
GLUE
NLI
various
accuracy
7. QQP
GLUE
paragraph detection
Quora
accuracy
8. RTE
GLUE
NLI
news, Wikipedia
accuracy
9. SST-2
GLUE
sentiment analysis
movie reviews
accuracy
10. WiC
SuperGLUE
word sense disambiguation
lexical databases
accuracy
11. CB
SuperGLUE
NLI
various
accuracy
12. COPA
SuperGLUE
QA
blogs, encyclopedia
accuracy
13. BooIQA
SuperGLUE
boolean QA
Wikipedia
accuracy
14. MultiRC
SuperGLUE
QA
various
accuracy
15. IMDB
SuperGLUE sentiment analysis
movic reviews
accuracy
Table 6: The details 15 datasets used our CL experiments. NLI denotes natural language inference, QA denotes
questions and answers task. First five tasks correspond the standard CL benchmark, all other tasks used in long-sequenee experiments.
Order
Model
Task Sequence
T5, LLaMA
dbpedia >> amazon yahoo ag
T5, LLaMA
dbpedia + amazon ag yahoo
T5, LLaMA
yahoo - amazon - ag -> dbpedia
mnli + cb - wic + copa + qqp - boolqa - rte - imdb +
4
T5
yelp + amazon - sst-2 - dbpedia - ag + multirc - yahoo
multire - boolqa + wic -, mnli - cb - copa qqp rte -> imdb - sst-2 + dbpedia + ag yelp - amazon + yahoo
5 T5
yelp amazon mnli cb copa - qqp - rte imdb
6
T5
sst-2 dbpedia ag yahoo multire boolqa wic
Table 7: Six different orders of task sequences used for continual learning experiments. Orders 1-3 correspond to the standard CL becnhmark adopted by prior works. Orders 4-6 are long-sequence orders spanning 15 tasks. following (Razdaihiedina al., 2023).
Task
Prompts
What is the logical relationship between the "sentence 1" and the "sentence 2"? Choose one from the option.
NLI
Whether the "first sentence" and the "second sentence" have the same meaning?
QQP
Choose one from the option.
SC
What the sentiment of the following paragraph? Choose one from the option. What is the topic of the following paragraph? Choose one from the option. According the following passage, is the question true false? Choose one from the option.
TC
BoolQA
According the following passage and question, is the candidate answer true or false? Choose one from the option.
MultiRC
Given a word and two sentences, whether the word is used with the same sense in both sentence? Choose one from the option.
WiC
Table 8: Instructions for different tasks.

---

MMLU-task
LLaMA-7B
Alpaca-LoRA
Alpaca-LoRA-CI.
Alpaca-inc-LoRA-CC.
Alpaca-OLoRA-C
math
27
25,9
20.4
23.4
21.6
health
38.2
40.9
24.8
30.7
36.1
physics
32.5
27.7
30.3
business
40.3
50.3
25.2
32.3
38.7
biology
33,5
38.1
21.8
29.1
35.7
chemistry
25.4
27.7
17.2
20.1
23.8
computer science
30.6
27.2
30.8
31.3
oconomics
31.3
31.4
23
27
28
engineering
28.3
32,4
22.8
23.4
24.1
philosophy
32.6
34.5
23
26.1
32.1
other
37.2
46.9
24.6
33.5
42.1
history
39
45.8
25.1
33.3
42.3
gengraphy
31.8
44.4
18.2
25.8
33.3
politics
28.2
33.6
psycholugy
39.4
40.4
22,8
28.3
35
culture
40.1
49.1
25.9
32.5
41
law
31.9
32.5
23.6
28.5
32.4
Weighted Avg.
34.4
37.5
23.3
28.6
33.6
Table 9: Detailed zero-shot results MMLU benchmark of different CL methods.


Reviews:
[{"review_id": "dcF8kLzQ4m", "reviewer": "Reviewer_PwhU", "summary": "", "questions": "The author suggests a continuous addition of new Lora components upon the emergence of new tasks, which introduces interesting aspects and prompts the following inquiries:\n\nCan the proposed Orthogonal Subspace Learning technique effectively enhance the learning process for a sequence of similar tasks? Although the method exhibits strong performance in mitigating forgetting issues, does it also facilitate the transfer of knowledge between these similar tasks?\n\nWith the linear increase in parameters as new tasks are incorporated, does the timing of task learning influence performance, potentially resulting in poorer outcomes for tasks learned later? Is there a discernible relationship between the order of tasks and the proportion of learnable parameters within the overall parameter count?\n\n", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation and an excellent summarization of our work.  We appreciate your time and effort in carefully reviewing our work. We try to address your remarks below:\n\n\n\n> **R1**: Code is unavailable\n\nWe acknowledge the concern. We commit to releasing the code and associated data simultaneously when the paper is made public.\n\n\n\n> **Q1:** Can the proposed Orthogonal Subspace Learning technique effectively enhance the learning process for a sequence of similar tasks? Although the method exhibits strong performance in mitigating forgetting issues, does it also facilitate the transfer of knowledge between these similar tasks?\n\nCertainly, **O-LoRA possesses the capability to transfer knowledge between different tasks**. This assertion is substantiated by our empirical results presented in Table 2, where the performance of O-LoRA surpasses that of \"PerTaskFT\". The latter method involves training a model specifically for each task and then computing their average accuracy. Below, we re-highlight the experimental results:\n\n```markdown\n|                    |             Standard CL Benchmark         |\n|                    | Order-1      | Order-2     | Order-3      |\n| ------------------ | ------------ | ----------- | ------------ |\n| PerTaskFT          | 70.0         | 70.0        | 70.0         |\n| O-LoRA             | 77.0         | 77.2        | 77.1         |\n```\n\nNevertheless, on the benchmark involving a large number of tasks, the performance of PerTaskFT excels. This discrepancy can be attributed to the fact that the benefits of knowledge transfer are outweighed by the catastrophic forgetting associated with longer task sequences. In the updated version of our paper, we plan to incorporate Forward Transfer Scores to provide further details on this aspect.\n\n\n\n> **Q2:** With the linear increase in parameters as new tasks are incorporated, does the timing of task learning influence performance, potentially resulting in poorer outcomes for tasks learned later? Is there a discernible relationship between the order of tasks and the proportion of learnable parameters within the overall parameter count?\n\nAn insightful query. Indeed, the sequential placement of a task exerts an influence on its learning efficacy. When a task is positioned earlier in a sequence, it generally manifests more optimal fitting results than if positioned later. Yet, this performance decline becomes conspicuously accentuated in particularly protracted task sequences.  Addressing continual learning in such extended sequences remains an open challenge. \n\nTo mitigate the suboptimal outcomes experienced by tasks introduced later in the sequence, a pragmatic approach would involve attenuating the stringent orthogonality constraints applied to them.\n\n\n\n> **Missing References:**\n\nThank you for the suggestion. The highlighted works, focusing on continual learning in specific NLP tasks, indeed represent significant contributions and provide valuable supplements to general continual learning methodologies. We will add these references in subsequent versions of our paper.\n\n\n\n> **Typos Grammar Style**\n\nThanks, we will correct this error in the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Effective mitigation of Catastrophic Forgetting\n- Well-structured and easy to follow\n- Minimal parameter costs and no user data storage", "reasons_to_reject": "- Code is unavailable", "questions_to_authors": "The author suggests a continuous addition of new Lora components upon the emergence of new tasks, which introduces interesting aspects and prompts the following inquiries:\n\nCan the proposed Orthogonal Subspace Learning technique effectively enhance the learning process for a sequence of similar tasks? Although the method exhibits strong performance in mitigating forgetting issues, does it also facilitate the transfer of knowledge between these similar tasks?\n\nWith the linear increase in parameters as new tasks are incorporated, does the timing of task learning influence performance, potentially resulting in poorer outcomes for tasks learned later? Is there a discernible relationship between the order of tasks and the proportion of learnable parameters within the overall parameter count?\n\n", "paper_topic_and_main_contributions": "The paper titled \"Orthogonal Low-Rank Adaptation for Continual Learning in Language Models\" introduces a novel approach, O-LoRA, aimed at addressing the issue of catastrophic forgetting in pre-trained language models when encountering multiple tasks sequentially. The authors propose a solution that leverages orthogonal low-rank adaptation to mitigate the performance degradation that typically occurs in such scenarios.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Minor Issue:\nIn line 420, the acronym \"LLM\" appears to refer to the Large Language Model, rather than the intended reference to the Language Learning Model.\n"}, "extra_text": {"Missing_References": "To my knowledge, the literature review is fine, but it would be even better to include more references in class-incremental language learning, e.g., continual relation extraction and continual intent detection.\n\n[1] Han, Xu, et al. \"Continual relation learning via episodic memory activation and reconsolidation.\" Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.\n[2] Wu, Tongtong, et al. \"Curriculum-meta learning for order-robust continual relation extraction.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021.\n[3] Bai, Guirong, et al. \"Incremental intent detection for medical domain with contrast replay networks.\" Findings of the Association for Computational Linguistics: ACL 2022. 2022.\n[4] Chen, Xiudi, Hui Wu, and Xiaodong Shi. \"Consistent Prototype Learning for Few-Shot Continual Relation Extraction.\" Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023."}}, {"review_id": "mBCFrYnQHE", "reviewer": "Reviewer_jxjZ", "summary": "", "questions": "A. What are the outcomes when evaluating O-LoRA's performance without the inclusion of the instruction tuning scheme? How do the baseline methods fare when augmented with the instruction tuning scheme? To what extent does the instruction tuning paradigm contribute to mitigating forgetting?\n\nB. How does LoRA's effectiveness manifest when integrated with the considered baselines such as EWC, Replay, and LwF? In comparison to rehearsal-free methodologies like SAM-based optimization [1] and dynamically expandable architectures utilizing progressive prompts [2], and parameter isolation [3] how does the proposed approach perform?\n\nC. The paper only reports average performance after training on the last task in the sequences. What is the behavior of the performance of the earlier tasks over the course of continual learning? Additionally, quantifying the forgetting metrics alongside the average performance, as demonstrated in [1] for forgetting and learning accuracy metrics, would provide insightful context.\n\nD. In the caption of Table 1, it is stated that TIF denotes the availability of task-id during inference. The rationale behind why EWC, A-GEM, MbPA++, etc., require task-id during inference remains unclear. Is this an oversight or are there specific details that have not been provided?\n\nE. In Figure 4 of the analysis, the paper visualizes the variation in hidden states across different layers with/ without O-LoRA constraints. Please specify how is this variation computed? Also, in case of the encoder, there is increased variation for both the considered scenarios. How does this align with the forgetting differences? Also, what makes the paper believe that less variation corresponds to lesser forgetting? Is there any evidence about smoothness of the representation?\n\nF. The paper contrasts the performance of various Pre-trained Language Models (PLMs) and draws the conclusion that larger networks mitigate catastrophic forgetting. However, there appears to be a confounding factor – the diversity of pre-training corpora across different PLMs. This outcome is already widely acknowledged within the community, as indicated in [1]. \n", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and have carefully considered your suggestions. We try to address your remarks below:\n\n\n\n> **R1:** About inconsistency in comparing O-LoRA with established baselines by not uniformly applying instruction tuning across all methods.\n\nWe'd like to clarify that during our experiments, **the baseline methods were trained using the same prompts as O-LoRA, ensuring uniformity in comparisons.** \n\nWe recognize that our experimental section lacked explicit mention of this, leading to potential misunderstandings. In subsequent versions of the paper, we will incorporate detailed descriptions regarding the implementation of the baselines to provide clarity.\n\n\n\n> **R2:**  About directly comparing O-LoRA's efficiency against these baselines without ensuring comparability of trainable parameters is not a fair comparison.\n\nIt's essential to note that **among the baselines compared in the paper, except for prompt-based methods such as L2P and LFPT5, all others are implemented based on LoRA**. This ensures a consistent parameter setting across O-LoRA and these baselines, facilitating an fair comparison.\n\nWe acknowledge the oversight in not detailing this in the original paper. We will furnish details regarding this aspect of baseline implementation in the upcoming revision of the paper. Thank you for highlighting this concern.\n\n\n\n> **R3 & QB:** About compare with SAM-based optimization and dynamically expandable architectures methods.\n\nIn our initial submission, we omitted comparisons with dynamically expandable architectures, primarily because these methods tend to train specific models for each task. Although they often excel on tasks they've been trained on, their effectiveness on unseen tasks is questionable.\n\nHowever, recognizing the importance of a comprehensive evaluation, we will incorporate comparisons with SAM-based optimization and dynamically expandable architectures in future revisions. \n\n\n\n> **R4:** About missing details about dataset size, training specifics (such as batch size, learning rate, epoch count, learning rate decay, etc.)\n\nWe have referenced the benchmark datasets at lines 304, 313, and 330 in our paper, pointing to the original publications where they were introduced.\n\nWe recognize the oversight regarding the detailed training specifics. In our subsequent revision, we will explicitly mention the training hyperparameters: a batch size of 64, learning rate set at 1e-03, epoch count at 1, and a weight decay of 0. Furthermore, we will make the code and related datasets publicly available concurrently with the paper's open access, ensuring reproducibility of our experimental results.\n\n\n\n> **QA**: What are the outcomes when evaluating O-LoRA's performance without the inclusion of the instruction tuning scheme? How do the baseline methods fare when augmented with the instruction tuning scheme? To what extent does the instruction tuning paradigm contribute to mitigating forgetting?\n\nWe didn't exclude the instruction in our experiments due to its role as a soft task identifier. **Given the same input, the intended tasks (e.g., sentiment classification, entity recognition, translation) could vary. Without the instruction, the language model would lack task specificity.** Hence, it's essential for our approach.\n\nTo control for instruction effects, all baseline methods employed the same instruction as O-LoRA. The results presented in Table 2 demonstrate that the observed enhancement from O-LoRA is attributable to the method itself, rather than the influence of the instruction.\n\n\n\n> **QC**: The paper only reports average performance after training on the last task in the sequences. What is the behavior of the performance of the earlier tasks over the course of continual learning? Additionally, quantifying the forgetting metrics alongside the average performance, as demonstrated in [1] for forgetting and learning accuracy metrics, would provide insightful context.\n\nThank you for the insightful suggestion. In subsequent revisions, we plan to incorporate additional metrics, including forward transfer score [2], backward transfer score [2], and forgetting metrics [1], to offer a more comprehensive perspective on the model's behavior during continual learning.\n\n\n\n> **QD**: In the caption of Table 1, it is stated that TIF denotes the availability of task-id during inference. The rationale behind why EWC, A-GEM, MbPA++, etc., require task-id during inference remains unclear. Is this an oversight or are there specific details that have not been provided?\n\nEWC, A-GEM, and MbPA++ were originally implemented based on classification models like Bert. Consequently, during the inference phase, they require a task-id to utilize the corresponding classification head for generating predictions for a given task.\n\n\n\n> **QE**: In Figure 4 of the analysis, the paper visualizes the variation in hidden states across different layers with/ without O-LoRA constraints. Please specify how is this variation computed? Also, in case of the encoder, there is increased variation for both the considered scenarios. How does this align with the forgetting differences? Also, what makes the paper believe that less variation corresponds to lesser forgetting? Is there any evidence about smoothness of the representation?\n\nFigure 4 illustrates **how the hidden states of previously tasks change after introducing a new task to the model.** Specifically, for a sequence like dbpedia (trained task) -> amazon (new task), the model trained on dbpedia is represented as $\\theta_d$ and post training on amazon becomes $\\theta_{da}$. The variation is quantified by evaluating the change in the representations for the dbpedia validation set $x_d$, i.e., $\\theta_{da}(x_d)-\\theta_d(x_d)$. A larger variation implies that the model extracts different features.\n\nFor the encoder, both approaches demonstrate significant representation changes, suggesting that both are extracting new features. However, this doesn't necessarily confirm if the model has retained the pertinent features from previous tasks. \n\nOn the decoder side, it determines which features the model should rely on for predictions. Lesser variation in decoder representations indicates that for previous tasks, the important features encoded by the model are largely preserved, which implies reduced forgetting.\n\nWe plan to introduce an analysis similar to that in [1] in our subsequent version to offer a deeper insight into this aspect.\n\n\n\n> **QF**: The paper contrasts the performance of various Pre-trained Language Models (PLMs) and draws the conclusion that larger networks mitigate catastrophic forgetting. However, there appears to be a confounding factor – the diversity of pre-training corpora across different PLMs. \n\nWe appreciate the reviewer's observation regarding the potential confounding factor of the diversity of pre-training corpora across different PLMs. \n\nHowever, **our conclusion is drawn based on a comparison of T5 models of varying sizes, all of which were trained on the same corpus**. Thus, this concern is not applicable in our case. As highlighted in Table 4, we re-emphasize our experimental results to clarify this point.\n\n```markdown\n|                    |             Standard CL Benchmark         |\n|                    | Order-1      | Order-2     | Order-3      |\n| ------------------ | ------------ | ----------- | ------------ |\n| T5-base            | 73.9         | 75.8        | 74.5         |\n| T5-large           | 77.0         | 77.2        | 77.1         |\n| T5-xl              | 78.9         | 79.0        | 77.19        |\n```\n\n\n\n> **Missing References:**\n\nWe appreciate your feedback. While our paper already references [2] and [3], we will incorporate [1] and [4] in our subsequent version and conduct a detailed comparison with them.\n\n\n\n> **Typos Grammar Style**\n\nThank you for pointing out these issues. We will make the necessary corrections in our next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses an important problem of sequential task learning, particularly within pre-trained models. It presents a realistic continual learning scenario involving privacy and storage constraints, and unavailability of the task identity during the inference. \n\n2. Toward the above-mentioned setup, the paper proposes an efficient solution based upon the widely adopted parameter-efficient finetuning technique like LORA. Also, the proposed Orthogonal LORA is efficient compared to previous rehearsal-based methods as it does not require increased computation as the number of tasks increases in the sequences, except for additional complexity from the auxiliary loss computation to enforce orthogonality constraints. \n\n3. The paper is well-written and easy to follow.\n", "reasons_to_reject": "1. One of the major weaknesses lies in the inaccurate assertion that the proposed O-LoRA can be utilized to address unseen tasks more effectively than established baselines such as EWC, A-GEM, and others. The paper adopts widely used instruction tuning as the training paradigm for O-LoRA which enables generalization to unseen/ novel tasks. To ensure a fair assessment against the considered baselines, it is imperative for the paper to apply instruction tuning uniformly across all methods. The inconsistency in the training methodology between O-LoRA and the other baselines skews the outcomes presented in Table 2, where O-LoRA exhibits notably superior performance. To enhance the rigor of experimentation, a concrete suggestion is as follows: initially compare O-LoRA without the instruction tuning paradigm against the baselines. Subsequently, introduce instruction tuning to all methods and observe the ensuing improvements. This approach would not only clarify the role played by the instruction tuning paradigm in addressing catastrophic forgetting but also yield insights into the efficacy of different continual learning methods.\nAnother claim the paper makes is that the proposed approach is parameter efficient as it uses the LoRA technique for sequential fine-tuning. \n\n2. By definition, the proposed solution experiences less forgetting as only a minimal set of parameters is updated while sequential learning. Conversely, methodologies like EWC, A-GEM, and LwF update all model parameters, thereby experiencing severe interference and increased forgetting. It is noteworthy that many of these methodologies originate from the period before the prevalence of pre-training and fine-tuning. In this context, the assertion that O-LoRA is parameter-efficient when measured against the baselines becomes untenable. Aligning with the earlier point, the paper should evaluate these methodologies by solely fine-tuning the LoRA parameters (ensuring comparability of trainable parameters across methods is vital for fair comparisons). \n\n3. The paper fails to compare their method with other rehearsal-free methods like SAM-based optimization for continual learning [1], and dynamically expandable architectures for continual learning [2, 3]. \n\n4. The paper lacks specific information regarding implementation, apart from indicating the models employed for experimentation. It omits crucial details about dataset size, training specifics (such as batch size, learning rate, epoch count, learning rate decay, etc.), and hyperparameters associated with the considered baselines (like regularization strength/ lambda for EWC, memory buffer size for replay, etc.). This omission raises a concern regarding the ability to replicate the reported results.\n", "questions_to_authors": "A. What are the outcomes when evaluating O-LoRA's performance without the inclusion of the instruction tuning scheme? How do the baseline methods fare when augmented with the instruction tuning scheme? To what extent does the instruction tuning paradigm contribute to mitigating forgetting?\n\nB. How does LoRA's effectiveness manifest when integrated with the considered baselines such as EWC, Replay, and LwF? In comparison to rehearsal-free methodologies like SAM-based optimization [1] and dynamically expandable architectures utilizing progressive prompts [2], and parameter isolation [3] how does the proposed approach perform?\n\nC. The paper only reports average performance after training on the last task in the sequences. What is the behavior of the performance of the earlier tasks over the course of continual learning? Additionally, quantifying the forgetting metrics alongside the average performance, as demonstrated in [1] for forgetting and learning accuracy metrics, would provide insightful context.\n\nD. In the caption of Table 1, it is stated that TIF denotes the availability of task-id during inference. The rationale behind why EWC, A-GEM, MbPA++, etc., require task-id during inference remains unclear. Is this an oversight or are there specific details that have not been provided?\n\nE. In Figure 4 of the analysis, the paper visualizes the variation in hidden states across different layers with/ without O-LoRA constraints. Please specify how is this variation computed? Also, in case of the encoder, there is increased variation for both the considered scenarios. How does this align with the forgetting differences? Also, what makes the paper believe that less variation corresponds to lesser forgetting? Is there any evidence about smoothness of the representation?\n\nF. The paper contrasts the performance of various Pre-trained Language Models (PLMs) and draws the conclusion that larger networks mitigate catastrophic forgetting. However, there appears to be a confounding factor – the diversity of pre-training corpora across different PLMs. This outcome is already widely acknowledged within the community, as indicated in [1]. \n", "paper_topic_and_main_contributions": "The paper tackles the challenge of continual learning in the context of pre-trained models, where tasks are encountered sequentially. Specifically, it focuses on the phenomenon of catastrophic forgetting, which arises when the continual learning of new tasks disrupts the performance of previously acquired tasks, leading to a decline in their effectiveness. The paper approaches the problem by employing the widely used parameter-efficient strategy, such as LORA, to facilitate the learning of new tasks and the overall technique involves training distinct sets of LORA parameters for each task. To address the problem of interference between the new and previously learned LORA parameters, the paper introduces an extension to the Orthogonal Gradient Descent method. This extension involves learning new tasks within distinct low-rank vector subspaces that remain orthogonal to one another, thereby minimizing interference. The effectiveness of this approach is demonstrated through empirical results on two distinct benchmarks, showcasing its superiority over the existing baseline.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Mehta, Sanket Vaibhav, Darshan Patil, Sarath Chandar, and Emma Strubell. \"An Empirical Investigation of the Role of Pre-training in Lifelong Learning.\" (2021).\n\n[2] Razdaibiedina, Anastasia, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. \"Progressive Prompts: Continual Learning for Language Models.\" In The Eleventh International Conference on Learning Representations. 2022.\n\n[3] Wang, Zhicheng, Yufang Liu, Tao Ji, Xiaoling Wang, Yuanbin Wu, Congcong Jiang, Ye Chao et al. \"Rehearsal-free Continual Language Learning via Efficient Parameter Isolation.\" In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10933-10946. 2023.\n\n[4] Yu, Tianhe, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. \"Gradient surgery for multi-task learning.\" Advances in Neural Information Processing Systems 33 (2020): 5824-5836.\n", "Typos_Grammar_Style_and_Presentation_Improvements": "1. [Line 482] Optimal rank for O-LoRA is reported in Table 5 instead of Table 9.\n\n2. [Lines 373-375] The paper mentions that MTL is the upper bound of continual learning. However, it is well-known that MTL is susceptible to task interference [4] in which case MTL might be sub-optimal and might cease to exist as upper bound. Please consider updating the description to cover this aspect.\n\n3. Do not include Figure 1 for the sake of just including it! This Figure is not referred to anywhere in the main text!\n"}}, {"review_id": "9FOLDhK2uj", "reviewer": "Reviewer_GrPc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback on our paper. We truly appreciate the time you invested in the review. We've carefully considered your insights and addressed the highlighted concerns. We hope our responses shed more light on the matter.\n\n\n\n> RA: The comparison with the state-of-the-art continual learning method [1] is missing. Particularly, in some evaluations, the state-of-the-art method outperforms the proposed method.\n\nWe appreciate your observation. In our initial submission, we overlooked the comparison with [1] due to its approach of training distinct models for each task. While such models excel in tasks they are trained for, their performance on unseen tasks can be inconsistent.\n\nHowever, recognizing the importance of a comprehensive evaluation, we will incorporate comparisons with [1] in future revisions. \n\n\n\n> RB:  The paper lacks in-depth analysis and comparison, including hyper-parameter optimality / sensitivity, forward / backward transfer, and generalization effects to unseen sequential tasks (as the authors mentioned that these effects were a unique strength of the proposed method).\n\nThank you for raising these concerns. \n\nRegarding **generalization to unseen tasks**, section 4.2 of our paper delves deeply into this. Our findings are illustrated in **Table 3**:\n\n```markdown\n|                    | MMLU         | CL Benchmark| \n| ------------------ | ------------ | ----------- |\n| LLaMA-7B           | 34.4         | /           | \n| Alpaca-LoRA        | 37.5         | /           | \n| Alpaca-LoRA-CL     | 23.3         | 46.7        | \n| Alpaca-inc-LoRA-CL | 28.6         | 33.1        | \n| ------------------ | ------------ | ----------- |\n| Alpaca-OLoRA-CL    | 78.9         | 79.0        |\n```\n\nWe utilized the MMLU benchmark, which is tailored for assessing generalization to such tasks.  As evidenced by our results, our method not only outperforms the baseline on new tasks but also maintains the generalization capabilities of the LLM. For a detailed discussion, please refer to the section 4.2 segment titled \"Impact on the Generalization Ability of LLMs.\"\n\nAppreciating your insightful feedback, we will expand our analysis in subsequent versions, adding insights on hyper-parameter optimality/sensitivity and introducing metrics for forward/backward transfer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. The paper is well-written and easy to follow.\n\nB. The soundness of the proposed method is sufficiently backed up by previous work.\n\nC. The experimental results present several interesting results, e.g., continually learned O-LoRA outperforms task-specific LoRA, which has not been observed in the standard fine-tuning scenarios.", "reasons_to_reject": "A. The comparison with the state-of-the-art continual learning method [1] is missing. Particularly, in some evaluations, the state-of-the-art method outperforms the proposed method.\n\nB. The paper lacks in-depth analysis and comparison, including hyper-parameter optimality / sensitivity, forward / backward transfer, and generalization effects to unseen sequential tasks (as the authors mentioned that these effects were a unique strength of the proposed method).\n\n[1] Razdaibiedina et al., Progressive Prompts: Continual Learning for Language Models, ICLR 2023.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the problem of catastrophic fogetting in continual learning. The authors propose O-LoRA, which trains a task-specific low-rank adapter in an orthogonal direction from the previously learned adapters to minimize the effect to them. The experimental results in the paper show that the proposed method outperforms various continual learning methods.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Kp5xZJLniB", "reviewer": "Reviewer_7Yor", "summary": "", "questions": "Please refer to reasons to reject above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for providing feedback. \n\nWe kindly request a few minutes of your time to carefully consider our clarifications and responses.\n\n\n\n> R1: **About  OLoRA VS separate LoRA**.\n\nLet's address the core concern first, specifically highlighting the advantages of our continual learning method over training separate LoRA models for different tasks.\n\n**1. Real-World Dynamic Environments:** In real-world scenarios, new tasks or variations of tasks often emerge over time. Training a separate LoRA model every time is both computationally costly and impractical, especially when quick adaptation is essential.\n\n**2. Knowledge Transfer:** Continual learning allows a model to transfer knowledge from one task to another. This can often lead to improved performance on new tasks, especially when the new tasks are related to previously learned tasks. Separate models don't have this advantage.\n\n**3. User Interface:** O-LoRA integrates all tasks into one model, interacting only through instructions, bypassing the need for task IDs. In contrast, task-specific LoRA demands explicit task IDs during inference, which restricts its usability and practicality.\n\n**4. Inference Efficiency :** Using separate models for each task, even with LoRA, can significantly increase inference time. Consider a system like ChatGPT: if every query required a different model, the response time would be impractical. Continual learning consolidates this, ensuring faster and efficient answers without switching between models.\n\nIn conclusion, while it is technically possible to train separate models for each task, Continual learning  [1] [2] presents a more efficient, practical, and sustainable approach, especially for scenarios with evolving tasks and constraints. \n\n\n\n**About catastrophic forgetting**\n\nCatastrophic forgetting, as described in [3] [4], is a phenomenon wherein a neural network, while learning new tasks, inadvertently overwrites or conflicts with the knowledge from earlier tasks. This interference results in reduced performance on those initial tasks. It is imperative to address this issue [1] [2], as neglecting it undermines the core principle of continual learning: the capacity to consistently amass knowledge without detrimentally affecting prior knowledge. Research by [6] illustrates that as the skills of models like ChatGPT expand, their proficiency in previously acquired skills tends to degrade.\n\n\n\n> R2: About ablation study of instructions tuning and orthogonal LoRA. \n\n**We didn't exclude the instruction in our experiments due to its role as a soft task identifier.** Given the same input, the intended tasks (e.g., sentiment classification, entity recognition, translation) could vary. Without the instruction, the language model would lack task specificity. Hence, it's essential for our approach.**To ensure fairness and account for potential instruction effects, all baseline methods utilized the same instruction format as O-LoRA.** The results presented in Table 2 demonstrate that the observed enhancement from O-LoRA is attributable to the method itself, rather than the influence of the instruction.\n\nIn Table 2, we present a comparison between two baselines: the method of individually training LoRA for each task (denoted as PerTaskFT) and the method of sequential LoRA learning (denoted as SeqLoRA). Let's revisit these results for clarity:\n\n```markdown\n|                    |             Standard CL Benchmark         |\n|                    | Order-1      | Order-2     | Order-3      |\n| ------------------ | ------------ | ----------- | ------------ |\n| SeqLoRA            | 44.6         | 32.7        | 53.7         |\n| PerTaskFT          | 70.0         | 70.0        | 70.0         |\n| O-LoRA             | 77.0         | 77.2        | 77.1         |\n```\n\n**SeqLoRA vs. O-LoRA**: The unconstrained sequential task learning approach, SeqLoRA, consistently underperforms compared to O-LoRA. This stark contrast underscores the severity of the catastrophic forgetting phenomenon, emphasizing its importance and urging attention.\n\n**PerTaskFT vs. O-LoRA**: When contrasted with the PerTaskFT method, which trains LoRA separately for each task, O-LoRA capitalizes on inter-task knowledge correlations. This leads to enhanced transfer capabilities and consequently better overall performance.\n\n\n\n\n\n**Key References :**\n\n- [1] Wang, Liyuan, et al. \"A comprehensive survey of continual learning: Theory, method and application.\" arXiv preprint arXiv:2302.00487 (2023).\n- [2] Ke, Zixuan, and Bing Liu. \"Continual learning of natural language processing tasks: A survey.\" *arXiv preprint arXiv:2211.12701* (2022).\n- [3] McClelland, James L., Bruce L. McNaughton, and Randall C. O'Reilly. \"Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory.\" *Psychological review* 102.3 (1995): 419.\n- [4] McCloskey, Michael, and Neal J. Cohen. \"Catastrophic interference in connectionist networks: The sequential learning problem.\" *Psychology of learning and motivation*. Vol. 24. Academic Press, 1989. 109-165.\n- [5] Wu, Tongtong, et al. \"Pretrained language model in continual learning: A comparative study.\" *International Conference on Learning Representations*. 2021.\n- [6] Chen, Lingjiao, Matei Zaharia, and James Zou. \"How is ChatGPT's behavior changing over time?.\" *arXiv preprint arXiv:2307.09009* (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The field of learning new tasks without leading to catastrophic forgetting is an interesting and challenging one. The proposed method seems to yield strong results.", "reasons_to_reject": "The main idea behind this paper seems to be flawed. Indeed, the authors argue that learning new tasks can lead to catastrophic forgetting, and that using orthogonal LoRA is necessary to avoid it. However, in the present setting, each task is learnt using its own set of LoRA matrices. How can the model forget, then, when its parameters are kept fixed? Why do we need each LoRA subspaces to be orthogonal, as, when we learn a new task, we keep every parameters linked to other tasks fixed?\n\nAnother major issue comes from the fact that this method is actually a combination of two: instructions tuning and orthogonal LoRA. Yet no ablation study seems to have been performed to investigate the contribution of each of them. My intuition is that orthogonal LoRA should perform similarly as regular LoRA on its own, given the idea described above.\n\nEdit: the authors have clarified the above objections, I have updated my scores accordingly.", "questions_to_authors": "Please refer to reasons to reject above.", "paper_topic_and_main_contributions": "This paper explores combining instructions tuning with orthogonal LoRA for multi-tasks learning. The authors keep each LoRA vector subspaces learnt for each task orthogonal to each others, with the aim of limiting forgetting previous tasks when learning a new one.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose O-LoRA, a new method for continual learning with large language models. For each new task, a new set of LoRA parameters is learned, but, different than the straightforward IncLoRA baseline, an orthogonality regularizer encourages the LoRA parameters to add new information compared to prior sets of parameters. Experiments vs. baselines demonstrate promising performance improvements. In addition, the proposed method is parameter efficient, and doesn't require task replay, which are important practical considerations, e.g., for privacy reasons.

Reviewers were mixed in their assessments, even after significant discussion.

A summary of the more negative leaning reviews/associated discussion:

===
Reviewer jxjZ maintains some concerns, including: 1) are the baselines LoRA or not? --- the authors say that "except for prompt-based methods such as L2P and LFPT5, all others are implemented based on LoRA" but the reviewer reasonably points out that the most plausible read of the results "suggests that all model parameters are fine-tuned for EWC and Replay." (which might cause more forgetting than just using LoRA); 2) How will the promised SAM-based results compare?; and 3) Were hyperparameters like LoRA r were optimized for the baseline?
~
Reviewer GrPc shares jxjZ's concerns about hyperparameter configuration optimization, but also, mentions yet another missing method from ICLR 2023 from Razdaibiedina et al. 2023 (which the authors promise to compare to, but experiments are not finished).

===

Among these concerns, I find the points about the baselines most compelling: it's not clear if they are LoRA versions (the authors didn't reply to the reply from jxjZ) and, as both reviewers mention, it's not clear how hyperparameters of baselines were optimized, nor is it clear how the author's method will compare to several missing baselines which the reviewers expected.

Overall: the authors could likely address many of the above concerns in revision for a potential camera ready, but I am hesitant to give a higher rating without seeing 1) the additional clarifications requested by jxjZ; 2) more details of how the baselines were hyperparameter tuned; and 3) how the missing baselines perform.
Decision: Accept-Findings