SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Models (2024)

Abstract: The continual learning (CL) ability is vital for deploying large language models (LLMs) in the dynamic world. Existing methods devise the learning module to acquire task-specific knowledge with parameter-efficient tuning (PET) block and the selection module to pick out the corresponding one for the testing input, aiming at handling the challenges of catastrophic forgetting and knowledge transfer in CL. However, these methods tend to address only one of the challenges, ignoring the potential of aligning the two modules to effectively address catastrophic forgetting and knowledge transfer simultaneously. To this end, we propose a novel Shared Attention Framework (SAPT), to align the PET learning and selection via the Shared Attentive Learning & Selection module. Extensive experiments on two CL benchmarks demonstrate the superiority of SAPT. Moreover, SAPT consistently demonstrates its superiority when we scale it to different model sizes (from 770M to 13B), different model architectures (T5 and LLaMA-2) and unseen tasks.1

heading: 1 Introduction
text: Endowing the continual learning (CL) ability for large language models (LLMs) (Brown et al., 2020; Raffel et al., 2020; Touvron et al., 2023) to learn different tasks sequentially is crucial for their deployment in the real-world, which allows them to dynamically adapt to novel tasks and acquire additional knowledge (Luo et al., 2023; Zhai et al., 2023; Wu et al., 2024). However, this scenario presents two significant challenges: (1) Catastrophic Forgetting (CF), referring to the loss of previously acquired knowledge when learning new tasks (McCloskey and Cohen, 1989), and (2) Knowledge Transfer (KT), involving the efficient
* Corresponding author 1Our source code is available at https://github.com/
circle-hit/SAPT.
ing of large language models based on PET blocks when the new Dialogue Generation task arrives. Dashed lines represent the working process of existing works while solid lines are for that of our SAPT in this work.
utilization of knowledge from past tasks to facilitate the learning of new ones (Ke and Liu, 2022).
Due to the heavy burden on computation resources, recent attempts study the CL of LLMs based on parameter-efficient tuning (PET) methods (Hu et al., 2021; Ding et al., 2022). Inspired by the parameter isolation CL methods (Rusu et al., 2016; Fernando et al., 2017), existing methods can be conceptualized as two pivotal components working in the pipeline fashion. As shown in Figure 1 (dashed lines), when a new Dialogue Generation task arrives, a private PET block is allocated by the learning module to acquire task-specific knowledge and then saved to the PET pool for the following selection module to pick it out when a test sample is coming. However, the designs of each module in current works exhibit certain limitations in effectively dealing with KT and CF challenges.
On one hand, the design of learning module is supposed to function to facilitate KT among different tasks. Unfortunately, for existing works, the learning of PET block is either performed seperately within each single task (Wang et al., 2023b),
11641
or kept orthogonal to each other to minimize interference (Wang et al., 2023a). Such isolation cuts off the potential transfer of acquired knowledge stored in the previous PET blocks and hinders them to assist the current acquisition of new knowledge.
On the other hand, the selection module plays the pivotal roles in mitigating CF because only when it is capable of automatically selecting the PET block to which the current input belongs can the LLM backbone successfully accomplish the current task. However, it would make LLMs vulnerable to CF by simply implementing such selection process via the summation (Wang et al., 2023a) or concatenation (Razdaibiedina et al., 2023) of all existing PET blocks or selecting them from a fixed PET pool (Wang et al., 2022b).
More importantly, they ignore the opportunity of aligning the two modules to address challenges of CF and KT simultaneously. The intuition is that (illustrated by solid lines in Figure 1), in order to facilitate KT in the learning of the new task, the learning module should rely on task correlations to leverage the most relevant knowledge in previous PET blocks. And such attentive process, expressed as shared attention in our study, could be naturally repeated by the selection module to resist CF through the combination of the corresponding PET blocks belonging to each testing input. As a result, the end-to-end alignment of these two modules is established via such shared attention.
To this end, we propose a novel Shared Attention Framework for Parameter-efficient conTinual learning (SAPT) of large language models. In SAPT, the Shared Attentive Learning & Selection Module (SALS) is devised, where each training sample is navigated to utilize the optimal combinations of existing PET blocks for completing the current task. This is achieved through an attention weight obtained via instance-level shared attention operation. Then inputs in the testing time are capable of following the same shared attention operation to reach the attention weight and pick out the appropriate PET blocks accordingly.
However, continually updating the SALS leads to the optimal attentive combination only for the newest task, resulting in the forgetting for that of previous ones. Thus, we introduce Attentive Reflection Module (ARM) to help SALS recall what the shared attention operation of inputs from previous tasks should be originally performed with pseudo samples. And the success of ARM offers a new perspective for the utilization of generated
pseudo samples instead of just blindly mixing them with samples of new tasks for multi-task training.
We conduct extensive experiments to evaluate SAPT on SuperNI (Wang et al., 2022a) and Long Sequence (Razdaibiedina et al., 2023) benchmarks. State-of-the-art performance is achieved by SAPT compared with recent PET-based CL methods. Moreover, SAPT also exhibits superior performance when we scale it to different model sizes (from 770M to 13B), different model architectures, including T5 (Raffel et al., 2020) (encoder-decoder) and LLaMA-2 (Touvron et al., 2023) (decoderonly) and previously unseen tasks.
The main contributions of this work are summarized as follows:
• We propose a novel framework SAPT, including SALS and ARM, to align the PET learning and selection process to effectively handle the CF and KT challenges simultaneously.
• A novel perspective for the utilization of pseudo generated samples is offered in ARM, exhibiting both improved effectiveness and efficiency than naive (generative) replay.
• Results of extensive experiments on the benchmark datasets demonstrate the effectiveness of SAPT to mitigate CF and facilitate KT.

heading: 2 Related Works

heading: 2.1 Parameter-Efficient Tuning
text: Recently, parameter-efficient tuning (PET) (Ding et al., 2022) has become an appealing research topic which aims at minimizing computational resources when adapting LLMs to specific tasks. Various approaches have emerged in this field, including adapter (Houlsby et al., 2019), prompt-based tuning (Lester et al., 2021; Li and Liang, 2021), BitFit (Zaken et al., 2022) and LoRA (Hu et al., 2021). Since LoRA has exhibited superior performance compared to many mainstream PET methods, our experiments will primarily concentrate on LoRA as a representative method. To ensure a fair comparison with previous prompt-based methods, we also implement a prompt-version of SAPT.

heading: 2.2 Continual Learning for LLMs
text: Conventional Continual Learning (CL) are divided into three categories. (1) Rehearsal-based methods introduce the fixed memory to store real samples (Lopez-Paz and Ranzato, 2017; Isele and
LLM
Attentive Learning Attentive Selection
Query Projection
LLM
Shared Attentive Leraning & Selection Module
Task
Task
Weighted Combination
Weighted Combination
Attention
Fine-tune Frozen
Attentive Reflection Module
Task
Attention
Query Projection
Attention Attention
Pseudo Pseudo
Figure 2: The overall architecture of our proposed SAPT. We assume that SAPT is currently at the time step 3 to learn the task T3. (1) In the SALS, as illustrated by the solid lines, the resulting attention weight a3 of task T3 is first obtained via the instance-level shared attention operation between the input x3 and PET key vectors {k1,k2,k3}, to perform weighted combination of all PET blocks {B1, B2, B3} for the attentive learning of the current task T3. And dashed lines display the process of attentive selection, following the same process of shared attention to reach the attention weight a3 and utilizing it to handle given inputs at the testing time. (2) In the ARM, for previous tasks T1 and T2, the current attention weights of them (â1 and â2), are pulled back to their original states (a1 and a2), with the introduction of generated pseudo samples x̂1 and x̂2.
Cosgun, 2018) or pseudo-generative examples (Shin et al., 2017; Sun et al., 2019) of previous tasks. (2) Regularization-based methods impose constraints on the loss function to penalize changes regarding the knowledge of previous tasks (Kirkpatrick et al., 2017; Li and Hoiem, 2017; Farajtabar et al., 2020; Wu et al., 2022; Chen et al., 2023). (3) Parameter isolation methods dynamically expand model capacity or isolate existing model weights to mitigate interference between new and old tasks (Rusu et al., 2016; Fernando et al., 2017).
Continual Learning for LLMs with PET. Based on PET methods, current approaches for the CL of LLMs inherit the idea of parameter isolation, exhibiting a pipeline fashion to learn and select PET blocks for each task. However, most of them assume task-ids are available at testing time so that they directly use the oracle PET block of each task and just skip the selection process (Qin and Joty, 2022; Zhang et al., 2022; Qin et al., 2023). These lines of works simplify the problems of CL and could not be applied for real-world application of LLMs where the task-ids are unavailable. Thus, another branches of attempts focus on the more practical settings where the process of PET selection must be involved due to the unavailable task-ids during testing time. However, they are limited in effectively dealing with CF and KT challenges. For the PET learning, Wang et al. (2023b)
allocate private prompt for each task and Wang et al. (2023a); Smith et al. (2023) constrain the learning of PET block to keep orthogonal. They restrict the knowledge transfer among different tasks. And simply implementing the PET selection via the summation (Wang et al., 2023a) or concatenation (Razdaibiedina et al., 2023) of all existing PET blocks or select them from a fixed pool (Wang et al., 2022b) make LLMs vulnerable to CF.
Our proposed SAPT stands out from them in that we attempt to align the learning and selection of PET blocks so that CF and KT can be effectively addressed simultaneously.

heading: 3 Problem Definition and Setup
text: Continual learning seeks to address challenges within ongoing sequences. Formally, a sequence of tasks {T1, . . . , TT } arrive in a streaming fashion. Each task Tt = {( xit, y i t )}nt i=1
contains a separate target dataset with the size of nt. At any time step t, the model not only needs to adapt to the t-th task, but also keep performances on all previous tasks.
In this study, we delve into the more challenging and practical settings, addressing: (1) Diverse task types: Unlike previous approaches that merely focus on classification problems (Wang et al., 2023a,b), the model would encounter a sequence of tasks encompassing various types, such as dialogue generation, information extraction, etc.
(2) Absence of task identifiers: During the testing phase, the model confronts samples without knowing which specific task they belong to.

heading: 4 Methodology

heading: 4.1 Overview of the Framework
text: We propose SAPT, a novel framework for the CL of LLMs, offering an effective solution to address the challenges of catastrophic forgetting (CF) and knowledge transfer (KT) simultaneously. The overall architecture of SAPT is illustrated in Figure 2, comprising two key components: (1) Shared Attentive Learning & Selection Module (SALS) and (2) Attentive Reflection Module (ARM). In SALS, attentive learning (solid lines) and attentive selection (dashed lines) are aligned through the shared attention operation. Then in ARM, we assist SALS in recalling the exact attentions of inputs from previous tasks with generated pseudo samples.

heading: 4.2 Shared Attentive Learning & Selection Module
text: We devise the SALS module to align the learning and selection processes for PET blocks, where challenges of catastrophic forgetting and knowledge transfer could be effectively addressed.
PET Methods. We adopt two representative PET methods, Prompt Tuning (Lester et al., 2021) and LoRA (Hu et al., 2021) in SAPT. The additional trainable parameters introduced by them are referred to as PET blocks. Please refer to Appendix A for more details of the two PET methods.
Attentive Learning. In order to transfer the knowledge acquired from previous tasks, when the t-th task arrives, parameters of all previous PET blocks {B1, B2, . . . , Bt−1} and the current one Bt are aggregated via weighted combination for the attentive learning of task Tt. Specifically, we allocate a key vector ki for each PET block Bi (i ∈ [1, t]) and calculate instance-level input-key attentions.2 Such input-key attention ensures the process of attentive learning to be PET-agnostic and compatible with both prompt tuning and LoRA in SAPT.
The process of shared attention begins when the j-th input of the current t-th task passes through the embedding layer of the LLM backbone to obtain Ejt (we will omit the superscripts j for simplicity). Since Et ∈ Rm×d and each key vector ki ∈ Rd
2 This process is called shared attention because it will be repeated by the following attentive selection.
are of different sequence length, we first perform the max-pool operation on the length dimension of Et, and obtain et ∈ Rd. Then et is fed to a sub-network to project it as a query into the spaces of the key vectors for better feature alignment. This consists of down and up projection:
hdownt = W down(et)
h up t = W up(NonLinear(hdownt ))
qt = LayerNorm(h up t )
(1)
where W down ∈ Rdp×d and W up ∈ Rd×dp are learnable projection parameters. Following Asai et al. (2022), we use SiLU (Elfwing et al., 2018) for the non-linear and apply Layer Norm (Ba et al., 2016) on hupt to stabilize the learning process.
Then, the attention weights at = {a1, a2, . . . , at} are calculated by the product between qt and each ki with softmax:
ai = eqtki/T∑t i=1 e qtki/T (2)
where T is a temperature factor to avoid making the attention weights over-confident and hindering the knowledge transfer. And the parameters of aggregated PET blocks can be obtained:
θB =
t∑
i=1
ai θBi (3)
where θBi is the parameters of PET block Bi. The training loss for the attentive learning of the
current task Tt is: Ltask = − ∑
(xt,yt)∈Tt logP
( yt | xt; θm, θB, θproj, θk )
(4) where θm, θB, θproj and θk are parameters of the LLM backbone, the aggregated PET block, the query projection layer and the set of all key vectors, respectively. And only those parameters belongs to the current t-th task are updated during the training, including θBt , θproj and θkt .
Attentive Selection. During the inference phase, when testing data from different tasks arrives, the correct PET blocks are supposed to be automatically selected to execute the corresponding tasks. Within the preceding attentive learning, each sample has already been guided to the optimal combinations of existing PET blocks through shared attention. Thus, the attentive selection process is inherently supposed to follow the same attention operation to pick out the relevant PET blocks for
the testing input accordingly. To be more specific, attentive selection involves the same computation process of Equations (1) - (3).
Shared Attentive Learning & Selection. In summary, the shared attention succeeds to align the attentive learning and selection of PET blocks, leading to the shared attentive learning & selection that is of the same computation process and exhibiting promising insights to deal with the CF and KT challenges simultaneously.

heading: 4.3 Attentive Reflection Module
text: With the sequential training of different tasks, the query projection layer in Equation (1) is continually updated. The introduction of the Attentive Reflection Module ensures that inputs from previous tasks can still correctly perform the corresponding shared attention to identify the combination of PET blocks specific to each of them. To achieve this, we employ generative replay to constrain the projection layer with pseudo-samples. This approach ensures that no real samples are involved, thereby saving the cost associated with maintaining a fixed memory (Sun et al., 2019; Qin and Joty, 2022).
At each time step t, a PET block Breft is trained to reconstruct input samples of task Tt. For each sample (input-output pair), only the input part is generated conditioned on an initial token [Gen]. Thus, we have { Bref1 , B ref 2 , . . . , B ref t } to obtain the generated pseudo-samples {G1, G2, . . . , Gt} (generated examples could be found in Appendix E.1).
To assist the query projection layer to reflect or recall the correct shared attention for samples from previous tasks at time step t, every instance x̂i from Gi is fed to the query projection layer and performs input-key attention operation following Equation (1) - (2) to obtain the current attention weight âi. To pull âi to what it should originally be, we minimize a KL divergence loss:
LKL =
t−1∑
i=1
n̂i∑
j=1
DKL(âi||ai) (5)
where n̂i is the number of pseudo samples from Ti. Here, ai is the average attention weights of the test samples from Ti, representing the overall attention weight of it. Notably, ai is preserved immediately after the completion of learning Ti, and the position of (i, t] in ai is padded with 0 when it participates the calculation in Equation (5).
It is worth to mention that our ARM exhibits both improved effectiveness and efficiency than
naive (generative) replay, which is verified by the experimental results in the following Section 6.
Finally, we jointly minimize the task loss and the KL loss in the multi-task learning fashion:
L = Ltask + λLKL (6)
where λ is a hyper-parameter that functions to balance the two parts.

heading: 5 Experiments

heading: 5.1 Dataset and Evaluation Metrics

heading: 5.1.1 Dataset
text: SuperNI Benchmark (Wang et al., 2022a): a benchmark of diverse NLP tasks and their expertwritten instructions, enabling rigorous benchmarking of the more practical settings for the CL of LLMs. Specifically, in the types of dialogue generation, information extraction, question answering, summarization, and sentiment analysis, we select three tasks for each type, forming a sequence comprising a total of 15 tasks to evaluate various methods. For each task, 1,000 instances from the dataset are randomly sampled for training and 100 instances for validation and testing.
Long Sequence Benchmark (Razdaibiedina et al., 2023): a continual learning benchmark of 15 classification datasets. Following Razdaibiedina et al. (2023); Wang et al. (2023a), we select 1,000 random samples for training each task and hold out 500 samples per class for validation and testing.
We explore two different task orders for each benchmark. Please refer to Appendix B for more details about the tasks and orders.

heading: 5.1.2 Metrics
text: Let ai,j be the testing performance (Accuracy for classification task and Rouge-L (Lin, 2004) for others) on the j-th task after training on i-th task, the metrics for evaluating are:
(1) Average Performance (AP) (Chaudhry et al., 2018). The average performance of all tasks after training on the last task, i.e., AT = 1T ∑T t=1 aT ,t;
(2) Forgetting Rate (F.Ra) (Chaudhry et al., 2018) measures how much knowledge has been forgotten across the first T − 1 tasks, i.e., FT = 1
T −1 ∑T −1 t=1 (max T −1 k=i ak,t − aT ,t);
(3) Forward Transfer (FWT) (Lopez-Paz and Ranzato, 2017) measures how much knowledge from previous tasks transfers to a new task, i.e., FWTT = 1T ∑T t=1(at,t − a0,t), where a0,t refers to the performance of training task t individually;
(4) Backward Transfer (BWT) (Ke and Liu, 2022) measures how much the learning of subsequent tasks influences the performance of a learned task, i.e., BWTT = 1T −1 ∑T −1 t=1 (aT ,t − at,t).

heading: 5.2 Baselines and Comparison Models
text: We evaluate SAPT against the following PETbased continual learning baseline methods: (1) SeqLoRA: sequentially trains the LoRA on the task orders. (2) Replay: replays real samples from old tasks when learning new tasks to avoid forgetting. (3) L2P (Wang et al., 2022b): uses the input to dynamically select and update prompts from a fixed prompt pool. (4) LFPT5 (Qin and Joty, 2022): continuously trains a soft prompt for each task with generative replay and an auxiliary loss. (5) ProgPrompt (Razdaibiedina et al., 2023): sequentially concatenates previous learned prompts to the current one during the training and testing time. (6) EPI (Wang et al., 2023b): trains prompts for each task and selects them via the distance between the input and distributions formed by labels of different classification tasks. (7) O-LoRA (Wang et al., 2023a): learns tasks in different LoRA subspaces that are kept orthogonal to each other and sums all LoRA weights up at testing time.

heading: 5.3 Implementation Details
text: SAPT is a model- and PET-agnostic CL method that is compatible with any transformer-based generative LLM. In our experiments, all methods are performed with instruction tuning (Wei et al., 2021; Ouyang et al., 2022) to leverage the task instruction provided in the two benchmarks. To ensure a fair comparison with recent works, we implement SAPT with both prompt tuning and LoRA based on the pre-trained encoder-decoder T5-large
model (Raffel et al., 2020). We also scale SAPT to the backbone with larger model size (up to 11B and 13B) and the decoder-only LLaMA-2 model (Touvron et al., 2023). For the baselines, since they only report the AP metric in their original papers, we carefully re-implement them with their official codes to report metrics of F.Ra, FWT and BWT, providing a thorough insight of how existing methods deal with CF and KT. For more detailed settings, please refer to the Appendix C.

heading: 6 Results and Analysis

heading: 6.1 Overall Results
text: Table 1 demonstrates the performance comparison of SAPT and recent PET-based continual learning baselines on the SuperNI and Long Sequence benchmarks. All results are averaged over the two different orders of each benchmark. Detailed results of each order and each task within a specific order are provided in Appendix D.
Our SAPT could effectively deal with the challenges of CF and KT simultaneously. Compared to both prompt-based methods (SAPTPrompt v.s LFPT5/ProgPrompt/EPI) and LoRAbased methods (SAPT-LoRA v.s Replay/O-LoRA), SAPT performs better in addressing the two critical challenges, CF (highest AP and lowest F.Ra) and KT (highest FWT and BWT) when learning different tasks sequentially. Moreover, for the replaybased methods, the better performance of SAPT over Replay and LFPT5 offers a new perspective for the utilization of pseudo samples instead of just blindly mixing them with samples of new tasks for joint training. Please refer to Appendix E.2 for more detailed results and analysis regarding the utilization of replayed samples.
The alignment of learning and selection of PET is better than previous pipeline fashion. SAPT outperforms the state-of-the-art pipeline method, EPI, which verifies the effectiveness of aligning the learning and selection with a shared attention weight. Since EPI is specifically designed for the CL of classification tasks where the selection of PET is based on the label information of each task, it can not be directly applied to the SuperNI benchmark covering various types of tasks other than classification. This manifests that SAPT is more practical to the real-world applications of LLMs. In addition, the best results of SAPT in terms of AP and F.Ra demonstrate the great potential that such attention-guided soft selection of PET are more resistant to CF, compared with previous methods of concatenation (ProgPrompt), summation (OLoRA) and top-1 selection (EPI).

heading: 6.2 Visualization on Shared Attention
text: Figure 3 displays the heat maps for shared attention during the training and testing time. We can observe that: (1) the learning and selection processes of PET blocks are exactly aligned that the two heatmaps nearly have the same layout. (2) KT do happens in the attentive learning process to assist SAPT acquire new knowledge. These further verify the effectiveness of SAPT to deal with CF and KT. Please refer to Appendix F for more
discussions and visualization results.

heading: 6.3 Ablation Study
text: We conduct ablation studies to verify the effectiveness of different modules proposed in SAPT-LoRA. Results are shown in Table 2.
Effect of Attentive Reflection. After removing the attentive reflection module (“– ARM”, implemented by discarding the LKL), the significant decline highlights its crucial role in assisting different input samples to recall the correct shared attention for the corresponding PET blocks they should originally combine. When replacing ARM with naive Replay (“+ Replay”), the decline of F.Ra further verifies our claim that ARM offers a more effective solution to apply pseudo samples. Please refer to Appendix E.2 for more detailed results and analysis regarding the efficiency of ARM module.
Effect of the Alignment. We transform the alignment of PET learning and selection in SAPT into an independent format. This involves initially performing attentive learning to obtain weights that represent the combination of existing PET blocks. Subsequently, a separate PET selector is trained with these weights and generated pseudo samples. The comprehensive decline in model performance validates our claim that the learning and selection processes of PET are inherently capable of aligning together to collaborate seamlessly.
Effect of Shared Attentive Learning & Selection. Furthermore, we remove the shared attentive mechanism based on the above pipeline settings, where each PET block is learned within a single task and the selector are required to pick the most confident top-1 block for inference. The model’s performance has suffered significantly, especially in terms of knowledge transfer. This demonstrates that leveraging acquired knowledge comprehensively, whether in PET learning or selection, is crucial for effectively addressing CF and KT.

heading: 6.4 Power of Scale
text: Scale to larger backbone. We empirically analyze how increasing the backbone T5 size affects the performance of SAPT. Figure 4 displays the performance of SAPT, O-LoRA and Replay in terms of AP, F.Ra and FWT, based on the ascending backbone sizes, Large (770M), XL (3B) and XXL (11B). Overall, with the increased sizes of the backbone model, SAPT could always demonstrate superior performance over baseline models in resisting catastrophic forgetting and facilitating knowledge transfer. It is worth noting that even with the largest backbone model, O-LoRA (11B) still falls short in terms of Average Performance compared to the smallest version of SAPT-LoRA (770M). This further underscores the crucial importance of selecting the pertinent PET blocks for each input sample in real-world application scenarios.
Scale to different architectures. The results of SAPT and baseline methods on the SuperNI Bench-
mark based on different sizes of T5 and LLaMA-2 are shown in Figure 5. It can be observed that SAPT is capable of effectively mitigating CF and promoting KT across different model architectures. Moreover, the average performance improves with the enhancement of the model’s basic capabilities (LLaMA-2 > T5). This further demonstrates the generality of our proposed SAPT. Please refer to Appendix G for more detailed results.
Scale to unseen tasks. We further select 3 tasks from each one of the above task category to assess the SAPT’s cross-task generalization ability. This is also a crucial dimension for evaluating CL algorithms. Table 3 shows the results. T5-ZS represent the zero-shot approaches for task adaptation, respectively. SAPT yields the best performances, which can be attributed to its superiority in effectively combining acquired knowledge to address novel tasks. This suggests that we should actively promote knowledge transfer between different tasks during the process of CL.

heading: 7 Conclusion
text: In this paper, we propose SAPT, a novel framework for the parameter-efficient continual learning
of LLMs. In SAPT, we ingeniously align the two key processes of parameter-efficient block learning and selection through the shared attention, allowing it to effectively alleviate catastrophic forgetting and promote knowledge transfer simultaneously. More importantly, SAPT works under the practical settings where no task-ids are provided for the inputs to select their corresponding parameters. Experimental results also demonstrate the applicability of SAPT across different parameter-efficient tuning methods, models of varying scales and architectures, highlighting its universality.

heading: 8 Limitations
text: There are several limitations to consider for future directions of continual learning of large language models. Firstly, when the learning sequence scales to hundreds of tasks, continually expanding the PET pool to allocate a PET block for each one of them would lead to large computation and storage costs. Thus, how to prune and merge similar PET blocks in the continual learning process can be an interesting direction to explore. Secondly, although SAPT exhibits the best performance of Backward Transfer (BWT), it still fails to allow subsequent tasks to impose the positive impacts on the learned ones. This could be a critical direction to further explore more advanced CL methods. Finally, even though our approach do not depend on identifying task-ids during the testing phase, it still necessitates the identification of tasks during training to establish distinct PET parameters for each task. Investigating techniques for training that is independent of task identification could prove to be a promising avenue for future research, which could favor the application of continual learning upon on the online streams of data.

heading: Acknowledgements
text: We thank the anonymous reviewers for their comments and suggestions. And we thank Libo Qin for the suggestions on the writing of this work.

heading: A Parameter-Efficient Tuning Methods
text: We adopt two representative PET methods, Prompt Tuning (Lester et al., 2021) and LoRA (Hu et al., 2021) in our proposed SAPT, which are referred to as PET blocks in this study.
In prompt tuning, a series of virtual tokens, called soft prompt P is prepended to the input text x, while keeping the LLM parameters frozen. In this case, during the training on the downstream tasks, gradient updates are preformed on the prompt parameters independently.
In LoRA, the pre-trained weight matrix of LLMs is updated with a low-rank decomposition. For a linear layer h = W0x, the forward pass with LoRA is modified to be:
h = W0x+BAx (7)
where W0 ∈ Rd×k, B ∈ Rd×r, A ∈ Rr×k, with the rank r ≪ min(d, k). The pre-trained weight matrix W0 remains fixed during training, while A and B contain trainable parameters.

heading: B Dataset Details
text: B.1 Datasets
Table 4 & 5 show details of the datasets we used for our experiments, along with their evaluation metrics. Overall, in SuperNI, we choose 3 tasks from dialogue generation (Dialog) (Zhang et al., 2018a; Zang et al., 2020; Peskov et al., 2020), information extraction (IE) (Santus et al., 2015; Nye et al., 2018; Mostafazadeh et al., 2020), question answering (QA) (Dasigi et al., 2019; Talmor et al., 2019), summarization (Sum) (Narayan et al., 2018; Gliwa et al., 2019; Kim et al., 2019) and sentiment analysis (SA) (Socher et al., 2013; Saravia et al., 2018), respectively.
For the Long Sequence benchmark, this includes five tasks from the standard CL benchmark (AG News, Amazon reviews, Yelp reviews, DBpedia and Yahoo Answers) (Zhang et al., 2015), four from GLUE benchmark (MNLI, QQP, RTE, SST2) (Wang et al., 2018), five from SuperGLUE benchmark (WiC, CB, COPA, MultiRC, BoolQ) (Wang et al., 2019), and the IMDB movie reviews dataset (Maas et al., 2011).
And unseen tasks from the SuperNI benchmark are displayed Table 6. They also from the five categories of Dialog (Wei et al., 2018; Cho and May, 2020; Aliannejadi et al., 2021), IE (Mausam et al., 2012; Zlabinger et al., 2020; Radev et al., 2020),
QA (Levy et al., 2017; Zhang et al., 2018b; Min et al., 2020), Sum (Henderson et al., 2014; Syed et al., 2020; Hasan et al., 2021) and SA (Sheng and Uthus, 2020; Lowphansirikul et al., 2020).
B.2 Task Sequence Orders
We report 4 different task orders used for our experiments in Table 7.
C Implementation Details
Our experiments are implemented with PyTorch (Paszke et al., 2019) and Transformer library (Wolf et al., 2020). The T5-Large is trained on a single NVIDIA Tesla A800 GPU and the larger backbones T5-XL, T5-XXL, LLaMA-2-7B and LLaMA-2-13B are performed on 4 NVIDIA Tesla A800 using DeepSpeed repository.
For our prompt-based methods, the length of prompts is set to 10. Following Lester et al. (2021), they are initialized from sampled vocabulary of the backbone model and trained using the Adafactor optimizer. On the SuperNI benchmark, we train SAPT-Prompt with 100 epochs, the constant learning rate of 3e-2 and the batchsize of 32 per GPU. As for the hyper-parameter λ in Equation (6), it functions to balance the share attention in the process of attentive learning for the newest task and that in the process of attentive reflection for previous tasks. The larger λ means that the attentive reflection contributes more to assist SALS in recalling the shared attention of previous tasks. However, excessive λ can impair attentive learning for the current task, thereby weakening knowledge transfer. Here, λ is set to 1, which is the relatively optimal balance of the attentive learning and reflection. The hidden dimension dp of the query projection layer is 100. On the Long Sequence benchmark, the model is trained for 10 epochs with a hierarchical learning rate, 3e-1 for prompts and 1e-2 for the query projection layer. We always keep the total batchsize to 32. And the λ and dp for order3 and order4 is (1.5, 200) and (1.3, 150), respectively. The attention temperature in Equation (2) is d× exp(1), where d is the LLM backbone dimension size.
For our LoRA-based methods, we use AdamW optimizer to train the model with the learning rate of 3e-4 for T5-Large, 1e-4 for those larger T5-XL and T5-XXL models, 5e-5 for LLaMA-2-7B and 1e-5 for LLaMA-2-13B. For T5 series, the batch size is set to 32 per GPU. On the SuperNI benchmark, the low rank r, λ and dp are 4, 0.5 and 100,
while they are set to 8, 0.1 and 100 for the Long Sequence benchmark. For LLaMA-2 family, and the batch size is 32 in total. The low rank r, λ and dp are both 4, 2 and 100 for the Superni and Long Sequence benchmarks. The attention temperature in Equation (2) is sqrt(d), where d is the LLM backbone dimension size.
To obtain pseudo samples for our ARM, the prompt length is 300 and is trained for 80 epochs utilizing Adafactor with learning rate of 0.5. And in LoRA, the low-rank r is 8. We train it with AdamW with the learning rate of 0.001 for 5k steps. Batch size is set to 16 for both methods.
Further, we carefully evaluate the official implementations of all baselines, in order to make the comparison as fair as possible. We strictly follow the hyper-parameter settings in their original code, where the prompt size is all set to 10 (except that for LFPT5 of 300) and the LoRA rank is set to 8. If this could not reach the expected performance, we carry out the hyper-parameter search of the learning rate and batchsize for them. Following Sun et al. (2019); Qin and Joty (2022), the volume of replay samples is 0.02 of the original training set for SAPT and all replay baseline methods (Replay and LFPT5). Please refer to Appendix E.2 for deeper analysis for the volume of pseudo samples. All the methods are evaluated for 3 random runs.

heading: D Fine-grained Results for the Main Experiments
text: We report the results of each task order on the two benchmark in Table 8 and Table 9. And results of the average performance at each time step is displayed in Figure 8. Overall, the our proposed SAPT demonstrates excellent capabilities in addressing CF and KT.

heading: E More Results and Analysis on Generated Pseudo-Samples
text: E.1 Examples of Pseudo Samples Table 12 shows several pseudo samples generated by SAPT for the SuperNI an Long Sequence Benchmark. Since there are tasks instructions in these two benchmarks, the input-output format of real samples is consists of three elements: [INS] task instruction, [IN] task input and [OUT] task output. And we only generate the input part, [INS] and [IN], to perform attentive reflection in SAPT, which is a novel ways of pseudo-samples usage and greatly different from previous works where
complete pseudo samples are generated and mixed with the current task data for multi-task learning. We can see that SAPT can indeed generate highquality pseudo samples to assist samples from previous tasks in correctly identify the combination of PET blocks specific to each of them.
ARM’s efficiency is demonstrated by its need to generate only the input part of samples, unlike previous generative replay methods (Sun et al., 2019; Qin and Joty, 2022) that required generating complete (input-output) pairs.
E.2 Different Volumes and Types of Replayed Samples
In SAPT, the Attentive Reflection Module (ARM) provides a novel perspective for utilizing generated pseudo-data. We conduct additional experiments to analyze the impact of using varying scales of pseudo-data and real data on SAPT and the baseline models Replay and LFPT5. The results are shown in Figure 6. We have the following two observations that are worth to discuss:
(1) Regardless of whether real data or pseudodata is used, SAPT demonstrates computational efficiency during replay, showing superior performance even with the minimum replay scale 2% compared to the maximum replay scale 100% of LFPT5 and Replay. It is worth mentioning that when the replay data volume of Replay is 100%, it corresponds to the setting of multi-task learning, which is commonly considered as the upper bound of continual learning. SAPT is able to surpass this upper bound, demonstrating its ability to flexibly handle different inputs, enabling them to be processed by corresponding parameters.
(2) For SAPT, there is no significant difference in performance between using real data and pseudodata. This firstly indicates the reliability of the pseudo-data we generated and the sufficient robustness of our proposed ARM, which can utilize pseudo data of different qualities to accomplish reflection on shared attention.
F Visualization on Shared Attention
We demonstrate the visualization on shared attention operation of SAPT-Prompt on the SuperNI (Figure 9) and the Long Sequence (Figure 10) Benchmark, and the SAPT-LoRA on the SuperNI (Figure 11) and the Long Sequence (Figure 12) Benchmark. And the resulting attention weights is obtained through the average attention weights of
the testing samples from a specific task. Overall, whether based on Prompt or LoRA, SAPT can maintain the alignment for the learning and selection process of PET blocks through shared attention on both benchmarks. Even as the task sequences become longer, it does not affect the ability to identify suitable combinations of PET modules. This directly demonstrates its effectiveness in addressing CF and KT.
Furthermore, both methods demonstrate varying degrees of knowledge transfer on the two benchmarks. Overall, the PET blocks in the current task contribute more significantly, as indicated by the darkest color of the diagonal elements. However, there are also interesting observations where the PET blocks for other tasks have weights higher than the current task, surpassing the higher similarity between these tasks (yelp & amazon, mnli & cb). Additionally, the knowledge transfer of
Prompt appears slightly more pronounced than LoRA, but overall, LoRA outperforms Prompt in terms of the overall performance. This may be attributed to LoRA’s superior representation and learning of task-specific knowledge in the low-rank space, aligning with the conclusions in previous works (Hu et al., 2021; Ding et al., 2022).

heading: G Scale to LLaMA-2 Model
text: The results of SAPT and baseline methods on the Long Sequence Benchmark based on different sizes of T5 and LLaMA-2 are shown in Figure 7. It can be observed that our proposed SAPT still exhibits superiority to effectively mitigating CF and promoting KT over baseline methods.
Selecting O-LoRA as the baseline method for experiments based on LLaMA-2 is because it is the only work among numerous baselines that experimented with LLaMA-2 in the original paper, while other baselines are almost originally implemented with T5 or BERT in their paper. Here we additionally supplement the experimental results of EPI, LFPT5 based on LLaMA2-7B and -13B. Results are shown in Table 10 and Table 11.

heading: Future Work
text: There are several limitations to consider for future directions of continual learning of large language models. Firstly, when the learning sequence scales to hundreds of tasks, continually expanding the PET pool to allocate a PET block for each one of them would lead to large computation and storage costs. Thus, how to prune and merge similar PET blocks in the continual learning process can be an interesting direction to explore. Secondly, although SAPT exhibits the best performance of Backward Transfer (BWT), it still fails to allow subsequent tasks to impose the positive impacts on the learned ones. This could be a critical direction to further explore more advanced CL methods. Finally, even though our approach do not depend on identifying task-ids during the testing phase, it still necessitates the identification of tasks during training to establish distinct PET parameters for each task. Investigating techniques for training that is independent of task identification could prove to be a promising avenue for future research, which could favor the application of continual learning upon on the online streams of data