Mitigate Negative Transfer with Similarity Heuristic Lifelong Prompt Tuning (2024)

Abstract: Lifelong prompt tuning has significantly advanced parameter-efficient lifelong learning with its efficiency and minimal storage demands on various tasks. Our empirical studies, however, highlights certain transferability constraints in the current methodologies: a universal algorithm that guarantees consistent positive transfer across all tasks is currently unattainable, especially when dealing dissimilar tasks that may engender negative transfer. Identifying the misalignment between algorithm selection and task specificity as the primary cause of negative transfer, we present the Similarity Heuristic Lifelong Prompt Tuning (SHLPT) framework. This innovative strategy partitions tasks into two distinct subsets by harnessing a learnable similarity metric, thereby facilitating fruitful transfer from tasks regardless of their similarity or dissimilarity. Additionally, SHLPT incorporates a parameter pool to combat catastrophic forgetting effectively. Our experiments shows that SHLPT outperforms state-of-the-art techniques in lifelong learning benchmarks and demonstrates robustness against negative transfer in diverse task sequences.1

heading: 1 Introduction
text: Drawing on the remarkable capacity of humans to amass new knowledge throughout their lifetime, lifelong learning (LL) systems aim to emulate this progressive learning trajectory by sequentially mastering various tasks, each contributing to the system’s cumulative knowledge base. However, it is not trivial for deep learning models to achieve this ideal due to inherent challenges. These include the need to (1) avoid catastrophic forgetting - where the acquisition of new information can lead to the erosion of previously learned knowledge, and to
* Corresponding author. 1Source code is available at https://github.com/
wcyno23/SHLPT.
(2) promote efficient knowledge transfer - where the model can leverage past learning experiences to aid in the understanding and performance of future tasks. Addressing these challenges is crucial for the development of LL systems that can adapt and grow in a manner akin to human learning.
Recent advancements in lifelong learning of language model (LM) have integrated the concept of prompt tuning to enhance its capabilities. These approaches maintain the pre-trained model’s parameters, while training a small set of additional prompts to adapt the model to various downstream tasks. The efficiency and lightweight nature of prompt tuning align well with the demands of LL, sidestepping the need for the onerous storage of entire model versions for every new task. This technique facilitates the model to accumulate knowledge over time, adapt flexibly to new tasks, and recall how to perform on older tasks with the aid of task-specific prompts. Moreover, substantial ef-
10944
fort is dedicated to the transfer of knowledge from past tasks. This includes methods such as prompt concatenation (Razdaibiedina et al., 2022), parameter sharing (Wang et al., 2022b,c), and weighted summation (Smith et al., 2023; Jiang et al., 2023a), which are pivotal in ensuring that knowledge is effectively retained and utilized throughout the lifelong learning process.
Current transfer learning methods commonly presuppose that earlier tasks can positively impact succeeding tasks. Razdaibiedina et al. (2022) leverages prompts from previous tasks in the process of learning new ones, and similarly, Zhu et al. (2022) utilizes prompts from former tasks as a foundation to harness prior knowledge. Nevertheless, we suggest that when there is a considerable dissimilarity between tasks, these approaches don’t consistently assure positive transfer; on occasions, they might even provoke negative transfer. This phenomenon is vividly illustrated by the empirical analysis in Table 1, which shows the transfer efficiency among diverse tasks within different soft prompt learning. On the other hand, how to effectively utilize negative transfer during the lifelong learning remains an open question for research. In this paper, we will further explore strategies for customized transfer learning tailored to the characteristics of different tasks, aimed at achieving more efficient knowledge accumulation while mitigating the potential impacts of negative transfer.
To address the issue of inconsistent knowledge transfer, we introduce a new approach to lifelong prompt tuning, which we refer as to SHLPT (Similarity Heuristic Lifelong Prompt Tuning). First, we construct a prompt pool for learned tasks, thereby reducing the risk of forgetting. Then, we segment our knowledge transfer module into three components: (1) assessing the similarity between the current task and previous tasks, (2) categorizing the previous tasks into similar and dissimilar subsets, and (3) applying different transfer algorithms to each subset accordingly. For the initial step, we calculate an attention-weighted combination of past prompt embeddings and incorporate this into the current task’s prompt. During optimization, the model assigns higher attention scores to tasks that are more beneficial. Then, we utilize this attention score as task similarity metric and split task set accordingly. For tasks deemed similar, we integrate their parameters to provide the current task with an optimized starting point. Conversely, for tasks that are dissimilar, we introduce a variety
of innovative regularization techniques aimed at guiding the pre-trained model towards accessing a broader range of knowledge. This nuanced approach allows the model to better adapt to each new task while preserving and effectively utilizing the knowledge from all previous learning experiences without negative transfer.
In this paper, we make several notable contributions: (1) We address the novel challenge of mitigating negative transfer and facilitating knowledge transfer from dissimilar tasks in lifelong learning, which is particularly important in sequences of low-similarity tasks. (2) We present SHLPT, an innovative lifelong prompt tuning technique that reduces forgetting and enables knowledge transfer across tasks with varying degrees of similarity. Our extensive experiments demonstrate that SHLPT surpasses existing methods on benchmark datasets. (3) We introduce a challenging benchmark characterized by low task similarity, which typically results in increased negative transfer. Our approach exceeds the performance of previous state-of-the-art methods in this context.

heading: 2 Related Work
text: Lifelong Parameter Efficient Tuning. Parameter efficient tuning tunes a subset of parameters of pretrained language models, and can largely reduce the computation costs and memory usage (Houlsby et al., 2019; Li and Liang, 2021; Hu et al., 2021; Lester et al., 2021; Ben Zaken et al., 2022). For lifelong prompt tuning, LFPT5 (Qin and Joty, 2021) uses distillation loss and a generative replay to learn a continuous prompt. Progressive Prompts (Razdaibiedina et al., 2022) learns a new prompt for every task, and progressively concatenate it with previous prompts. LPT (Liang et al., 2023) employs a trainable binary mask on the overall prompt to selectively choose parameters for different tasks. L2P (Wang et al., 2022c) initializes a prompt pool and selects a certain number of prompts from it for each task. Through this approach, parameters among different tasks can be shared and isolated simultaneously. Base on this parameter pool architecture, CODA-prompt (Smith et al., 2023) employs an attention-based prompt selection strategy, Diana (Dai et al., 2023) and HiDe-Prompt (Wang et al., 2023a) decompose prompts into a hierarchical structure. In addition to lifelong learning on prompt tuning, O-LoRA (Wang et al., 2023b) introduces Orthogonal regularization to LoRA, facilitate
learning in mutually orthogonal subspaces for different tasks; CLASSIC (Ke et al., 2021b) adds task masks to the Adapter layer and utilizes contrastive loss to transfer knowledge between similar tasks. Similarity Heuristic Methods. Similarity heuristic lifelong learning methods use task similarity to identify which tasks can transfer knowledge and minimize interference from dissimilar tasks. CAT (Ke et al., 2020) compares the performance between transfer model and reference model to determine whether use the transfer model. B-CL (Ke et al., 2021c) and CTR (Ke et al., 2021a) utilize capsule network and routing algorithm to cluster similar tasks along with their shareable features. CLASSIC (Ke et al., 2021b) creates views form the hidden space information of previous similar tasks, and uses the contrastive loss to help current tasks to learn shared knowledge.

heading: 3 Preliminary

heading: 3.1 Lifelong Learning Setup
text: In lifelong learning, the language model will be sequentially finetuned across a series of 𝑛 tasks, labeled 𝑇1, ..., 𝑇𝑛. These tasks could originate from various domains and types. The training objective is to minimize the expected loss of all learned tasks with no access to data from previous tasks. Let 𝐷1, 𝐷2, ..., 𝐷𝑛 be the set of datasets corresponding to tasks 𝑇1, 𝑇2, ..., 𝑇𝑛, respectively. Each dataset 𝐷𝑖 encompasses a collection of data (𝑋 𝑖, 𝑗 , 𝑦𝑖, 𝑗), where 𝑋 𝑖, 𝑗 = [𝑥1, 𝑥2, ..., 𝑥𝑙] is the input text with length 𝑙, and 𝑦𝑖, 𝑗 is the corresponding output. The model is trained sequentially on these datasets with loss function 𝐿𝑖 specific to each task.
In this paper, we categorize LL scenarios into two types based on task similarity. In the first type (de Masson D’Autume et al., 2019; Razdaibiedina et al., 2022), the task sequence 𝑇1, ..., 𝑇𝑛 exhibits a relatively high degree of similarity, meaning that the likelihood of any task 𝑇𝑡 being significantly dissimilar from previous tasks 𝑇𝑖 (𝑖 < 𝑡) is low. The second type represents the opposite situation. The latter is more likely to emerge during the early stages of a real-world system when it has not yet accumulated a substantial knowledge base. We suppose that dissimilar tasks sequences are more prone to negative transfer, and for the first time, we create a lifelong learning benchmark composed of tasks that are dissimilar and likely to induce negative transfer for research purposes.

heading: 3.2 Prompt Tuning
text: Prompt tuning, as introduced by Lester et al. (2021), presents a resource-efficient methodology for adapting language models without necessitating extensive fine-tuning of the original model. This technique utilizes a small set of trainable parameters known as a prompt, denoted 𝑃 = [𝑝1, 𝑝2, . . . , 𝑝𝑙𝑝 ], which serves as a prefixed sequence to the input token embeddings 𝑋𝑒 = [𝑒(𝑥1), 𝑒(𝑥2), . . . , 𝑒(𝑥𝑙)]. Here, 𝑙𝑝 denotes the length of the prompt, 𝑒 symbolizes the embedding function, and each vector 𝑝𝑖 ∈ R𝑑 . For simplification, we use 𝑋 to replace 𝑋𝑒 below.
In the process of prompt tuning, the model employs the combined sequence [𝑃; 𝑋] ∈ R(𝑙𝑝+𝑙)×𝑑 as the input for the LM. The objective function for adapting to a specific downstream task is given by L𝐷𝑜𝑤𝑛 ( [𝑃; 𝑋]) = − log 𝑝 (𝑦 | [𝑃; 𝑋]), which seeks to maximize the likelihood of the correct output 𝑦 given the input embedding 𝑋 and the prompt vectors 𝑃. Notably, this fine-tuning procedure exclusively modifies the prompt parameters 𝑃, leaving the remaining weights of the LM fixed.

heading: 3.3 Empirical Study of Negative Transfer in Prompt Tuning
text: Definition of Negative Transfer. Transfer learning leverages data or knowledge from source tasks to enhance target task’s learning performance. However, the effectiveness of transfer learning is not always guaranteed (Zhang et al., 2022); the performance on the target task may even worsen after transfer learning, a situation that is referred to as negative transfer (Pan and Yang, 2009). Let S be one or more source tasks, T a target task, 𝜖T target task’s test error, 𝐴(S,T) a transfer learning algorithm between S and T , 𝐴(∅,T) the same algorithm without source tasks’ information. The test error reduction after transfer learning can be formulated as
𝑟T = 𝜖T (𝐴(∅,T)) − 𝜖T (𝐴(S,T)) (1)
A positive test error reduction 𝑟T indicates a successful transfer learning result, while a negative value indicates negative transfer. Negative Transfer in Prompt Tuning. Vu et al. (2022) shows many tasks can benefit each other under soft prompt transfer learning setting. In order to examine the negative transfer phenomenon in this setting, we conduct a empirical study on the transferability of soft prompt across a collection of
language tasks (details on the tasks are available in Appendix A, while experimental specifics can be found in Appendix C). Here, a prompt learned on one source task is transferred and used as the initial prompt for a target task. As shown in Figure 1, the negative transfer is a common occurrence in the setting of soft prompt transfer, particularly when the source and target tasks are highly dissimilar – for instance, Yahoo→RTE resulted in a 5.3% drop in accuracy (from 78.67 to 73.33). When the source and target task pairs are similar, transfers tend to yield positive results. For instance, Amazon→Yelp, both tasks involve sentiment analysis of reviews, leading to an 2.4% increase in accuracy. Interestingly, positive transfer can sometimes occur even between dissimilar tasks (e.g., CB→QQP). To better predict and avoid negative transfer, we design a similarity estimator that can discern transfer potential more effectively.
We further evaluated the few-shot (16 shot per class) performance of two task similarity agnostic transfer learning methods on four tasks from Standard CL Benchmark (de Masson D’Autume et al., 2019): AGNews, Yahoo, DBpedia, Amazon. Results in Table 1 indicate that these two methods also cannot guarantee positive transfer among all tasks (e.g., DBpedia→AG News, Amazon→Yahoo). Negative Transfer Benchmark. Based on above results, we have developed a benchmark for lifelong learning research that consists of tasks prone to negative transfer (Appendix B). This benchmark requires lifelong learning algorithms to effectively avoid negative transfer or further benefit from tasks involving negative transfer.
Theoretical Analysis. We further analyze the causes of negative transfer from theoretical bound
for domain adaptation (Ben-David et al., 2010) .
𝜖T (𝐴(S,T)) ≤ 𝜖S (𝐴(S,T)) + 1 2 𝑑HΔH (𝐷𝑠 , 𝐷𝑡 ) + 𝜆 (2)
where 𝐷 is domain distribution, 𝑑HΔH (𝐷𝑠, 𝐷𝑡 ) is the divergence between source domain S and target domain T , 𝜆 is a problem-specific constant. Based on the theoretical bound, negative transfer in LL can be attributed to the following reasons: (1) large error in the source tasks. This is caused by the few-shot setting or the learning difficulty of source tasks. (2) large distribution divergence. This is because the tasks are too dissimilar. (3) Unsuitable transfer learning algorithms. Transfer learning algorithms cannot adapt to all scenarios, an unsuitable algorithm may leads to negative transfer.

heading: 4 Method
text: To this end, we propose SHLPT (Similarity Heuristic Lifelong Prompt Tuning), which intends to achieve the following goals for a robust lifelong learning system: (1) avoid forgetting previous knowledge, (2) transfer knowledge from both similar and dissimilar tasks, (3) reduce the probability of negative transfer during the transfer stage.
SHLPT is composed of the following three structures: (1) A prompt pool preserving previous task’s prompts {𝑃1, 𝑃2, ..., 𝑃𝑡 } to avoid forgetting of past learning in a lifelong learning context. During the testing phase, identification of the task is necessary. (2) A task similarity estimator which can estimate the similarity between current tasks and previous tasks. We operate under the premise that "the more similar the tasks, the more effective the knowledge transfer." Based on estimated similarity, the collection of previous prompts is further categorized into two subsets: prompts from similar tasks 𝑃1𝑠 , 𝑃 2 𝑠 , ..., 𝑃 𝑚 𝑠 , and those from dissimilar tasks 𝑃1𝑑 , 𝑃 2 𝑑 , ..., 𝑃 𝑛 𝑑 . (3) Two transfer learning algorithms address transfer problems in different scenarios. One to disseminate shared knowledge extracted from similar tasks, and the other to differentiate core features from dissimilar tasks. Each structure is detailed below.

heading: 4.1 Attention-based Similarity Estimator
text: To tailor the transfer learning algorithm for appropriate tasks, we first conduct a similarity assessment and partitioning of the previous task. When training task 𝑡, the previous task’s prompt embeddings are recorded as {𝑃1, 𝑃2, ..., 𝑃𝑡−1}. Rather than calculate similarity between different tasks,
we calculate similarity between task 𝑡’s every instance and previous tasks, which is more robust to data variance (Wu et al., 2022). Given that we lack access to data from past tasks, our similarity assessment relies on the prompts of previous tasks. Let 𝛼𝑖 (𝑋) to denote the instance-wise attention value for previous task 𝑖 given 𝑋 . It is calculated by extracting features from three elements: previous task prompts{𝑃1, 𝑃2, ..., 𝑃𝑡−1}, current task prompt 𝑃𝑡 and instance embedding 𝑋 . The max pooling operation is first applied to 𝑋 ∈ R𝑙×𝑑 and 𝑃𝑖 ∈ R𝑙𝑝×𝑑 , and transforms them into compact forms ?̂?, 𝑃𝑖 ∈ R𝑑 , respectively. ?̂? is then fed to a two-layer network to map it to the prompt space:
?̃? = 𝐿𝑎𝑦𝑒𝑟𝑁𝑜𝑟𝑚(𝑆𝑖𝐿𝑈 ( ?̂?𝑊𝑑𝑜𝑤𝑛)𝑊𝑢𝑝), (3) where 𝑊𝑑𝑜𝑤𝑛 ∈ R𝑑×𝑟 and 𝑊𝑢𝑝 ∈ R𝑟×𝑑 are trainable projection weights, SiLU (Elfwing et al., 2018) is employed as an activation function and LayerNorm (Ba et al., 2016) is used to address the issue of gradient explosion, following Asai et al. (2022). Finally, the similarity 𝛼𝑖 of previous task 𝑖 can be obtained through tempering softmax of inner product between ?̃? and 𝑃𝑖
𝛼𝑖 (𝑋) = 𝑒 ?̃??̂?𝑖/𝜏𝑠𝑖𝑚∑𝑡−1
𝑗=1 𝑒 ?̃?𝑃 𝑗/𝜏𝑠𝑖𝑚
(4)
where 𝜏𝑠𝑖𝑚 is a modifiable temperature parameter controlling the separation of similarity. Lian et al. (2020a) points out that the parameter 𝜏 in tempering softmax is an important factor controlling the output distribution. If 𝜏 is too large, the output is close to the uniform distribution. Conversely, if 𝜏 is too small, there will be a gradient vanishing problem. The temperature should be neither too large nor too small (Lian et al., 2020b).
Having obtained the task similarity between current instance and previous tasks, previous tasks can be easily partitioned based on it. We compare similarity value 𝛼𝑖 with threshold 𝛿. If 𝛼𝑖 > 𝛿, the prompt corresponding to task 𝑖 is classified into similar task prompt set {𝑃𝑠1 , 𝑃𝑠2 , ..., 𝑃𝑠𝑚}, otherwise it will be moved into dissimilar set {𝑃𝑑1 , 𝑃𝑑2 , ..., 𝑃𝑑𝑛}. Here 𝑠𝑖 and 𝑑 𝑗 denotes the original indicate of 𝑖-th similar task and 𝑗-th dissimilar task, respectively.{
𝑃 → {𝑃𝑠1 , 𝑃𝑠2 , ..., 𝑃𝑠𝑚}, if 𝛼 > 𝛿 𝑃 → {𝑃𝑑1 , 𝑃𝑑2 , ..., 𝑃𝑑𝑛}, if 𝛼 ≤ 𝛿 (5)
We use a predefined threshold to clearly partition the previous task into two sets, enabling the model to avoid negative transfer by applying the suitable transfer algorithm for different task scenario.

heading: 4.2 Similar Task Transferring
text: Towards previous tasks similar to current tasks, we consider transferring by prompt initialization perspective. Previous work has shown prompt tuning is sensitive to parameter initialization and usually suffer from slow convergence (Lester et al., 2021; Vu et al., 2022; Wang et al., 2022a; Shi and Lipani, 2023). Thus, we use a mixture of similar tasks’ prompt embeddings to initialize prompt, and sum it with a newly allocated prompt 𝑃𝑡 for the current task. This method not only enhances model performance on current task, but also reduces the overall training time. The final prompt is derived as:
𝑃(𝑋) = 𝑡−1∑︁ 𝑗=1 𝛼 𝑗 (𝑋)𝑃 𝑗 + 𝑃𝑡 (6)
Here, only the current prompt 𝑃𝑡 and weights in similar estimator network are trainable. The
prompt 𝑃 𝑗 of previous tasks are fixed, thus preventing forgetting or backward regression on those tasks. The dissimilar tasks’ prompts are explicitly excluded to reduce their interference. Mixture value 𝛼𝑖 is obtained from following steps. First, per task similarity is input into a threshold function to let dissimilar tasks’ similarity value to 0.
𝛼𝑖 = { 𝛼𝑖 , if 𝛼𝑖 > 𝛿 0, if 𝛼𝑖 ≤ 𝛿
(7)
Then, a normalization operation is applied on it to let the overall sum equal to 1. At the end of current task training stage, all training samples’ similarity 𝛼 are averaged and used to calculate final prompt embedding 𝑃 of current task. Then 𝑃 is added to the prompt pool.

heading: 4.3 Dissimilar Task Transferring
text: Exclude dissimilar tasks from specific transfer learning algorithm can also mitigate negative transfer. Furthermore, we explore the use of an alternative transfer learning algorithm capable of leveraging dissimilar tasks to facilitate positive transfer effects. This is a novel problem that do not mentioned in previous lifelong learning researches.
Rather than transfer knowledge from prompt parameters (Asai et al., 2022; Wang et al., 2022a), our approach leverages the knowledge embedded in a pre-trained model with selected prompts. Since higher layers of a pre-trained model often exhibit more task-specific behavior (Liu et al., 2019), we introduce two novel loss functions that based on language model’s inner representation. These functions are designed to differentiate the behaviors of the current task from those of dissimilar tasks.

heading: 4.3.1 Hidden States Contrastive Loss(HSC)
text: Recognizing that dissimilar tasks may offer limited shareable knowledge with the current task, our approach emphasizes the divergence in the output representations of these tasks. we prepend the transferred current task’s prompt 𝑃(𝑋), as well as the prompts from the dissimilar tasks {𝑃𝑑1 , 𝑃𝑑2 , ..., 𝑃𝑑𝑛}, and input the combined results into a pre-trained language model. We then calculate the last hidden states of the decoder, ℎ for the current task, and ℎ1, ..., ℎ𝑛 for the 𝑛 dissimilar tasks, all based on the same instance. The pairs (ℎ, ℎ1), ..., (ℎ, ℎ𝑛) are treated as negative examples, while (ℎ, ℎ) is treated as a positive example. The
hidden states contrastive(HSC) loss is defined as
L𝐻𝑆𝐶 = −log exp(cos(ℎ, ℎ)/𝜏ℎ𝑠𝑐 )exp(cos(ℎ, ℎ)/𝜏ℎ𝑠𝑐 ) + ∑𝑛
𝑘=1 exp(cos(ℎ, ℎ𝑘 )/𝜏ℎ𝑠𝑐 ) (8)
where cos refers to cosine similarity, 𝜏ℎ𝑠𝑐 refers to temperature parameter. The HSC loss is designed to diverge the hidden states representation between current task and dissimilar tasks for a same input.

heading: 4.3.2 Activation States Contrastive Loss(ASC)
text: Previous work has shown activation states of neurons in transformers’ feed forward network are associated to specific behaviour (Geva et al., 2021; Dai et al., 2022), and can be used to calculate task similarity (Su et al., 2022). The feed forward network FFN can be formulated as
FFN(ℎ) = 𝑓 (ℎ𝑊𝑖)𝑊𝑜 (9)
where 𝑓 is an activation function, ℎ is the hidden states and 𝑊𝑖 ,𝑊𝑜 are parameter matrices.
We denote activation value 𝑓 (ℎ𝑊𝑖) as 𝑠. The activation states, computed as 𝑠𝑖𝑔𝑛(𝑠), take the form of binary vectors where each element indicates the status of a particular neuron.
Since the activation value 𝑠 contains more information, we use it for our implementation. We then add a mask 𝑚 to filter out neurons activated by instance X.
𝑚 = 1 − 𝑠𝑖𝑔𝑛(𝑠0), 𝑠 = 𝑠 ⊙ 𝑚, (10)
where 𝑠0 is the activation value when no prompt is prepended, ⊙ refers to element-wise multiply. As previous probing experiments has shown higher layers’ feature are more task-specific (Liu et al., 2019), we construct our activation contrastive loss based on last FFN layers’ activation states from current task: 𝑠 and dissimilar tasks: {𝑠1, ..., 𝑠𝑛}
L𝐴𝑆𝐶 = −log exp(cos(𝑠, 𝑠)/𝜏𝑎𝑠𝑐 )exp(cos(𝑠, 𝑠)/𝜏𝑎𝑠𝑐 ) + ∑𝑛 𝑘=1 exp(cos(𝑠, 𝑠𝑘 )/𝜏𝑎𝑠𝑐 ) (11)
where 𝜏𝑎𝑠𝑐 refers to temperature parameter. The activation states contrastive loss aims to reduce the overlapping rate of activation states between current task and dissimilar tasks.
Finally, the overall loss is computed by
L = L𝐷𝑜𝑤𝑛 + 𝜆1L𝐻𝑆𝐶 + 𝜆2L𝐴𝑆𝐶 (12)
where L𝐷𝑜𝑤𝑛 is the standard prompt tuning loss on the downstream task. In addition, we use the hidden states and activation states at first position of the decoder output to compute contrastive losses.

heading: 5 Experimental Setup

heading: 5.1 Datasets and Metrics
text: We utilize three benchmarks to evaluate the model performance: Standard CL Benchmark is a widely used benchmark for lifelong language learning models’ evaluation. It consists of four text classification datasets on different tasks or domains (Zhang et al., 2015): AGNews (topic classification), Yahoo (QA categorization), DBpedia (Wikipedia article classification), Amazon (sentiment analysis). Large Number of Tasks consists of 15 classification tasks and is used to evaluate lifelong learning methods’ performance on long sequences of tasks (Razdaibiedina et al., 2022). It includes four datasets from standard CL benchmark, four datasets from GLUE benchmark (Wang et al., 2018), five datasets from SuperGLUE benchmark (Wang et al., 2019a), Yelp reviews (Zhang et al., 2015) and IMDB reviews (Maas et al., 2011). Negative Transfer Benchmark is a benchmark that we introduced to evaluate the robustness of a lifelong learning system under sequences of dissimilar/negative transfer tasks. We construct the benchmark in the following steps: First, we use initialization from the source task as the transfer algorithm and test which source/target task pair exhibits negative transfer in a collection of datasets: MNLI, QQP, RTE from GLUE benchmark (Wang et al., 2018), WiC, CB, COPA, BoolQ, MultiRC from SuperGLUE benchmark (Wang et al., 2019a), SQuAD 2.0 (Rajpurkar et al., 2018), Yahoo, Yelp and Amazon (Zhang et al., 2015). The experiment result is shown in Figure 1 and Appendix C. Then, we construct three dissimilar task sequences, with the requirement that preceding tasks induce negative transfer on subsequent tasks.
The detailed information regarding the task sequences are provided in the Appendix B. We use normalized F1 score (McCann et al., 2018) for SQuAD, accuracy for other datasets. The task details and metrics are provided in the Appendix A.

heading: 5.2 Baselines and Training Details
text: We compare SHLPT with the following baselines, including recent SOTA methods. Finetune: continually finetunes the whole model parameters on sequences of tasks. Online EWC (Schwarz et al., 2018): utilizes a regularization loss to constrain updates on crucial parameters associated with previous tasks. ER (Chaudhry et al., 2019): replays
samples from previous tasks when training future tasks. Per-task Prompts: trains each task with a separate prompt and keeps the remaining parameters fixed. This represents prompt tuning without transfer. L2P (Wang et al., 2022c): maintains a prompt pool and selects prompts from it using an instance-wise query. CODA-Prompt (Smith et al., 2023): implements instance-wise prompts through a weighted summation of prompts from the pool. ProgPrompt (Razdaibiedina et al., 2022): train a new prompt for each task and progressively concatenate it with prompts from old tasks.
To ensure a fair comparison with ProgPrompt and SHLPT, task identity is provided during test stage for L2P and CODA-prompt. While the task identity is not mandatory for these two methods, providing it can enhance their performance. The above methods are all implemented on original T5large model. We report the mean results over three runs with different random seeds. The temperature 𝜏𝑠𝑖𝑚 for similarity estimator is set to 2 × 104. The temperature 𝜏 for each contrastive loss is set to 1 while the weights 𝜆1 and 𝜆2 of the losses are set to 0.1 and 0.5 respectively. The similarity threshold 𝛿 is set to 0.06 for Standard CL Benchmark and Large Number of Tasks, and 1.5 for Negative Transfer Benchmark. We include further analysis of the sensitivity of SHLPT to the similarity threshold, as well as additional training details, in Appendix D.

heading: 6 Results and Analysis

heading: 6.1 Results on Existing Benchmark
text: We first evaluate our method and other baselines on two existing benchmark: Standard CL Benchmark and Large Number of Tasks. Considering the task order may impact the results, we conduct experiments under three different task orders. Table 2 shows that our method SHLPT outperforms all baselines in these two existing benchmarks. SHLPT improves the recent SOTA with an increase of accuracy by 2.6% on Standard CL Benchmark and 0.95% on Large Number of Tasks. Since prompt-based methods can all avoid forgetting, we attribute the improvement of SHLPT to its better inter-task transfer effect.
In addition to the average accuracy metrics, we also provide the backward transfer scores and forward transfer scores in Appendix H. These scores are employed to evaluate the effectiveness of mitigating forgetting and negative transfer.

heading: 6.2 Results on Negative Transfer Benchmark
text: We also assess SHLPT using a challenge Negative Transfer Benchmark that we proposed. Our motivation is to test robustness to sequences of dissimilar tasks. When similarity is low, negative transfer is more likely to occur. This is a more realistic lifelong learning scenario, requiring the model to mitigate negative transfer and transfer knowledge from dissimilar tasks. We conduct experiments on three different sequences that exhibit negative transfer and the overall results are shown in Table 3. SHLPT achieves an improvement of 1.2% in average score over CODA-Prompt. These findings indicate that SHLPT exhibits greater robustness when confronted with sequences of tasks involving negative transfer. The backward transfer scores and forward transfer scores are reported in Appendix H.

heading: 6.3 Ablation Studies
text: We present the ablation results in Table 4. "-ASE", "-ASC", "-HSC" and "-STT" denote models without Attention-based Similarity Estimator, without ASC loss, without HSC loss, without Similar Task Transferring, respectively. For "-ASE", we replace the similarity estimator with one that outputs the same similarity across all old tasks (for example,
{0.25, 0.25, 0.25, 0.25}). “-STT” and SHLPT have same result in Seq1-3, because there is no similar task in Negative Transfer Benchmark. We observe substantial drops when eliminating “STT” in Seq4, indicating that it contributes to certain transfer benefits. Table 4 shows that every component is effective and the full SHLPT gives best results.

heading: 6.4 Knowledge Transfer from Dissimilar Tasks
text: We further investigate whether the proposed two contrastive losses can facilitate positive knowledge transfers from dissimilar tasks. The experiments are conducted on eight task pairs where negative transfer is observed when transferring through initialization. Table 5 demonstrates that in the majority of cases, these two losses have positive transfers on dissimilar tasks. Notably, the combination of both losses yields the best transfer effects (+2.1%). This confirms SHLPT’s effectiveness in transferring knowledge across diverse task sequences.

heading: 6.5 Task Order Analysis
text: As Table 2 shows, task order may affect the performance of SHLPT. For example, SHLPT’s performance is relatively low on Order5. We further
investigate the possible reasons for this order’s impact on performance, as outlined below.
In Order5, SHLPT’s performance on RTE is inferior compared to others, as shown in Table 6. The difference in RTE’s previous tasks between three orders is that Order5 includes MultiRC, which is dissimilar to RTE in task type, and negative transfer (MultiRC → RTE) is observed in Table 10. This may indicate that SHLPT has poor transfer performance when transferring from MultiRC to RTE. In conclusion, the task order can impact SHLPT’s performance, as SHLPT has different transfer learning results based on different previous task sets.
The additional analysis about the training curves of task similarities and the impact of ASC loss on activation states is provided in Appendix E and Appendix F.

heading: 7 Conclusion
text: In this paper, we introduce a novel method SHLPT for lifelong language learning. SHLPT aims to alleviate negative transfer by learning task similarity in one step and employing suitable transfer method for different old task scenarios (similar or dissimilar). Experimental results show SHLPT outperforms baseline methods via better knowledge transfer on two existing benchmarks and our custom Negative Transfer Benchmark.
Limitations
Though our approach has achieve significant results, there are still some limitations. Firstly, SHLPT requires task identity at training and inference stage. The identification and mitigation of negative transfer in task agnostic lifelong learning setting remain undiscovered. Secondly, we do not evaluate SHLPT on multilingual tasks (Wang et al., 2020) or other application tasks (Lian et al., 2014; Wang et al., 2019b; Jiang et al., 2023b), and negative transfer may also occur in these scenario. Last, the scalability of SHLPT across different language models, especially LLM, has not been thoroughly researched. We leave these for our future work.

heading: A Dataset Details
text: We present detailed information about datasets we used in Table 7. Following previous studies (Asai et al., 2022; Razdaibiedina et al., 2022), we use datasets from http://goo.gl/ JyCnZq for Standard CL Benchmark, while using HuggingFace datasets library (https://github. com/huggingface/datasets) for the remaining datasets.

heading: B Task Sequence Details
text: We use six differ orders of sequences in existing benchmark experiments, and the sequence information is presented in Table 8. For Negative Transfer Benchmark, we utilize three sequences composed of different datasets. The benchmark requires that preceding tasks induce negative transfer on subsequent tasks. Therefore, we do not alter the order as it may involve positive transfer. The sequence information is shown in Table 9.

heading: C Soft Prompt Transfer Results
text: The detailed results of our empirical study experiment are shown in Table 10. We perform transfer by initializing the target prompt from the source prompt. For classification tasks, we sample 100 samples per class to form the training set and validation set, and 100 samples per class to form the test set. For SQuAD dataset, we sample 400 samples to form the training set and validation set, and 400 samples to form the test set.
D Implementation Details
We implement all methods with PyTorch (Paszke et al., 2019) and huggingface transformers (Wolf et al., 2020) library. All the experiments are run on eight NVIDIA 3090 GPUs. We set the max token length to 256 for all datasets. Following Razdaibiedina et al. (2022), we use the available validation set as the test set and create validation set from the training set. If not specifically mentioned, for classification tasks, we sample 75 samples per class to form the training set and validation set, and 100 samples per class to form the test set, following Qin and Joty (2021). For SQuAD dataset, we sample 300 samples to form the training set and validation set, and 400 samples to form the test set. We use AdamW optimizer (Loshchilov and Hutter, 2017) with a weight decay of 0.01 and a batch size of 8. All results are averaged over three runs with random seeds {42, 142, 242}.
For baselines that tuning all parameters (Finetune, online EWC, ER), we use the learning rate of 1 × 10−4. For prompt-based methods, we use the learning rate of 0.3.
For SHLPT, we perform grid search on 𝜏𝑠𝑖𝑚 within {2×104, 2×105, 2×106, 2×107}, 𝜏ℎ𝑠𝑐 and 𝜏𝑎𝑠𝑐 within {0.03, 0.5, 0.8, 1}, and 𝜆1 and 𝜆2 within {0.03, 0.1, 0.5, 0.8, 1.5}. We set prompt token length to 150 and train the model with 80 epochs in Large Number of Tasks benchmark, while for others, we train 50 epochs. Early stopping mechanism is employed on all experiments. For similarity threshold 𝛿: as the Standard CL Benchmark and Large Number of Tasks contain some similar task pairs, we perform a grid search within the range of {0.02, 0.04, ..., 0.20}. We opt for a relatively low value range as we aim to ensure that no similar tasks are mistakenly categorized as dissimilar. Then we select 0.06 for these benchmarks. For Negative Transfer Benchmark, the previous task are all dissimilar from current task, so we search in a relatively high value range: {0.50,...,1.50}. And we select 1.5 for this benchmark.
Table 11 reports the searching results of threshold on Standard CL Benchmark in Order1. The results exhibits relatively minor changes as the threshold approaches 0. When the threshold increases, we notice that there is a certain performance decrease observed in the tasks towards the end of the task sequence. This is because the threshold value affects how and when SHLPT partitions previous tasks, thus further impacting the performance of the transfer algorithm. The tasks within the Standard CL Benchmark exhibit relatively high similarity, hence a lower threshold can prevent similar tasks from being partitioned as dissimilar.

heading: E Similarity Training Curve
text: We conduct an experiment on datasets from Standard CL Benchmark and show how similarity is learned in the similarity estimator. Figure 3 illustrates the variation of similarity between the Amazon task and four source tasks during SHLPT training. The similarity is quickly learned within a few steps, and the most similar task, Yelp, also aligns with intuition (both are sentiment analysis tasks for reviews).

heading: F Effects of ASC Loss on activation states
text: To investigate how much ASC loss affects activation states, we visualize the cosine similarity of
activation states between prompts trained on different tasks (Figure 4). The similarity is calculated as follows: we average the similarity values of activation states from different prompts for all samples on each dataset (row). For the figure on the right, we added ASC Loss during the training process to diverge activation states from other tasks. As shown in Figure 4, activation states vary across different tasks, and adding ASC Loss reduces the similarity of these tasks’ activation states, thereby promoting more diverse neuron activation and a more varied utilization of pre-training knowledge.

heading: G Standard Deviations
text: Table 12, 13, 14 report the standard deviations of the results from Table 2 (on Standard CL Benchmark and Large Number of Tasks), 3 (on Negative Transfer Benchmark), 4 (ablation studies). Based on the standard deviation results, we find that the performance of SHLPT on the Large Number of Tasks benchmark is significantly better than other baselines. While on the Standard CL Benchmark and Negative Transfer Benchmark, SHLPT also shows a considerable improvement. We believe that the variations in improvement across differ-
ent benchmarks may stem from this reason: the Large Number of Tasks benchmark contains more datasets, thus resulting in a more significant cumulative performance gain for each task through transfer learning.

heading: H Backward Transfer Scores and Forward Transfer Scores
text: We further compare SHLPT and the baseline methods in terms of backward transfer scores and forward transfer scores, as defined in Wang et al. (2024). Table 15 reports the average results across three orders/sequences on three benchmarks.
The prompt-based methods, such as L2P, show no forgetting phenomenon, resulting in a backward transfer score of 0. Conversely, other baselines have negative backward transfer scores, as subsequent tasks induce forgetting, leading to impaired performance in previous tasks.
We use the forward transfer scores to measure negative transfer in lifelong learning, similar to the negative transfer gap in Wang et al. (2019b). A negative forward transfer score indicates negative transfer, while a positive score indicates positive transfer. As shown in Table 15, our method SHLPT achieves the best forward transfer score and exhibits the most effective mitigation of negative transfer in three benchmarks.

heading: Future Work
text: We leave these for our future work