Scalable Language Model with Generalized Continual Learning
Authors: Bohao PENG
Zhuotao Tian
Shu Liu
Ming-Chang Yang
Jiaya Jia
Venue: ICLR 2024 poster (2024)
Keywords: continual learning
natural language processing
large language model
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Continual learning has gained increasing importance as it facilitates the acquisition and refinement of scalable knowledge and skills in language models. However, existing methods typically encounter strict limitations and challenges in real-world scenarios, such as reliance on experience replay, optimization constraints, and inference task-ID. In this study, we introduce the Scalable Language Model (SLM) to overcome these limitations within a more challenging and generalized setting, representing a significant advancement toward practical applications for continual learning. Specifically, we propose the Joint Adaptive Re-Parameterization (JARe), integrated with Dynamic Task-related Knowledge Retrieval (DTKR), to enable adaptive adjustment of language models based on specific downstream tasks. This approach leverages the task distribution within the vector space, aiming to achieve a smooth and effortless continual learning process. Our method demonstrates state-of-the-art performance on diverse backbones and benchmarks, achieving effective continual learning in both full-set and few-shot scenarios with minimal forgetting. Moreover, while prior research primarily focused on a single task type such as classification, our study goes beyond, with the large language model, i.e., LLaMA-2, to explore the effects across diverse domains and task types, such that a single language model can be decently scaled to broader applications. The code and models will be released to the public.

Published as a conference paper at ICLR 2024
SCALABLE LANGUAGE MODEL WITH GENERALIZED CONTINUAL LEARNING
Bohao PENG* Zhuotao TIAN Shu LIU*
Mingchang YANG*
Jiaya JIA
The Chinese University of Hong Kong SMartMore
ABSTRACT
Continual learning has gained increasing importance as it facilitates the acquisi- tion and refinement of scalable knowledge and skills in language models. How- ever, existing methods typically encounter strict limitations and challenges in real-world scenarios, such as reliance on experience replay, optimization con- straints, and inference task-ID. In this study, we introduce the Scalable Language Model (SLM) to overcome these limitations within more challenging and gen- eralized setting, representing a significant advancement toward practical appli- cations for continual learning. Specifically, we propose the Joint Adaptive Re- Parameterization (JARe), integrated with Dynamic Task-related Knowledge Re- trieval (DTKR), to enable adaptive adjustment of language models based on spe- cific downstream tasks. This approach leverages the task distribution within the vector space, aiming to achieve smooth and effortless continual learning process. Our method demonstrates state-of-the-ar performance on diverse backbones and benchmarks, achieving effective continual learning in both full-set and few-shot scenarios with minimal forgetting. Moreover, while prior research primarily fo- cused on single task type such classification, our study goes beyond, with the large language model, i.e., LLaMA-2, to explore the effects across diverse do- mains and task types, such that single language model can be decently scaled broader applications. The code is available on the project website
INTRODUCTION
Human-level intelligence demonstrates the remarkable ability to continuously acquire new knowl- edge and skills while retaining previously learned information. Although deep learning in language models has achieved significant advancements recently, it still faces challenges in retaining and ac- cumulating knowledge when dealing with sequential tasks. It is also known as the "catastrophic forgetting" phenomenon, which refers the potential loss of previously learned information caused by the distribution shift during the fine-tuning process for novel tasks (McCloskey & Cohen 1989).
Despite considerable efforts tackle the aforementioned challenges, recent studies on continual learning in language models still encounter significant limitations. Specifically, shown in Fig. I (a), the replay-based methods (Rebuffi al., 2017] Romanov et al. (2018), require access to the previously learned data, leading to additional demands on resources for continual training. This approach also raises potential privacy concerns. Then, the regularization-bant
approaches Huang et al. 202T))[Aljund et al. (2018) (Fig. (b)) exhibit vulnerability in long task sequences and strug- gle to strike balance between forgetting and adaptability to specific tasks. And, certain architecture- based methods (Razdaibiedina et al.) 2023) (Fig. (c)) rely task-ID during inference, which poses challenges in practical scenarios where obtaining task-IDs for individual runs may not be feasible. Besides, most previous methods have primarily focused single task type, such text classifi- cation, neglecting the broader spectrum of language-related tasks (Qin & Jory (02T). These issues deprecate the efficacy and greatly hinder the practical applications of continual learning.
In this paper, our objective is to extend the application of continual learning to a more practical and generalized setting without relying on experience replay, optimization constraints, inference task- ID, which enables agile adaptation to novel tasks. To this end, we propose the Scalable Language
hhtps://github.com wttphhhh..coompo/oosli.
Cooresponnencc Zhuotao Tiant tianzhutaaoggaaii com

---

Published as a conference paper at ICLR 2024
(a) Replay
(b) Regularization
(c) Task-id needed
ji
Language Model
Language Model
B
Language
Model
Sample
Trnining
Training
Comppneeti
loss a Cll +1l2)
Camponent
Task
Task
Gll
Price Tasks
Optimization
Componinn
Ausiliary Information
Figure I: Illustration depicting the framework comparison of various previous methods.
Model (SLM), which efficiently scales base language model novel tasks in different domains without compromising the performance of the witnessed ones.
SLM incorporates vector space retrieval into the language model, which aids in achieving scalable knowledge expansion and management, ultimately enhancing its capabilities and skill set. It com- prises two primary components: Joint Adaptive Re- prramtterriiito (JARe) and Dynamic Task- related Knowledge Retrieval (DTKR). Assuming that each task is associated with distinct distri- bution in the vector space (Finn et al. 2017). the DTKR technique is utilized to identify the most relevant knowledge for each input instance. The relevant knowledge is preserved as compilation of weight increments that leverage low-rank adaptation techniques to mitigate computational ex- penses (Hu et al. 2021). Then, these weight increments are employed by JARe techniques to achieve adaptive re-parameterization of the pre-trained model, with the objective of effectively aligning with specific downstream tasks according to the task distribution.
Extensive experiments demonstrate remarkable efficacy and stability of our method on widely rec- ognized benchmarks, reaching state-of-the-a performance on various models, including BERT, T5 and the latest LLaMA-2 (Devlin et al. 2018: Qin & Joty| 202T; Touvron al. 2023). Our method achieves impressive up 80% reduction in forgetting, with only a minimal 0.5% performance degradation on the BERT benchmark. Unlike previous literature that primarily focuses on a sin- gle task like classification, our study pushes the boundaries by exploring continual learning across multiple task types in various domains. This comprehensive analysis highlights the superior gener- alization ability of our approach, making it applicable wider range real-worke applications.
In summary, the primary contributions of this paper can be summarized as follows:
We propose the Scalable Language Model (SLM) model-agnostic solution for scalable acquisition of knowledge and skills. SLM eliminates dependencies on experience replay, optimization constraints, and inference task-IDs in generalized continual learning setting.
SLM incorporates vector space retrieval into the language model, with two primary compo- nents: Joint Adaptive Re-parameterization (JARe) and Dynamic Task-related Knowledge Retrieval (DTKR). Extensive experiments conducted on standard continual learning bench- marks demonstrate its remarkable superiority over previous state-of-the- methods.
Our study goes beyond previous literature by exploring continual learning across multiple task types from diverse domains, showcasing the superior generalization ability.
2 PRELIMINARIES
Continual learning aims to facilitate ongoing knowledge acquisition from sequential tasks while mitigating the issue of catastrophic forgetting. Specifically, the language model exposed to a se- quence of M tasks denoted as T (T+... ...., TM J. Each task T consists of a collection of training samples {( ) where x² represents the input instance, and yf denotes its corresponding la- bel. Assuming that the language model is parameterized by 0 and the loss function L, the learning objective across all tasks is to minimize the generalization error:
M
arg min
Σ
((fo(x').v')
(ar' ET!
However, current continual learning approaches always encounter practical limitations and chal- lenges due to their stringent constraints, which are difficult to achieve in real-life scenarios.
2

---

Published as a conference paper at ICLR 2024
Language Model
Joint Adaptation
A03
Input
k3
A0
0
2
22
k1
0
Fenture
K- Neighbors
Extractor
0
(a) Overview
9: 4.ERR
[0:]
[60,]]
(b) Task-related Retrieval
(c) Joint Re-Paramcterization
Figure 2: Illustration depicting our proposed method. qs k; E R indicate the query and key. where h E with the channels and g as the groups. The weight increment is denoted as A0; SLM first retrieves relevant knowledge based on the task distribution and then adapts the pretrained model through joint re-parametrization to align with the corresponding task.
9
Generalized continual learning. We propose addressing this challenging problem in a more generalized setting, which effectively eliminates auxiliary operations by solely leveraging new task data, and encompasses a wider range of task types. Our goal is to achieve incremental knowledge acquisition and retention without relying on experience replay of past data, model optimization straints, or artificial auxiliary information. Furthermore, unlike prior methods that are primarily limited single tasks such classification, we extend the scope of our approach encompass di- verse domains and task types within the broader spectrum of language- related tasks. This expansion allows for a more comprehensive and practical application of our proposed methodology.
relaaee
SCALABLE LANGUAGE MODEL
In this study, introduce two novel techniques, namely Joint Adaptive ne-paramettrization (JARe) and Dynamic Task-related Knowledge Retrieval (DTKR), which are detailed in Sec. 3.T and Sec. 3.2 respectively. JARe dynamically adjusts the model's weights to suit various task contexts, leveraging the knowledge priors obtained from DTKR. This adaptive mechanism enables effective scaling of the language model as illustrated in Fig. 2 Consequently, we refer to any language model that efficiently integrates and extends novel knowledge using JARe and DTKR techniques as the Scalable Language Model (SLM).
3.1 JOINT ADAPTIVE RE- EE-PARAMETERIZATIOO
Recent research has shown that optimizing small subset of the model or incorporating minimal trainable parameters enables the pre-trained model adapt to downstream tasks (Li & Liang 2021 Houlsby al. 2019). Based on this, recent
Efficient tuning for continual learning.
continual learning methods have proposed to incrementally incorporate new parameters like prompts for sequential tasks while keeping the pre-trained models frozen (Razdaibiedina al. 2023 Qin &
Joty 2021 Wang et al. 2022 Madotto et al. 2020). However, they still face certain limitations:
Appending new parameters without pre-training may result in convergence challenges, perfor- mance degradation, and increased cost. Especially when scaling up large language models and
long prompts (Li Liang 2021 Hu et al. 202I). it can introduce additional training challenges.
The new parameters are commonly stacked and accumulated together without distinguishing or relying on task-IDs before being incorporated into the model. These approaches (Razdaibiedina al. 2023 Qin & Jory 2021 Wang et al., 2022) still lack the capability to adaptively adjust the importance of each element based on the task distribution.
More discussions regarding the parameter-efficient tuning methods can be found in Appendix A.5
Joint adaptive re-paramcterization. To address these challenges, propose an alternative model-agnostic approach called Joint Adaptive Re- parameterrization (JARe), which adaptively re- parameterizes pretrained models to effectively adapt to downstream tasks based the joint task distribution. Let fo represent the pretrained model, which is parametrized with the initial parame- ters 0. The goal during fine-tuning is adapt the language model to specific downstream task TT
3

---

Published as a conference paper at ICLR 2024
using gradient- based learning. This adaptation is guided by the following objective:
arg min
Σ (T), (T.
0' = + + A0,
(2)
0'
where Ly denotes the loss function specific to task T, and A represents the weight increment. We regard the process of assigning the corresponding weight increment from memory to fit a specific instance as the "adaptive re-purameterization"
Directly preserving all weight increments of the pre-trained models would result in excessive source consumption. Therefore, following Hu et al.] (202I), we only selectively update minimal weight matrices in the dense layers and leverage low-rank adaptation technique to achieve additional cost savings. Consider a specific pre-trained weight matrix of the linear layer Wo. It is updated as:
(3) where Wo E Rddk is frozen, B E gdxr and A E x are trainable parameters, and < min(d, k). Thus each task only requires minimal trainable parameters and utilizes accept- able memory. More implementation details can be found in A.6 in the appendix.
y y x = (Wo AW) = (Wo + BA)x,
Subsequently, we introduce the process of adaptively re- re--arameterziing the pre-trainee trained models based on the joint task distribution. In the context of a specific task T the corresponding task distribution denoted as Pt- Thus, after learning sequence of tasks, a set of weight increments (_0).... A0 M0 is derived, where each increment is associated with one of the M distributions, namely {Pi,..., Par ) Given a specific instance æ drawn from the distribution p. i.e. æ ~ pretrained model fo to the corresponding distribution, resulting in fo fe+28,
p, the objective is to adapt the
Given the discrete nature of preserved values, direct computation precise weight increments in continuous space is infeasible. Consequently, we resort to utilizing set of interrelated elements to approximate and estimate similar, similar to the linear interpolations used in meta-learning Tri- antafillou et al.] (2021). To be specific, first employ the K-nearest neighbors (KNN) algo- rithm to select a subset of K weight increments from the most relevant distributions, denoted as (A01, ... A where K < ... Then, the pre-trained models re-parametrized towards the target task as shown in Fig 2 c). which can be formulated as:
ΣK D(p.Pi) AO; ΣK D(p.Ps)
0' = 0 + A0, = 0
Here, D(-) represents the function that measures the correlation between two distributions. prac- tice, we approximate the correlation by using query-key similarity distance.
Discussion. single dataset can also be allocated and partitioned into multiple distributions. In practical scenarios, there situations where the model may inadvertently retrieve unrelated or in- correct information, resulting the erroneously selected information and worse performance. The proposed JARe effectively alleviates this issue by employing joint re- parameterizzaion that reaches a consensus among multiple feasible directions for optimization, thus mitigating the negative im- pacts. Moreover, it is noteworthy that even different datasets can often share transferable knowledge. This approach leverages the shared common knowledge among closely related tasks to enhance the model's performance and improve its generalization ability.
3.2 DYNAMIC TASK-RELATED KNOWLEDGE RETRIEVAL
Overview. This section outlines the process of retrieving the most relevant knowledge. As previ- ously mentioned. the sequentially learned knowledge can be represented as collection of weight increments {_0), AdM }. Subsequently, each A0, is correlated with a key vector k; E R² E 1,.... M), which serves to estimate the centroid of its corresponding task distribution p;- This forms the key-value pair, i..,, [k; A0.). During the inference phase, given query obtained from the input, the proposed Dynamic Task-related Knowledge Retrieval (DTKR) identifies the most relevant pairs based on the correlations between the query and key vectors and then re-parameterizes the pre- trained model using the corresponding values as Eq. 4 As for the training phase, we divide into the preparation stage and the fine-tune stage. The preparation stage exclusively serves the purpose of keys generation. In the subsequent fine-tune tune stage, the keys are frozen, and the values are utilized for fine-tuning fine-tuning specific tasks, which follows the same procedure the inference phase.
4

---

Published as a conference paper at ICLR 2024
Keys generation and knowledge retrieval. To begin, we initialize set of learnable parameters with a (semi) orthogonal matrix, following the methodology described in Saxe et al. (2013): Wang et al.] (2022). This initialization yields collection of initial keys, ensuring orthogonaityy between any two keys within the set. After that, given a tokenized input æ, we employ Sentence-BERT (Reimers & Gurevych 2019), denoted fs. extract its semantic features. This extraction process maps the original text æ to hidden feature space, resulting in the generation of the query vector q. Mathematically, this process can be represented as q fxx (æ R q R where / represents the sequence length and с denotes the number of channels. It is important to note that, to maintain consistency the mapping process during training. f. remains frozen and unchanged.
Then, we calculate the correlations between the query and keys, and employ the --nearest neighbors algorithm to retrieve the top K most similar keys Ky = (ki,..., kx} ). where K S M. The cosine similarity distance is utilized as the metric to measure the distance between the query and the keys.
During the preparation stage, the selected keys K2 undergo optimization to improve their alignment with the distribution of input instances and perform centroid estimation. The other unselected keys remain unchanged and are not affected, which can be written as:
k' t Vacossq k).
Kgg
(5)
where y is the learning rate and cos(.) represents the cosine similarity.
However, directly utilizing such an operation for keys generation may inadvertently result in getting stuck in local optimum, as elaborated in Appendix A.8] This occurs when only a subset of keys is constantly selected and optimized throughout the entire process, while the remaining keys are ignored and never updated. To address this problem, we propose two strategies:
Group- basup- retrieval. Insipired by Vaswani et al. (0017), rather than retrieving directly from the entire keys set, we first partition the set into multiple equal groups. Simultaneously. the query vector q E Ro is also segmented into equal parts follows:
.h.....'' Lg q q(i-1)e/gie///
(6)
where 9 E R2/9, and g represents the number of groups, which is a hyperparameter. The re- trieval process conducted independently within each qi in distinct groups, while the outcomes are subsequently aggregated across multiple groups. Group retrieval enables the model to simul- tancously capture diverse patterns and relationships presented in the input data by attending to different aspects and subsets of features. Additionally, this approach enhances the robustness of the retrieval system by compensating for any potential failure of any group to capture relevant information. As result, it facilitates a more comprehensive and expressive representation.
Random keys mask. To mitigate the retriever's tendency to overly prioritize specific keys, we in- troduce method called random keys mask. This technique involves randomly masking certain keys during the training process, which encourages the model to allocate attention to other ele- ments. In practice, this objective is accomplished by randomly assigning some cosine similarity results to Inf, effectively excluding them from retrieval during training. Specially,
S. cos(g': k.z) (1 -Bp)) - Inf B(p),
where B(p) represents a Bernoulli random variable that takes the value 1 with probability p.
Notably, only keys are optimized during the preparation stage as Eq. [ and values are unchanged and still remain as the initialized learnable parameters. After the keys have been generated during the preparation stage, they are subsequently frozen and the associated values are adopted as adaptable weight increments to align the language models with the forthcoming tasks of continual learning. The overall training pipeline is illustrated in Algorithm
4
EXPERIMENTS
4.1 EXPERIMENT SETUP
Datasets. We evaluate across various benchmarks with different backbones, demonstrating strong generalization capabilities. We first test method the widely adopted continual learning bench- marks for language models following de Masson D' A'Auumme et al. (2019), which use five text clas- sification datasets (Zhang al. 2015 Chen al. 2020) including AG News (news classification),
5

---

Published as a conference paper at ICLR 2024
Algorithm The training pipeline of Scalable Language Model
Input: Training sets {T),... J, ; T {(r) vt ) Output: Grouped key-value pairs Vi.....g = [[k, ]
for .,,.., .M do
2: Initialize the t-th task's grouped key-value pairs V 2!....
3: for - T do # The preparation stage for task T
4:
Feature extraction and Group partition [4' . Lg s 9 = (a:)) via Eq.
5:
Calculate similarities S.,j = cos king) . B(p) - Inf fff B(p via Eq.
6: K K2 U... Kg, where K. K similar keys of group j E
7:
Update Ki.j E K. by ks.j +- k..j + VV..,, cos(a ks.s) as Eq, end for
8:
9:
for
(x); y.) E T do The fine-tune stage for task T
10:
Retrieve most related weight increments {A0] ... A0x) with similarity distances D Obtain the weight increment A Σ D,-A9./211 D, used Eq.
11:
Calculate sample loss C. [(fe+a0(z))) v/) Back- proppapr the gradients AoC; to update (AQ [0k] end for
13:
15:
Vi....9 + Vi....g U UV VI.....
end for
Yelp (sentiment analysis), DBPedia (Wikipedia article classification), Amazon (sentiment analysis) and Yahoo Answers (Q&A classification).
In our experiments with BERT-base backbone (Devlin et al., 2018). we follow the approaches of IDBR and ProgPromt (Razdaibiedina al., 2023 Huang et al. 2021 employing four different task orders from the five tasks. We adopt the full supervised continual setting, where the training set and test set are the same as MbPA++ and LAMOL (de Masson D' Autume al.) [019]] Romanov ct al. 2018), consisting of 1 15,000 training examples and 7,600 test examples for each task. On the contrary, we conduct the few-shot continual learning setup with T5-large backhone (Raffel et al., 2020). following the approach of LFPT5 (Qin & Joty) 2021). This setup involves sampling 16 examples per class in the training and validation sets evaluate the performance of our proposed method on limited training resources.
We further extend our method large generation language models with LLaMA-2 backbone (Tou- vron et al. 2023) and introduce a new benchmark that spans multiple domains and task types. This benchmark includes three types of tasks: question answering (medical), multiple-choice exami- nation (mmlu), and sentiment classification (finance) (Li et al. 2023 Hendrycks et al.) 2021b a These tasks are drawn from domains such as medical, history, finance, and more. For each task, we randomly allocate 85% of the data to the training set and the remaining portion to the test set.
Methods Compared. In order to compare and evaluate the performance of our method, we have selected several baselines. The selected baselines include: Fine-lune (de Masson D' Autume et al.) Wang et al. 2020), Replay (Razdaibiedina
2019
al.] (023),, MBPA ++ (de Masson D' A'ttum al 2019), IDBR (Huang al., 2021), LFPT5 (Qin & Joty| 202|l and ProgPromt (Razdaibicdina 2023 Detailed descriptions of these methods can be found in A.12 in the Appendix.
ct al.
4.2
IMPLEMENTATION DETAILS
Backbones. Our proposed method, Scalable Language Model (SLM), is model-agnostic ap- proach to continual learning that can be applied to various backbones. our study, we specifically selected three different models: encoder-only BERT-base model (Devlin et al., 2018). encoder- decoder T5-large model (Qin & Joty 202T), and decoder-only LLaMA2-7B model Touvron et al.) 2023), covering various scales and architectures. To ensure consistency, we replicate all models
from HuggingFace Transformers (Wolf et al. 2020) with corresponding pretrained weights.
Configuration. We conducted trials using the BERT and T5 backbones with NVIDIA GeForce RTX 3090 GPUs. We set the batch size to 8 and the maximum sequence length to 512 for these experiments. Additionally, for experiments involving the LLaMA2-7B backbone, we uti- lized NVIDIA A A000 GPUs with batch size of 2. To enhance training efficiency, we employed

---

Published as a conference paper at ICLR 2024
What sentiment of this tweet: The cooperation will double The Switch converter capacity.
Medical Q&A
have severe head ache back side.
It could be due to stress, anxiety, lack of sleep, dehydration, eye strain or due to migraine, If... You can take paracetumol for pain relief. If it persists for more than days, you need to consult your doctor.
[This is:] Positive. Finance Sentiment
Dear user, Thanks using Chat Doctor. Cooperation with Switch will double the speed of data transfer and will improve overall performance.
[This tweet is:] Neutral
Order: Medical - MMLU - Finance
Order: Finance- MMLU
Medical
Scalable Language Model (Ours)
Fine-tune (Baseline)
Comparison between our method and the baseline with the LLaMA backbone. We employ the continual training strategy to train chat robot with diverse skills, and evaluate its performance using examples from the first task it learned. The baseline exhibits catastrophic forgetting.
Figure 3:
DpepSpeed (Rasley et al. 2020) as training optimization. AdamW is employed as the opti- mizer (Loshchilov & Hutter 2017) for our experiments. For the preparation stage, we set the learning rate Ir le and the random mask rate p 20% for all scenarios. Specifically, we set the learning rate to 2e -4 for fully continual learning using the BERT and LLaMA2 backbones. For the few-shot continual learning scenario with the T5 model, set the learning rate to 2e" 2 The weight decay is set to 0.01. More configuration details can be found in Appendix A.4
4.3 RESULTS ON CONTINUAL LEARNING BENCHMARKS
Table 1: Results on BERT benchmark. The results are aver- aged over 2 runs. "TI": whether task-1D is available during in- ference. "DR": whether require data replay. and & denote sults from Huang al. (2021] and Razdaibiedina et 2023
Table 2: Results on the continual learn-
ing with T5 backbone. All selected meth- ods don't use task-ID during inference.
We report the averaged results over 3 runs. f denotes results from Qin & Joty 2001]
Order
Method
TI
DR
7
Avg
Order
Finetunet
14.8
27.8
26.7
4.5
18.4
Method
1
2
3
Avg
Replayt
67.2
64.7
64.7
44.6
57.8
MBPA++
74.9
73.1
74.9
74.1
74.3
Finetunet
18.9
24.9
41.7
28.5
IDBR'
75.9
76.2
76.4
76.7
76.3
Promptt
18.9
24.9
41.7
28.5
SLM
79.2
78.8
79.0
79.2
79.1
EWCf
39.0
38.0
44.8
40.6
LFPTS+
47.6
52.6
$7.9
52.7
ProgPrompts
78.0
77.7
77.9
77.9
77.9
SLM
73.1
72.9
73.3
73.1
SLM-TI
80.0
In our evaluation, we initially fine-tune the pretrained models to adapt them to sequential tasks during the training stage. Then, we assess the performance of these models on the test sets associated with each task and report the averaged scores. Experiments without the inclusion of specific notation don't provide task-ID during inference. Further, Appendix I shows detailed task orders, A.3 presents the dataset details, and A.9 investigates the number of learnable parameters.
BERT benchmark. Tab. showcases the performance of our proposed method on the BERT continual learning benchmark. Our method achieves new state-of-the-art (SOTA) result, surpassing the alternatives, even without relying on experience replay task-ID. Task-ID utilization simplifies the problem. particularly for methods that introduce new parameters (Razdaibiedina et al. 2023 Qin & Joty| 2021). It resembles fine-tuning on multiple tasks with distinct parameters. However, the practical determination of the input source remains challenging, such as in applications like online chatbot services where advanced knowledge of upcoming tasks may not be accessible. While our method does not depend on the task-ID, incorporating it yields a slight improvement, resulting in a remarkable performance of 80% as a first achievement.
T5 benchmark. We conducted experiments on the few-shot continual learning benchmark for the T5 model, following Qin & Joty (202T). The results of our experiments are presented in Tab. 2 where we compare the performance of SLM with other methods. All selected methods do not require the task-ID, and only LFPT5 necessitates slight experience replay. In accordance with Qin et al. (2021) Qin & Joty (2021). we employ the text-to-text formulation for all T5 experiments, where classification labels are mapped into words. We employ accuracy as the comparative metric, considering only the first word selected as the answer from the generated output.
LLaMA benchmark. We extend our method to the large language model, utilizing the decoder- only LLAMMA-7B (Tourron et al. 2023) as the backbone. In study, incorporate three types of
7

---

Published as a conference paper at ICLR 2024
tasks: question answering (medical), multiple-choice examination (mmlu), and sentiment classifica- tion (finance) across various domains. For the multiple-choice and classification tasks, we evaluate performance using accuracy. And utilize BERTScore, following Zhang et al. (2019), to assess the medical answers generation quantity. Specially, we assign a score of O to the answers that do not align with the given tasks. The performance comparison with the baseline is presented in Tab. and Fig. 3 provides more intuitive sampled examples. We conduct the replay methods following previous related work (He et al., 2021; Huang al. 2021 with 1% sampled instances. It is evi- dent that after fine-tuning sequential tasks, the baseline model has almost completely forgotten the first-learedd knowledge and skills, suffering from catastrophic forgetting. And as the interval be- tween tasks increases, the severity of forgetting tends to worsen. Indeed, our method demonstrates outstanding performance without significant forgetting. More examples can be found in Fig. 3 and
Fig. G in the Appendix.
Table 3: Results on LLaMA benchmark. Finance: finance news sentiment classification.
MMLU:
multiple choice questions across multiple domains. Medical: medical question answering.
Order
Order
Method
Avg
Finance + MMLU Medical
Medical + MMLU Finance
Finetune
18.0
25.5
85.3
1.6
13.6
87.2
38.5
Replay
71.5
23.3
85.0
83.7
23.6
86.8
62.3
SLM
89.0
72.4
85.4
85.1
72.5
89.1
82.3
4.4 ANALYSIS
Table 4: The comparison of forgetting which is calcu- lated each time after completing the training on new task of the BERT benchmark.
7
Method Order
SLM
IDBR
5
Avg
Avg
After 2 tasks
0.0
0.0
0.0
0.0
0.8
After 3 tasks
0.0
0.6
0.4
0.3
2.4
Figure 4: t-SNE visualization of keys distribution. Different spans indicate distinct groups, and the same tasks are linked by lines.
After 4 tasks
0.2
0.4
0.8
0.5
2.7
After 5 tasks
0.5
0.5
0.5
0.5
2.9
Influence of task sequence length forgetting. In addition accuracy, forgetting is another important indicator for assessing continual learning. Following the approach of Huang et al. (2021) and [Chaudhry al. (2018), we conduct experiments on the BERT benchmark and measure forget- ting F& after training on task t using the following formula:
F& E==1...-1ff
1
max 1Ef1...1-1)
a1,j -t,j.
(8)
where at,j is the accuracy on task j after trained on task 1. We report the forgetting evaluated on each new task and report the results compared with IDBR in Tab. A Our method demonstrates remark- able improvement of up to 22.8% compared to the previous state-of-the-art (SOTA) approaches and all indicators are less than 0.5%. It effectively minimizes the forgetting of previously learned knowledge while acquiring new knowledge. Additional discussions are in Appendix A.7
Visualization of the keys' distribution. To investigate the partitioning of distinct knowledge do- mains and assess the impact of the grouping strategy, we adopt t-SNE (Van der Maaten & Hinton 2008) to visualize the distributions of the keys, demonstrated in Fig. In this figure, different cluster spans indicate different groups, and the keys belonging to the same task are connected by lines within each group. We can observe that different groups correspond to varied distributions, demonstrating the effectiveness of the grouping strategy in capturing diverse patterns and improving robustness. This is crucial because single group may fail to retrieve the related information, and the presence of multiple groups helps mitigate this limitation.
Effects of JARe. Multiple ablation experiments were conducted to examine the impact of our pro-
posed Joint Adaptive Re-Parramteerization (JARe), and the results are presented Tab. 5 Specif- ically, we replaced the weight increments in our DTKR with prompts and adapters (Li & Liang 2021 ZZhng et al. 2023). The "Separate Fine-tune" approach involves individually fine-tuning on different tasks instead of continual learning among multiple tasks. By demonstrating marginal deviation of only 0.8% %, the proposed JARe manifests its superiority over the competitors.
8

---

Published as a conference paper at ICLR 2024
Table Results of the ablation studies on various storage values on BERT benchmark.
Table 6: Zero-shot evaluation on open bench- marks to assess the phenomena of forgetting and knowledge transfer.
Order
Method
4
5
6
Avg
Task
Method
Arc-c
Arc-e
Piqa
Wino
DTKR Prompt
54.7
55.8
49.4
53.3
DTKR + Adapter
71.2
71.2
70.2
70.9
Finetune
31.8
42.6
67.9
64.3
DTKR + JARe
79.2
78.8
79.0
79.0
SLM
44.7
76.0
76.3
67.7
Separate Fine-tune
79.8
LLaMA2
43.9
74.4
76.7
66.4
We further evaluate our method in zero-shot setting on four open bench- marks (Arc-c, Arc-e, Piqa. Wino) (Clark et al. 2018 Sakaguchi et al. 2021 Bisk al.) 2020) following Gao et al. (02T). We first fine-tune the LLaMA-2 backbone following the order: Finance, and then evaluate the models on the above four benchmarks.
Zero-shot evaluation.
Medical MMLU Ther results are shown in Tab. 6 and more detailed comparison can be found in A0 It can be seen that directly utilizing fully fine-tune will result in a deterioration of the overall performance because of catastrophic forgetting. In constract to deterioriting the performance, our method even slightly improves the baseline on several tasks. It demonstrates the dual capability of our method to alleviate forgetting and effectively transfer knowledge.
5 RELATED WORK
Continual Learning, also known as lifelong learning or incremental learning, aims to improve a learning system to progressively acquire and preserve knowledge from various tasks. Existing meth- ods for continual learning can be broadly classified into three primary categories: (1) Replay-based methods: periodically replay past experiences and knowledge from the observed tasks and data (Re-
buffi et al. 2017, Romanov et al., 2018). The experiential data can be sampled from the previous tasks (de Masson D Autume et al. 2019 Rebuffi al. (017)) or synthesized using generative mod- els (Romanov et al. 2018 Shin et al. (2)17) (2) Regularization- based methods: impose constraints the parameter changes of the model to prevent forgetting of previously learned tasks (Aljundi) (et al. 2018 Huang et al. 2021). (3) Architecture-based methods: employ distinct components and separate sets of parameters within the model for different tasks (Rusu et al. 2016 Mallya &
Lazebnik 2018 Razdaibiedina et al. 2023).
Vector space model. Compared traditional retrieval methods, such the keyword-based or the rule-hased, the Vector Space Model (VSM) has emerged as a prominent paradigm in informa- tion retrieval (Berry et al.) 1999 Wong et al. 1987 (Singhal et al., (001). The VSM represents queries as vectors in a high-dimensional space. This representation enables the application of var- ious similarity measures, such as cosine similarity,
determine the relevance between documents and queries (Zhang & Lu (003).. Previous methods have endeavored to incorporate vector space retrieval into diverse endeavors (Peng al. 2023 Danisman & Alpkocak 2008) Wang al., 2022), and Wang et al. (2022) adopts VSM for in- co-tent learining. In contrast, our work introduces the use of VSM to enable dynamic transfering and adaptation of models for downstream tasks, incorporating meta-learning techniques similar to the "model soup" (Wortsman et al.) 2022).
CONCLUSION
This paper presents Scalable Language Model (SLM), which enables incremental learning of se- quential tasks while effectively mitigating catastrophic forgetting in generalized setting. Notably, our approach eliminates the requirement for experience replay, optimization constraints and infer- ence task-ID, enhancing its applicability to practical scenarios. We propose the integration of Joint Adaptive Re-Parameterizattin (JARe) with Dynamic Task-related Knowledge Retrieval (DTKR) to adaptively re-parameterize pretrained models based on the distance between task distributions. Our approach demonstrates remarkable stability and effectiveness across diverse model scales, leading to state-of-the-an performance multiple benchmarks encompassing different tasks types.
The weakness of our method lies in the introduction of an additional retrieval framework, which may lead increased computational and memory storage costs. However, when compared to the resource requirements of large models used for inference generation, this additional consumption is relatively small. Further quantitative analysis regarding this weakness can be found in Section A.

---

Published as a conference paper at ICLR 2024
REFERENCES
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) forget. Proceedings of the European confer- ence on computer vision (ECCV), pp. 139-154, 2018.
Michael Berry, Zlatko Drmac, and Elizabeth R Jessup. Matrices, vector spaces, and information retrieval. SIAM review, 41(2):335-362, 1999.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piga: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference Artificial Intelligence, 2020.
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. Proceedings of the European conference on computer vision (ECCV), pp. 532-547, 2018.
Jiaao Chen, Zichao Yang, and Diyi Yang. Mixtext: Linguistialll-informed interpolation of hidden
space for semi-supervised text classification. arXiv preprint arXiv:2004.12233 2020.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc. the ai2 reasoning challenge. arXiv preprint arXiv: 1XXiv:1803.05550 2018.
Taner Danisman and Adil Alpkocak. Feeler: Emotion classification of using vector space model. In AISB 2008 convention communication, interaction and social intelligence, volume 1, pp. 53, 2008
Cyprien de Masson D' Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. Episodic memory lifelong language learning. Advances Neural Information Processing Systems, 32, 2019.
Jacob Devlin, Ming-Wes Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800. 2018.
Chelsea Finn, Picter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference machine learning, pp. 1126-1133 1135. PMLR, 2017.
Lco Gao. Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version vO. 0. Sept, 2021.
Tianxing He, Jun Liu, Kyunghyun Cho, Myle Ott, Bing Liu, James Glass, and Fuchun Peng. An- alyzing the forgetting problem in pretrain finrriiin-iettting of open-domain dialogue response models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computa- tional Linguistics: Main Volume, pp. 1121-1133, 2021.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
Dan Hendryeks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- lional Conference on Learning Representations (ICLR), 2021b.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: 106.000009655 2021.
10

---

Published as a conference paper at ICLR 2024
Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang. and Diyi Yang. Continual learning for text classification with information disentanglement based regularization. arXiv preprint arXiv:2104.05489, 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.0011 2021.
Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. Chatdoctor: A medical chat model fine-tuned large language model meta-ai (Ilama) using medical domain knowledge. Cureus, 15(6), 2023.
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 1.0051011, 2017.
Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Seungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Funjoon Cho, and Zhiguang Wang. Continual learning in task-oriented dialogue systems. arXiv preprint 1xXix:20/2./5500 2020.
Arun Mallya and Svetlana Lazebnik. Packnet: pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018.
Adding multiple tasks to a single network by iterative
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The scquential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Bohao Peng, Zhuotao Tian, Xiaoyang Wu, Chengyao Wang, Shu Liu, Jingyong Su, and Jiaya Jia. Hierarchical dense correlation distillation for few-shot segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2023.
Chengwei Qin and Shafiq Joty. Lfpt5: unified framework for lifelong few-shot language learning based on prompt tuning of t5. arXiv preprint arXiv:21 10.07298, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555 2020.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505-3506, 2020.
Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. Progressive prompts: Continual learning for language models. arXiv preprint 1rX11:2301.12314, 2023.
Sylvestre- Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprint 19Xiv:1988.10006 2019.
Alexey Romanov, Anna Rumshisky, Anna Rogers, and David Donahue. Adversarial decomposition of text representation. arXiv preprint 1808. 2018.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 160iv:1606.044 2016.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications the ACM, 64(9):99-106, 2021.
11

---

Published as a conference paper at ICLR 2024
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
Amit Singhal et al. Modern information retrieval: A brief overview. IEEE Data Eng. Bull., 24(4): 35-43, 2001.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An cnttruction-oollowiin llama model. https://githut: httpg/tttt.coooooooooooo htt///atsuuuue lachhllabssee st anford alpaca, 2023.
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykow. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0925 092888 2023.
Eleni Triantafillou, Hugo Larochelle, Richard Zemel, and Vincent Dumoulin. Learning universal template for few-shot dataset generalization. In International Conference on Machine Learning, pp. 10424-10433, PMLR, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911), 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning prompt for continual learning. In Pro- ceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 139-149, 2022.
Zirui Wang, Sanket Vaibhav Mehta, Barnabás Póczos, and Jaime Carbonell. Efficient meta lifelong- learning with limited memory. arXiv preprint arXiv:2010.0255 2020.
Thomas Wolf, Lysandre Debut, Victor Sanb, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 conference empirical methods in natural language processing:
system demonstrations, pp. 38 45, 2020.
S. K. Michael Wong, Wojciech Ziarko, Vijay V. Raghavan, and Patrick CN Wong. On modeling of information retrieval concepts in vector spaces. ACM Transactions on Database Systems (TODS), 12(2):299-321, 1987.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 13965-23999 PMLR, 2022.
Dengsheng Zhang and Guojun Lu. Evaluation of similarity measurement for image retrieval. In International Conference on Neural Networks and Signal Processing, 2003. Proceedings of the 2003, volume 2, pp. 928-931. IEEE, 2003.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init atten- tion. arXiv preprint 2tf.:2333.119999 2023.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluat- ing text generation with bert. arXiv preprint (rXiv:/904.09675. 2019.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Advances in neural information processing systems, 28, 2015.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 TASK SEQUENCE ORDERS
In standard continual learning benchmarks, the BERT and T5 models (Devlin et 2018 Raffel et al. 2020) utilize total of 7 orders. as described by |Huang et al. (2021 Qin & Joty (002 The
specific orders are presented in Tab. a follows:
In all benchmark experiments, we initially train the pretrained model on the specific dataset, follow- ing the predefined orders mentioned above. Subsequently, we evaluate the fine-tuned model on all test sets simultaneously to test the model's performance and alleviating forgetting ability.
Table 7: Different orders of the task sequences that used for the standard continual learning benchmarks with the BERT and T5 backbones. The 1-3 orders are used for T5 models, while the 4-7 orders used for the BERT models.
Order
Model
Task Sequence
T5
db amazon yahoo ag db - amazon - ag + yahoo yahoo + amazon - ag - db
T5
3
T5
4
ag > yelp + amazon - yahoo db yelp - yahoo - amazon db ag BERT - yahoo + ag - amazon + yelp
BERT
5
BERT
7
BERT
yelp -> ag + db - amazon yahoo
In the LLaMA benchmark, we use two orders due to resource constraints. The specific orders are listed follows:
*
Order 8: Medical - MMLU + Finance
Order 9: Finance > MMLU -> Medical
A.2 EXAMPLES DEMO
Large Language Model (LLM) has achieved a significant success in recent years, demonstrating their distinguished ability to excel in various tasks. Furthermore, numerous applications are have been proposed that leverage fine- tuning on the pretrained large language models to adapt them to specific domains (Taori et al., 2023 Li et al.) 2023). However, such operation only let the LLM grasp single domain-specific skills while potentially causing catastrophic forgetting of its general abilities.
The objective of this study is to enable the large language model (LLM) to acquire diverse skills and knowledge across multiple domains, while also possessing the potential for lifelong learning capability. More examples about the comparison of our method and the baseline, which involves direct fine-tuning of the pretrained L.LM sequence tasks, are presented in Fig. 5 and Fig.
*
Fig.
Medical - MMLU -> Finance.
Fig. 6
Finance - MMLU - Medical.
The results clearly demonstrate that while fine-tuning enables the model to acquire specific knowl- edge, it suffers from catastrophic forgetting, which can only answer following the formats of the last task. This is even detrimental to LLM's general abilities, which supported by the zero-shot evaluation the open benchmarks in Sec. 4.4 In contrast, method significantly alleviates these issues, allowing the pretrained LLM to acquire wide range of skills across multiple domains without experiencing catastrophic forgetting.
Disclaimer of liability. The examples provided is for demonstration purposes only and should not be relied upon as professional advice.
13

---

Published as a conference paper at ICLR 2024
Train Order
Medical
MMLU
Finance
Medical
My bottom teeth have been hurting lot for week. It really hurts. My teeth are crooked, and my gums are swollen. started brushing my teeth for couple of days and plan to never stop. How do make the pain go away?
Question
[Chat Doctor:] Thanks for your query. have reviewed your concems. tooth pain could be due a tooth or gum infection. recommend consulting oral physician for thorough examination and radiograph to identify any infections. If'l were your treating doctor, I would prescribe antibiotics as amoxicillin and metronidazole, along with analgesics like diclofenac, Additionally, performing saline gargling may provide relief. hope this information helps. Take care.
Medical
SLM
[This tweet is:] neutral.
Finance
Finetune
MMLU
During science class, students combined kg of iron filings with I kg sand
mixture. Which of following methods would be best for separating the iron filings from the sand?
A. Stir the mixture into hot water to dissolve the sand.
Question
B. Strain the mixture through paper filter collect the sand.
Drag magnet through mixture to attract iron filings.
D. Shake the mixture to cause the iron filings to settle to the bottom.
[The answer is:] C
MMLU
SLM
[This is:] neutral.
Finance
Finetune
Finance
What is the sentiment of this tweet? Please choose un unswer from /negative/neutrally posttiii:e:.//:siieee According to CEO Kai Telanne. the company 's newspapers achieved the best financial result ever.
Question
[This tweet is:] positive.
Finance
SLM
[This tweet is:] positive.
Finance
Finetune
Figure 5: Demo showcases the chat robot with the LLaMA2 backbone, which undergoes continual fine-tuning on the following datasets: Medical - MMLU - Finance order.
14

---

Published as a conference paper at ICLR 2024
Train Order
Finance
MMLU
Medical
Medical
got lot of stretch marks, mostly on legs. know I'm not over weight but what could do to make them less noticeable. What might be causing them?
Question
[Chat Doctor:] Hi, stretch marks caused by the breakdown of collagen and elastin fibers in the skin. This can be due to various reasons genetic
Medical
SLM
like pregnancy, weight gain, weight loss, hormonal changes, factors. These marks can be improved by using creams and ointments containing vitamin E, retinol, cocoa butter, and glycolic acid. These can be applied over the affected areas twice daily. Apart from this, the stretch marks can be treated by laser therapy.
[Chat Doctor:] Hi, Stretch marks due to the sudden growth of the body, and it is very common. It very difficult remove them. You can apply retinoid cream at night and vitamin cream in the morning. Hope I have answered your query. Let me know if can assist you further.
Medical
Finetune
MMLU
Rhea is moon of Saturn. Which of these facts about Rhea best indicates that it does not have water cycle which water changes state?
A. Its radius is 765 km.
Question
B. Its density is about 1.3 kg/m².
C. Its period of rotation is about 4.5 Earth days.
temperature is between C and C220"C.
[The answer is:]
MMLU
SLM
[Chat Doctor:] Dear Min/m have gone through your query and read your symptoms. In my opinion
Medical
Finetune
Finance
What is the sentiment of this tweet? Please choose an answer from Tnegative/ncutra/posstive
Customers wide range of industries use stainless services worldwide.
Question
[This tweet is:] neutral.
Finance
SLM
600 e
[Chat Doctor:] Hi, Welcome Chat Doctor, Stainless steel is type of steel alloy that contains least 11% chromium. It is highly resistant corrosion...
Medical
Finetune
Figure 6: Demo showcases the chat robot with the LLaMA2 backbone, which undergoes continual fine-tuning on the following datasets: Finance - MMLU - Medical order.
15

---

Published as a conference paper at ICLR 2024
A.3 DATASETS
BERT and TS benchmarks. More details about the five datasets used for the BERT and T5 bench- marks are listed in Tab. For the BERT benchmarks, we adopt a fully supervised training approach as described in the works by HHung et al. (202T); de Masson D Autume et al. (2019). As for the experiments conducted with T5 backbones, we use the few-shot setting where 16 examples are sampled for each class, following the methodology outlined in (Qin & Joty 2021
Table 8: Details of the datasets used for the BERT and T3 benchmarks. The datasets used for the BERT benchmark involve fully supervised training, while the T5 benchmark employs the few-shot setting.
Training Samples
Dataset
Alias
Class
Type
BERT
T5
AGNews
ag
News
115,000
64
Yelp
yelp
Sentiment analysis
115,000
80
Amazon
amazon db
Sentiment analysis
115,000
80
DBPedia
14
Wikipedia classification
115,000
224
Yahoo
yahoo
10
Yahoo Q&A
115,000
160
LLaMA benchmarks. this study, we utilize three distinct datasets et al. 2023
Hendrycks
et al. 2202bb a) for conducting experiments on the LLaMA benchmark. We adopt instruction tuning similar to Taori et al. (2023), while replacing the training datasets with our selected datasets. We present the sampled examples in Tab. to show more details, including the instructions used in the experiments. It can be seen that in the LLaMA benchmark, the models should learn to adapt to distinct tasks across various domains with different generation formats.
Table 9: The examples of the input instances in the LLaMA benchmark.
Dataset
Instruction
Input
Output
Medical
If you are doctor, please answer the medical questions based on the pa- tient's description.
am suffering from bad breath, this makes me don't dare to talk and get closer people. I can't find way to get rid of it.
Hi, simple and effec- tive ways to freshen your breath. Brush and floss more frequently. Scrape your tongue. Avoid foods that sour your breath. Chat Doctor.
MMLU
Question: Since green plants make their own food, they called?
D
Choices: [predators, prey, de-
composers, producers]
Finance
What is the senti- ment of this tweet? Please choose answer from (negative / neutral / positive)
According to CEO Kai positive.
Telanne the company 's
newspapers achieved the best financial result ever.
A.4 IMPLEMENTATION DETAILS
Task-ID. In this work, we mainly focus on the scenarios where inference is conducted without the task ID. In such cases, don't know that which task dataset is the input come from for each input instance. particular, for architecture- based methods, having knowledge of the task ID significantly simplifies the problem by enabling direct determination of the target parameters, which
16

---

Published as a conference paper at ICLR 2024
is similar to fine-tune separate tasks independently. However, in the practical scenarios, it is always unable to determine the input task-ID directly. Such as a customer service chatbot, it doesn't t have the feasibility to provide the model with the task source from the user.
Labels. For the T3 models, we employ a mapping technique to convert the classification labels into words, following the methodology outlined in Raffel et al. (2020). The same operation is also applied to the MMLU and Finance tasks in the LLaMA benchmark. During the evaluation phase, select only the first word and compare it with the labels to measure accuracy, following Raffel et al. (2020). The excess part of the generation results will be ignored. Regarding the Medical task, we utilize the entire generated outputs with maximum length of 512 and compare them with the labels using the BERTScore metric introduced by ZZhang et al. (2019) following Li et al. (2023).
Optimization hyperparameter. Adam (Lohhhhllvv & Hutter 2017 adopted as the optimizer in all experiments. The details of the optimization hyperparameter are listed in the Tab. T0
The details of the optimization hyperparameter. When the number of warm-up steps specified as a floating-point value, it represents ratio of the total training steps.
Table 10:
Benchmark
BERT
T5
LLaMA
learning rate
2c-
4
2e
2
2c
4
batch size
8
4
2
epoch
5
100
3
warmup steps
100
100
0.03
weight decay
0.01
0.01
0.01
A.5 DISCUSSION OF VARIOUS PEFT METHODS
In this section, we delve into further details and compare different parameter efficient fine-tuning (PEFT) methods as the retrieved targets using the vector space retrieval framework. We replace our Joint Adaptive Peeaameetterinnioonn (JARe) with alternative components and perform ablation experiments on various individual tasks.
Prior continual learning methods have made attempts to introduce PEFT techniques like prompt tuning, which involves tuning prompts for better adaptation to new tasks. (Razdaibiedina et al.) 2023 Qin & Joty| 2022]] Wang et al.) 2022). Specially, for a novel incremental task Tx. the learining objective is to minimize the log probability of training examples:
cim (0p,.) =
Σ (2.8)ET
log p(y
[Pt, 2l.
where P& learnable prompt with its corresponding parameters fz- Similar to prompt tuning, another alternative is to replace prompts with adapters, such as prefix-tuning (Li & Liang] [021]) which also provide a flexible and modular approach to incorporate task-specific information without modifying its base parameters. We conduct the ablation experiments on the different single tasks, and the results are shown in Fig.
Adaptability. In contrast to JARe, replacing it with prompts and adapters only allows them to be retrieved without the ability to dynamically adjust the importance and significance of each element based the distribution distance in the vector space. As a result, all the responsibility is placed on the pretrained model itself to determine the importance of attention without the additional task distribution information, which, although lost, can be valuable for effective adaptation.
Limited trainable parameters. Directly introducing more learnable parameters through prompts and adapters did not lead to significant improvements and fitting abilities. As more prompts and adapters are added, the input length increases significantly. However, the rate of improvement grad- ually slows down (Hu et al. 2021). 122I..o To solve this problem, Razzdaiiiedina et al. (2023) proposes to introduce a res-mlp layer, specifically,
P = MLPx(PP) + PE.
(10)
17

---

Published as a conference paper at ICLR 2024
Tode: MLRs
Taski oeaazn
Tol stoede
Tol:yep
Figure 7: Comparison with different finetune methods with the vector space retrieval framework. All the experiments are conducted on a single task.
where (.)x is learnable MLP layer for the task Tk- However, the inclusion of task-ID infor- mation is inevitable when determining which MLP layer use. But in the practical scenarios, such as a customer service chatbot, it impossible to provide the task-ID from the users.
Large language model. The direct incorporation of prompt-tuning or adapters into large language models can lead to convergence difficulties and training instability. This issue primarily arises from the fact that all newly introduced parameters are randomly initialized without pretraining, which can make the model fragile when dealing with a large number of parameters. To alleviate this problem. Zhang et (2023) proposes to introduce gate variable, specifically.
P' = P..
(11)
where gate E R is learnable parameter that is initialized as zero. This initialization ensures that the introduced parameters have influence the original model at first and provides slow warm-up process. But the gate variable also limits the influences of the prompts and determining its optimal value can be challenging.
In our practical experiments, we discovered an alternative approach where the introduced prompts be initialized with tokens from the pretrained embedding layers. This initialization strategy can be effective in improving the performance and stability of the model during training. Specifically,
P. <- (EE)
(12)
where E E R represents the pretrained embedding tokens, and y denotes the random selection function that returns a token randomly.
However, while these strategies may improve stability, they often do not fully overcome the upper limit bottleneck. Additionally, they can make models and inputs redundant and increase the time cost of the inference with more and more incremental tasks added.
A.6 MODEL. RE-PARAMETERIIATTON
In the Sec. 3.1 we have introduce that we utilize a single group of weight increments to re- parameterize the pretrained models to adapt to specific downstream task following Hu et al. (202T) ) Specifically, we freeze all the pretrained parameters without further optimization and introduce limited number learnable parameters to store the weight increments during training. We empoly the low-rank adaptation techniques to reduce more costs, as more details are shown in Fig. Re- cent research further shows that optimizing all pretrained parameters for fine-tuning is unnecessary. Instead, selectively optimizing limited set of parameters can achieve comparable performance to fully fine-tuning tine-tuning (Hu et al., 220TT: Li & Liang 200TT: Zhang et al.] (003).. Another question arises regarding which part of the pretrained models should be selected for optimization. In this work,
18

---

Published as a conference paper at ICLR 2024
Table 11: Ablation experiments results of different weight types.
Tasks
Weight
amazon
yahoo
db
Average
Query
61.1
94.3
74.7
99.2
82.3
Key
61.4
94.1
74.6
99.1
82.3
Value
62.3
94.3
75.3
99.2
82.8
Out
62.9
94.7
75.5
99.3
83.1
y W'X Bios
I Linear
W + AW W BA
Concat
Scaled
Query Linear
Key Linear
E Linear
g
BERT
f
f
Figure 8: Illustration showcases the selection of parameter parts to store weight increments, along with the application of low-rank adaptation techniques for re- re-parameterizing the model.
focus on optimizing the weight matrix within the out linear layer in the attention module as shown in Fig. 8 For saving more memories, only select single part from the attention module, and additional ablation experiment results, showcasing the impact different parts, are presented in Tab.
11
A.7 FORGETTING EVALUATION
Apart from the performance, the ability to mitigate forgetting is another crucial indicator for as- sessing continual learning method. To evaluate this ability, conduct separate tests to assess our model's performance on all previously learned tasks after training of each single incremental task. This evaluation reflects the extent which the model retains past knowledge. The results for four different task orders on BERT benchmark are shown in Fig. [ It is worthy to notice that as the sequence of learned tasks grows, our proposed method exhibits no significant degradation, demonstrating its remarkable ability to mitigate forgetting.
A.8 KEYS GENERATION
Local optimization. In this section, will delve into further details about the key generation process, focusing strategies address the issue of local optimization. When selecting and up- dating keys using gradient descent, it is possible that only certain keys are optimized while others left untouched, leading situation of being stuck in local optimization. To tackle this issue, we propose two casy yet effective strategics: Group-based retrieval and Random keys mask. These strategies aim to capture diverse patterns and relationships within the input data by attending to different aspects and subsets of features. To evaluate the impact of these two strategies, we con- ducted ablation experiments, and the results presented in Tab. 12 Specifically, with JARe, the retrieved keys are not constrained to belonging to the same task as the query, and more details we have discussed abovee. So calculate the accuracy follows:
K
Acc Σ E8(H E Kg k E GET
/
IT..
(13)
2
where 8(.) condition function that returns 1 if the condition satisfied. In other words, We calculate the percentage of inputs for which the retrieved keys from the same task distribution
19

---

Published as a conference paper at ICLR 2024
Order-4
Crder-5
Clads
Orden7
Figure 9: Accuracy of our method evaluated after training on different task sequence lengths. is observed that as the sequence length increases, there is no obvious degradation in accuracy, indicat- ing the significant ability of our method to alleviate forgetting.
stitute more than half of the total retrieved keys. It can be seen that our proposed strategies have significantly improved the performance, particularly in the case of the group partition operation.
Table 12: Results of the ablation experiments
proposed keys generation strategies.
Task dbpedia
ID
Group
Mask
Avg
ag
amazon
yahoo
yelp
v
98.0
98.0
98.6
86.3
99.5
96.1
96.8
96.7
97.7
82.6
99.5
94.6
i11
91.6
92.2
91.1
78.9
93.9
89.5
Time consumption. For each incremental task, we split the process into two stages: (1) Prepa- ration stage: generating keys for each stored value [k. A0). In this stage, generate keys that correspond to the stored key-value pairs. These keys play a critical role retrieving the correct information during the subsequent fine-tuning process. (2)Finetune stage: fine-tune the models and preserve corresponding values. In this stage, we fine-tune the models to adapt them to the specific requirements of the downstream task. So another important consideration is the computational time required for key generation. We display the training time in Tab. 13 All experiments are conducted on NVIDIA GTX 3090 GPUs, batch size per device is set 4 and epoch 3. The results show that the training time for key generation is approximately 11 minutes, which is minimal and has negligible impact on the overall process.
Table 13: Time consumption involved in generating the keys.
Tasks
ag
amazon
dbpedia
yahoo
yclp
Avg
Time(min) 10.2
8.7
10.5
14.3
15.4
11.8
Similarity matrix. To provide a visual representation of the relationships and similarities among all the generated key vectors, we present the visualization of the similarity matrix in Fig. 10 We randomly sampled some groups, where each group consists of 5 tasks and each task is associated with four keys. The visualization reveals that the keys belonging to the same task generally exhibit similar distributions, resulting in higher similarities among them. This characteristic ensures that the keys from particular task be easily distinguished from those of other tasks.
20

---

Published as a conference paper at ICLR 2024
Similarity Matrix
Similarity Matrix
444
-22
Similarity Matrix
Figure 10: Visualization of similarity values between different generated key vectors.
A.9 THE NUMBER OF LEARNABLE PARAMETERS
The size of the introduced learnable parameters does not always follow "less is better" principle. Insufficient learnable parameters for certain tasks may result underfitting. leading to unsatisfac- tory performance. Compared to the prompt tuning and adapters discussed in the previous sections, another advantage its ability to dynamically adjust the size of learnable parameters within the range of the lower bound and the size of the entire model the upper bound. This adaptability ensures its ability to dynamically adjust to diverse tasks with varying requirements.
is important note that while the size the learnable parameters is critical factor in evaluating a method, it does not have any impact on the inference process in terms of delays or computational burdens. This advantage derived from the complete decoupling of the continual learned knowl- edge from the pretrained model. The cost of the retrieval process remains constant, and the retrieved values are solely utilized for model re-panr parameteriizaion, without affecting the input or model scales.
To assess the impact of learnable parameter size on performance across different tasks, we conducted experiments with various hyperparameter settings, specifically modifying the size of the learnable parameters. These experiments were conducted individually for each task to evaluate the influence on performance with the BERT as the backbone.
All the experimental results are presented in Tab. 14 revealing several interesting findings and con- clusions. Different tasks necessitate varying sizes of learnable parameters, and increasing the pa- rameter size yields distinct improvements depending task, increasing the parameter size beyond certain point does not provide notable benefits same as "ag" task. Our method allows probability to assign different sizes to the respective tasks based on their specific requirements.
the task at hand. In the case of the "yahoo"
Regarding the memory usage for storing these additional parameters, provide statistical analysis Tab, The introduced parameters in our approach are considered acceptable and relatively small compared to the size of the original pretrained large models.
21

---

Published as a conference paper at ICLR 2024
Table 14: Ablation experiments investigate the impact of learnable parameter size on perfor- mance. For all the conducted experiments, we maintained consistent configuration of 768 channels and 12 layers for the learnable parameters.
Task dbpedia
ID
Group
TopK
Rank
Avg
ag
amazon
yahoo
yelp
i
3
2
2
59.5
93.1
74.3
99.2
63.0
77.8
ii
3
4
61.0
94.1
74.8
99.2
64.2
78.7
iil
3
4
4
61.0
94.1
75.4
99.3
65.6
79.1
IV
6
8
62.5
94.4
75.4
99.4
66.5
79.7
6
12
63.2
94.5
75.4
99.3
67.1
80.0
Table 15: The memory consumption of the additional parameters and their proportion relative to the original model.
ModelSize
Additional Parameter
Proportion
Bert
512M
3.6M
0.7%
LLaMA
12.6G
33M
0.3%
A.10
ZERO-SHOT EVALUATION ON VARIOUS TASKS
Table 16: Zero-shot evaluation on open bench-marks assess the phenomena of forgetting and knowledge transfer.
Task
Method
Are e
Arc c
Piqa
Wino
Finetune
31.8
42.6
67.9
64.3
Finace
SLM
44.7
76.0
76.3
67.7
Finetune
30.0
39.7
63.6
66.3
MMLU
SLM
49.4
76.7
76.6
66.2
Finetune
73.2
73.8
76.5
66.9
Medical
SLM
44.3
75.0
77.8
67.8
Table 16 presents more detailed zero-shot evaluation of our method using the LLaMA2 backbone finetuned various downstream tasks. It has been observed that fine-tuning small-scale datasets that differ significantly from the training data can have negative impact on the LLM's generality and adaptability. Our aim is address this issue and mitigate the catastrophic forgetting.
A.11 WEAKNESS DISCUSSION
We humbly acknowledge that the proposed method indeed introduces a cost associated with the retrieval process. However, we find the additional cost be acceptable because:
1. Compared to the subsequent inference models, the retrieval stage model used is notably smaller, lighter, and operates at a faster speed. This distinction is particularly significant for the T5 and Ilama models.
2. In the case of generation models with the decoder architecture, each inference only pro- duces single token, necessitating multiple inferences to generate complete sentence. However, the retrieval process is executed only once. Therefor, given f, as the retrivak time, the inference time, the tokens number, the proportion of time consumed is:
4r
100%
f, + n +
We conducted an experimental comparison to measure the time consumption of different parts of various tasks using Ilama single A 100 GPU. And the results shown in Tab. 17
22

---

Published as a conference paper at ICLR 2024
Table 17: Infference time of the retrieval framework and generalization model.
Task
Retrieval
Generation
Sum
Proportion
Finance
7.2ms
147.6ms
154.8ms
4.7%
MMLU
9.7ms
1mm
170.9ms
5.7%
Medical
8ms
2600ms
2608ms
0.3%
In terms of storage, show the memories used for the additional parameters in Tab. [5] Moreover, it is worth noting that while our method requires additional parameters, these parameters are only used to store the weight increments. They do not incur any computational cost or increase the complexity of the original models.
A.12 COMPARED METHODS
Below the detailed descriptions of the methods we have chosen compare:
Fine-tune (de Masson D Autume et al. 2019 Wang al. 2020 Fully fine-tune all model param-
eters to adapt to sequential downstream tasks without additional episodic or modular components.
Replay (Razdaibiedina et al., 2023): incorporates mechanism to replay samples from previous tasks stored in the memory butfer during whole model fine-tuning, ensuring that the model retains knowledge from old tasks.
MBPA+ (de Masson D' Autume et al. 2019): augments the BERT model with an episodic mem- ory module, storing all seen examples. It performs experience replay during training and uses K-nearest neighbors for local adaptation at test time.
IDBR (Huang al. 202T): divides the representation learning process into task-specific and task- generic spaces to attain effective representation for BERT model. This method involves continual training of the model while incorporating data replay and regularization loss.
LFPT5 (Qin & Joty| 2021): leverages prompt tuning (PT) from T5 simultaneously train the model as a task solver and a data generator. leverages experience replaying during the learning process, requiring only limited amount of resources.
ProgPromt (Razdaibiedina al.) (023): utilizes prompt tuming to adapt models for individual downstream tasks by employing a distinct set of prompts for each task and sequentially concate- nating them with previously learned prompts. During inference, Progressive Prompts assumes that the task identifier is known, enabling the model to appropriately select the corresponding prompts.
23


Reviews:
[{"review_id": "XoeBpgSxeB", "reviewer": "Reviewer_3jYk", "summary": "The authors present a novel approach aimed at continual learning of new tasks without the requirement of revisiting examples from preceding tasks, nor necessitating the taskID during the inference phase on the examples from the trained tasks. The proposed method hinges on the learning of keys and values, represented by low-rank weight matrices. Initially, the keys are learned for a task by aligning them closely with the input features derived using a frozen encoder from examples of the task. Subsequently, based on the keys, the corresponding low-rank matrices are fine-tuned. The method demonstrates superior performance compared to previous approaches.", "questions": "1. Is the preparation phase conducted collectively for all tasks, or is it executed sequentially—preparation followed by fine-tuning—for each task individually?\n2. For clarification, during each task, are the keys and low-rank weights updated utilizing the examples specific to that task? Upon the introduction of a new task, are these keys and low-rank weights frozen, or are they further fine-tuned?\n3. What is the initial count of keys, and is it dependent on the number of future tasks? If yes, this assumption appears unreasonable as the total count of tasks is undisclosed initially.\n4. How is the aggregation across groups achieved, especially given that in each group, the query is of reduced size, and are keys of reduced size too?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method exhibits good results on continual learning benchmarks as evidenced by the experiments \n2. The simplicity of the method, along with the lack of additional overhead, stands out.", "weaknesses": "1.The method's description appears somewhat convoluted, making it challenging to grasp the benefits fully. Relevant questions highlighting each part are in the Questions section.\n\n2. A basic baseline employing an average feature extractor across examples of a task as a key and training a singular low-rank weight for each task as value is absent. In this method, each task would have just one key and one low-rank weight, and retrieval for examples from a new task or an existing task could be executed using either Top-1 or Top-k with existing keys.\n\n3. The retention of performance on trained tasks seems less beneficial since for inference on an example originating from one of the trained tasks, the already trained model could simply be employed. The principal advantage of these continual systems lies in their capability to be deployed for inference as zero-shot evaluation. However, the results from Table 6 appear unconvincing as the baseline LLAMA model already exhibits decent performance. A more robust experiment could entail learning a sequence of tasks continually and evaluating the zero-shot performance with every newly learned task, please see training datasets from T0 Held-in as depicted in Figure 2 from https://arxiv.org/abs/2110.08207 for experimental setup.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oSeu6zbRCM", "reviewer": "Reviewer_US6d", "summary": "- This paper proposes the Scalable Language Model (SLM) which accumulates knowledge over a sequence of tasks in a continual learning setup.\n- In SLM, for each task, the task knowledge is preserved in vector space as multiple (key, value) pairs where the keys are the vector representations of the centroid of the distributions of different groups within the task distribution, and the values are the corresponding weight increments which indicate the change/delta in weight parameters over the pre-trained model parameters after training on the task dataset. Here, low-rank adaptation techniques are used to store the weight increments to avoid computational overhead.\n- SLM integrates vector space retrieval into the language model to achieve scalable knowledge expansion and management \n- Specifically, SLM uses (a) Dynamic Task-related Knowledge Retrieval (DTKR) to fetch the most relevant knowledge (weight increments) for an input instance from the vector space based on its distribution, and (b) Joint Adaptive Re-parameterization (JARe) to adaptively consolidate the fetched weight increments and re-parameterize the model parameters to align them with the downstream task distribution.\n- In this way, SLM eliminates reliance on \"regularization constraints\" over weight parameters, \"data replay\" and \"inference task-IDs\", which makes it generalizable and scalable.\n- Extensive experimental setup demonstrates the efficacy and stability of the model. The proposed SLM model achieves an 80% reduction in catastrophic forgetting while only losing 0.5% of the performance on the BERT benchmark.", "questions": "- As the pretrained model $f_s$ is frozen and $f_{\\theta}$ is always initialized with the same pre-trained initial weights, the keys of tasks with similar distributions can overlap with each other. In such cases, their values i.e., weight increments, should also be similar to each other. To understand this numerically, have authors done any such analysis to study the similarity between (key, value) pairs across tasks?\n- Related to the above question, during the inference phase if DTKR provide similar (key, value) pairs from different tasks, then is there any mechanism to avoid this as they will provide redundant information and can be ignored for other relevant but non-overlapping knowledge?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Originality\n\t- The authors have proposed a novel continual learning (CL) method, the Scalable Language Model (SLM), which eliminates the use of \"regularization constraint\" and \"data replay\" using vector space retrieval of relevant past knowledge, thus making it scalable across a variety of downstream tasks.\n\t- Although it seems that SLM aligns with the CL methods which incorporate additional trainable parameters for each encountered task in the sequence, however, SLM does not append additional parameters in the model itself, the knowledge acquired from the past tasks (weight increments) can be selectively utilized for future tasks thus enabling efficient knowledge transfer, and does not require the task-ID at the inference time which makes it more generalizable.\n- Quality\n\t- The motivation is well-founded and the claims are sound.\n- Clarity\n\t- Mathematical formulation is very detailed and explanatory.\n\t- Paper is clearly presented and easy to follow.\n- Significance\n\t- The proposed SLM model is effective and consistently demonstrates state-of-the-art performance by outperforming baseline methods on diverse backbones and benchmarks.\n\t- Moreover, the SLM method goes beyond single task type and explores continual learning setup with diverse task types and data domains thus boosting its scalability claims.", "weaknesses": "- Quality\n\t- The number of data points in a task should also be considered while using the weight increments from that task during the retrieval step, otherwise, weight increments of the task with 100 data points will have the same importance/impact as the task with 10000 data points. Just like it happens in the weighted averaging strategy used in the FedAvg algorithm in federated learning settings.\n\t- The computational complexity of the proposed model seems high as the training is done for one data instance at a time and that too involves vector space retrieval of top-K relevant weight increments for each training example. Therefore, it would be fair if the authors could provide a comparison of time spent in training the proposed SLM method as compared to the baseline methods.\n- Clarity\n\t- It would further improve the understanding of the proposed method if authors could also provide an algorithm for the inference phase.\n\t- It is not clear how authors have performed the initialization of (key, value) pairs for each task as mentioned in algorithm 1. It would be better if the authors could provide the mathematical formulation of this initialization strategy.\n- Significance\n- Typographical errors\n\t- [Section-3.1, Line-1] \"researchs have\" -> \"research has\" or \"works have\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "34mxkJdW9u", "reviewer": "Reviewer_phuQ", "summary": "This paper tackles the continual learning of large language models. The major idea is to save the individual task adaptation parameters in an efficient manner by using LoRA, then during the adaptation time, they proposed a way to mix the LoRA parameters. To mix such LoRA parameters, this paper suggests defining a metric function that measures the similarity of the task by using the extracted feature from an external model, i.e., sentence BERT. The paper shows better results than prior continual learning works in LLMs in several domains.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "**Strengths**\n\n(1) The overall presentation is very good, and the writing is very clear.\n\n(2) The method is sound and shows effective performance.\n\n(3) This direction resembles the recent interests of model soup [1], and also some prior works in meta-learning that linearly interpolates the learned modulations to adapt the current model [2]. I think it would be great to discuss the connection.\n\n------------\n\nReference\\\n[1] Wartsman, Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, ICML 2022\\\n[2] Triantafillou et al., Learning a Universal Template for Few-shot Dataset Generalization, ICML 2021", "weaknesses": "**Weaknesses**\n\n(1) I disagree with the claim regarding the weakness of replay-based methods in the introduction. \n- While I do agree that replay-based methods require additional storage to save the previous datasets, this paper also requires saving the parameters of the previous task's low-rank parameters and saving the keys. \n- Since text dataset saving does not require much storage, **I believe comparing the storage is needed** for a fair comparison.\n\n(2) Comparing with the upper bound performance will be interesting by assuming the ideal update is made (i.e., D(p, p_i)=0 for non-relevant tasks and D(p, p_i)=1 for the current task). Maybe such results are not an upper bound, as the method has benefits on forward transfer.\n\n(3) Also, as the proposed and (c) task-id needed scenarios assume additional parameters (for adaptation), I believe outperforming replay-based, and regularization (e.g., EWC) is somewhat trivial. Furthermore, the proposed method requires an additional model like sentence-Bert.\n\n(4) Due to the fact that data replay is not a big issue (as mentioned in (1)), I think adding more baselines related to data replay is needed, e.g., online SGD or other replay-based approaches [1].\n\n------------\n\nOverall, I quite liked the paper's approach, and the paper has the strength to be accepted to the venue. While there are weaknesses (e.g., additional parameters and network like Sentence-BERT), I believe they can be solved by re-claiming the weakness of prior methods and by highlighting the weakness of the proposed method in the manuscript, e.g., in the discussion section. Therefore, I recommend, weak acceptance, assuming that the weakness will be addressed throughout the rebuttal.\n\n------------\n\n**Reference**\\\n[1] He at al., Analyzing the Forgetting Problem in Pretrain-Finetuning of Open-domain Dialogue Response Models, EACL 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ua7LBO3xT6", "reviewer": "Reviewer_hwoc", "summary": "This presents Scalable Language Model (SLM), a new continual learning method for language modeling. It combines frozen large language models with Joint Adaptive ReParameterization (JARe) to adaptively adjust for new tasks. Experimental results show strong performance compared to popular continual learning methods.", "questions": "NA", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This is first paper to utilize a large language model such as llama2-7b to study continual learning problems on a larger scale. This should bring the research in the domain to the next level.\n2. Experimental results are strong.\n3. The proposed method is novel, and is indeed a new paradigm for continual learning. Prior methods heavily rely on memory replay, and this method explores a very different direction.", "weaknesses": "1. It may be harder to reproduce the method compared to prior ones, as it is quite different from existing methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)