Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal (2024)

Abstract: Large language models (LLMs) suffer from catastrophic forgetting during continual learning. Conventional rehearsal-based methods rely on previous training data to retain the model’s ability, which may not be feasible in real-world applications. When conducting continual learning based on a publicly-released LLM checkpoint, the availability of the original training data may be non-existent. To address this challenge, we propose a framework called Self-Synthesized Rehearsal (SSR) that uses the LLM to generate synthetic instances for rehearsal. Concretely, we first employ the base LLM for in-context learning to generate synthetic instances. Subsequently, we utilize the latest LLM to refine the instance outputs based on the synthetic inputs, preserving its acquired ability. Finally, we select diverse high-quality synthetic instances for rehearsal in future stages. Experimental results demonstrate that SSR achieves superior or comparable performance compared to conventional rehearsal-based approaches while being more data-efficient. Besides, SSR effectively preserves the generalization capabilities of LLMs in general domains.

heading: 1 Introduction
text: Large language models (LLMs) have demonstrated remarkable performance across various natural language processing (NLP) tasks (Touvron et al., 2023b; OpenAI, 2023). In real-world applications, LLMs are often updated in a continual learning (CL) manner (de Masson d'Autume et al., 2019), where new instruction tuning data is incrementally introduced over time. However, a significant issue that limits the effectiveness of LLMs is catastrophic forgetting, which refers to the LLM’s tendency to forget previously acquired knowledge when learning new instances (Kirkpatrick et al., 2017; Li et al., 2022; Luo et al., 2023).
∗Corresponding author.
To mitigate catastrophic forgetting, a line of work focuses on rehearsing previous training instances (de Masson d'Autume et al., 2019; Rolnick et al., 2019; Scialom et al., 2022). These rehearsal-based methods maintain the model’s ability by training on real data from previous training stages. However, the real data may not always be desirable in practical applications. For instance, when conducting continual learning based on a publicly-released LLM checkpoint (e.g. Llama-2chat), the availability of the original training data may be non-existent. This raises an interesting research question: Can we maintain the LLM’s ability during continual learning without using real data in previous training stages?
We propose the Self-Synthesized Rehearsal (SSR) framework to mitigate catastrophic forgetting in continual learning. As shown in Figure 1, unlike standard rehearsal-based continual learning that samples training instances from previous stages as rehearsal data, SSR framework uses the LLM to generate synthetic instances for rehearsal. Specifically, we first use the base LLM to generate
1416
synthetic instances, conducting in-context learning (ICL) with few-shot demonstrations. These demonstrations can be collected from the previous data or human-constructed containing similar knowledge to the previous data. Then, the latest LLM is used to refine the outputs of synthetic instances to retain the latest LLM’s ability. Finally, we select diverse high-quality synthetic instances for rehearsal in the future stages.
Extensive experiments on the task sequences derived from the SuperNI dataset (Wang et al., 2022) demonstrate that SSR has superior or comparable performance compared to the conventional rehearsal-based approaches, with higher data utilization efficiency. Besides, experiments on AlpacaEval and MMLU (Hendrycks et al., 2021) show that SSR can also effectively preserve the generalization capabilities of LLMs in general domains. We release our code and data at https: //github.com/DeepLearnXMU/SSR.

heading: 2 Related Work
text: Learning a sequence of datasets continually while preserving past knowledge and skills is a crucial aspect of achieving human-level intelligence. Existing approaches to continual learning can be broadly categorized into three main categories: (i) regularization-based, (ii) architecture-based, and (iii) rehearsal-based methods. Regularization techniques (Kirkpatrick et al., 2017; Cha et al., 2021; Huang et al., 2021; Zhang et al., 2022) control the extent of parameter updates during the learning process, preventing interference with previously learned tasks. Nonetheless, these methods typically rely on hyperparameters that need to be carefully tuned for optimal performance. Architecturebased approaches (Xu and Zhu, 2018; Huang et al., 2019; Razdaibiedina et al., 2023) often take a different approach by learning separate sets of parameters dedicated to individual tasks. This enables the model to specialize and adapt its parameters for each task, avoiding interference between tasks and preserving task-specific knowledge. However, these approaches will introduce additional training parameters, which may not be very flexible and feasible for various LLMs.
Therefore, we focus on rehearsal-based methods (de Masson d'Autume et al., 2019; Rolnick et al., 2019), which are also called replay-based methods. These methods typically involve the storage of a subset of data from previous tasks. These stored
data are used for future rehearsal through techniques such as experience replay (Rolnick et al., 2019) and representation consolidation (Bhat et al., 2022). Prior rehearsal-based approaches for language models mainly focus on using a little bit of precedent data (Scialom et al., 2022; Mok et al., 2023; Zhang et al., 2023b). However, these approaches often ignore discussion on real-world applications where previous data may be limited or unavailable. Although data-free knowledge distillation methods (Yin et al., 2020; Smith et al., 2021) introduce auxiliary generative models for data construction, they are primarily designed for classification tasks, which may not be effective in LLMs, where a wide range of NLP tasks are involved. Additionally, similar to introducing teacher models (Miao et al., 2023; Cheng et al., 2023; Huang et al., 2024), it can be challenging and time-consuming to train additional generative models. Self-distillation methods (Zhang et al., 2023a) may be useful, but catastrophic forgetting of the latest LLMs and the knowledge discrepancy among LLMs from distinct stages are still inevitable challenges.
In this work, we propose a rehearsal-based continual learning framework in which LLMs can be trained on self-synthesized data to retain the knowledge of the previous stages, with several demonstrations used during data construction. Unlike other approaches, our framework does not depend on additional generative models for data construction or require previous real data for rehearsal. This offers advantages in terms of data efficiency and application flexibility.

heading: 3 Rehearsal-Based Continual Learning
text: In continual learning, the LLM is sequentially updated for T stages, with each stage t having its corresponding instruction data d(t). To mitigate catastrophic forgetting, in each stage t, rehearsalbased methods (Scialom et al., 2022; Mok et al., 2023) sample some training instances of previous stages to expand the training data in the current stage. Formally, the augmented training data D(t) can be formulated as follows:
D(t) = d(t) ⋃ t−1∑
i=1
(rd(i)), (1)
where r represents the rehearsal ratio determining the percentage of sampled training instances. Finally, we use D(t) to fine-tune the LLM θ(t−1), obtaining the updated LLM θ(t). Particularly, in
the first stage, we fine-tune the base LLM θ(0) on D(1) = d(1). By doing so, the catastrophic forgetting problem of LLM can be effectively alleviated, which has been verified in previous studies (Scialom et al., 2022; Mok et al., 2023; Zhang et al., 2023b).

heading: 4 Our Framework
text: In this section, we detail the proposed SelfSynthesized Rehearsal (SSR) framework, which involves three main steps: 1) in-context learning based instance synthesis, 2) synthetic output refinement, and 3) rehearsal with selected synthetic instances, as illustrated in Figure 2.
In-Context Learning Based Instance Synthesis Rehearsal-based methods utilize the training instances to cache the knowledge acquired by the LLM from previous stages. Nevertheless, in realworld scenarios where a publicly-released LLM checkpoint is used, the availability of original training data may be limited. To address this limitation, we try to generate rehearsal training instances synthetically. To ask the LLM to follow abstract instructions, we leverage the in-context learning (ICL) capability of LLMs for instance synthesis.
Formally, during each training stage t, we first acquire K demonstrations {(xk, yk)}Kk=1. To retain previously acquired knowledge, these demonstrations can be collected from the previous instruction data d(t−1) or manually constructed containing similar knowledge to d(t−1). We concatenate all demonstrations and utilize the base LLM to generate the synthetic instance (x̂, ŷ) = LLM(concatKk=1(xk, yk); θ (0)). By reordering the
demonstrations and sampling multiple times, we can easily obtain different synthetic instances. It should be noted that we use the base LLM θ(0) rather than the latest LLM θ(t−1) to conduct ICL. This is because the ICL ability of LLMs tends to exhibit a significant degradation after supervised fine-tuning (SFT) on specific tasks, as analyzed in (Wang et al., 2023).
Synthetic Output Refinement Through the above process, we obtain a series of synthetic instances, some of which, however, may be of low quality with unreliable outputs. To address this issue, we use the latest LLM θ(t−1) to refine the output of each synthetic instance: ȳ = LLM(x̂; θ(t−1)). By doing so, we can ensure that each refined synthetic instance (x̂, ȳ) retains the knowledge acquired by the latest LLM.
Rehearsal with Selected Synthetic Instances Finally, we select the refined synthetic instances for rehearsal. During this process, to ensure the diversity and quality of selected synthetic instances, we first adopt a clustering algorithm (e.g. K-means) to group {(x̂, ȳ)} into C clusters. Then we calculate the distance between each synthetic instance and its corresponding cluster centroid, and finally select a certain amount of synthetic instances near cluster centroids as the rehearsal data.
Formally, we use d̂(t−1) to represent the set of selected synthetic instances. Thus, the augmented training data in stage t can be formulated as
D̂(t) = d(t) ⋃ t−1∑
i=1
d̂(i), (2)
where d̂(i) denotes the selected synthetic data similar to the previous training data d(i). Note that d̂(1), d̂(2), ..., d̂(t−2) have been generated in previous stages, thus we will not regenerate them in stage t. Finally, we use D̂(t) to fine-tune θ(t−1), updating the LLM as θ(t). In this way, the LLM can preserve previously learned knowledge even without real data from previous stages.

heading: 5 Experiments

heading: 5.1 Setup
text: Datasets We conduct several groups of experiments on the SuperNI dataset (Wang et al., 2022), a vast and comprehensive benchmark dataset for instruction tuning. First, to simulate a typical continual learning process, we choose a subset of 10 tasks from SuperNI, encompassing various categories and domains. Each task is trained in a separate stage for empirical studies. For each task, we randomly sample 2,000 instances for training and 500 for evaluation. Please refer to Appendix A for the details of these tasks. To simplify the empirical study, we adopt default continual learning orders on {5, 10} SuperNI tasks: QA → QG → SA → Sum. → Trans. (→ DSG → Expl. → Para. → PE → POS).
Base LLMs Our main experiments involve three base LLMs: Llama-2-7b (Touvron et al., 2023b), Llama-2-7b-chat (Touvron et al., 2023b), Alpaca7b (Taori et al., 2023).
Baselines We compare SSR with the following baselines:
• Multi-task Learning (MTL). This is the most commonly used baseline, where all tasks are trained simultaneously.
• Non-rehearsal. It is a naive baseline that the LLM is fine-tuned with only the instruction data d(t) in each stage t.
• RandSel(r) (Scialom et al., 2022). We randomly sample r = {1, 10}% of the original instruction data for each previous task. Note that as mentioned in (Scialom et al., 2022), the abilities of language models can be effectively preserved with r = 1%.
• KMeansSel(r). Unlike the above approach, we first employ K-means clustering to group real instances into 20 clusters and then select
r = {1, 10}% of instances with the highest similarities to the cluster centroids. Here we adopt SimCSE (Gao et al., 2021) to obtain instance representations before clustering.
Evaluation Metrics Due to the diversity and the open-ended sequence generation characteristic of SuperNI tasks, we adopt the ROUGE-L metric (Lin, 2004) to evaluate the performance of LLM on each task. This metric shows a good alignment with human evaluation, as demonstrated by Wang et al. (2022). Besides, we follow Lopez-Paz and Ranzato (2017) to consider the following metrics based on ROUGE-L. Here a(i)j denotes the ROUGE-L performance on the task j in training stage i.
• Average ROUGE-L (AR). It is used to quantify the final average performance of LLM across all T tasks in stage T , which is defined as follows:
AR = 1
T
T∑
i=1
a (T ) i . (3)
• Forward Transfer (FWT). It evaluates the LLM’s generalization ability on unseen tasks, measuring the average zero-shot performance a (i−1) i on the next task i in each stage i− 1:
FWT = 1 T − 1 T∑
i=2
a (i−1) i . (4)
• Backward Transfer (BWT). It is a metric used to evaluate the impact of learning subsequent tasks on a previous task. For each task i except for the final one, it compares the final performance a(T )i to the online performance a (i) i in stage i:
BWT = 1 T − 1 T−1∑
i=1
(a (T ) i − a (i) i ). (5)
A negative BWT indicates that the LLM has forgotten some previously acquired knowledge.
Implementation Details During training, we utilize LoRA (Hu et al., 2021) with query and value projection matrices in the self-attention module to train LLMs, setting the LoRA rank to 8 and the dropout rate of 0.1. We employ the Adam optimizer with an initial learning rate of 2e-4. The
global batch size is 32 for all our experiments. Besides, we set the maximum length of the input to 1,024 and the counterpart of the output to 512. Following Luo et al. (2023), we train each LLM for 3 epochs and use the final checkpoint for evaluation.
To conduct ICL, we utilize 1% of the training data from SuperNI tasks as demonstrations, considering K = 2 demonstrations and sampling multiple times to obtain diverse synthetic instances. When clustering instances, we use K-means clustering with C = 20 clusters for synthetic instances of SuperNI tasks, which is similar to KMeansSel(r).

heading: 5.2 Experiments on 5 SuperNI Tasks
text: Table 1 presents the experimental results on 5 SuperNI tasks. Overall, regardless of the continual learning order and the base LLM, SSR consistently outperforms all rehearsal-based baselines, exhibiting an improvement of approximately 2 scores in both the AR and BWT metrics. This result shows the superiority of SSR in mitigating catastrophic forgetting. Particularly, SSR closely approaches MTL which sets the upper bound of the AR performance. Besides, compared to rehearsal-based baselines, RandSel(r) and KMeansSel(r), SSR is
more data-efficient with only 1% real data utilization for ICL and only synthetic data of previous stages for rehearsal.
After further analysis, we draw the following conclusions:
Non-rehearsal vs. rehearsal The non-rehearsal baseline shows the poorest, indicating severe catastrophic forgetting. Besides, it exhibits the highest metric variance, signifying its lack of robustness in different CL orders. In contrast, SSR and rehearsalbased baselines maintain better and more consistent performance regardless of the CL order.
Effect of r The appropriate rehearsal ratio r varies depending on the continual learning orders. A higher r is beneficial in certain cases, as observed in CL orders 2 and 3. However, this is not always the case. In CL order 1, regardless of the instance sampling strategy employed, the rehearsal-based baselines with r = 1% consistently outperform their r = 10% counterparts, respectively.
RandSel(r) vs. KMeansSel(r) When comparing RandSel(r) and KMeansSel(r), we can observe that K-means clustering-based selection of previous
data for rehearsal may slightly enhance the model performance when using only r = 1%, demonstrating the importance of data representativeness. However, when r is set to 10%, significant differences in model performance may not be observed for LLMs such as Llama-2-7b and Alpaca-7b.

heading: 5.3 Experiments on 10 SuperNI Tasks
text: To further investigate the effectiveness of SSR in longer continual learning sequences, we evaluate SSR and all baselines on 10 SuperNI tasks. Table 2 shows that SSR surpasses all rehearsal-based baselines across all metrics. Moreover, as illustrated in Figure 3, SSR consistently achieves better performance in terms of AR and BWT compared to rehearsal-based baselines throughout the entire continual learning process. Although SSR with Llama-2-7b as the base LLM falls behind RandSel(10%) in terms of FWT in the early stage, it gradually strengthens its performance as the number of training stages increases, eventually surpassing RandSel(10%). Please refer to Appendix C for more details.

heading: 5.4 Experiments on the Generalization Capability Preservation of Alpaca-7b
text: To further analyze the preservation of LLM’s generalization ability in a broader domain beyond SuperNI tasks, we utilize Alpaca-7b to conduct continual learning on 5 SuperNI tasks and then investigate whether SSR can preserve the abilities of Alpaca7b gained from the Alpaca-52k dataset1. Here, Llama-7b (Touvron et al., 2023a) serves as the base LLM θ(0), and Alpaca-7b is considered as the updated LLM θ(1) after fine-tuning on Alpaca-52k. Therefore, we also generate synthetic instances similar to Alpaca-52k for SSR and use Alpaca-52k for rehearsal-based baselines.
We evaluate the LLM from three perspectives: 1) General instruction-following ability. We use AlpacaEval 2.02 as an automatic evaluator. Concretely, we measure the LLM’s performance in terms of the win rate, comparing the LLM’s generations with those generated by gpt-4-turbo. To minimize financial costs, we utilize ChatGPT as the evaluation annotator. 2) General language understanding ability. We leverage the MMLU (Hendrycks et al., 2021) benchmark, where accuracy (Acc.) is used as the evaluation metric. 3) Task-specific ability. We evaluate the AR performance of the LLM on 5 SuperNI tasks. Please refer to Appendix D for more details.
From Table 3, we observe that SSR not only achieves the best on the 5 newly learned tasks but also maintains comparable or even superior performance on AlpacaEval and MMLU. These findings suggest that SSR effectively preserves the
1https://huggingface.co/datasets/ tatsu-lab/alpaca
2https://github.com/tatsu-lab/alpaca_ eval
generalization ability of Alpaca-7b throughout the continual learning process, even in the absence of Alpaca-52k as rehearsal data. This highlights the great potential of SSR in general domains.

heading: 5.5 Analysis
text: Effect of in-context learning To investigate the effect of in-context learning on SSR, we conduct experiments for Llama-2-7b on 5 SuperNI tasks, introducing the following variants: (a) Llama-27b⇒Llama-7b. This variant validates the scenario where we acquire a public-released fine-tuned LLM checkpoint, but the original base LLM is unavailable. Thus we employ a different LLM Llama-7b to conduct ICL. (b) Llama-2-7b⇒Alpaca-7b. Similar to the above one, but using Alpaca-7b. (c) train demos⇒new demos. In this variant, we utilize demonstrations that are not included in the previous training data but belong to the same SuperNI task, simulating manually constructed demonstrations to conduct ICL. (d) w/ input-only demos. This variant utilizes only the instance inputs in previous stages as demonstrations, simulating the scenario where real instances lack output annotations.
Table 4 illustrates that SSR can perform well even without the original base LLM or demonstrations from previous training data to conduct ICL. This provides convenience for replacing some ICL components in practical application scenarios. Comparing SSR and its variants (a) to (b), we notice a slight decrease in performance when conducting ICL using Alpaca-7b. This highlights the
limitation of this fine-tuned LLM in terms of ICL capability. Besides, ICL with input-only demonstrations also yields comparable performance, indicating that output annotations are also not essential for ICL, further verifying the robustness of SSR.
Effect of synthetic output refinement In Section 4, we claim that synthetic output refinement provides more reliable synthetic outputs from the latest LLM. To verify the effectiveness, we conduct an experiment where SSR is implemented without synthetic output refinement.
As illustrated in Figure 4, this results in lower AR values and significant BWT inferiority, highlighting the negative impact of data noise originating from the base LLM. In contrast, by incorporating the synthetic input with the refined output, SSR can maintain the predictive distribution of the latest LLM during rehearsal, preserving the acquired knowledge.
Effect of K-means clustering on synthetic instance selection In terms of application flexibility, we utilize an unsupervised K-means clustering algorithm, fitting and predicting solely with synthetic instances. To explore the effect of K-means clustering, we compare SSR with the following variants: (a) SSR w/o KMeans: Random selection of synthetic instances. (b) SSR w/ sup. KMeans:
K-means clustering-based synthetic instance selection, with the supervision of real instances to fit the K-means clustering and then predict on synthetic instances.
Figure 5 shows that the supervised K-means clustering method leads to a slight improvement in AR and reduces forgetting with larger BWT. Thus, incorporating real instances during clustering may allow for a more representative selection. Nonetheless, clustering is not essential in the absence of supervision, because SSR with random selection for synthetic instances can outperform SSR, sometimes even surpass SSR with supervised K-means. This indicates a certain level of robustness of SSR.
Real instances vs. synthetic instances Our main experiments demonstrate surprising results that rehearsal with synthetic instances may surpass those with real instances. For the comparison of real and synthetic instances, we consider the following variants: (a) RandSel (10%): Real inputs and outputs for rehearsal. (b) RandSel (10%) w/ syn. op.: Real inputs and synthetic outputs for rehearsal. Concretely, we regenerate the outputs of randomly sampled previous instances by the latest LLM, with
similar operations to SSR. Figure 6 demonstrates that RandSel (10%) with real inputs and synthetic outputs for rehearsal, outperforms RandSel (10%) utilizing only real instances. Meanwhile, SSR, which leverages both synthetic inputs and outputs for rehearsal, achieves intermediate performance, sometimes even surpassing the other two. This indicates that real instances are not always essential and appropriate for the continual learning of LLMs. As depicted in Figure 7, real instances often lead to a slower descent in loss. Therefore, they may not be conducive to optimization due to the distribution gap between distinct datasets. Conversely, synthetic instances, with lower model perplexity, embody the LLM’s realtime acquired knowledge, which aids in smoothing the data distribution and discovering better local optima for the LLM.
Effect of synthetic instance quantity Here, we define r̂ = |d̂(i)|/|d(i)| as the SSR ratio, which represents the proportion of selected synthetic data compared to the original training data size. By default, we retain synthetic instances at an SSR ratio of r̂ = 10%. However, as depicted in Figure 8, increasing r̂ can lead to further improvements in the final AR, highlighting the potential of SSR. Moreover, it is important to note that the training cost and memory limit should also be taken into consideration when determining an appropriate r̂.
SSR vs. regularization-based and architecturebased methods In this paper, we focus on generating synthetic rehearsal data for instruction tuning. Prior work (Zhang et al., 2023b) has demonstrated that rehearsal-based approaches are generally superior to regularization-based and architecture-based ones for instruction tuning of language models. Table 5 presents experimental results using two classical regularization-based baselines (L2 and
EWC) for Llama-2-7b under CL order 1, with SSR still demonstrating its superiority. Besides, these lightweight strategies can be easily combined with SSR, potentially leading to further improvements in model performance. Moreover, architecturebased approaches heavily rely on additional taskspecific parameters, which may not be practical in real-world applications where the inference time of LLMs is a crucial consideration.

heading: 6 Conclusion
text: In this work, we propose Self-Synthesized Rehearsal (SSR), a continual learning framework for mitigating catastrophic forgetting in LLMs, to effectively preserve knowledge without relying on real data during rehearsal. Through extensive experiments, SSR demonstrates its data efficiency and superior performance to conventional rehearsal-based approaches. Besides, it preserves LLM’s generalization capability both in specific and general domains, with flexibility and robustness in real-world scenarios. Overall, SSR presents a promising solution for continual learning of LLMs in real-world settings, with implications for maintaining the acquired abilities of LLMs.
Limitations
Although SSR demonstrates superior performance in terms of AR and BWT, it may not always achieve the best FWT score, as shown in Figure 3(b). However, as discussed in Subsection 5.4, SSR effectively preserves the generalization capabilities of LLMs in general domains, highlighting its practical value. For the final FWT results on the 5 SuperNI tasks, please refer to Table 7 in Appendix B. Additionally, synthetic instances generated by LLMs may potentially contain unsafe content due to data bias during training.

heading: Acknowledgements
text: The project was supported by National Key R&D Program of China (No. 2022ZD0160501), National Natural Science Foundation of China (No. 62276219), and the Public Technology Service Platform Project of Xiamen (No.3502Z20231043). We also thank the reviewers for their insightful comments.

heading: A Details of the Selected 10 SuperNI Tasks
text: Table 9 lists all of the selected 10 SuperNI tasks for our main experiments. To simplify the description, we utilize abbreviations to represent these tasks in this paper. The SuperNI dataset can be found at https://github.com/allenai/ natural-instructions.

heading: B More Details of Experiments on 5 SuperNI Tasks
text: Table 6 lists all of the continual learning orders on 5 SuperNI tasks conducted in our experiments.
Table 7 shows the final FWT results on 5 SuperNI tasks. Our SSR framework surpasses r = 1% rehearsal-based baselines but falls behind r = 10% counterparts. However, as illustrated in Figure 3, the FWT performance of SSR will finally surpass the r = 10% rehearsal-based baselines as the number of training stages increases.

heading: C More Details of Experiments on 10 SuperNI Tasks
text: Figure 9 depicts the heatmaps of the ROUGE-L performance for Llama-2-7b across 10 SuperNI tasks. A visual inspection reveals that the non-rehearsal baseline rapidly forgets previously learned tasks when subsequent tasks are learned. In contrast, after learning a task in its respective stage, SSR demonstrates minimal color change in future stages, indicating the least amount of forgetting.
Table 8 highlights that SSR retains its superiority in terms of AR and BWT for Llama-2-7b-chat and Alpaca-7b on 10 SuperNI tasks. However, its FWT performance is comparable or inferior to that of rehearsal-based baselines. It is worth noting that Alpaca-7b tends to exhibit poorer performance regardless of the CL approaches employed.
Model AR FWT BWT
Llama-2-7b-chat
MTL 62.68 - - Non-rehearsal 47.05 22.24 -20.73 RandSel(1%) 60.70 25.77 -4.17 KMeansSel(1%) 61.53 26.62 -4.24 RandSel(10%) 58.46 27.99 -3.34 KMeansSel(10%) 59.27 27.52 -3.12 SSR 63.34 27.70 -1.27
Alpaca-7b

heading: D More Implementation Details of Experiments on the Generalization Capability Preservation of Alpaca-7b
text: The continual learning order is as follows: (Alpaca52k →) QA → QG → SA → Sum. → Trans. To conduct ICL, we utilize 1% / 0.1% of the training data from SuperNI tasks and Alpaca-52k as demonstrations, respectively. When clustering instances, we use KMeans with C = 20 clusters for synthetic instances of SuperNI tasks and C = 520 for those of Alpaca-52k. We retain synthetic instances with the SSR ratio r̂ = 10% / 1% for SuperNI tasks and Alpaca-52k, respectively.

heading: Future Work
text: In contrast, after learning a task in its respective stage, SSR demonstrates minimal color change in future stages, indicating the least amount of forgetting. Table 8 highlights that SSR retains its superiority in terms of AR and BWT for Llama-2-7b-chat and Alpaca-7b on 10 SuperNI tasks. However, its FWT performance is comparable or inferior to that of rehearsal-based baselines. It is worth noting that Alpaca-7b tends to exhibit poorer performance regardless of the CL approaches employed. Model AR FWT BWT
Llama-2-7b-chat
MTL 62. 68 - - Non-rehearsal 47. 05 22. 24 -20. 73 RandSel(1%) 60. 70 25. 77 -4. 17 KMeansSel(1%) 61. 53 26. 62 -4. 24 RandSel(10%) 58. 46 27. 99 -3. 34 KMeansSel(10%) 59. 27 27. 52 -3. 12 SSR 63. 34 27. 70 -1. 27
Alpaca-7b