Large-scale Lifelong Learning of In-context Instructions and How to Tackle It
Year: 2023

Abstract:
Jointly fine-tuning a Pre-trained Language Model (PLM) on a pre-defined set of tasks with in-context instructions has been proven to improve its generalization performance, allowing us to build a universal language model that can be deployed across task boundaries. In this work, we explore for the first time whether this attractive property of in-context instruction learning can be extended to a scenario in which tasks are fed to the target PLM in a sequential manner. The primary objective of so-called lifelong in-context instruction learning is to improve the target PLM’s instanceand task-level generalization performance as it observes more tasks. DYNAINST, the proposed method to lifelong in-context instruction learning, achieves noticeable improvements in both types of generalization, nearly reaching the upper bound performance obtained through joint training.

heading: 1 Introduction
text: A number of recent studies have shown that Pretrained Language Models (PLMs) only need to undergo a brief fine-tuning process to achieve remarkable instance-level generalization performance within the observed task (Brown et al., 2020). Compared to instance-level generalization within seen tasks, however, the zero-shot cross-task generalization capability of PLMs to unseen tasks is just starting to be explored (Sanh et al., 2021; Wei et al., 2021). In order to build a language model that can generalize across task boundaries and thus be deployed to various task scenarios, in-context instruction learning draws inspiration from how humans can familiarize themselves with a variety of language-related tasks only by following instructions (Mishra et al., 2022). To emulate the human learning process, in-context instruction learning trains the target PLM with both input-output pairs
∗Work done while interning at Amazon Alexa AI (magicshop1118@snu.ac.kr)
†Corresponding Authors
and a set of in-context instructions that contain additional task-specific information.
The introduction of in-context instructions is spearheading a noticeable progress in cross-task generalization within Natural Language Processing (NLP) research. The recent advances in-context instruciton learning (Ouyang et al., 2022; Mishra et al., 2022; Wang et al., 2022) indicate that finetuning a PLM with task-specific instructions has a positive impact on its cross-task generalization capability. One common limitation shared by the existing works is that they require a static, pre-defined set of tasks to jointly train the target PLM. Any learning paradigm bound by the assumption that all of the tasks are pre-defined and non-changing not only incurs a huge memory and computational cost but also raises serious data privacy concerns (De Lange et al., 2021). In this paper, we aim to address such concerns by studying whether it is possible to sequentially fine-tune the target PLM on a stream of large-scale instruction-paired tasks in a lifelong manner.
In the same spirit as joint in-context instruction learning, the primary objective of lifelong in-context instruction learning is to gradually improve both instance- and task-level generalization capabilities as the target PLM observes more train tasks. With the help of lifelong in-context instruction learning, deployment of a universal language model on edge devices with limited memory and computation power becomes easier. In such a resource-constrained setting, it is infeasible to jointly train a language model from scratch on all user data every time a new task is introduced. Moreover, it is difficult to deploy a separate model for every task or utilize an extremely large model with multi-task capability. Instead, we can deploy a continuously evolving language model and train it sequentially on a task stream.
To study the problem of lifelong in-context instruction learning within the context of instance-
12573
and task-level generalization, we adopt SuperNaturalInstructions (Sup-NatInst) (Wang et al., 2022), which is the largest dataset with in-context instructions to date, and restructure it accordingly. Because we believe that cross-language generalization is beyond the scope of this paper, we use the English subset of Sup-NatInst, and from here on, we use Sup-NatInst to refer to the English subset instead of the entire dataset. More details on the characteristics of Sup-NatInst and the data restructuring process are provided in Section 3.
Our proposed approach to lifelong in-context instruction learning, DYNAINST, combines parameter regularization and experience replay. The regularizer employed by DYNAINST is designed to induce wide local minima in the target PLM. Deep neural networks with wide local minima are known to achieve improved generalization performance and become more robust against task distribution shifts (Cha et al., 2020); these two advantages of wide local minima are well-aligned with the objectives of lifelong instruction learning, making it a particularly attractive choice of regularization. To design a memory- and compute-efficient experience replay framework, we devise Dynamic Instruction Replay, which is comprised of Dynamic Instance Selection (DIS) and Dynamic Task Selection (DTS). DIS and DTS flexibly determine which instances and tasks are stored and replayed, respectively. Our experimental results demonstrate that DYNAINST outperforms strong baselines in both instance- and task-level generalization under various experimental scenarios.
Our contributions can be summarized as follows:
• This is the first work to study the potential of lifelong in-context instruction learning as an efficient framework towards building a continuously evolving universal language model.
• We propose DYNAINST, a hybrid approach to lifelong in-context instruction learning that integrates a wide local minima-inducing regularizer and Dynamic Instruction Replay. With extensive experimental results, we verify that DYNAINST outperforms existing baselines from continual learning.
• We present a series of empirical analyses and ablation studies that offer further insights into lifelong in-context instruction learning and the inner-workings of DYNAINST.

heading: 2 Related Works

heading: 2.1 Lifelong Learning
text: Lifelong learning (De Lange et al., 2021; McCloskey and Cohen, 1989) concerns with the problem of learning from a continuous stream of data (Parisi et al., 2019; Chen and Liu, 2018). Thus, what distinguishes lifelong learning from the conventional paradigm of joint training is the sequential characteristic of the learning process, in which only a subset of input data are fed to the model at once. There are largely three different settings for lifelong learning: class, domain, and task incremental settings (De Lange et al., 2021). Here, we focus on the methods for task incremental setting, which is most relevant to the investigated framework of lifelong in-context instruction learning.
Based on how information from each task is stored and utilized later in the task stream, task incremental methods can be categorized into three: parameter regularization-, rehearsal-, and architecture expansion-based methods. Regularizationbased methods (Li and Hoiem, 2017; Aljundi et al., 2018; Liu et al., 2018; Kirkpatrick et al., 2017) discourage re-visiting of inputs from previous tasks and instead introduce an auxiliary regularization term. Rehearsal-based methods (Rebuffi et al., 2017; Lopez-Paz and Ranzato, 2017; Chaudhry et al., 2018b; Shin et al., 2017) store a small number of instances and explicitly reuse the stored instances when training on future tasks. They inevitably result in some memory consumption, but this cost is offset by the clear advantage on the performance side. Lastly, architecture expansionbased methods (Mallya and Lazebnik, 2018; Serra et al., 2018) add new parameters to the backbone architecture each time a new task is presented.
Although lifelong learning is studied mostly within computer vision or robotics-related tasks, lifelong learning with NLP data has also been attracting significant interest. Many of the lifelong
learning works in NLP focus on learning a sequential stream of data that belong in the same task, such as sentiment classification (Chen and Liu, 2018) or task-oriented dialog systems (Mi et al., 2020; Madotto et al., 2021). Recent research efforts aim to explore a more challenging lifelong learning setting that encompasses more than one task (Sun et al., 2019; Kanwatchara et al., 2021), only uses a limited number of instances per task (i.e., fewshot setting (Qin and Joty, 2021)), or generalizes to out-of-distribution data (Lin et al., 2022).

heading: 2.2 Learning with In-Context Instructions
text: The idea of introducing in-context instructions was first proposed by Goldwasser and Roth (2014) who explored whether an automated agent can understand and execute the instruction that is transformed into a comprehensible expression through semantic parsing. In the NLP community, utilizing in-context language-based instructions continues to rise in popularity as an effective method of improving the generalization capability of PLMs. The core concept of in-context instruction learning is to utilize task-specific instructions that provide some description or hint to the task at hand. For instance, the NaturalInstructions-v1 (NI-v1) dataset (Mishra et al., 2022), a predecessor to Sup-NatInst used in our work, contains 64 NLP tasks, all paired with real-world instructions from Amazon Mechanical Turk (AMT) (Paolacci et al., 2010). The instruction schema of NI-v1 and Sup-NatInst resemble each other in that they include the following: taskspecific definition, positive examples, negative examples, and some explanation for why an example is positive or negative.
To the best of our knowledge, this is the first work to explore whether a PLM can be trained sequentially on a stream of instruction-paired tasks. While Yin et al. (2022) seemingly consider a similar problem, their framework, ConTinTin, and ours fundamentally differ in several aspects. The key differences between ConTinTin and our framework are summarized in Table 1. Because ConTinTin includes the joint training step prior to sequentially adapting the jointly-trained model, it is not fully continual. During the adaptation step, ConTinTin observes evaluation tasks and thus ignores the trained model’s generalization capacity to unseen tasks, an important aspect of instruction-based learning. Lastly, it is not as large-scale as ours, and it assumes that for all of the pre-train tasks, the
same number of instances are available. To tackle the ConTinTin framework, Yin et al. (2022) propose a method dubbed InstructionSpeak, which we consider as a baseline approach in Section 5.2.

heading: 3 Problem Definition
text: In this section, we describe Sup-NatInst and how it is restructured for the purpose of lifelong in-context instruction learning. Then, we formulate evaluation metrics for quantifying the instance- and task-level generalization capabilities. The visual summary of problem definition is provided in Figure 1.

heading: 3.1 Data
text: Dataset Sup-NatInst, which contains 757 train tasks and 119 evaluation tasks, is the largest and most comprehensive among existing datasets for in-context instruction learning. For the sake of computational efficiency, we randomly sample 500 out of 757 train tasks. The results reported in the original paper indicate that only minor performance change occurs after the model observes approximately 400 train tasks; therefore, it is reasonable to assume that the results obtained after 500 tasks are sufficient for analyzing the characteristics of large-scale lifelong in-context instruction learning.
The default instruction scheme of Sup-NatInst includes four components: task definition, positive examples, negative examples, and explanation. Unless specified otherwise, all explored approaches leave out negative examples because they have been shown to deteriorate the generalization performance of the target model (Wang et al., 2022). An example of instruction and instance in Sup-NatInst can be found in Section A1 of Appendix. Data Restructuring From here on, we refer to some arbitrary target PLM model as F . We use t to
denote a task, which implicitly includes an instruction, and (x, y) to denote the input and output of a task-specific instance. During the lifelong learning process, the 500 train tasks with L number of labeled instances per task (Ttr = [titr] 500 i=1, where titr = {(xi1, yi1), ..., (xiL, yiL)}) are sequentially fed into F . In this work, we study two different settings for the choice of L: the static instance setting, where the same L number of instances per task are used for training, and the random instance setting, where a changing number of instances are used for each task. Because in real life, it is hard to guarantee that the same number of instances are available for each task, the random instance setting may be considered more realistic. In the random instance setting, we use a random integer value between 1 and L for each train task.
The primary objective of the lifelong learning process is to gradually improve the trained model’s instance- and task-level generalization performance, as more train tasks are visited by F . To measure the instance-level generalization performance, we leave out 100 instances in each train task for evaluation process and treat them as test instances within train tasks: t̃itr = {(x̃i1, ỹi1), ..., (x̃i100, ỹi100)}. To measure the task-level generalization performance, we utilize 100 test instances in each one of 119 evaluation tasks (Teval = [t̃ieval] 119 i=1, where t̃ i eval = {(x̃i1, ỹi1), ..., (x̃i100, ỹi100)}).

heading: 3.2 Evaluation Metrics
text: All metrics are measured with the Rouge-L score (Lin, 2004), which quantifies the sentence-
level structural similarity; thus, a high Rouge-L score corresponds the to improved performance of a language model. The Rouge-L score is used as the default metric in the original Sup-NatInst paper as well. We denote the Rouge-L score of the j-th task after training on the k-th task as: Ak(tj). GENInst measures the degree of instance-level generalization and is formulated as: 1k ∑k i=1Ak(t̃ i tr). This is equivalent to the Rouge-L score averaged across test instances of observed train tasks. GENTask measures the degree of task-level generalization and is formulated as: 1119 ∑119 i=1Ak(t̃ i eval). This is equivalent to the Rouge-L score averaged across test instances of unseen evaluation tasks.

heading: 4 Methodology
text: We introduce DYNAINST, our approach to lifelong in-context instruction learning. DYNAINST is a hybrid method that combines parameter regularization and experience replay. In Section 4.1, we elaborate on the use of wide local minima-inducing regularizer for lifelong instruction learning. Then, in Section 4.2, we describe how instances and tasks are dynamically stored and replayed through Dynamic Instruction Replay. The lifelong instruction learning process with DYNAINST is illustrated in Figure 2. We also provide line-by-line description of DYNAINST in Algorithm 1.

heading: 4.1 Wide Local Minima
text: Promoting wide local minima in neural networks has been widely accepted as an effective way of achieving improved generalization performance (Pereyra et al., 2017). In addition, in (Cha
et al., 2020), it is shown that not only does wide local minima help with generalization performance, but it can also be used to combat task distribution shifts in a sequential learning process. The multifaceted benefits of wide local minima are wellaligned with the objectives of lifelong instruction learning. Therefore, we incorporate an implementation of wide local minima-inducing regularizer proposed in Cha et al. (Cha et al., 2020) by modifying the plain cross entropy loss as follows:
LDyna = Lce + γ · Lwlm,
where Lwlm = 1
L ·
L∑
i=1
DKL(ŷi || Unif.). (1)
ŷi is the softmax output of the model (F (xi)), and γ is the coefficient used to control the strength of Lwlm. DKL and Unif. are the Kullback-Leiber (KL) divergence of two distributions and the uniform distribution, respectively. Essentially, Lwlm is designed to drive ŷi closer to the uniform distribution. By doing so, Lwlm effectively discourages the model output from becoming overconfident. Because penalizing overconfident model outputs allows F to avoid overfitting, the resulting F trained with LDyna becomes more robust to distribution shifts and thus obtains higher generalization (Pereyra et al., 2017). In practice, this regularization term is implemented by maximizing the entropy of the model predictions. We additionally discuss its relationship to maximum entropy regularization used in Soft Actor-Critic from reinforcement learning in Section A2 of Appendix.

heading: 4.2 Dynamic Instruction Replay
text: Dynamic instruction replay (DIR) can largely be divided into two processes: Dynamic Instance Selection (DIS) and Dynamic Task Selection (DTS). To implement DIS and DTS, we introduce Replay Bank that consists of N number of task-specific Instance Banks with known task boundaries. The size of Replay Bank N is thus equal to the number of task-specific Instance Banks. Each Instance Bank of size M in Replay Bank contains M number of train instances per task. The maximum cap on the size of each memory bank is enforced to limit the memory consumption of DYNAINST. DIS: Storing all the train instances within each task in Instance Bank leads to an excessive amount of memory consumption. Therefore, after learning each task, it is preferable to selectively store instances that will be revisited later down the task
Algorithm 1: DYNAINST Require: Target model F , Number of Tasks K, Instance Bank of Size M , Replay Bank of Size N , Number of Replayed Tasks Rt, Number of Replayed Instances RI , Number of Train Epochs per Task E
for k = 1 : K do if k == 1 then
Train F on tk with Lce + Lwlm for E epochs → WLM
else Select Rt most difficult tasks from
Replay Bank → DTS Sample RI number of instances
from each selected task Finetune F on the sampled instances
of selected tasks and tk with Lce + Lwlm for E epochs → WLM
Replay Bank.push(tk) Select M/2 instances with lowest H Select M/2 instances with highest H Store the selected instances in Instance
Bank of tk → DIS if | Replay Bank | > N then
Replay Bank.pop(tk−N−1)
stream. In DYNAINST, the stored instances are 1) used to determine which tasks must be prioritized for replay and 2) replayed with future tasks. As a criterion for instance selection, we adopt the entropy of model predictions as defined below:
H(ŷi) = − ∑
i
p(ŷi|xi;F ) log(p(ŷi|xi;F )) (2)
From here on, we refer to this quantity as the model’s predictive entropy. Predictive entropy is commonly adopted for sample selection across various research fields (e.g., active learning (Gal et al., 2017) and neural architecture search (Na et al., 2021) that can benefit by identifying a subset of instances that best represents the dataset as a whole.
After finetuning F on the k-th task, DYNAINST first measures H of all train instances in the k-th task. Based on H, DYNAINST stores a mixture of high and low entropy instances in the task-specific Instance Bank. Given Instance Bank of size M, we split it into two and allocate each half to high and low entropy instances. This hybrid approach to DIS allows easier and more difficult examples to be represented evenly in Instance Bank within a fixed memory budget. After determining which
instances to store in Instance Bank, Instance Bank of the k-th task is added to Replay Bank. Once the number of Instance Banks exceeds the pre-set size of Replay Bank N , Instance Bank of the oldest task is removed from Replay Bank. DTS: Instead of replaying all stored tasks in Replay Bank, we only replay the more difficult tasks that the model struggles to learn. To quantify the difficulty of a task, we rely on the instances stored in the corresponding Instance Bank. Based on the Rouge-L score of a task measured using the stored instances, Rt number of tasks with bottom RougeL scores are replayed with the current task. When replaying a task, we randomly sample Ri number of instances from its Instance Bank to replay. In essence, within DYNAINST, DIS and DTS complement each other to identify which tasks and instances should be replayed to maximize the generalization performance of the target model in a memory- and compute-efficient manner.

heading: 5 Experiments

heading: 5.1 Experimental Set-up
text: For all experiments, BART-base model (Lewis et al., 2020) is used as the target model F . All of the implementation is done through Huggingface (Wolf et al., 2019) and PyTorch (Paszke et al., 2019), and NVIDIA V100 GPU is used to run the experiments. The following hyperparameters are shared across all baselines and DYNAINST: AdamW optimizer with the learning rate of 5e-5 and constant learning rate scheduling, 2 epochs of training per task, and effective batch size of 16. As for the hyperparameters specific to DYNAINST, the Replay Bank size (N ) is set to 50, and the Instance Bank size (M ) is adjusted to store 50% of train instances depending on the value of L. The values of Rt and Ri are set to 10 and 2, respectively. All approaches are run using five different random seeds to create different task and instance streams.

heading: 5.2 Baselines
text: • Naive: sequentially finetunes the target model on a stream of tasks with no additional technique. • Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017): is a benchmark parameter regularization method for continual learning. EWC retains past knowledge by preventing significant changes in important parameters. • LAMOL0.5,0.02 (Sun et al., 2019): is a benchmark continual learning method in NLP that relies on experience replay. The sampling ratio in LAMOL (0.5, 0.02) denotes the percentage of instances replayed from each one of previous tasks. • InstructionSpeak (Yin et al., 2022): is a continual learning method designed for the ConTinTin framework. InstructionSpeak consists of two main processes: History Training and Negative Training. History Training replays the past two tasks with the current task, and Negative Training utilizes negative samples as if they were positive ones.

heading: 5.3 Main Results
text: In Figure 3, we compare GENInst and GENTask of compared approaches under the static setting. Likewise, in Figure 4, we report the results of using random number of instances. The default number of instances for the static setting is set to 100, and for the random setting, the number of instances per task is randomly sampled from 1 to 100. In addition to the lifelong learning approaches, we visualize the performance obtained by jointly training on all of the tasks, equivalent to the upper bound performance for lifelong in-context instruction learning, with pink dotted lines with star markers.
In both figures, (a) and (c) show visualize how mean GENInst and GENTask averaged over five different random seeds change over time. Under the static setting, DYNAINST outperforms all baselines no matter how many tasks are used for lifelong learning. Under the random setting, DYNAINST
comes in as a close second to LAMOL0.02 in the beginning when only 100 tasks are used, but soon outperforms all baselines as more tasks are added. These results clearly demonstrate that DYNAINST better utilizes the increased number of train tasks. DYNAINST appears to be particularly effective at improving the task-level generalization performance, outperforming the second-best baseline by a significant margin on the Rouge-L score under both settings.
In (b) and (d), we report means and standard deviations of GENInst and GENTask after observing all 500 train tasks. Changes in random seeds seem to have a similar amount of effect on all compared methods, with no single method having a particularly small or large error bar. Under the random setting, we observe a slight increase in performance variation. We believe that this occurs because in the random setting, there are two sources of variation: changing number of instances and task ordering.
In Table 2, we report the performance of the model trained with DYNAINST and that trained with joint training on separate evaluation categories. The evaluation categories as defined by Wang et al. (2022) are in Section A1 Appendix. Due to the page constraint, from here on, we only report the results after training on all 500 tasks. It appears that the models trained with DYNAINST and joint training struggle with similar evaluation categories. Thus, it is reasonable to assume that the performance discrepancy among tasks is caused by the inherent task distribution skew within Sup-NatInst.
In Section A3 of Appendix, we report the results of using up to 20 instances per task. Once again, DYNAINST achieves the highest Rouge-L score, closest to the upper bound performance. In addition, we provide forgetting and intransigence analyses and an example of instances identified through hybrid DIS in Sections A4 and A5 of Appendix, respectively.

heading: 6 Ablation Studies

heading: 6.1 Separate Components
text: To validate the efficacy of each technical component in DYNAINST, we perform a component-wise analysis of DYNAINST and report the results in Table 3. It is apparent that parameter regularization with Lwlm and experience replay with DIR each contributes to improving the generalization performance of the target model trained with DYNAINST.

heading: 6.2 Hyperparameter Sensitivity
text: We now analyze the sensitivity of DYNAINST to the following hyperparameters: the strength of Lwlm (γ), the number of replayed tasks (Rt), and the size of the Instance Bank (M ). We test out one hyperparameter at a time and fix the rest of them as default values. The results are reported in Table 4. What is particularly noteworthy is that reducing M to 10 preserves the performance of DYNAINST; this result indicates that DYNAINST is capable of achieving high generalization performance even with a limited number of stored instances.
In addition, we observe that increasing Rt does not necessarily improve the performance of DY-
NAINST. We conjecture that the reason behind this phenomenon is that replaying relatively easier tasks by increasing Rt may hinder the target model from learning more difficult tasks. On the contrary, joint training, which uses all train tasks at once, does not experience performance degradation as the number of train tasks increases. Note that in joint training, all instances are shuffled in a task-agnostic manner, effectively blurring the task boundaries. Therefore, we would expect the discrepancy in task difficulties to have less influence on the generalization performance of the model.

heading: 6.3 DIS and DTS Design Choices
text: Lastly, we study how different design choices for DIS and DTS influence the pefroamcne of DYNAINST. The results can be found in Table 5. For DIS, we investigate three additional entropy-based instance selection methods - random, minimum, and maximum instance selection - as well as the upper bound performance obtained by storing all of the train instances. It is clear that the hybrid DIS best approximates the upper bound performance. Such a result validates that the hybrid selection is most capable of identifying instances that are representative of the task as a whole.
The default criterion for task selection in DTS is the absolute Rouge-L score per task. One alternative approach to DTS is to utilize the relative change in the Rouge-L score, effectively replaying the tasks that are forgotten the most by the target model. The results in Table 5 show that using the relative change in the Rouge-L score leads to a meaningful degree of performance drop compared to default DTS, consolidating the effectiveness of DTS based on the absolute Rouge-L score.

heading: 7 Conclusion
text: In this work, a fully lifelong learning of in-context instructions was investigated for the first time. We proposed DYNAINST, a novel hybrid approach to lifelong in-context instruction learning, and verified its superiority to existing baselines under various experimental scenarios. Potential directions for future research include extending our investigation to blurred or unknown task boundaries and analyzing whether DYNAINST outputs biased predictions.

heading: Acknowledgements
text: This work was supported in part by the Institute of Information & communications Technology Planning & Evaluation (IITP) and the National Research Foundation of Korea (NRF) grants funded by the Korean government (MSIT) (2022-0-00959, No. 2022R1A3B1077720, No. 2022R1A5A708390811).
Limitations and Potential Risks
The two limitations of DYNAINST are that it requires known task boundaries, and that it does not concern with corrupted or noisy training instances. In a realistic industry setting where the task definition is quite ambiguous, and a non-negligible amount of human bias and noise are introduced during the data collection process, these limitations of DYNAINST may degrade its performance. However, considering that this is the first time lifelong instruciton learning has been studied, these limitations can be considered interesting directions for future research.
Like any language model, the model trained with DYNAINST may output unfair and/or offensive predictions due to the bias embedded in the dataset. Improving the fairness of instruction-tuned language models is beyond the scope of this paper; nonetheless, if these problems remain neglected, we will risk deploying language models that are heavily biased and discriminatory.

heading: A Appendix

heading: A1 Sup-NatInst Instruction Schema and Evaluation Categories
text: A1.1 Instruction Schema The instructions in Sup-NatInst are contributions of 88 NLP practitioners, and the collected instructions are reviewed through Amazon Mechanical Turk (AMT) (Paolacci et al., 2010). All instructions in the Sup-NastInst dataset follow the same instruction schema that consists of: task definition, positive examples, negative examples, and explanation. The description and example of each component can be found in Figure A1.
A1.2 Evaluation Categories All of the evaluation tasks in the Sup-NatInst dataset fall into one of the following 12 categories: Textual Entailment (TE), Cause Effect Classification (CEC), Coreference Resolution (CR), Dialogue Act Recognition (DAR), Answerability Classification (AC), Word Analogy (WA), Overlap Extraction (OE), Keyword Tagging (KT), Question Rewriting (QR), Title Generation (TG), Data to Text (DT), and Grammar Error Correction (GEC).

heading: A2 Similarities in DYNAINST and Soft Actor-Critic
text: Soft Actor-Critic (Haarnoja et al., 2018) from reinforcement learning and DYNAINST both employ maximum entropy regularization. Interestingly enough, while DYNAINST and Soft Actor-Critic are used in two different domains, their motivations behind utilizing maximum entropy regularization bear some resemblance. By maximizing the expected reward and the entropy of the actor simultaneously, Soft Actor-Critic incentives the actor to improve exploration in the exploration-exploitation trade-off. The resulting model becomes more robust against estimation errors. Similarly, with the WLM regularizer, DYNAINST drives the target model towards a flatter or wider minima, effectively making the model more robust against task and data distribution shifts.
A3 Results of Using 20 Instances (L = 20)
The instance- and task-level generalization performance of models trained by using up to 20 instances per task are reported in Table A3 (static)
and Table A4 (random). Even with fewer train instances, DYNAINST achieves the best generalization performance among compared approaches.

heading: A4 Forgetting and Intransigence
text: Analyses for 100 Instances (L = 100) and 20 Instances (L = 100)
In this section, we analyze the lifelong learning process through the following measures borrowed from the continual learning literature (Chaudhry et al., 2018a). For these measures, it is preferable to obtain lower numbers. Forgetting measures the stability of the lifelong learning process by quantifying the degree of catastrophic forgetting. When Sk(tj) is the stability of j-th task after training on k-th task, Sk(tj) is defined as: Sk(tj) = maxl∈j,...k−1Al(tj)−Ak(tj). Intransigence measures how much knowledge from past tasks is utilized by the model when learning the current task. Otherwise known as plasticity, the intransigence measure of j-th task is defined as: Ij = A∗(tj)−Aj(tj) where A∗(tj) denotes the Rouge-L score of F trained only on the j-th task.
The forgetting and intransigence analyses when using L = 100 instances can be found in Table A1 (static) and Table A2 (random). The intransigence measure seems to have little correlation with the generalization performance of the model. For instance, EWC achieves relatively high intransigence under both settings but exhibits poor generalization performance. Such results may imply that a high level of forward transfer could be preventing the model from learning to generalize across tasks because it only promotes rapid learning of the current task, thus overfitting the target model to a single task. While the forgetting measure seems to be indicative of the model’s generalization performance under the static instance setting, it again becomes less reliable under the random instance setting.
We conjecture that this conflict between the generalization metrics and continual learning metrics is a result of the difference in learning objective between conventional continual learning and lifelong in-context instruction learning. The objective of conventional continual learning is to battle catastrophic forgetting within observed train tasks, and thus forgetting and intransigence metrics focus on quantifying how much knowledge is retrained and transferred among train instances. On the contrary,
lifelong in-context instruction learning wishes to improve instance- and task-level generalization performance on unobserved instances and tasks. Because the two learning paradigms and their evaluation metrics prioritize different aspects of the trained model, the two sets of metrics end up showing little correlation with each other. The forgetting and intransigence analyses under the 20-shot setting in Table A5 (static) and Table A6 (random) show similar tendencies.

heading: A5 Analysis of Stored Instances
text: Here, we analyze which examples are being stored in Instance Bank and whether the difficulty of an instance quantified with the predictive entropy is aligned with human perception of difficulty. In Toxic Language Classification task, which is a binary classification task that identifies whether a tweet contains toxic language or not, the following examples were selected by DIS. Prior to discussing the examples, we apologize for their vulgarity. • High-entropy (difficult) instance: Input: “Monkey see monkey doo,” | Output: “No.” It appears that the model found this example to be difficult to classify because of the following reasons. First, it does not contain explicitly toxic language. Although the word “monkey” may be used as a derogatory term against certain races, within the above input sentence, it appears innocuous. Second, it is difficult to determine the intent of the writer in the first place. Because the input, which is an idiomatic expression, reads rather incoherently, the intent of the writer remains rather ambiguous without more context. • Low-entropy (easy) instance: Input: "Then the Mexican bitch wanna speak some other fuckin language & then give attitude. Can’t speak right ass! Tf" | Output: “Yes.” It is rather clear why the model found this to be an easy example. The input sentence contains explicitly negative words such as “bitch, “fucking,” and “ass," and is openly expressing the writer’s racist views.
Component Description Example
Definition Defines the scope of the target task in natural language and provides a complete description of how an input should be mapped to an output. "Given an utterance and recent dialogue context containing past 3 utterances (wherever available), output ‘Yes’ if the utterance contains the small-talk strategy, otherwise output ‘No’. Small-talk is a cooperative negotiation strategy. It is used for discussing topics apart from the negotiation, to build a rapport with the opponent”
Positive Example
Samples of correct input-output pairs.
• Input: “Context: … ‘That's fantastic, I'm glad we came to something we both agree with.’ Utterance: ‘Me too. I hope you have a wonderful camping trip.’” • Output: “Yes”
Negative Example
Samples of incorrect or invalid input-output pairs.
• Input: “Context: … ‘Sounds good, I need food the most, what is your most needed item?!’ Utterance: ‘My item is food too’.” • Output: “Yes”
Explanation Short explanation for why an example falls under the positive or negative example category.
• Positive: “The participant engages in small talk when wishing their opponent to have a wonderful trip.” • Negative: “The utterance only takes the negotiation forward and there is no side talk. Hence, the correct answer is ‘No’.”
Instance Additional nstances for training and/or evaluation.
• Input: “Context: … ‘I am excited to spend time with everyone from camp!’ Utterance: ‘That’s awesome! I really love being out here with my son. Do you think you could spare some food?’ ” • Output: “Yes”
Figure A1: Description of Sup-NatInst (Wang et al., 2022) instruction schema and sample instance.
Methods Forgetting Intransigence T100 T200 T300 T400 T500 T100 T200 T300 T400 T500
Naive 49.24 51.66 52.95 52.84 52.68 -3.57 -4.44 -4.32 -4.38 -4.18 EWC 49.26 51.34 52.85 53.01 52.92 -3.58 -4.26 -4.29 -4.79 -4.98 LAMOL0.5 51.99 55.49 55.49 54.87 54.42 -2.50 -4.36 -4.11 -3.61 -3.35 LAMOL0.02 52.43 56.26 55.99 55.24 54.81 -4.83 -6.88 -6.45 -5.89 -5.76 InstSpeak 51.14 53.34 53.49 53.19 53.02 -3.56 -4.94 -4.66 -4.49 -4.48 DYNAINST 46.29 50.89 51.85 52.36 51.59 -2.88 -3.37 -2.69 -2.18 -1.63
Table A1: Forgetting and intransigence analysis when using L = 100 instances per task. Unlike generalization performance, it is preferable to achieve lower numbers for both forgetting and intransigence metrics. The best and second-best metrics in each column are marked in bold and underline, respectively.
Methods Forgetting Intrasigence T100 T200 T300 T400 T500 T100 T200 T300 T400 T500
Naive 49.19 52.63 53.39 53.49 53.56 -8.48 -9.31 -8.89 -8.34 -8.46 EWC 48.17 52.34 52.98 52.65 53.16 -7.59 -8.64 -7.68 -6.89 -7.28 LAMOL0.5 50.46 54.06 54.34 53.96 53.88 -7.66 -8.16 -7.22 -6.08 -6.14 LAMOL0.02 52.01 55.69 56.18 55.39 55.15 -7.15 -7.95 -7.52 -6.07 -6.23 InstSpeak 49.46 53.45 54.13 53.60 53.44 -5.63 -6.89 -6.08 -5.55 -5.43 DYNAINST 50.75 52.30 53.25 53.27 53.34 -3.41 -4.63 -4.02 -2.81 -3.32
Table A2: Forgetting and intransigence analysis when using L ∈ [1, 100] instances per task. Unlike generalization performance, it is preferable to achieve lower numbers for both forgetting and intransigence metrics. The best and second-best metrics in each column are marked in bold and underline, respectively.
Methods GENInst GENTask T100 T200 T300 T400 T500 T100 T200 T300 T400 T500
Naive 23.73 28.60 31.77 30.25 33.09 25.01 30.85 37.63 34.96 36.59 EWC 24.48 27.48 31.71 30.87 32.27 25.78 29.62 36.88 34.40 35.14 LAMOL0.5 24.52 30.42 30.39 30.66 32.05 27.20 33.85 35.02 35.43 37.56 LAMOL0.02 27.12 29.62 32.78 30.65 33.34 28.27 31.71 34.85 32.75 32.17 InstSpeak 24.62 26.49 30.98 30.62 33.68 26.84 25.85 34.17 34.51 35.09 DYNAINST 29.36 31.09 32.95 32.64 34.55 31.23 34.35 37.79 35.75 37.96
Joint 29.93 36.29 39.62 42.50 43.41 32.28 34.44 37.49 38.93 38.25
Table A3: Comparison of zero-shot instance- and task-level generalization performance when using L = 20 instances per task (static instance setting). The best and second-best Rouge-L scores in each column are marked in bold and underline, respectively.
Methods GENInst GENTask T100 T200 T300 T400 T500 T100 T200 T300 T400 T500
Naive 20.07 26.07 30.58 29.68 30.56 20.11 26.26 33.60 31.73 32.12 EWC 23.74 26.48 30.78 29.61 31.04 25.17 27.76 35.74 34.25 32.07 LAMOL0.5 26.38 29.05 31.62 30.51 33.43 28.63 31.53 35.83 35.12 36.09 LAMOL0.02 26.12 29.95 31.99 32.05 31.44 30.92 33.25 35.04 34.48 36.65 InstSpeak 24.11 25.89 31.37 30.81 32.55 25.89 27.01 35.23 33.25 34.53 DYNAINST 25.28 30.29 32.45 33.00 33.44 31.29 34.09 36.99 36.07 37.55
Joint 27.33 31.76 34.51 34.93 35.79 30.34 34.61 37.68 37.24 38.58
Table A4: Comparison of zero-shot instance- and task-level generalization performance when using L ∈ [1, 20] instances per task (random instance setting). The best and second-best Rouge-L scores in each column are marked in bold and underline, respectively.
Methods Forgetting Intransigence T100 T200 T300 T400 T500 T100 T200 T300 T400 T500
Naive 43.88 46.89 49.99 51.02 50.89 -11.44 -10.39 -12.20 -11.53 -11.73 EWC 44.20 47.09 49.38 50.25 50.48 -11.73 -10.19 -11.65 -10.81 -11.35 LAMOL0.5 47.27 49.01 50.77 50.88 51.16 -7.84 -5.64 -7.76 -6.99 -7.58 LAMOL0.02 49.11 51.25 52.38 52.60 52.55 -14.03 -10.96 -12.17 -10.37 -10.55 InstSpeak 46.11 49.18 50.69 51.18 51.25 -8.74 -8.19 -9.49 -9.01 -9.36 DYNAINST 42.69 45.36 46.53 47.07 47.64 -9.71 -8.05 -9.31 -8.16 -8.56
Table A5: Forgetting and intransigence analysis when using L = 20 instances per task (static instance setting). Unlike generalization performance, it is preferable to achieve lower numbers for both forgetting and intransigence metrics. The best and second-best metrics in each column are marked in bold and underline, respectively.
Methods Forgetting Intrasigence T100 T200 T300 T400 T500 T100 T200 T300 T400 T500
Naive 48.38 51.45 52.25 52.14 52.57 -15.12 -15.45 -15.49 -14.53 -15.65 EWC 50.97 54.61 57.11 58.25 58.72 -17.82 -17.99 -17.83 -17.31 -18.91 LAMOL0.5 50.33 53.25 54.30 55.76 56.31 -10.85 -10.64 -10.42 -10.32 -11.89 LAMOL0.02 49.22 54.57 55.94 56.01 56.56 -7.63 -8.09 -9.17 -8.70 -9.42 InstSpeak 50.69 55.99 56.56 56.90 57.25 -12.42 -14.75 -14.78 -14.50 -15.69 DYNAINST 43.19 53.56 55.45 55.56 56.07 -4.70 -8.22 -9.16 -8.27 -8.69
Table A6: Forgetting and intransigence analysis when using L ∈ [1, 20] instances per task (random instance setting). Unlike generalization performance, it is preferable to achieve lower numbers for both forgetting and intransigence metrics. The best and second-best metrics in each column are marked in bold and underline, respectively.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: However, considering that this is the first time lifelong instruciton learning has been studied, these limitations can be considered interesting directions for future research. Like any language model, the model trained with DYNAINST may output unfair and/or offensive predictions due to the bias embedded in the dataset. Improving the fairness of instruction-tuned language models is beyond the scope of this paper; nonetheless, if these problems remain neglected, we will risk deploying language models that are heavily biased and discriminatory