CITB: A Benchmark for Continual Instruction Tuning
Authors: Zihan Zhang
Meng Fang
Ling Chen
Mohammad Reza Namazi Rad
Venue: EMNLP 2023 Findings (2023)
Keywords: Instruction tuning
continual learning
benchmark
evaluation
TL;DR: A benchmark for continual instruction tuning.

Abstract: Continual learning (CL) is a paradigm that aims to replicate the human ability to learn and accumulate knowledge continually without forgetting previous knowledge and transferring it to new tasks. Recent instruction tuning (IT) involves fine-tuning models to make them more adaptable to solving NLP tasks in general. However, it is still uncertain how instruction tuning works in the context of CL tasks. This challenging yet practical problem is formulated as Continual Instruction Tuning (CIT). In this work, we establish a CIT benchmark consisting of learning and evaluation protocols. We curate two long dialogue task streams of different types, InstrDialog and InstrDialog++, to study various CL methods systematically. Our experiments show that existing CL methods do not effectively leverage the rich natural language instructions, and fine-tuning an instruction-tuned model sequentially can yield similar or better results. We further explore different aspects that might affect the learning of CIT. We hope this benchmark will facilitate more research in this direction.

CITB: A Benchmark for Continual Instruction Tuning
Zihan Zhang', Meng Fang', Ling Chen', Mohammad-Reza Namazi-Rad" "University of Technology Sydney UUniversity of Liverpool
Universitty of Wollongong
Zi Ziha Zhang- uts. edu. au, Meng Fang@1 verpool ac. uk Ling. Chenêuts, edu. au, mr mrad@uow. edu. au
Abstract
Instruction Tuning
Multi-task training
Continual learning (CL) is paradigm that aims to replicate the human ability to learn and accumulate knowledge continually with- out forgetting previous knowledge and transfer- ring it to new tasks. Recent instruction tuning (IT) involves fine-tuning models to make them more adaptable to solving NLP tasks in general. However, it is still uncertain how instruction tuning works in the context of CL tasks. This challenging yet practical problem is formulated Continual Instruction Tuning (CIT). this work, we establish CIT benchmark consist- ing of learning and evaluation protocols. We curate two long dialogue task streams of dif- ferent types, InstrDialog and InstrDialog++, to study various CL methods systematically. Our experiments show that existing CL meth- ods do not effectively leverage the rich natu- ral language instructions, and fine-tuning an yield
Summarization
Test on unseen tanks Natural Language Inference
Suntiment
Analysis
tuned
Question
Answuring
Continual Instruction Tuning
Stage 1: Multt- task tratning
Stage 2: Sequential single task learning
Evaluate on:
D test
+ D
test
test
Figure Illustration of proposed continual instruction tuning (CIT). Unlike previous works, evaluate the instruction-tun model on the initial training, unseen, and newly learned tasks.
instruction-tunet sintee model sequentially similar or better results. We further explore different aspects that might affect the learning of CIT. We hope this benchmark will facilitate more research in this direction'.
to update themselves or adapt to a changing envi- ronment (Zhang et al., 2023; Bubeck al., 2023). Continual learning (CL.) aims to enable informa- tion systems to learn from continuous data stream across time (Biesialska al., 2020). Therefore, it is promising leverage CL for instruction-tunce PLMs to continually adapt to new domains and tasks without costly re-training. Despite its impor- tance, it is non-trivial to alleviate catastrophic for- getting, a phenomenon in which previously learned knowledge or abilities are degraded due to over- written parameters (McCloskey and Cohen, 1989). Moreover, enabling knowledge transfer is also sential since many tasks similar and have com- mon knowledge (Ke al., 2021).
1 Introduction
Recent studies have shown that multi-task instruc- tion tuning (IT) makes language models better zero- shot learners (Wei et al., 2022: Sanh et al., 2022: Wang et al., 2022; Chung et al., 2022: Longpre et al., 2023). IT fine-tunes pre-trained language models (PLMs) on various tasks with natural lan- guage instructions (Fig. 1) and can achieve remark- ably well generalization to unseen tasks.
Despite their impressive performance, these instruction-cuned PLMs still fall short on domain- specific tasks due to the limited exposure rele- vant knowledge and vocabulary from the training corpus (Luo et al., 2023). Moreover, PLMs are static after deployment, and there is no mechanism
Unfortunately. there is little work on applying CL for IT and has only been explored in rather specific settings Scialom et al. (2022) continu- ally fine-tune a TO (Sanh al., 2022) on eight new tasks with memory replay avoid forgetting. Despite effectiveness, they need to store large num- ber of instances per task in memory, which too costly when scaling to a larger number of tasks.
'Code and data are available at petps://ggttee com/ hyintell/CITe.

---

In addition, they do not study knowledge trans- fer between tasks. Yin al. (2022) propose to use history task instructions to reduce forgetting and enable knowledge transfer. However, they do not compare with commonly adopted CL methods, which makes the effectiveness of other CL methods unknown. Moreover, they only evaluate the model on the newly learned tasks while largely ignoring previously learned tasks during the multi-task train- ing stage (Fig. 1). They also overlook the intrinsic ability of the instruction-tune tuned model on unseen tasks. Lastly, both of them use different evalua- tion metrics and setups, which creates an obstacle to comparing different techniques and hinders the development of this field.
and ablation studies to analyze the lack of cur- rent practices, and propose future direction.
Related Work
Instruction Tuning. Much effort has been made recently to use natural language instructions solve multiple tasks concurrently to align with human preferences (Touvron et al., 2023; Zhou et al., 2023; OpenAl, 2023). Unlike simple and short prompts (Liu et al., 2021), natural language instructions (Fig.2) can be more comprehensive, including components such as task definition, in- context examples (Brown et al., 2020), and expla- nations. Through IT, PLMs learn to complete tasks by following instructions, which enables them solve new tasks by following instructions without learning (i.e., generalization ability). Ideally, we expect the instruction-tune model to understand any given task instruction so that an end user can directly leverage the model to solve the task with- out the need to annotate a large dataset and train it. Unfortunately, despite the instruction-tuned models such FLAN (Wei et al., 2022; Longpre et al., 2023), TO (Sanh et al., 2022), and Tk-Instruct (Wang et al., 2022) showing strong generalization performance to their evaluation tasks, there is still a sizeable gap compared with supervised training. which limits the usage of the models. From a prac- tical point of view, a desirable instruction-tuned model should be able to extend its ability by contin- ually learning those under-performen tasks or any new task, while not forgetting the old ones.
To this end, we first formulate this practical yet under-explored problem as Continual Instruction Tuning (CIT). Then, we propose first-ever bench- mark suite to study CTT systematically. Our bench- mark, CITB, consists of both learning and evalua- tion protocol and is built on top of the recently proposed SuperNT dataset (Wang et al., 2022). We create two CIT task streams: InstrDialog stream, which consists of 19 dialogue-related tasks spanning three categories; InstrDialog++ stream, which includes all the tasks in InstrDialog stream and 19 additional tasks selected from broad cat- egories and domains. Using the two long task streams, we implement various CL methods to study forgetting and knowledge transfer under the setup of CIT. We find that directly fine-tuning an instruction-tune model sequentially yields com- petitive performance with existing CL methods. With further investigation, we find that rich natural language instructions enable knowledge transfer and reduce forgetting, which is barely fully lever- aged by current CL methods. We conduct com- prehensive experiments to explore what effects the learning of CIT. We hope our CITB benchmark will serve as a helpful starting point and encourage sub- stantial progress and future work by the community in this practical setting. To summarize, our main contributions are:
Continual Learning. In contrast to multi-task learning, continually fine-tuning a model on tasks might lead to catastrophic forgetting (McCloskey and Cohen, 1989), where the model forgets previ- ously acquired knowledge after learning new tasks. In CL literature, approaches overcoming catas- trophic forgetting can be grouped into three cate- gories (Biesialska al., 2020; Ke and Liu, 2023). Regularization- methods use an additional loss to prevent important parameters of previous tasks from being updated (Kirkpatrick et al., 2017;
We formulate the problem of CIT and establish benchmark suite consisting of learning and evaluation protocols.
De Lange et al., 2019). Replay-based methods store and replay a small subset of training data from previous tasks to prevent forgetting (Rebuffi et al., 2017; Scialom et al., 2022); Architecture- based methods introduce task-specific components for new tasks and isolate parameters of old tasks (Madotto et al., 2021: Zhu et al., 2022). How- ever, the effectiveness of these CL methods for
We curate two long task streams of various types based on the SuperNI dataset to study different setups of CIT.
We implement various CL methods of differ- ent categories, conduct extensive experiments

---

Definition:
instances D' (
) E X x J") which can split into the training Dr irain validation Déev and test sets D Each instance filled into an instruction template such that different tasks can be transformed into unified text-to-text format (Fig.2). IT aims to learn a model I' x X - )" that predict the output y given the task instruc- tion I' and an input ut. In general, the model is first trained on a mixture of tasks ( (Teeea) and then evaluated for its zero-shot generalization ability on held-out tasks (Tunscen). where Tscen n Tunseen - D. The model is expected to learn to follow instruc- tions via the training tasks and then solve new tasks with only the help of task instructions.
In this task, you are given a sentence and question, you would be asked
which is contained the
sentence provided.
Positive Example 1
Input: "Sentence: Heat from the sun causes the most evaporation of water
lake. Question: Which of these causes the MOST evaporation of water from a lake?"
Output: "Heat from the Sun"
Explanation: output correct as ... the 'Heat from the sun
Negative Example 1
Input: "Sentence: Gas has no definite volume and no definite shape. Question: Which state matter has no definite volume and definite shape?"
Output: "Matter"
Explanation: "The correct answer to hence it is incorrect."
Input: The
eye a(n) convex shape. What shape is the lens of the eye?"
Question:
4 Continual Instruction Tuning Benchmark
Expected Output: "convex"
An example of natural language instruction that consists of a descriptive task definition, one positive and one negative in-context example with explanation (Wang et al., 2022). Given task instruction and the input of test instance, model needs to produce the desired output.
Figure 2:
In this section, we first formalize the CIT problem (Fig.1). 1). Then, we present the learning and evalua- tion protocol of our framework CITB. Lastly, we describe the data for creating the benchmark.
4.1 Continual Instruction Tuning
In contrast to static IT, which only learns fixed set of tasks (Tseen), the model should be able to keep learning new tasks without catastrophically forgetting previously learned knowledge and fa- cilitate knowledge transfer if possible. Let us ex- pand the definition such that we have a set of T tasks Txoq {t,,.... ,tr) } that arrives sequentially. Note that the tasks in the stream can be any type and are not restricted specific categories or do- mains. Similarly, each task t, E Tsey has a natural language instruction 1', training irain validation D' dev and test sets D's Likewise traditional CL, the goal of CIT is to learn single model from Tseq sequentially.
CIT remains unknown. CIT differs from traditional CL in heavily relying comprehensive instruc- tions. Can previous methods fully leverage the rich instructions avoid forgetting while facilitat- ing knowledge transfer between tasks? Moreover, many proposed CL methods target tasks specific types (e.g., text classification, relation extraction) (Huang et al.. 2021; Qin and Jory, 2022; Zhu et al., 2022), while CIT can learn broad tasks of different categories because of the natural language instruc- tions?. Yin et al. (2022); Scialom et al. (2022); Mok et al. (2023) study similar problem using IT in the CL setting, but no benchmarks are built for different categories of CL methods. To tackle CIT, it is essential to establish an unified bench- mark to compare existing approaches and promote the development of this field. However, to our best knowledge, CIT is still immature and no public benchmark is available.
CIT Traditional CL. While sharing similar desiderata with traditional CL. CIT differs in that: (I) it pays more attention to effectively leveraging the rich natural language instructions to prevent catastrophic forgetting and encourage knowledge transfer; (2) because of the multi-task nature of instructions, all tasks can be formatted in the uni- fied text-to-text format, therefore CIT can learn any task domain instead of few specific tasks domains; (3) after learning a few tasks, the model should have learned how to follow instructions complete tasks, therefore, we expect fewer training instances required and higher knowledge transfer for future tasks.
3 Preliminaries
Instruction Tuning (IT). Following previous studies (Wei et al., 2022; Sanh et al., 2022; Wang consists of its natural
al., 2022), each task language instruction I' and a set of N input-output
AAll tasks can be filled into a natural language instruction template and transformed into text-to-text format (Wei et al., 2022).

---

4.2 Learning Protocol of CIT Benchmark
models have shown strong zero-shot generaliza- tion to unseen tasks (Wang et al., 2022), our initial model trained in stage might also have zero-shot generalization to some unseen tasks Tunsoen, where Tinit O Tseq O Tunseen = 0. Let Diesten be the test sets of all tasks in Tunseen- We evaluate the model on Dusson if it is available. To sum up, once a new task is learned, the model will be evaluated on:
A non-ins pee-insrructoo-tuuee model (c.g,, T5; Raffel et al. 2020) may struggle to understand instructions if trained only on task sequentially. It is also against our motivation extend a model's ability that is already instruction-tuned. Therefore, we separate the learning process into two stages.
Stage 1: Initial Multi-task Fine-tuning. To teach model a better understanding of task instruc- tions, we first fine-tune the model on instruction data. Suppose we have another group of M tasks Tim = [4,,... . , that also equips with natu- ral language instructions, where iitt n Tsqq = 0. We fine-tune base pre-trained model on the train- ing set (i.e., Dinit = U Di of the mixed M tasks to get an instruction-tume model, denoted as finit After training, most of the training data D train is unavailable for subsequent sequential learning, but a memory Mini (Mint < [Dinit 1) that stores a small portion of training instances is accessible. We use this model as the starting point to conduct the subsequent learning.
D Die + Dinit +
(1)
test
In CIT, it is more critical for the instruction-tunee model to maintain its existing abilities than learn new ones because can solve multiple tasks by fol- lowing instructions. Otherwise, if it forgets many tasks, there no point in using such a model than task-specific one. Therefore, it is essential to evaluate on Disit and Dunseen
Evaluation Metries. Due to the diversity of the tasks in CIT and the open-ended generation nature of the text-to-text format, we follow Wang et al. (2022) to use ROUGE-L (Lin, 2004) to measure the aggregated performance of cach task. They have shown that ROUGE-L generally works well for both generation and classification tasks.
Stage 2: Sequential Single Task Fine-tuning. To keep extending knowledge of the instruction- tuned fini, we fine-tune it on the training set D train of each task Ij in the stream T Similarly, when learning the task ty, the training data of previ- ous tasks in the stream (i.e., Drai - - D'u < j < T) is unavailable, but a small memory Maaq can be used for training.
Following Lopez-Paz and Ranzato (2017) and Biesialska et al. (2020), we also use CL-related metrics to measure the learning procedure. Let ajs be the ROUGE-L score of the model on the test set of task 4 right after training on task Ly, we define the following:
Average ROUGE-L (AR), which measures the average performance of the model on all tasks after the final task (T is learned:
4.3 Evaluation Protocol of CIT Benchmark
Evaluation Process. After learning each task tý (1 j < T) in stream Tsqq. we consider three datasets to measure the model's performance. (1) Similar to the standard CL, we evaluate the model on the test sets of all previously learned tasks in the stream, the test set of the current task, and the test set of the next task, denoted as Des- This helps us measure whether the model forgets previ- ous knowledge and whether it is helpful to learn future tasks; (2) We evaluate the model on the test sets of the M tasks that are used in stage to teach the model how to follow instructions, denoted as D Dinit M Dis This is where different from the conventional CL. In CL, previous works only evaluate downstream tasks the stream but not the tasks during the pre-training phase because such data is generally not accessible to end-users (Ke et al., 2022). (3) Since multi-task instruction-tuned
T
AR, = Σ aT
(2)
T
We use Final ROUGE-L (FR) to measure the per- formance of the model on Digit and Dunseen respec- tively, after the final task by is learned. Forward Transfer (IFWT), which measures how much the model can help learn the new task. FWT also tests the model's zero-shot generaliza- tion to new tasks:
1
FWT,
Σ 0:11,
(3)
Backward Transfer (BWT), which measures the impact that continually learning on subsequent tasks has on previous tasks:

---

from the remaining training task set. We intention- ally select tasks from broad categories, including sentence ordering, style transfer, toxic language detection, and others. In total, we have 38 tasks of 18 categories (3 categories from InstrDialog and 15 categories from the new 19 tasks), i.c., [7xey] = 38.
1 BWT, T-1 Σ (oT 0is)
Notably, positive BWT indicates that subsequent tasks can improve the performance of previous tasks, while negative value implies knowledge for- getting.
The remaining training tasks can be used for stage 1 initial multi-task fine-tuning ($4.2). In sum- mary, the number of initial fine-tuning tasks avail- able is M iinir| 718, and we use the official 119 test task sets as Tunseen to evaluate whether the performance deteriorates for unseen tasks after learning new tasks. For all tasks, we fill instances in natural language instruction template and trans- form them into a unified text-to-text format ($3). Unless otherwise specified, we use the instruction template consisting of the task definition and two positive examples for all tasks because it generally yields the best performance (Wang et al., 2022). See an example of natural language instructions Fig.2, and the selected tasks Table 5. We study the effect of the instruction template in $6.3.
4.4 Data Curation
In this work, adopt the recently proposed Su- perNI (Wang et al., 2022) dataset establish the benchmark. SuperNI consists of more than 1,600 NLP tasks, spanning a diverse variety of 76 broad task types, such as language generation, classifica- tion, question answering, and translation. More- over, each task is equipped with an instruction and a set of instances, and all the instances can be trans- formed into the text-to-text format. Therefore, the dataset is suitable for studying CIT. The official training set of SuperNI² consists of 756 English tasks spanning 60 broad NLP categories, while 11 tasks from 12 categories are used for zero-shot eval- uation. We keep the official 119 evaluation tasks untouched and create two CTT task streams from the 756 training tasks.
5 Experiments
Using our CITB benchmark, we conduct experi- ments on various popular CL methods of different kinds. We describe our experiment setups and com- pared methods in this section.
InstrDialog Stream. Dialogue is important field to study continual learning because new tasks, domains or intents are continuously emerging (Madotto et al., 2021). To investigate how a model learns new dialogue data under the sctup of CIT, we carefully curate all dialogue-related tasks from the training set of SuperN1 to form the CIT task stream. Specifically, we use 4 tasks from dialogue state tracking. 11 tasks from dialogue generation, and 4 tasks from intent identification, resulting in a total of 19 dialogue tasks, i.c., |Tsegl 19. We remove tasks that are excluded by the official task splits
5.1 Setup
Model. We use the LM-adapted version of T5- small (Raffel et al., 2020), which is further trained with language modeling objective. We initial- ize T5 model from HuggingFace?* Since it is costly to fine-tune all 718 tasks, we randomly select 100 tasks from Tinit and fine-tune T5 to ob- tain an instruction tuned model finit ($4.2), which has learned to understand some instructions and can act as a good starting point to conduct subsequent learning. Note that the 100 randomly selected train- ing tasks do not overlap with InstrDialog and In- str strDialog++, but their task categories might over- lap with the categories in InstrDialog++
InstrDialog++ Stream. Because of the multi- task nature of instructions, an instruction-tun model can learn any new task with different types ($4.1). To study how different types of tasks and how long-task curriculum affects CIT, we first include all 19 dialogue tasks from the InstrDialog stream, then we manually select the other tasks
TrrainDev/Tess Splits. Since the number of in- stances in each task is imbalanced and a large num- ber of training instances do not help generalization in IT (Wang et al., 2022), we follow Wang et al. (2022) use a fixed size of 500/50/100 instances per task as the train/dev/test set for the InstrDialog
hhttps://github wwt.e/occcaaaaaaaeeeae allnai/ natural-instruction
ctto./ccmaaaaaaoaaa aalaaa//eeaa///e//aa///a/aaaa natural-inse wtt.iirruuutiioosseeeeeetteteeee
ttt hhttps://hugginggaate www.hugggggggc.....c. co/ so/google/
t5- 35-small- 1- 1m--mmapppp

---

InstrDialog
Tiiit FR
Tunseen FR
Method
AR
FWT
BWT
Mem.
+P (Tun)
Time
FT-no-init
29.62.1
8.0
10.8
17.
16.5
(1)
0.50.0
AdapterCI
8.1
9.40.7
-1.99.9
T*0.02 (0.02)
0.4g.03
Init
22.51
43.5
36.5'
FT-init
35.70.2
18.50.7
-4.60.2
38.60.3
32.30.e
(1)
0.60.01
L2
35.60.1
17.50.5
1..
39.40.4
3.29
1(1)
0.60.1
EWC
34.50.6
16.80.4
-6.1.5
37.0o.1
32.50.5
2(1)
1.10.2
AGEM (10)
33.20.4
19.1
-7.31.0
38.61.1
32.40.o
(T+M))10
0(1)
1.10.1
AGEM (50)
34.90.9
18.11.0
-6.0.99
37.70.1
32.61.0
(T+M)*50
0
1.30.1
Replay (10)
38.40.7
23.70.0
- -1.3u.
42.70.7
32.40.4
(T+M)*10
0
1.40.04
Replay (50)
40.40.0
22.90.1
1.61.
47.1o.3
31.81.0
(T+M)*50
(
3.20.5
Multi
42.10.6
44.71.3
32.8p.9
01)
1.10.2
Table I: Performance of different methods the InstrDialog stream. Means and standard deviations are reported. means zero-shot performance. "Mem." means the number instances stored in the memory for each task; T is the total number of tasks the stream and M the number of tasks used for initial training. "+P" means the percentage of additional parameters added for each task, measured by the total parameters of the hase model; "Tun" is the portion of tunable parameters during training. "Time" the average hours for each method to complete the task stream. Best numbers in bold.
stream. For InstrDialog++, since it has a longer task sequence, to save computational cost, we use 100/50/100 instances per task instead. For Timi we use 100/50/100 instances per task and 100 in- stances per task for Tunsocn- We study the effect of different numbers of training instances in $6.3.
Architectuval-basedl methods introduce task- specific parameters to the base model to pre- vent subsequent tasks from interfering with pre- viously learned parameters. We adopt AdapterCL (Madotto et al., 2021), which freezes the pre- trained model and trains residual Adapter (Houlsby et al., 2019) for each task independently.
5.2 Baselines and Compared Methods
We implement commonly used CL methods from three categories (Biesialska et al., 2020) to bench- mark CIT.
Apart from the three categories of CL, also implement instruction-bused baselines because all previous CL methods are not designed for CIT. No prior work had tried to fine-tune an instruction- tuned model sequentially without any mechanism for preventing forgetting or encouraging knowl- edge transfer. However, it is commonly considered performance lower bound in CL literature. To this end, we propose to continually fine-tune the ini- tial instruction-tunce model ($5.1) on subsequent tasks, named FT-init. As direct comparison, we initialize a new T5 model, which is not tuned on any instruction data, and we continually fine- tune it (FT-no-init). In addition, we also report the performance of the initial instruction-tun -uned model (Init), which is the starting point before subsequent learning. Lastly, we jointly fine-tune a T5 model us- ing all the data, including the training data used in stage I and the training data of all subsequent tasks in the stream (Multi). This is often regarded as the performance upper bound in CL and does not have catastrophic forgetting and knowledge transfer.
Regularization-base methods rely on a fixed model capacity with an additional loss term to con- solidate previously gained knowledge while learn- ing subsequent tasks. We use L2 and EWC (Kirk- patrick et al., 2017), which uses a fisher informa- tion matrix to reduce forgetting by regularizing the loss to penalize the changes made important parameters of previous tasks.
Replay-based methods store a small subset of training instances from previous tasks in a memory. The data are replayed later to reduce forgetting. We adopt Replay, which saves random instances from each task in memory and then jointly trains the model new task data and the old data in the memory; AGEM (Chaudhry et al., 2019), which adds constraint prevent parameter update from increasing the loss of each previous task. The loss previous tasks is calculated using the instances stored in the memory.

---

InstrDialogt+
Tint FR
T Tunseen
Method
AR
FWT
BWT
FR
FT-no-init
31.32.4
26.80.3
1..214
28.90.6
31.22.1
AdapterCI.
20.70.2
14.30.9
-7.31.1
Init
30.5t
43.5
36.5t
FT-init
34.80.3
29.80.5
-2.0066
40.80.2
35.90.5
L2
36.10.2
27.9g.3
-4.01.2
41.10.4
34.91.1
EWC
38.60.4
28.60.5
-4.01.8
41.30.3
34.0p.7
AGEM (10)
39.30.6
28.90.2
-3.81.1
40.30.9
34.1
Replay (10)
l..
29.00.4
-3.60.8
44.00.5
30.10.6
Multi
44.90.5
44.60.9
33.90.5
Table 2: Performance of different methods the InstrDialog stream. t means zero-shot performance.
5.3 Implementation Details
worse than Init. We conjecture that AdapterCL fails to learn instructions effectively because it is initial- ized from a non-instruction--uuee model (T5) and the few tunable parameters restrict it from learning complex instructions.
For both InstrDialog stream and InstrDialog++ stream, we conduct continual instruction tuning using the same initial instruction-tunee model for all methods except FT--n--init, AdapterCL, and Multi. For these methods, we initialize a new model. For AGEM and Replay, we experiment on a memory size of 10 and 50, i.e., we set the memory size Mini to 10 and 50, same as Masq. We jointly train the data in Minits Msequ and the new task data for these two methods. For AdapterCI, use a bottleneck of 100. Due to limited computing resource, we randomly permute the task streams and run all experiments using three random seeds. We refer this as task order 1. We study the effect of task orders in $6.3. The selected tasks for task stream InstrDialog and InstrDialog++ are listed in Table 5. The task orders are listed in Table 6. More details in Appendix A.
Second, among all baselines, Replay generally have the best performance. All methods except Replay (50) have negative BWT, meaning that they all suffer from catastrophic forgetting. Further- more, forgetting on Tinit and Tunseen is even worse, which demonstrates that the ability of the initial instruction tundd model Init has deteriorated after learning new dialogue tasks. We also find storing more examples in memory improves Replay but does not significantly help AGEM. It might be be- cause the constraints added to the loss are roughly the same, no matter how many instances are stored. Despite used additional parameters, regularization- based L. and EWC perform similar to other base- lines. Multi overall performs well, with the high- est AR and improved FR on Tim,, however, it also forgets tasks in Tunscon Replay (50) has higher FR on Tinit than Multi because the 5,000 instances stored in Mini are jointly trained multiple times when learning subsequent tasks ($5.3), leading better data fitting.
6 Results and Analysis
this section, we report the performance vari- ous baselines discussed in $5.2 on our benchmark.
6.1 Results on InstrDialog Stream
Table 1 shows each method's overall performance and resource requirement after continually learning the InstrDialog stream. We have the following observations:
Third, FT-init performs surprisingly well on all metrics and is competitive to L2, EWC, and AGEM. This finding contradicts the common sense in CL that simply fine-tuning model sequentially would lead to catastrophic forgetting because all parameters can be freely updated in learning new tasks (Kirkpatrick al., 2017). Even for FT-no-init, which not tuned on any instruction data, shows increased AR after learning the 19 dialogue tasks. This raises a question: do various CL methods truly
First, all methods except AdapterCL have im- proved AR, compared the zero-shot performance (22.5) of the starting point model Init. This shows CIT can extend a model's knowledge. In con- trast, although AdapterCL parameter-efffiieet and does not rely on memory, performs even

---

mitigate forgetting and promote knowledge trans- fer in CIT? We hypothesize that the rich natural language instructions lead to the remarkable per- formance of the baselines ($6.3).
FT-init
FT-no-init
Instr.
Tinit
Tunseen
Tinit
Tunseen
id
3.6
2.8
1.0
1.7
def
25.8
23.6
20.3
22.4
6.2 Results InstrDialog++ Stream
def+2p
40.8
35.9
28.9
31.2
The performance of all methods after learning the InstrDialog++ steam is shown in Table 2. We observe most of the same findings in $6.1, except that:
def+2p+2n
37.7
35.7
27.2
33.1
Table 4: Effect of instruction templates on Tinis and
Tunseen-
First, Init has a higher (30.5 vs. 22.5) zero-shot performance on this long stream than on InstrDia- log, as in Table 1. We analyze that the categories of the selected 100 training tasks ($5.1) overlap with the categories in the stream, which enables more knowledge transfer of tasks between the same cat- egory because of the similar natural language in- structions. For example, both sets have tasks from sentiment analysis and toxic language detection. contrast, Init did not learn dialogue tasks, thus showing lower generalization on InstrDialog,
Adaptart.
Repley-12 11
20
10
Taak in the Streem
Figure of each method during learning the Instr- Dialog stream (task order 1).
see improved performance for almost all methods compared to Table especially on Tini and Tunseen. For FT-init and FT-no-init, the improvements of FWT and BWT are particularly significant, reaching the best among all CL meth- ods.
Second,
CL methods ($6.1 & $6.2). We also explore differ- ent aspects that might affect CIT.
Rich instructions enable knowledge transfer and reduce forgetting in CIT. We use the same setup as Table 2, except for using different instruction templates. Results in Table 3 confirms our hypothe- sis that the remarkable performance of FT-init and FT-no-init comes from the rich natural language instructions. When only using the task Id (e.g., task565_cines lrrrrcaanswerreeeeee
Combining the results on the two streams from Table and 2, we find that catastrophic forget- ting exists in CIT. However, learning longer task stream and diverse tasks of different types leads to better knowledge transfer and lower forgetting.
as instruc- tion without descriptive task definition or in- context examples, simply fine-tuning the model sequentially yields low AR, FWT, and BWT, which aligns with the findings in CL literature. Addition- ally, providing in-context examples (2p or 2p+2n) generally improves performance. However, al- though task performance (AR) is improved with two additional negative examples, witness de- creased knowledge transfer (FWT and BWT). For the model that not fine-tuned on any instruction data (FT-no-init), we find worse than FT-init, showing the benefits of the initial multi-task train- ing on instruction data.
FT-init
FT-no-init
Instr.
AR
FWT
BWT
AR
FWT
BWT
id
13.9
10.0
-30.3
11.5
8.0
-27.7
def
38.0
21.7
-6.9
33.8
-7.3
def+2p
34.8
29.8
-2.8
31.3
26.8
-5.2
def+2p
40.4
28.1
-7.1
33.8
-7.3
+2n
Table 3: Effeec instruction templates on InstrDia- log++. "id": only use the short task id; "def": only use descriptive task definitions; "def+2p": use task def- inition and two positive examples; "def+2p+2n": use additional two negative examples.
found Tim and Tunsson in Table 4, where the model catastrophi- cally forgets its initial abilities after learning long stream of tasks. Providing descriptive task defini- tions significantly boosts task performance as well
Similar observations
6.3 Ablation Studies
In this section, we investigate the reason why instruction-base baselines (FT-init and FT-no-init) perform as well as or even better than conventional

---

FT-init
EWC
AGBM-50 50
20
FT-no-init
FT-ntt
FT-to-rrt
FWT
8
15
Reslay-10
10
10
200
400
Task the Stream
training instances per task
Figure 4: AR of each method during learning the Instr- Dialog stream (task order 2).
Figure 6: Effect of training instances per task on FWT.
FT-init
0
Ad
FT-no-ini no-nnit
AGEM 50
EWC
BWT
FT-ntt
407
FT-tb-rtt
Reslay-10
-10
200
400
training instances task
Figure Effect of training instances per task on BWT.
Figure 5: AR of each method during learning the Instr- Dialog stream (task order 3).
ing knowledge forgetting will eventually appear if learning longer enough tasks.
as facilitates knowledge transfer. Moreover, it also
A large number of training instances do not help knowledge transfer. We vary the number of instances per task used for learning the InstrDialog stream from [10, 20, 100, 200, 500]. As shown in Fig.6 and 7, FWT and BWT gradually decrease when the number of training instances is scaled large values. It aligns with the findings by Wang et al. (2022) in standard IT, where large number of instances do not help generalization to unseen tasks, we also find it is true in CIT. Additionally, we find instruction-tune models (FT-init) have better generalization to new tasks (FWT) than the model not fine-tuned any instruction data (FT-no-init). This shows that, after learning few tasks, the model have learned how to follow instructions complete tasks, thus fewer training instances are required for new tasks.
maintains the model's generalization ability on un-
seen tasks. Combining the results in Table 1, 2, 3, and 4, find that those conventional CL meth- ods do not fully leverage the instructions reduce forgetting and facilitate knowledge transfer while learning continuously because the naive FT-init and FT-no-init can also achieve the same. This calls for novel CL methods designed for CIT.
Task types and learning order matter to CIT. To explore how task types and orders in a stream affects CIT, we randomly permute the InstrDialog stream to get two new task orders and conduct the same learning as Table 1. We present the interme- diate learning trends of all 19 tasks in Fig.3, Fig.4 and Fig.5. One can see from the plots that all base- lines are highly affected by task orders, fluctuating dramatically when different tasks are learned first. We argue that it is because the task difficulties and similarities vary a lot. Learned tasks transfer knowl- edge through the instructions to new tasks of the same type, therefore facilitating its learning. For example, the last task in order (Fig.3) is type of dialogue generation, which is the dominant task type in the stream (11/19, $4.4), therefore base- lines are improved. However, all baselines reach below Multi after learning all 19 tasks, demonstrat-
7 Conclusion
In this work, we establish a benchmark for contin- ual instruction tuning, with two 19 and 38 long task streams to be learned sequentially. We implement and compare various continual learning methods of different types using the benchmark study their effectiveness under this new domain. We conduct extensive ablation studies to analyze the lack current practices, and propose a future direction.

---

Limitations
ter Lee. Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro. and Yi Zhang. 2023. Sparks of artificial general in- telligence: Early experiments with gpt-4.
We identify our limitations as follows. First, due to limited resources, all experiments in this work use T5-small (LM-adapted) as the backbone, which might not entirely reflect continual instruction tun- ing in general. As Wang et al. (2022) points out, there is sizable gap between the smaller mod- els and the 11B or 3B models in generalizing to new tasks. Second, when creating the two CIT task streams, we only use English tasks from the Su- perNI dataset (Wang et al., 2022). In future, it can be extended to multilingual task streams to study cross-language continual instruction tuning. Third, we follow the SuperNI dataset to use ROUGE-L aggregated metric to evaluate all tasks. Al- though it acts good proxy for the model's over- all performance, it might not serve as an effective measurement for some specific tasks. Fourth, while we selected diverse tasks to form the InstrDialog and InstrDialog++ task streams, we did not analyse the characteristics of these tasks (Kim et al., 2023). In future, we consider to select better source tasks and study how source tasks affect CIT.
Arslan Chaudhry, Mare' Aurelio Ranzato, Mareus Rohrbach, and Mohamed Elhoseiny, 2019, Efficient lifelong learning with a GEM. In International Con- ference on Learning Representations.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling eestruction-finetunee language models. arXiv preprint arXiv:2210. 10.1 1416.
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2019. Continual learning: A comparative study on how to defy forgetting clas- sification tasks. arXiv preprint arXiv: 1909.08383, 2(6):2.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. efficiet transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Acknowledgements
Yufan Huang, Yanzhe Zhang. Jiaao Chen. Xuezhi Wang. and Diyi Yang. 2021. Continual learning for text clas- sification with information disentanglement based regularization.
This work is supported by TPG Telecom. We would like to thank anonymous reviewers for their valuable comments.
Proceedings of the 2021 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, pages 2746, Online. As- sociation for Computational Linguistics.
References
Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà. 2020. Continual lifelong Iearn- ing in natural language processing: A survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6523-6541, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Zixuan Ke, Haowei Lin, Yijia Shao, Hu Xu, Lei Shu, and Bing Liu. 2022. Continual training of lan- guage models for few-shot learning. arXiv preprint 1rXivv2210.05549.
Zixuan Ke and Bing Liu, 2023, Continual learning of natural language processing tasks: A survey.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1807-1901. Curran Associates, Inc.
Zixuan Ke, Bing Liu, Nianzu Ma. Hu Xu, and Lei Shu. 2021. Achieving forgetting prevention and knowl- edge transfer in continual learning. In Advances in Neural Information Processing Systems.
Joongwon Kim, Akari Asai, Gabriel Ilharco, and Han- nanch Hajishirzi. 2023. Taskweb: Selecting better source tasks for multi-task nlp.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz. Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Grabska- Barwinska, Demis Hassabis, Clau- dia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13)33511-522
Sébastien Bubeck, Varun Chandrasekaran, Ronen El- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Pe-

---

Chin-Yew Yow Lin. 2004, ROUGE: package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Our, pages 74-81, Barcelona, Spain. Association for Computational Linguisties.
Sylvestre- Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. 2017. icarl: In- cremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR).
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. CoRR, 15s/2107713586.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Amaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang. Mike Tian-Jian Jiang. Han Wang. Mattco Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey. Rachel Bawden, Thomas Wang, Tr- ishala Neeraj. Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan, Teven Le Scao, Stella Biderman, Lco Gao, Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In International Conference on Learning Representations.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. The flan collection: Designing data and methods for effective instruction tuning.
David Lopez-Paz and Marc' Aurelio Ranzato. 2017. Gradient episodie memory for continual learning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Ziyang Luo, Can Xu, Pu Zhao, Xiubo Geng, Chongyang Tao, Jing Ma. Qingwei Lin, and Daxin Jiang. 2023. Augmented large language models with parametric knowledge guiding.
Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. 2022. Fine-tuned language models are continual learners. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 6107-6122, Abu Dhabi, United Arab Emirates, Association for Computational Lin- guisties.
Andrea Madotto, Zhaojiang Lin, Zhempeng Zhou, Se- ungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Eu- njoon Cho. Pascale Fung. and Zhiguang Wang. 2021. Continual learning task oriented dialogue systems. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 7452-7467, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller. Cynthia Gao, Vedanuj Goswami, Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Di- ana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- bog. Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten. Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez. Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine- tuned chat models.
Michael McCloskey and Neal J Cohen. 1989. Catas- trophic interference in connectionist networks: The sequential learning problem. In Psychology of learn- ing and motivation, volume 24, pages 109-165. Else- vier.
Jisoo Mok. Jaeyoung Do, Sungjin Lee, Tara Taghavi, Seunghak Yu, and Sungroh Yoon. 2023. Large-scar scale lifelong learning of in-context instructions and how to tackle it. In Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages I 12573-1258) Toronto, Canada, Association for Computational Linguistics.
OpenAl. 2023. Gpt-4 technical report.
Chengwei Qin and Shafiq Joty. 2022. Continual few- shot relation learning via embedding space regular- ization and data augmentation. In Proceedings of the 60th Annual Meeting of the Association for Compu- lational Linguistics (Volume 1: Long Pupers), pages 2776-2788 Dublin, Ireland. Association for Compu- tational Linguistics.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Puro- hit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma,
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):1-67

---

Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022. huperrVaturallnsstuciion
size Minit to 10 and 50, same as Mseq We jointly train the data in Minit- Musg. and the new task data for these two methods.
Generaliza- tion via declarative instructions on 1600+ NLP tasks. In Proceedingss of the 2022 Conference Empiri- cal Methods in Natural Language Processing, pages 5085-5109 Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
For L.2 and EWC, we tune the regularization term from [0.001, 0.01, 0.1], and use 0.01. For AdapterCL, we use learning rate of le-3. For other methods, we use 1e-5. For the InstrDialog stream, use a batch size of 4 for EWC and for all other methods; for the InstrDialog+4 stream, we use batch size of 16. For all experiments except AdapterCI, we train maximum epoch of 15 and perform early stopping with patience to avoid overfitting; for AdapterCL, we use a larger epoch of 20 with patience of 5.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe Le. 2022. Finetuned language mod- els are zero-shot learners. In International Confer- ence Learning Representations.
Wenpeng Yin, Jia Li, and Caiming Xiong. 2022. Con- TinTin: Continual learning from task instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume
The selected tasks for task stream InstrDialog and InstrDialog++ are listed in Table 5. The task orders are listed in Table 6. All experiments done in an RTX3090 Ti with 24GB VRAM.
1: Long Papers), pages 3062-3072, Dublin, Ireland. Association for Computational Linguistics.
Zihan Zhang, Meng Fang, Ling Chen, Mohammad-Res Reza Namazi-Rade and Jun Wang. 2023, How do large language models capture the ever-changing world knowledge? review of recent advances.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma. Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. Lima: Less more for alignment.
Qi Zhu, Bing Li. Fei Mi, Xiaoyan Zhu, and Minlie Huang. 2022. Continual prompt tuning for dialog state tracking. In Proceedings of the 60th Annual Meeting of Association for Computational Lin- guistics (Volume 1: Long Pupers), pages 124-1137, Dublin, Ireland. Association for Computational Lin- guistics.
A Implementation Details
For both InstrDialog stream and InstoDialog++ stream, we conduct continual instruction tuning using the same initial instruction-tuned model for all methods except FT--n--nnit, AdapterCL, and Multi. For these methods, we initialize new T5 model. Since AdapterCL needs train task- specific adapter for each task, is too costly to train the 100 adapters for it, therefore we initialize it from a T5 model. For AdapterCL, we use a bottleneck of 100.
The initial instruction-tuned model ($5.1) is trained on 100 tasks with 100 instances per task (in total 10,000). We a maximum of 15 epochs with learning rate 1e-05. We perform checkpoint selection using the development set.
For AGEM and Replay, we experiment on a memory size of 10 and 50, i.e., we set the memory

---

Task Category
Number
Task Name
Domain
Dialogue Generution
tack565 ciree answer generation
Dialoguc
lusk574 dialogue sentence centenc__enerationn generation
Dialogue
3
task576 curiosity dialogs, wnwwer peneration
Dialogue, Commonsense
task611 manual multi dialogue lsk6399 mulii ware user uilerancee generation
Dialogue
Dialogue
task diplomacy __stet____ peneratiion lask1600 smcalllowssentenne generition
Dialogue, Game
Dialoguc, Commonsense
task 1603 smcalflow seaifowwseetene,.... sentence, generation
Dialogue
274__covvaa3___. sentence, generation 1729 personachot, geveratee next
Dialogue
10
Dialogue
task1730 personachat cou___choe__ee__e_ _extttte__tt___
Dialogue
tad229 searycmmoonnene
Intent Identification
12
Story
motiv text, peneration
lack573, air diallgueeeeeaaeee classiiiciittooo
Dialogue
14
task848 pubimodip classificatton
Medicine
convail_sentencee convai) sentence, generation
Dialogue
Dialogue State Tracking
166k766 cnigglist Laap__uaaaan_aeaeeeee classification
Dialogue
13441384
dialog classification
Dialogue, Commonsense
18
task1500_ddte 1et.5000ddul_ccom soo_dduc__clssiiii clasilication
Dialogus, Public Plazes
tnck1501 dsic3 anewer generstion
Dialoguc, Public Plaess
Style Transfer
20
[ax9977 negutive_lo possttive____iiiive_eei....eeeee style_ transfer
Reviews
Sentence Ordering
wigwwwewwwwrr peneration missing_sty stop
Natural Science
Word Semanties
22
tas4459 matres, static, cuuui_ccssiiicatiinn
News
Text Culegorization
29
tack379 agnews, topic, classification
News
Pos Tagging
24
task347 hyhridga, incorrect, answer generation
Wikipadia
Fill in The Blank
25
task 160 numer sense. muliiple choice_eeee sariiie___g_geeee sp__gemrttion task lask115) [15] s_s mapx min
Commonseuse
26
Mathernatics
Program Execution
27
tss6636_exit extract and_sort sort wowtoonnttutnttoteeeee www.uuunnilpplleeeee hitptabbsppbbttiaee ata_list
Malhsematiss
Question
tack301 record question generation
lnkk02__bbbe il La__ill_snnggge single_supprring
29
Commonsense
question generation
Misc.
30
usk306 jeopurdy. answwr preser__encaation double
Knowlolge Base
31
1421427 country region world
Countries
Coherence Clasilicatiinn
32
task298, moryyllaee correct, end classification
Story
Question Answerfigg
33
lask864 audiv_ undi sudglliingloopp.mmm question amswering
Mathematics
34
lnsk598 coad answer generation
Law
Summarization
35
task1553 dhilyyali summarization
News
Commonsense Classification
36
lnsk1203 atomic, classfication, preacl preact task967 niletaker incorrect fact generation, bused, paragr_p uak1607__thoo text cla_ssssiitiiatoonn
Commoneense
Wrong Candidate Generation
37
Commonsense
Toxic Langunge Detection
38
Social
Table 5: List of tasks selected from SuperNI (Wang et al., 2022)
Task Order
Task's IDs in order
Orderl
848 611 565
1714
574
1590
1730 294 576 1600 1500 639 1729 1501
1713
766
1603
1384
573
Order2
848
1603 1714
565
611
1590
1600
639 294 1500 1384 1713 1501 576 574
1729
766
573
1730
Order3
1713
576
1384
294
573 611
1729 1600
574 1590 848 639 766 1501 565
1603
1730
1500
1714
Table 6: Task orders for three runs of the InstrDialog Stream.


Reviews:
[{"review_id": "zxMacgKhQA", "reviewer": "Reviewer_okNW", "summary": "", "questions": "Q1) Was there any heuristic of choosing new tasks included in the InstrDialog++ Stream, or was it randomly chosen?\nQ3) One reason why there wasn't much knowledge transfer might be due to the low task correlation between the chosen stream tasks & the remaining tasks. Additional analysis using TaskWeb[1] might be helpful to take a look into this.\nQ4) Expert LM[2] is another considerable baseline, where a single expert LM is trained in order to mitigate catastrophic forgetting while preserving good unseen task performance when retrieved properly. It would be great if it is added to the baseline.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer okNW,\n\nWe appreciate your valuable time, review and suggestions. We want to address your concerns in detail as follows.\n\n> Q1. One of the reasons why CL methods do not work might be due to the small model size used in the experiments. Although it is understandable that it might have been due to limited resources, it is still questionable if the same result will hold for larger models as well.\n\n**A1:** It is a very good suggestion and thanks for your understanding. We tried T5-large as a larger model on the InstrDialog task stream and found consistent results with T5-small. We would like to borrow more computing resources to use larger models to add more comparisons. We plan to experiment using the T5-3B, Pythia-1.4B and Pythia-2.8B and will add ablation studies on the effect of model size. \n\n> Q2. Was there any heuristic of choosing new tasks included in the InstrDialog++ Stream, or was it randomly chosen?\n\n**A2:** We intended to explore the effect of various task categories. Based on such a strategy, we first select task categories essentially different from dialogues by nature (such as sentence ordering and style transfer) to distinguish with the InstrDialog stream. Then, from these chosen categories, we randomly select tasks.\n\n> Q3. One reason why there wasn't much knowledge transfer might be due to the low task correlation between the chosen stream tasks & the remaining tasks. Additional analysis using TaskWeb[1] might be helpful to take a look into this. \n\n**A3:** Thanks for your suggestion. \nWe also note that all tasks in the InstrDialog stream are relatively similar to other task categories (i.e., have a relatively higher correlation than non-dialogue-related tasks). However, the knowledge transfer is still not very high (Table 1). We will consider using TaskWeb to analyse the impact of different tasks. \n\n> Q4. Expert LM[2] is another considerable baseline, where a single expert LM is trained in order to mitigate catastrophic forgetting while preserving good unseen task performance when retrieved properly. It would be great if it is added to the baseline.\n\n**A4:** Thanks for your suggestion.\nHowever, technically speaking, training a separate expert LM per task does not precisely conform to the continual learning (CL) setting because there has to be only a single model for all sequentially learned tasks in CL [1]. We will consider adding this baseline to our benchmarks for completeness.\n\n\n[1] Continual Lifelong Learning in Natural Language Processing: A Survey, COLING 2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. CITB is a well established benchmark that could further be used in future research.\n2. Extensive experiments are conducted to show that previous CL methods are not very effective in more challenging settings (InstrDialog++) opposed to the argument from previous papers such as ContinualT0.", "reasons_to_reject": "One of the reasons why CL methods do not work might be due to the small model size used in the experiments. Although it is understandable that it might have been due to limited resources, it is still questionable if the same result will hold for larger models as well.", "questions_to_authors": "Q1) Was there any heuristic of choosing new tasks included in the InstrDialog++ Stream, or was it randomly chosen?\nQ3) One reason why there wasn't much knowledge transfer might be due to the low task correlation between the chosen stream tasks & the remaining tasks. Additional analysis using TaskWeb[1] might be helpful to take a look into this.\nQ4) Expert LM[2] is another considerable baseline, where a single expert LM is trained in order to mitigate catastrophic forgetting while preserving good unseen task performance when retrieved properly. It would be great if it is added to the baseline.", "paper_topic_and_main_contributions": "The authors test different continual learning methods on their proposing benchmark, CITB. CITB is consisted of InstrDialog and InstrDialog++ while using SuperNI as the seen task group. While there were a number of works studying continual learning of instruction-tuned LMs, this work organized a well established benchmark to extensively study continual learning. The findings are surprising in that CL methods do not mitigate catastrophic forgetting in the context of instruction tuning.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Kim, J., Asai, A., Ilharco, G., & Hajishirzi, H. (2023). TaskWeb: Selecting Better Source Tasks for Multi-task NLP. arXiv preprint arXiv:2305.13256.\n[2] Jang, J., Kim, S., Ye, S., Kim, D., Logeswaran, L., Lee, M., ... & Seo, M. (2023). Exploring the benefits of training expert language models over instruction tuning. arXiv preprint arXiv:2302.03202."}}, {"review_id": "JLzfaRrBtC", "reviewer": "Reviewer_Nb3H", "summary": "", "questions": "1. What is the practical scenario of Continual Instruction Tuning?\n2. In Table 1 and Table 2, the Init model outperforms the Multi model on unseen test set. I would expect these two models have comparable performance, especially when the init training set is much larger than the sequence training set. Could the authors provide some insights on this?\n3. For replay method, does the memory have fixed size? Or is each task allocated fixed number of samples to store in memory, i.e. the memory size grows as tasks arrive?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Nb3H,\n\nWe appreciate your valuable time, review and suggestions. We want to address your concerns in detail as follows.\n\n> Q1. One of main contribution of the paper is the CITB benchmark. I would expect analysis on task characteristics such as the instruction style, negative and positive ratio and similarity among tasks within and across init, seq and unseen test set. For example, in the same task category (e.g. dialogue generation), is the instruction similar among task? Do tasks mostly differ in the domain (covariate shift) or labels (label shift)? This can help to draw insight on the unexpected effectiveness of finetuning and the intriguing case where the Init model outperforms the Multi model on unseen test set.\n\n**A1:** Thanks for your suggestion. We have briefly provided the characteristics and statistics of our benchmark tasks in Appendix Fig.5 and Table 5. \nFor all experiments, by default, we use the instruction template consisting of the task definition, two positive examples and no negative examples (L378) because it has been proven to be the most effective by SuperNI.\nHaving said that, we will consider adding more statistics and analysis of our benchmark.\n\n> Q2. There should be a metric to account for how the learning of new task affects the performance on the init and unseen test set after training each task in the sequence. Currently, the paper only report the performance after training the final task.\n\n**A2:** This might be a misunderstanding of our evaluation protocol. As discussed in Section 4.3 L283, we evaluate the model on the **D_seq**, **D_init**, and **D_unseen** test sets **after each task is learnt**. Table 1 and Table 2 only show the final results after the last task is learnt due to space limitations.\nWe have shown the intermediate learning performance (after each task) in Figures 2, 6, 7. \nHowever, we acknowledge that we do not have intermediate performance on the init and unseen tasks while training each task in sequence (similar to Figures 2, 6, 7). We will add these plots when we can modify our paper.\n\n> Q3. Table 2 is missing a multi-task baseline (Multi-Init) on seq training set where the model initialized with Init model.\n\n**A3:**\nThere might be some misconceptions here. The **Init** model in **Table 2** is initialised from the fresh T5 model checkpoint before learning the InstrDialog++ stream (not the same mode in Table 1).\n\nPlease correct us if we misunderstood your question. If we understand correctly, initialising the model from the **Init** model in **Table 1** when learning the InstrDialog++ stream (Multi-Init) does not make sense. The InstrDialog++ stream already contains all 19 tasks in the InstrDialog stream (with another 19 non-dialogue-related tasks, L361). Therefore, it does not make sense to initialise the model from the Table 1 Init model, which has already learned the 19 dialogue tasks.\n\n> Q4. As stated in the limitation, the paper only evaluates on T5-small model. It's challenging to confidently assert that the current observations are applicable for larger model.\n\n**A4:**  It is a very good suggestion and thanks for your understanding. We tried T5-large as a larger model on the InstrDialog task stream and found consistent results with T5-small. We would like to borrow more computing resources to use larger models to add more comparisons. We plan to experiment using the T5-3B, Pythia-1.4B and Pythia-2.8B and will add ablation studies on the effect of model size.\n\n> Q5. What is the practical scenario of Continual Instruction Tuning?\n\n**A5:** In practice, pre-trained models such as T5 and Llama are versatile in many tasks but might still be poor enough to follow human instructions to perform tasks.\nTo improve the LM’s instruction-following ability, a common approach is to fine-tune the LM on instruction data. For instance, fine-tune T5 on dialogue/chat data so that they are more chat-friendly and can perform dialogue-related tasks. However, existing efforts only fine-tune the LM on one specific domain. To obtain an instruction-tuned LM on other domains, they need to start from the initial pre-trained LM and repeat the fine-tuning process, leading to **multiple copies of the instruction-tuned model** (such as T5-chat, T5-law). In Continual Instruction Tuning, we only have **one model** and continually improve the model on multiple domains.\n\n> Q6. In Table 1 and Table 2, the Init model outperforms the Multi model on unseen test set. I would expect these two models have comparable performance, especially when the init training set is much larger than the sequence training set. Could the authors provide some insights on this?\n\n**A6:** We use the official 119 test tasks from the SuperNI dataset as the unseen task (L372) and randomly select 100 tasks from the non-overlapping tasks to train the Init model (L395). For the Multi model, the model is trained jointly on the 100 tasks and the InstrDialog or the InstrDialog++ tasks.\n\nWe conjure the reason why the Init model outperforms the Multi model on the 119 unseen tasks is that the Multi model might be overfitting and thus perform worse. We have checked the 119 unseen tasks. Although all tasks are not seen, their task categories might overlap with those in the 100 initial training tasks and InstrDialog and InstrDialog++.  For example, there are 10 dialogue-related tasks in the 119 unseen tasks. Since our base model is relatively small, training on more related data might hurt its generalization ability on unseen data and thus perform worse than the Init model.\n\n> Q7. For replay method, does the memory have fixed size? Or is each task allocated fixed number of samples to store in memory, i.e. the memory size grows as tasks arrive?\n\n**A7:** For the replay method, the memory size grows as tasks arrive. However, we experimented with two settings – we set the memory size as 10 and 50 respectively for each task, both in the initial multi-task training and sequential single-task training stage. We found that using the small replay size can already perform relatively well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors design a thoughtful evaluation protocol to evaluate on both test set of the pretrained and finetuned phase as well as unseen test set.\n- Authors conduct rigorous experiments to study the behaviour of different CL methods in the CIT setting. ", "reasons_to_reject": "- One of main contribution of the paper is the CITB benchmark. I would expect analysis on task characteristics such as the instruction style, negative and positive ratio and similarity among tasks within and across init, seq and unseen test set. For example, in the same task category (e.g. dialogue generation), is the instruction similar among task? Do tasks mostly differ in the domain (covariate shift) or labels (label shift)? This can help to draw insight on the unexpected effectiveness of finetuning and the intriguing case where the Init model outperforms the Multi model on unseen test set.\n- There should be a metric to account for how the learning of new task affects the performance on the init and unseen test set after training each task in the sequence. Currently, the paper only report the performance after training the final task.\n- Table 2 is missing a multi-task baseline (Multi-Init) on seq training set where the model initialized with Init model.\n- As stated in the limitation, the paper only evaluates on T5-small model. It's challenging to confidently assert that the current observations are applicable for larger model.", "questions_to_authors": "1. What is the practical scenario of Continual Instruction Tuning?\n2. In Table 1 and Table 2, the Init model outperforms the Multi model on unseen test set. I would expect these two models have comparable performance, especially when the init training set is much larger than the sequence training set. Could the authors provide some insights on this?\n3. For replay method, does the memory have fixed size? Or is each task allocated fixed number of samples to store in memory, i.e. the memory size grows as tasks arrive?", "paper_topic_and_main_contributions": "This paper formulate and establish a benchmark for continual instruction tuning (CIT) problem. The benchmark contains two task sequences constructed from SuperNI dataset: (i) InstrDialog consisting of 19 dialog tasks and (ii) InstrDialog++ of 38 tasks from broader categories. The authors also propose evaluation protocol to assess the performance on both old, upcoming and unseen tasks. The experiments and analysis on several well-studied CL methods reveal an unexpected effectiveness of the simple fine-tuning method over other CL methods.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "60RIPVB2c3", "reviewer": "Reviewer_y4HB", "summary": "", "questions": "1. In Section 5.2, why don't you include ConTinTin which you also mentioned in the Related Works? I think it's the closest work to this paper and its method can be used in the CIT problem. If applicable, showing its results on your benchmark will be great.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer y4HB,\n\nWe appreciate your valuable time, review and suggestions. We want to address your concerns in detail as follows.\n\n> Q1. I don't see a major point to reject this paper. However, I think the paper can be greatly improved if there could be in-depth analysis of why/how rich instructions can enable knowledge transfer and reduce forgetting in CIT. This is a very important question for the study of CIT as this is where CIT and other CL scenarios behave differently. While this question may be quite difficult to answer, a possible way to approach it may be tracking how the PLM itself change in the process of CIT. For example, [1] tracks PLM's representation ability in the class-incremental learning process and finds that the generation objective is beneficial to continual learning by preserving PLM's representation ability in the CL process. A similar tracking approach may be applied in this work; or you may investigate the changing trajectory of other internal properties of PLM in the CIT process.\n\n**A1:** Thanks for your suggestion. We will provide more investigations and explanations of how rich instructions can enable knowledge transfer and reduce forgetting in CIT. Apart from your suggestion to use [1] to track the PLM's representation ability in the learning process, we also plan to use [2] to investigate task correlations and similarities. Our InstrDialog stream consists of all dialogue-related tasks, while the InstrDialog++ stream provides more diverse task categories. Using these two CL task streams and the above tools, we plan to provide more explanations and analysis. \n\n> Q2. In Section 5.2, why don't you include ConTinTin which you also mentioned in the Related Works? I think it's the closest work to this paper and its method can be used in the CIT problem. If applicable, showing its results on your benchmark will be great.\n\n**A2:** Since ConTinTin did not release their code, we tried to re-implement their method according to their paper (L446). However, we cannot reproduce the results as reported in their paper. Therefore, we did not use ConTinTin in the benchmark for a fair comparison.\n\n\n[1] Class-Incremental Learning based on Label Generation, Shao et al., ACL 2023.\\\n[2] TaskWeb: Selecting Better Source Tasks for Multi-task NLP,  arXiv 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Strictly formulating the CIT problem and providing the corresponding benchmark is very beneficial to the community. This paper did a solid work on this side by exploiting the SuperNI dataset and designing comprehensive metrics (i.e., the model is evaluated on the initial training, unseen, and newly learned tasks to reveal different aspects in CIT).\n2. One interesting finding in this paper is that unlike other CL scenarios, the forgetting is not a major issue in CIT when we directly update the model sequentially. The authors attribute this to rich instructions and cleverly verify this via an ablation experiment.\n3. Overall, the paper is nicely written and the experiments are extensive for a benchmarking paper.", "reasons_to_reject": "I don't see a major point to reject this paper. However, I think the paper can be greatly improved if there could be in-depth analysis of why/how rich instructions can enable knowledge transfer and reduce forgetting in CIT. This is a very important question for the study of CIT as this is where CIT and other CL scenarios behave differently. While this question may be quite difficult to answer, a possible way to approach it may be tracking how the PLM itself change in the process of CIT. For example, [1] tracks PLM's representation ability in the class-incremental learning process and finds that the generation objective is beneficial to continual learning by preserving PLM's representation ability in the CL process. A similar tracking approach may be applied in this work; or you may investigate the changing trajectory of other internal properties of PLM in the CIT process.\n\n[1] Class-Incremental Learning based on Label Generation, Shao et al., ACL 2023.", "questions_to_authors": "1. In Section 5.2, why don't you include ConTinTin which you also mentioned in the Related Works? I think it's the closest work to this paper and its method can be used in the CIT problem. If applicable, showing its results on your benchmark will be great.", "paper_topic_and_main_contributions": "The paper formulates the Continual Instruction Tuning (CIT) problem which studies how instruction tuning works in the context of continual learning (CL) tasks. A first-ever CIT benchmark is established which includes two long task streams of different types to study various CL methods systematically. The benchmark results show that existing CL methods do not effectively leverage the rich natural language instructions. Also, strikingly, unlike other CL scenarios, simply fine-tuning the model sequentially doesn't trigger a serious forgetting problem. The authors attribute this to the rich semantic information provided by the task instructions.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Please refer to my comments in \"Reasons To Reject\".", "Typos_Grammar_Style_and_Presentation_Improvements": "1. Line 059: Despite effectiveness -> Despite the effectiveness\n2. Line 113: the lack of -> the defects of "}, "extra_text": {}}, {"review_id": "Jf9VxrQPUT", "reviewer": "Reviewer_eWi8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer eWi8,\n\nWe appreciate your valuable time, review and suggestions. We want to address your concerns in detail as follows.\n\n> Q1. As the framework is looking at instruction tuning, generative replay methods may be a great addition to be included in the benchmark.\n\n**A1:** Thanks for your suggestion. Generative replay methods generate pseudo samples instead of storing true samples in the memory, which is beneficial when raw data is unavailable. We will consider exploring generative replay methods as a future work.\n\n> Q2. The paper has no mention of code that will be provided or open-sourced, nor supplementary materials provided for the code. Unfortunately, without code, the contribution as a benchmark is degraded significantly.\n\n**A2:** Thanks for your interest. As a benchmarking paper, we will release all data, code and models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper proposed a new continual instruction tuning (CIT) benchmark that consists of dialogue tasks.\n* The paper provides adequate baseline results of CL methods on the CIT problem and shows that instruction tuning is a great way to provide forward/backward transfers.\n* No obvious concerns regarding to the benchmark construction.", "reasons_to_reject": "* As the framework is looking at instruction tuning, generative replay methods may be a great addition to be included in the benchmark.\n* The paper has no mention of code that will be provided or open-sourced, nor supplementary materials provided for the code. Unfortunately, without code, the contribution as a benchmark is degraded significantly.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper provides a continual instruction tuning (CIT) benchmark. The paper provides two long task streams for instruction tuning and systematically studies the existing continual learning methods.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "See reasons to reject."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper titled "Continual Instruction Tuning: A Benchmark Study" presents a benchmark for the continual instruction tuning (CIT) problem. The benchmark consists of two long task streams constructed from the SuperNI dataset, and the authors propose an evaluation protocol to assess performance on old, upcoming, and unseen tasks. The experiments and analysis on several well-studied continual learning (CL) methods reveal an unexpected effectiveness of the simple fine-tuning method over other CL methods.
Overall, the reviewers think that the authors design a useful evaluation protocol to evaluate on both the test set of the pretrained and finetuned stages as well as the unseen test set. However, the paper lacks detailed analysis on task characteristics such as the instruction style, negative and positive ratio, and it only evaluates on a small model, and it is challenging to confidently assert that the current observations are applicable for larger models.
Decision: Accept-Findings