Lifelong Sequence Generation with Dynamic Module Expansion and Adaptation
Authors: Chengwei Qin
CHEN CHEN
Shafiq Joty
Venue: EMNLP 2023 Main (2023)
Keywords: lifelong learning
sequence generation
dynamic module
forward knowledge transfer
TL;DR: We propose dynamic module expansion and adaptation for lifelong sequence generation, outperforming existing methods in different settings.

Abstract: Lifelong sequence generation (LSG), a problem in continual learning, aims to continually train a model on a sequence of generation tasks to learn constantly emerging new generation patterns while avoiding the forgetting of previous knowledge. Existing LSG methods mainly focus on maintaining old knowledge while paying little attention to knowledge transfer across tasks. In contrast, humans can better learn new tasks by leveraging previously acquired knowledge from similar tasks. Inspired by the learning paradigm of humans, we propose Dynamic Module Expansion and Adaptation (DMEA), which enables the model to dynamically determine the architecture for acquiring new knowledge based on task correlation and select the most similar previous tasks to facilitate adaptation to new tasks. In addition, as the learning process can easily be biased towards the current task which might cause more severe forgetting of previously learned knowledge, we propose dynamic gradient scaling to balance the learning of the current task and replayed tasks. With extensive experiments, we demonstrate that DMEA can consistently outperform existing methods in different LSG settings.

Lifelong Sequence Generation with Dynamic Module Expansion and Adaptation
Chengwei Qin Chen Chen Shafiq Joty**
Nanyang Technological University
Salesforce AI
cchengwei0030e ntu, chen1 4360e.44ee..e ntu, srjotyentu).edue edu. sg
Abstract
in recent interview said: "Most models are static. But the world changes every minute, every second. Dealing with a dynamic world is a new area that's s up and coming." (Source)
Lifelong sequence generation (I.SG), a prob- lem in continual learning, aims continually train a model on a sequence of generation tasks to learn constantly emerging new generation patterns while avoiding the forgetting of previ- ous knowledge. Existing LSG methods mainly focus on maintaining old knowledge while pay- ing little attention knowledge transfer across tasks. In contrast, humans can better learn new tasks by leveraging previously acquired knowl- edge from similar tasks. Inspired by the learn- ing paradigm of humans, we propose Dynamic Module Expansion and Adaptation (DMEA), which enables the model to dynamically deter- mine the architecture for acquiring new knowl- edge based task correlation and select the most similar previous tasks to facilitate adapta- tion to new tasks. addition, as learning process casily be biased towards the cur- rent task which might cause more severe for- getting of previously learned knowledge, we propose dynamic gradient scaling to balance the learning of the current task and replayed tasks. With extensive experiments, we demon- strate that DMEA can consistently outperform existing methods in different LSG settings.
A potential solution is to formalize sequence generation as lifelong sequence generation or LSG (Sun et al., 2020), where the model is expected to learn sequentially from a stream of generation tasks with potentially different data distributions. In such cases of distribution shift, the model might forget previously acquired knowledge upon learning new tasks, phenomenon known as catastrophic forget- ting (McCloskey and Cohen, 1989). Previous LSG methods (Mi et al., 2020; Sun et al., 2020; Madotto et al., 2021) mainly explore different ways to al- leviate forgetting. Recently, Zhang et al. (2022) propose Adaptive Compositional Modules (ACM) which dynamically adds modules for new tasks depending on whether there are reusable previous modules, achieving SOTA performance on LSG.
Despite its effectiveness, ACM has several key limitations. First, it mainly focuses on mitigating forgetting of previously acquired knowledge while paying little attention to transferring learned knowl- edge to new tasks which is as important for con- tinual learning preventing forgetting (Ke et al., 2020). In fact, hallmark of human intelligence is that humans can better learn new tasks by lever- aging previously acquired knowledge from simi- lar tasks (Lake al., 2017). They can not only determine whether previously acquired skills are sufficient to solve a new task, but also exploit the most similar learned skills to facilitate the learning of the task; see Appendix A. for an illustration. Second, ACM does not consider the correlation be- tween Icarned tasks and the new task when adding modules, which might hinder finding the optimal architecture (case study in Appendix A.9). Finally, the learning process ACM can be biased towards the new task as the gradient norm of the new task on reused modules is typically much larger than
1 Introduction
With the recent advancements in pre-trained lan- guage models (LMs), current sequence generation methods have achieved impressive performance on variety of generation tasks (Radford et al., 2019; Raffel et al., 2020). Typically, these models are trained on a fixed corpus, assuming the un- derlying data distribution to be static (Ham et al., 2020; El-Kassas et al., 2021). However, real cog- nitive tasks are generally more complex involving changing contexts and dynamic environments. The ever-changing data distribution causes the models face challenges in acquiring new knowledge, while retaining the prior knowledge. Speaking about what is next for NLP, Kathleen McKeown

---

that of replayed tasks, which may affect previously acquired knowledge; see Appendix A.2 for an ex- planation.
With extensive experiments and analysis. we demonstrate the effectiveness of our method com- pared to existing ones in different LSG settings.
Inspired by the learning paradigm of humans and to address the above limitations of ACM, in this work we propose Dynamic Module' Expansion and Adaptation (DMEA). We divide the learning process of new task into three stages: expansion, selection and adaptation. In the expansion stage, DMEA determines whether to reuse modules of previous tasks or insert new modules for learning novel knowledge, Inspired by Zhang et al. (2022), it utilizes differentiable architecture search (Liu et al., 2019) to enable the model to dynamically determine the architecture for solving the new task. The learnable coefficients in architecture search are initialized based on the cosine similarity of word frequency distributions between learned tasks and the new task, aiming to discover the optimal model architecture. After searching, the module with the largest coefficient in every layer is chosen for the new task. In the selection stage, DMEA selects the top-K most similar previous tasks through input subspace (Lin et al., 2022b). Finally, in the adap- tation stage, it utilizes the selected similar tasks to facilitate adaptation to the new task. The output of seleeted similar tasks is fused with that of the new task using learnable coefficients in every trans- former layer to enable forward knowledge transfer. This is indeed an instance of mixture-of-expprts (Masoudnia and Ebrahimpour, 2014).
Related Work
Lifelong Learning (LL) aims to continually learn knowledge from a sequence of tasks with different distributions. The goal is twofold: alleviate catas- trophic forgetting (McCloskey and Cohen, 1989) of learned tasks, and facilitate knowledge transfer (Lopez-Paz and Ranzato, 2017) across tasks.
Catastrophic forgetting typically means that the model forgets previously acquired knowledge af- ter learning new tasks. Prior LL methods mainly focus on mitigating this problem and can be di- vided into three categories. First, regularization- based methods constrain the update of parameters that are important to learned tasks retain pre- vious knowledge (Kirkpatrick et al., 2017; Li and Hoiem, 2017: Zenke et al., 2017; Ritter et al., 2018). Second, architecture-basel methods dynamically adjust the model architecture to acquire new in- formation while preventing the forgetting of previ- ously learned tasks (Rusu et al., 2016; Chen et al., 2016; Fernando et al., 2017; Madotto et al., 2021; Zhang et al., 2022). Finally, memory-bused meth- ods keep a number of key samples from previous tasks in memory to alleviate forgetting (Rebuffi et al., 2017: Shin et al., 2017: Chaudhry et al., 2019; Qin and Joty, 2022a). The memory data can be ei- ther real examples (Han et al., 2020) or generated by language models (Sun et al., 2020; Qin and Joty, 2022b).
In addition, when the model learns new task, DMEA also incorporates eseudo-sample replay (Sun et al., 2020) to further mitigate catastrophic forgetting. To address the "bias to the new task" in the gradient update, we introduce dynamic gradi- ent scaling balance the learning of the new task and replayed tasks. To verify the effectiveness of DMEA, we conduct extensive experiments on vari- ous generation tasks in different 1.SG settings. The empirical results show that DMEA can consistently outperform previous ofate-of-the- baselines.
More recently, researchers have considered ex- ploring knowledge transfer in LL, i.e., learning on a task can benefit from learning on another task by transferring related knowledge. This includes CTR (Ke et al., 2021) and CUBER (Lin et al., 2022a). Despite their effectiveness, these methods mainly focus on classification tasks, while generation tasks typically have more complex label space. Note that this line of research is different from transfer learning (Ruder et al., 2019), which mainly focuses on exploring better ways to reuse learned knowl- edge which is usually static, e.g., a frozen language model. In contrast, the acquired knowledge is con- tinually accumulated in lifelong learning.
In summary, our main contributions are:
To the best of our knowledge, we are the first to explore solving L.SG from the perspective of human learning. We propose DMBA, a novel method based on dynamic module expansion and adaptation, to alleviate catastrophic forgetting and facilitate knowledge transfer in I.SG.
Lifelong Sequence Generation (LSG) enables the model to learn sequentially from a stream of gen- cration tasks. Sun et al. (2020) propose LAMOL which formalizes different types of tasks as ques-
'Following Zhang et al. (2022), we use an Adapter (Houlsby et al., 2019) as the insertable module.

---

tion answering and utilizes pseudo-sample replay alleviate forgetting. Chuang et al. (2020) fur- ther improve LAMOL by knowledge distillation (Hinton et al., 2015). AdapterCL (Madotto et al., 2021) inserts task-specific modules into every trans- former layer to learn new tasks while keeping the pre-trained LM and previous modules frozen. On the basis of AdapterCL, Zhang et al. (2022) in- troduce ACM which dynamically adds modules for learning new tasks depending on whether there are reusable previously inserted modules. Though ACM can enable knowledge transfer to some extent via module sharing, there is no explicit mechanism to encourage knowledge transfer across tasks, a common phenomenon of human learning.
T
Expansion
Layer /
Layer I
Layer 1
Module
Layer 1
Selection
Layer 0
Layer 0
newly
module selected for the new task
previous tasks
module
dynamic
inbalization of coafficiants
weighted
discarded
average
Summary. Existing work in L.SG mainly focuses on mitigating the catastrophic forgetting of previ- ously learned knowledge while paying little atten- tion to knowledge transfer across tasks, In contrast to these lines work, we aim to explicitly encour- age forward knowledge transfer in LSG inspired by the way humans learn (Lake et al., 2017).
Figure 1: In the expansion stage, after inserting new module (purple dashed hexagon) into each layer, DMEA dynamically determines the architecture by differen- tiable architecture search. Specifically, the outputs of all modules the same layer are fused through dynam- ically initialized learnable coefficients. The weighted average is then passed to the next layer of the model for learning. Note that only newly added modules (purple dashed hexagons) are learnable modules in this stage. After several epochs of training, the module with the largest coefficient in every layer (green polygon) is se- lected for the new task. Thee selected module can be either a previous module (rectangle) or the newly added one (hexagon). Finally, newly added modules that are not selected (red dashed hexagons) will be discarded.
3 Problem Formulation
LSG involves learning from stream of sequence generation tasks
(T) ...., ....,"), where every task T has its own training set Dirinn validation set Dialid and test set Dist Every dataset D contains a set of examples {(x,, ) where X, and Y, denote the input and output texts, respectively. At time step k the model is trained on the training set D of task T and has no access to real samples of previously learned tasks.
task, the model is optimized to decode Y given X and Q. Denoting the concatenation of X, and Y as A, the autoregressive training objective is:
n
After the training on D the model is pected to perform well on all the tasks learned so far, i.e., T Th and will be evaluated on the test set Diest of each task (11 < i k) with corre- sponding evaluation metrics separately. Therefore, to achieve the goal of LSG, the model is required to alleviate the forgetting of acquired knowledge and better learn new patterns through possible forward knowledge transfer.
Lissk
log
(1)
j=m+1
where n is the total number of tokens in A and (Ai,... Am) is the concatenation of X and Q, and denotes the model parameters.
4 Methodology
Inspired by how humans learn new task (Fig. 5), DMEA divides the learning process into three stages. The expansion stage ($4.1) first determines the model architecture dynamically. The selection stage ($4.2) then selects the top-K most similar previous tasks which are utilized in the final adap- tation stage ($4.3) facilitate adaptation the new task. We also employ saeudo-sample replay along with a dynamic gradient scaling method to balance the learning of the new and replayed tasks.
3.1
Data Format
Given an input-output text pair (X, Y) for a task, the model learns to decode the output text Y af- ter reading the input X. Following Zhang al. (2022), natural language question Q describing the purpose of each task (task instruction) is in- serted after the input to form a triple (X, Y);
see Appendix A.3 for an example. To learn new

---

4.1 Expansion Stage
Selection
T
Adaptation
Humans are able to determine whether previously acquired skills are sufficient to solve a new task. Our method DMEA aims to mimic this learning process in the expansion stage. It can dynamically decide whether to reuse modules of previous tasks insert new module in every transformer layer learn novel knowledge. Inspired by Zhang et al. (2022), we utilize differentiable architecture search (Liu et al., 2019) to achieve this goal.
Learned Tasks
Layer I
Layer 1
Input Subspace
Layer 0
module simiar
tusion
Specifically, assuming that there are k modules (i.e., Adapter (Houlsby et al., 2019)) {m²,.... ... m)) in layer of the transformer model before learning new task TV, we temporarily insert a new module 1 into this layer at the beginning of the expan- sion stage. For each forward pass, after calculating the output it of every module m² in the layer sepa- rately, we fuse all outputs {M1; hi (1) through learnable coefficients {M
Selected Simllar Tasks
medille
fooed
auppi
Figure 2: In the selection stage, DMEA selects the top- most similar previous tasks through input subspace to facilitate adaptation to new task. During adaptation. the output of the selected similar tasks is fused with that of the new task in every layer to enable forward knowledge transfer. Note that only modules selected for the new task (green polygons) are learnable modules in the adaptation stage. addition, DMEA introduces dynamic gradient scaling to balance the learning of the new task and replayed tasks.
..., X/1 1 as follows.
k+1
M'
1/g
k+1 s== e.a
The selected module in layer can be either from previous modules (m!....,', m,.)) or the newly added one m and will be tuned in the adaptation stage to accommodate new knowledge. We then discard newly added modules that are not selected. Note that only newly added modules and coefficients are learnable in the expansion stage; the pre-traines pre-trained LM and previous modules are kept frozen.
The weighted average '' is then passed to the next part of the model for learning. After training the model on D Dirain for several epochs using Liaann (de- fined in $4.3), we select the module with the largest coefficient in every layer for the new task T Different from Zhang et al. (2022) which initial- ize (X, M with predefined hyperparame- ters, we propose to dynamically initialize learnable coefficients based on the correlation between the learned tasks T ...., Ti-1 and new task T5. De- noting the word frequency distribution of T as f and all previous tasks sharing the module mi as 2/- the learnable coefficient X/ is initialized as:
4.2 Selection Stage
As humans, we can better acquire new knowledge by recognizing and utilizing knowledge from pre- viously learned tasks that are similar (Lake et al., 2017). Based on the observation that the norm of one task's 's gradient projection onto the subspace of another task can characterize the correlation be- tween them when the model architecture is static (Lin et al., 2022b), we further extend it to dynamic modules. Specifically, we obtain the input subspace of each task using modules of it and select the top- K most similar previous tasks by input subspace similarity to facilitate adaptation to the new task TV. The model architecture induced from the ex- pansion stage is used for selection and adaptation. Similar Lin et al. (2022b), we adopt Sin- gular Value Decomposition (SVD) to obtain the input subspace of each task. After training the model on D Di for several epochs in the ex- pansion stage, we randomly select n samples
max cos(f), ph+t), 1 1stsk T'Ez min \
X
(3)
t t= +1
1SiSk
where cos is the cosine similarity function and f is calculated based on the training set Dimin- In this way, a previous module shared by tasks with higher word frequency distribution similarity to the new task has larger initial coefficient, increasing the tendency to reuse it. In addition, the coeffi- cient X of the newly added module m/+1 is initialized to the minimum value of the initial coef- ficients/ (2/.... X of previously added modules } to encourage module reuse.

---

.n)...,,n) from D Drain and obtain their rep- resentations (,,, E Rn by forward- propagating them through the network. We use the final-layer representation of the last non- non-padding token the input as the sample representation.
added modules) and learnable coefficients while keeping the pre-trained language model and other modules frozen.
As there no saved real sample of previously learned tasks when the model adapts to new task, we also incorporate pseudo-sample replay (Sun et al., 2020) to alleviate the forgetting of acquired knowledge. We achieve this by simultancously training the model as a task solver (Lusk in $3.1) and as a data generator. When training as data generator, the model learns to generate the triple (X, Q, Y) given a mask-specific generation token G as input. Then before learning a new task, the model can generate pseudo samples of previous tasks, which are combined with new data for train- ing to mitigate forgetting. Denoting the concatena- tion of G, G,X, Q and Y as A', the data generation loss is expressed as:
After obtaining the representation matrix R [X.,.., X E RRxxn for task TV, we apply SVD to Rè, i.e., R (V'), where U [u), -. umn] C mm.x is composed of left-singular vectors u/. E/ E IR is a rectangular diagonal matrix with singular values on the diagonal, and V = - lu]..... w/ E R is composed of right- singular vectors v.. To obtain the input subspace S of TV, we select the first left-singular vec- tors in U to form the bases B [u]..... [ for 51, where k is determined by the requirement: [1 2 <'1| with Ry being the k-rank ap- proximation of Ri, F being the Frobenius norm, and el being predefined threshold.
m
For the new task T', the norm of its sub- space projection onto the subspace of a previously learned task 7 could characterize the similarity 2j between these two tasks. More formally,
dalta
Σ
log po (A,14
(6)
where m is the total number of tokens in A'. The overall loss that DMEA optimizes for adapting to a new task is:
Q3 Projs (S) l2
(4)
|
Cuain - Lrask + L&
(7)
where Projs: (5 BBB denotes the sub- space projection. After getting the similarity scores Qj., 1 <<< < of all previous tasks, we pick K tasks Tsim- (T) ..., T with the top-K highest scores facilitate adaptation to the new task T
=
where a is the weight of data generation loss. After the expansion stage, if the new task reuses some modules of previously learned tasks, the model will generate some pseudo samples of these tasks and train the model using Crain on the combi- nation of new data and pseudo data. As the model has not seen new data before, the gradient norm of the new task reused modules is much larger than that of replayed tasks. The learning process can easily be biased towards the new task which may affect previously acquired knowledge.
4.3 Adaptation Stage
For adaptation to Ti, assume that Tall (T) T T³) contains a total of r modules {m)..... m in layer 1. During the training on DJ
train using Ciain (see Eq. (7)), for each sample in D we fuse the output h/g of each mod- ule my E {m/.... {m/...,,, m by learnable coefficients {at,... 2 to enable forward knowledge transfer:
Therefore, to balance the learning of the new task and replayed tasks, we introduce dynamic gra- dient scaling. Specifically, assuming that the new task T reuses modules (m1:.., of pre- vious task T in all layers, we randomly select q examples from DJ and pseudo samples of T separately and forwards them through the model obtain the gradient of T and T using Cirain with regard to reused modules (mz..... ). denoted as g and g", respectively. The dynamic scale factor mi is then calculated as:
Σ
(5)
2a=1
The learnable coefficients (o/:... .a....., J are equally initialized to 1.0. Similar to the expansion stage, the fused output /e' is passed the next part of the model for learning. After training, the learnable coefficients will be saved for inference. Note that we only tune modules selected in the expansion stage (can be modules of previous tasks or newly
||و ||2
1)e -t +1
(8)
1

---

Methods
Finetune
EWC
LAMOL
Metac
Adapter +LAMOL
AdapterCI.
ACM
DMEA
MTL
Tume Whole Model?
x
Pseudo sample Replay?
x
/
4
56.6
65. 3
65. 20.4
64. 52..2
63. 42o.
61.420a
65.8
67.1.0.2
Similar Tasks
#2
51.840.
61.020.2
65. 25.2....4
65.0 65.0=e2
64.629.4
63. 63.4
64.9402
65.6:0.2
4
$7.8+0.1
65.749.2
65.4-0.4
64.2.u.s
63.4
64.444
65.5+.u.
4
31.440.4
46.6
65.0e0.
65.5-o.1
65. 65.1
63. 63.4
65,440.
66.250.3
Average
42.
53.526.
65 65.b2o.
05.3gn2
84.620.4
63 13.0
61.9g4
65.8
67.1
10.
37.8
55.7
65.9
2..2
26.1
56.7
50.7
#2
33.1
38.410.4
62.620.3
62.4
61.820.
64.2.0.
64.9104
65.6.0.
67.520.1
Random Tasks
25.7
43.220.a
54.8u0.1
55.4
53.620.
55, 65.6
56.32n.
$7.3
60.4
#4
34. 34.2
48.9
64.7
65.3
62.5g.
65 65.4go.
06.2g09
67.4gn
09.8
Average
31.7+4.0
42.0
59.5e4.9
5..8-4.
57.8ea.1
60.3:a.2
6L.0asa
62.0
64.4
Table 1: The average performance score for each task sequence after learning all tasks. Bold indicates the best score. 'MTL' stands for 'multi-task learning', serving the upper bound for LSG. In each scenario, DMEA is significantly better than ACM with p-value < 0.05 (paired I-test). Note that while LAMOL and Metac are not directly comparable to other adapter-based methods as their learnable parameters are orders of magnitude larger, DMEA still outperforms them in most cases. The comparison of learnable parameters and computational resources between ACM and DMEA is reported in Appendix A.8.
where is the number of completed training epochs. After dynamic gradient scaling, the total loss for jointly learning T and T is:
quences for LSG on random tasks. For each sce- nario, we randomly select four different orders? (Appendix A.4) and run experiments for every or- der five times with different random seeds (20 runs per scenario). For each order, we report the average of all learned tasks' performance scores following Zhang et al. (2022); see Appendix A.5 for details of task task-specifie evaluation metries.
Liotal = C Cirain + -ie Cirain
Note that in the early stage of training, the value of l is small. ?h is greater than 1 to balance the gradient of the new task T' and the replayed task Tí. When the model has seen enough new data in the late stage of training (no need to balance), Th is approximately equal to 1 as the value of t is large.
5.2 Methods Compared
Following Zhang al. (2022), use GPT-2 (Radford et al., 2019) as the backbone model and Adapter (Houlsby al., 2019) as the insertable module, and compare with the following methods:
5
Experimental Setup
In this section, we first describe investigated tasks and then introduce methods compared in our work.
Finetune tunes the whole GPT-2 model only on the training data of the new task during the LSG process.
5.1 Tasks
Four representative sequence generation tasks are investigated in our work: natural language genera- tion, summarization, task-oriented dialogue and
EWC (Kirkpatrick et al., 2017) constrains the update of parameters that are important to previ- ously learned tasks to alleviate forgetting.
SQL query generation. Following Zhang et al.
LLMOOL (Sun al., 2020) tunes the whole GPT- 2 model with aseudo-sample replay.
(2022), we consider two different scenarios: (i) LSG on similar tasks where the model learns a sequence of tasks of the same type but different domains, and (ii) LSG on random tasks where the model learns knowledge from different types of tasks. For LSG on similar tasks, we use five different domains from two natural language generation datasets (RNNLG (Wen et al., 2015) and E2ENLG (Novikova et al., 2017)) form the task sequences. We further incor- porate summarization (CNNDM (See et al., 2017)). task-oriented dialogue (MultiWOZ (Budzianowski et al., 2018)) and SQL query generation (Wik- iSQL (Zhong et al., 2017)) to form the task se-
Metac-Adapt (Metac) (Wang et al., 2023) adapts LAMOL towards better semantic space for generating pseudo samples.
AdA only inserts adapter modules for the first task and tunes these modules with pseudo-sample replay while keeping the back- bone model frozen.
AdapterCI. (Madotto et al., 2021) inserts task-
>Zhang et al. (2022) sample data from the original set for data balance. To ensure a fair comparison among all methods, we resample new data for experiments.

---

Method
Similar Tasks Random Tasks
Time Step
AdapterCI.
ACM
DMEA
DMEA
57.3
55.8(+0.0)
56.0(+0.1)
56.3++0.3)
transler 64.9
56.5
58. 6).66)0.00
59.1(-044
59.5(+0.6)
Similar
M.O. scaling
65.5
56.8
4
61.2(+0.0)
62.5(-0.6)
63.2(+0.9)
W.O. initialization
65.4
57.0
63.4(+0.0) 5
64.8(+0.3)
65.8(+1.0)
2
55.4(+0.0)
56.3(-0.9)
57.1(+2.1)
Table 2: The average performance score for different ablations: (i) without forward knowledge transfer, (ii) without dynamic gradient scaling, and (iii) without dy- namic initialization. All components improve the per- formance of method.
58. (8.40(0.0)
58 (8.9(+0.3)
59.7(+1.3)
Random
64.0(+0.0)
64.3(+0.7)
65.4(+1.4)
64.2(+0.0)
64.9(+0.6)
65.6(+1.1)
Table 3: The average performance score and forward knowledge transfer (FKT) of different methods at every time step. FKT is reported parentheses.
specific adapter modules for every new task while keeping the backbone model and previous mod- ules frozen.
its adaptive compositional architecture. However, ACM has no explicit mechanism to encourage for- ward knowledge transfer in I.SG, which is actu- ally the human learning paradigm. Our proposed DMEA consistently outperforms ACM by dynami- cally leveraging previously acquired knowledge facilitate adaptation to new tasks.
ACM (Zhang al., 2022) dynamically adds adapter modules for new tasks depending on whether there are reusable previous modules to improve the performance and parameter ef- ficiency of AdapterCL. It is the state-of-the-art on I.SG.
6.2 Ablation Study
6 Results and Analysis
We conduct several ablations to analyze the con- tribution of different components DMEA. particular, we investigate three variants of DMEA (a) without selecting similar previous tasks for for- ward knowledge transfer (w.o. transfer), (b) remov- ing dynamic gradient scaling (w.o. scaling), and (c) without dynamically initializing learnable coef- ficients (w.o. initialization). For each scenario, i.e., similar tasks or random tasks, we randomly pick one sequence for experiments. Table 2 reports the average performance score after learning all tasks for different ablations.
6.1 Main Results
Table shows the average performance score for each task sequence after learning all tasks (see Ap- pendix A.7 for the performance of each task). From the results, we can see that DMEA outperforms pre- vious baselines in all LSS settings, which demon- strates the superiority of our method. Note that while the learnable parameters of LAMOL are or- ders of magnitude larger, DMEA still achieves better performance than LAMOL in 7 out of runs, show- ing its effectiveness in LSG.
From the results, we can observe that all compo- nents contribute to the average performance. Re- moving forward knowledge transfer leads to a sig- nificant performance drop in both scenarios, in- dicating that selecting top-K most similar previ- ous tasks can indeed discover and transfer useful learned knowledge to facilitate adaptation to the new task. The adoption of dynamic gradient scal- ing yields a moderate performance boost as it can balance the learning of the new task and replayed tasks mitigate catastrophic forgetting. Dynamic initialization of Iearnable coefficients also facili- tates performance improvement, demonstrating the effectiveness of leveraging the similarity of word frequency distributions between tasks.
Simply fine-tuning the model with new samples leads to poor performance due to catastrophic for- getting. Although EWC adopts Fisher information matrix to alleviate forgetting, its performance is still much worse than other memory based base- lines, indicating the importance of pseudo-sample replay. When learning from a sequence of simi- lar tasks, AdaO performs better than AdapterCI. as AdapterCI. applies parameter iso- lation to different tasks which might prevent pos- itive knowledge transfer across tasks. However, this is not the case when learning from random tasks: AdapterCL achieves much better results than Adapter+LAMOL as AdapterCL can avoid catas- trophic forgetting by assigning different learnable parameters each task. The performance of ACM is superior to Adapter=LAMOL and AdapterCL in both scenarios, showing the effectiveness of
6.3 Further Analysis
Quantify Forward Knowledge Transfer. Fol- lowing Ke et al. (2020), we define metrics quanti-

---

Metries
Similar Tasks
Random Tasks
Module Type
ACM
Input Subspace 65.3
57.3
Prefix-tuning
62.6
63.4
Frequency
56.9
LoRA
63.1
64.2
Representation
65.2
56.9
w.o, transfer
64,9
56.5
Table 5: The average performance score of ACM and DMEA with different module types.
Table 4: The average performance score using different similarity metries.
the insertable module, showing its robustness module type.
fying forward knowledge transfer (FKT) at every time step as:
Longer Sequence. As mentioned in $5.1, we mainly conduct experiments on sequences consist- ing of tasks following Zhang et al. (2022). To verify whether DMEA can still outperform the base- lines when learning from larger number of tasks, we further combine all tasks investigated in this work to form longer sequence of tasks. We eval- uate ACM and DMEA this longer sequence with different orders and report the average perfor- mance score for each order after learning all tasks in Fig. We can observe that DMEA is still supe- rior to ACM when learning from longer sequences.
FWT- 1 Ras dr.
(10)
where Ri, is the performance score on T after learning T and d, refers to the performance of training T individually, which is actually the re- sult of AdapterCI.. For each scenario, we randomly select one sequence for analysis and report the av- erage performance score along with FKT at each step in Table From the results, we can see that DMEA consistently outperforms ACM in terms of the average performance score and FKT at all steps, demonstrating that DMEA can better facilitate posi- tive knowledge transfer.
Quality of Pseudo Data Fig. 4 shows several pseudo samples generated by DMEA. We can see that DMRA can indeed generate high-quality pseudo samples to mitigate the forgetting of previously learned knowledge. However, the generated pseudo data could also be noisy as shown at the bottom of the figure, which might hinder further performance improvement.
Input Subspace vs. Other Similarity Metrics. The ablation (w.o transfer) $6.2 demonstrates the importance of seleeting similar Icarned tasks. To further investigate whether different similarity metries influence the performance of DMEA, we conduct controlled experiments with two new met- rics: (a) cosine similarity of word frequency dis- tributions between different tasks (frequency), and (b) cosine similarity of the representations of se- lected samples from different tasks² (representa- tion). For each scenario, we use the same sequence as $6.2. From the results in Table 4, we can ob- serve that selecting similar previous tasks by input subspace consistently outperforms using other sim- ilarity metrics, demonstrating its superiority.
Other Types of Tasks To explore whether the performance gain of DMEA is consistent on other types of tasks, we further include three new tasks: sentiment analysis (SST (Socher et al., 2013)), semantic role labeling (SRL (He et al., 2015)) and question answering (SQuAD (Rajpurkar al., 2016)). We randomly select tasks from the original task set three times and combine them with new tasks to form three task sequences. From the results shown in Table 6, we can observe that DMEA performs better than ACM on all sequences, showing its robustness to task types.
To verify whether the performance gain of DMEA is consistent across different types of modules, we extend the experi- ments prefix-tuning (Li and Liang, 2021) and LoRA (Hu et al., 2022). We randomly pick four sequences for experiments and report the average result in Table 5, we can see that DMEA still out- performs ACM when using other architecture as
Robustness to Module Type
Different Pseudo-dats Sampling Ratios Fol- lowing Zhang et al. (2022), we set the pseudo-data sampling ratio to 0.2. To validate whether differ- ent pseudo-data sampling rates influence the per- formance gain of DMEA, we conduct controlled experiments with sampling rates (0.05, 0.1, 0.4). We randomly pick three sequences for experiments and report the performance comparison between
FFor a pair of tasks, we compute the cosine similarity for every representation pair and use the average as the similarity.

---

63.0
OVBA
ACV
Sequence
62.5
Method
Average
62.0
(i)
(ii)
(iii)
$1.5
ACM
68.4
63.6
71.8
67.9
21.0
DMEA
69.5
64.4
73.0
69.0
Table 6: The average performance score for every se- quence after learning all new types of tasks.
Figure 3: The average performance score for every order after learning tasks of the longer sequence.
Sampling Ratio 0.05
0.1
0.4
ACM
61.7
62.0
High-quality Data
DMEA
62.5
63.1
The Cricketersl, eafType]restauranl, French). prieeRange/mdderatt [moderate]. Rear|Rainhow Vegetarian Cafe] what the natural language form? The Cricketers is French restaurant next to the Rainbow Vegetarian Cafe with moderate prices Freneh taste.
Table 7: The average performance score of ACM and DMEA with different pseudo-data sampling ratios.
Noixy Data
the table has columns week number, date, opponent, result, record and words min, select, where, and, col, table, caption -- who the opponent was weekend where the record was what the translation from english to sql? select opponent from table where record
Limitations
avg, op.
Although effective, DMEA has couple of limita- tions:
DMEA mainly focuses on the setting where every task has plenty of training samples. contrast, humans can easily learn to perform new tasks with only few data, which is a hallmark of human intelligence. We leave how to explore lifelong sequence generation in few-shot settings as future work.
Figure 4: Some examples of generated pseudo data. We color the task instruction blue and output text in gray. Missing/wrong information is colored in red.
ACM and DMEA in Table 7. consistently outperforms ACM in all cases, demon- strating its effectiveness.
We can see that DMEA
DMEA does not consider machine translation, a sequence generation task that might involve vo- cabulary changes. One potential solution is multilingual pre-trained language models.
In addition, we show case studies of learned model architecture, model output, dynamic gradi- ent scaling and task selection, generalization of dynamic initialization, and potential real-world ap- plications in Appendix A.9 ~ A.14, respectively.
References
7 Conclusion
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, lñigo Casanueva, Stefan Ultes, Osman Ra- madan, and Milica Gašié. 2018. MultiWOZ a large- scale multi-domain Wizard-of-Or dataset for task- oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
In this work, we have introduced DMEA for lifelong sequence generation (LSG).. DMEA leverages task correlations to dynamically determine the suitable architecture required acquire novel knowledge new task and selects the most similar previous tasks through input subspace to facilitate knowl- edge transfer. It uses pseudo-sample replay along with dynamic gradient scaling to balance the learn- ing of the new task and replayed tasks to further alleviate forgetting. With extensive experiments and analysis have shown that DMEA consis- tently outperforms previous methods in different 1.SG settings. In the future, we would like to inves- tigate ways to improve the quality of pscudo data and explore more metrics for task similarity.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2019. Efficient lifelong learning with A-GEM. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. 201 Net2net: Accelerating learning via knowledge transfer. In 4th International Conference ing Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Learn-

---

Yung-Sung Chuang, Shang-Yo Yu Su, and Yun-Nung Chen. 2020. Lifelong language knowledge distillation. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing (EMNLP), pages 2914-2924, Online. Association for Computa- tional Linguisties.
Zixuan Ke, Bing Liu, Nianzu Ma, Hu Xu, and Lei Shu. 2021. Achieving forgetting prevention and knowl- edge transfer in continual learning. Advances in Neural Information Processing Systems, 34:22443- 22456.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz. Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Brabska-Barwinska, et al. 2017. Over- coming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences. 114(333332
Wafaa S El-Kassa,, Cherif R Salama, Ahmed A Rafea, and Hoda K Mohamed. 2021. Automatic text sum- marization: A comprehensive survey. Expert Systems with Applications, 165:1 13679.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei Rusu, Alexander Pritzel, and Daan Wierstra. 2017. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint urXiv:1701.086 1701.0833.
Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenen- baum, and Samuel J. Gershman. 2017. Building ma- chines that learn and think like people. Behavioral and Brain Sciences, 40:e253.
Donghoon Ham, Jeong-Gwai Lee, Youngsoo Jang, and Kee-Eung Kim. 2020. End-to-end neural pipeline for goal-oriented dialogue systems using GPT-2. In Proceedings of the 58th Annual Meeting of the Associ- ation for Computational Linguistics, pages 583-592, Online. Association for Computational Linguistics.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pages 4582- 4597, Online. Association for Computational Lin- guistics.
Xu Han, Yi Dai, Tianyu Gao, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2020. Contin- ual relation learning via episodic memory activation and reconsolidation. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 6429-6440, Online. Association for Computational Linguistics.
Zhizhong Li and Derek Hoiem. 2017. Learning without forgetting. IEFE transactions on pattern analysis and machine intelligence. 40(12):2935 2947.
Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. 2022a. Beyond not-forgetting: Continual learning with backward knowledgs transfer. In Advances in Neural Information Processing Systems.
Luheng Hc, Mike Lewis, and Luke Zettlemoyer. 2015, Question-answer driven semantic role labeling: Us- ing natural language to annotate natural language. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 643-653, Lisbon, Portugal. Association for Compu- tational Linguistics.
Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. 2022b. TRGP: Trust region gradient projection for continual learning. In International Conference on Learning Representations.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015, Distilling the knowledge in a neural network. arXiv preprint arXiv: 150300231, 2(7).
Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2019. DARTS: Differentiable architecture search. In Inter- national Conference on Learning Representations.
David Lopez- Paz and Mare' Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. Ad- vances in neural information processing systems, 30.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In In- ternational Conference on Machine Learning, pages 2790-270 2799. PMLR.
Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Se- ungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Eu- njoon Cho, Pascale Fung, and Zhiguang Wang. 2021, Continual learning in task-oriented dialogue systems. In Proceedings of the 2021 Conference Empiri- cal Methods in Natural Language Processing, pages 7452-7467, Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Saeed Masoudnia and Reza Ebrahimpour. 2014. Mix- ture of experts: a literature survey. Artificial Intelli- gence Review, 42(2):275-293
Zixuan Ke, Bing Liu, and Xingchang Huang. 2020. Continual learning of a mixed sequence of similar and dissimilar tasks. In Advances in Neural Informa- tion Processing Systems, volume 33, pages 18493- 18504, Curran Associates, Inc.
Michael McCloskey and Neal Cohen. 1989. Catas- trophic interference in connectionist networks: The

---

sequential learning problem. In Psychology of learn- ing and motivation, volume 24, pages 109-165. Else- vier.
Hippolyt Ritter, Aleksandar Botev, and David Barber, 2018. Online structured laplace approximations for overcoming catastrophic forgetting. In Advances in Neural Information Processing Systems 31: An- nual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018 Montréal, Canada, pages 3742-3752.
Fei Mi, Liangwei Chen, Mengjie Zhao, Minlie Huang, and Boi Faltings. 2020. Continual learning for natu- ral language generation in task-oriented dialog sys- tems. In Findings of the Association for Computa- tional Linguisties: EMNLP 2020, pages 3461-3474, Online. Association for Computational Linguistics.
Sebastian Ruder, Matthew
Peters, Swabha Swayamdipta, and Thomas Wolf. 2019. Transfer learning in natural language processing. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Tutorials, pages 15-18, Minneapo- lis, Minnesota. Association for Computational Linguistics.
Jekaterina Novikova, Ondfej Dušek, and Verena Rieser. 2017. The E2E dataset: New challenges for end- to-end generation. In Proceedings of the 18th An- nual SIGdial Meeting on Discourse and Dialogue, pages 201-206, Saarbrücken, Germany. Association for Computational Linguistics.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Des- jardins. Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. 2016. Progressive neural networks. arXiv preprint NNNiv:660.04677
Jonas Pfeiffer, Andreas Rücklé Clifton Poth, Aishwarya Kamath, Ivan Vulié. Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. AdapterHub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing: System Demonstrations, pages 46-54, Online. Association for Computational Linguistics.
Abigail See, Peter Liu, and Christopher D Man- ning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04360 04336...
Chengwei Qin and Shafiq Joty. 2022a. Continual few- shot relation learning via embedding space regular- ization and data augmentation. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 2776-2789, Dublin, Ireland. Association for Compu- tational Linguistics.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. 2017. Continual learning with deep generative replay. In Advances in Neural Information Process- ing Systems 30: Annual Conference on Neural In- formation Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 2990-2999.
Chengwei Qin and Shafiq Joty. 2022b. LFPTS: A uni- fied framework for lifelong few-shot language learn- ing based on prompt tuning of In International Conference on Learning Representations.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013, Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference Empiri- cal Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188):9
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):1-67
Yan-Keng Sun, Cheng- Hao Ho, and Hung Yi Lee. 2020. Lamol: Language modeling for lifelong language learning. In International Conference on Learning Representations.
Pranav Rajpurkar. Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383 23922 Austin, Texas. Association for Computational Linguisties.
Han Wang, Ruiliu Fu, Xuejun Zhang, Jun Zhou, and Qingwei Zhao. 2023. Metacognitive adaptation to enhance lifelong language learning. IECCE TRANS- ACTIONS on Information and Systems, 106(1))86 90.
Tsung- Hsien Wen, Milica Gašić, Nikola Mrkšić, Pei- Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned LSTM-based natural lan- guage generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1711-1721, Lisbon, Portugal. Association for Com- putational Linguistics.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. 2017. icarl: In- cremental classifier and representation learning. In 2017 IEEE Conference on Computer Vision and Pat- tern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 5533-55445 5542. IEEE Computer Society.

---

Dun
Raeriig
Beating
Siring
Leurner
Reld
Human Learning Paradigm
Boow
Doew
Learned skills
New skill
Learned skills
Figure 6: The effect of large gradient norm. 9old- Inews and 9avm represent the gradient of replayed tasks, the gradient of the new task, and the aggregated gradient, respectively. The blue arrows show the projection of Jaum onto Goid- If the gradient norm of Inew is large (right part of the figure), this projection might deviate too much from Gold-
M
svistar
New skil
Figure Given three learned skills, Le., swimming, running and boating, humans can determine that these skills are not sufficient for diving. And after realizing that swimming is the most similar learned skill, they only need to learn the new aspect, i.e. how to safely jump off diving platform to master the new diving skill.
nume/The Vaults), eatType|publ, priceRange|modesee customer rating[ out SJ, near[Cuf Adriatic] what the natural language form? moderately priced pub, named The Vaults, is located near Caf Adriatic. It has customer rating of out of
Figure An example of task instruction for E2ENLG. We color the task instruction in blue.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistae, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- natural language processing. In Proceedings the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
A.2 Effect of Large Gradient Norm
As shown in the Fig. 6, if the gradient norm of the task Inew is large, the projection of the aggregated gradient Jeum onto the gradient of re- played tasks Gold might deviate too much from Golds leading to more severe forgetting.
formers: State-of-the
A.3 Task Instruction Example
Following Zhang et al. (2022), we insert a natural language question describing the purpose of every task (task instruction) after the input of each sample. Fig. shows example of the task instruction for E2ENLG (Novikova et al., 2017).
Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual learning through synaptic intelli- gence. Proceedings of the 34th International Con- ference Machine Learning, ICMI. 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 3987-3995. PMLR.
A.4 Task Orders
Yanzhe Zhang, Xuezhi Wang, and Diyi Yang. 2022. Continual sequence generation with adaptive compo- sitional modules. Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 3653-3667, Dublin, Ireland. Association for Computational Lin- guistics.
We present different task orders for two LSG sce- narios in Table 8.
A.5 Task-specific Evaluation Metries
We report details of task-specific evaluation metrics in Table 9.
A.6 Implementation Details
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning, arXiv preprint arXiv:1 Xiv:1709.00103
All methods are implemented with Py- Torch/Transformers library (Wolf et al., 2020). We adopt AdapterHub (Pfeiffer et al., 2020) to implement adapter modules. For hyperparameters, mainly follow the settings in Zhang al. (2022) have a fair comparison. In the expansion stage, we train the model for 6 epochs before selecting modules. In the adaptation stage, we set the number (n) of samples selected to obtain the
A Appendix
A.1 Illustration of Human Learning
We show the illustration of human learning in Fig. 5.

---

Order
Task Sequence
Similar #1
e2e
res
hotel
laptop
Avg
<2e hotel laptop
ACM
49.6
65.7
65.8
71.6
71.1
64.8
Similar Tasks
e2e tv laptop hotel
DMEA
49.2
67.1
68.1
72.5
72.0
65.8
3
hotel e2e laptop
Similar #2
ele
res
laptop
hotel
Avg
laptop hotel res c2c
ACM
48.7
74.0
64.4
72.6
65.0
64.9
mwoz cnn c2e res hotel #
DMEA
47.9
74.9
64.9
74.2
65.9
65.6
e2c sql hotel mwoz # 2
Random Tasks
cnn hotel sql c2e mwoz # c2c mwoz laptop sql tv
3
Similar #3
res
hotel
e2e
laptop
tv
Avg
ACM
65.6
67.3
48.5
72.0
69.3
64.5
DMEA
66.9
66.6
49.5
73.7
70.9
65.5
Table 8: Different task orders for each scenario. 'e2e' stands for E2ENLG. 'res", 'hotel', 'laptop" and 'tv' are four domains in RNNLG (restaurant, hotel, laptop and television). "sql', 'enn' and 'mwoz' respectively stand for "wikiSQL" (SQL query generation), 'CN- NDM' (summarization) and 'MultiWO:" (task-oriented dialogue).
Similar #4
laptop
hotel
res
c2c
Avg
ACM
73.1
66.8
66.9
72.4
47.6
65.4
DMEA
74.6
67.6
67.4
48.3
66.2
Random #1 mwoz
cnn
e2e
res
hotel
Avg
ACM
81.6
26.0
47.5
64.1
56.7
DMEA
81.6
48.1
65.4
57.5
Random #2
c2e
sql
hotel
mwwz
res
Avg
Dataset
Metric
ACM
48.4
62.7
64.6
64.0
64.9
DMEA
48.7
64.9
64.6
65.6
RNNLG
Random #3
can
hotel
sql
e2e
mwoz
Avg
E2ENLG
A-RG
ACM
26.3
63.2
62.1
47.7
82.4
56.3
CNNDM
DMEA
26.6
65.0
63.1
48.4
83.5
57.3
WikiSQL
IfEM
Random #4
c2c
mwoz
laptop
sql
tv
Avg
MultiWOZ
dsEM
ACM
48.6
80.5
70.3
68.2
66.2
DMEA
49.0
71.7
64.4
68.9
67.4
Table Details of task specifcc evaluation metrics. 'A- RG', 'ITEM" and 'dsEM' respectively stand for 'average ROUGE-1, ROUGE-2 and ROUGE-L scores', 'exact match of logical forms' and 'exact match of dialogue state".
Table 10: The performance of each task for every se- quence after learning tasks.
Method Avg Para Num Avg Time (min)
ACM
4.6M
218.1
DMEA
4.8M
223.5
input subspace as 100. The threshold e set as 0.95 for selecting left-singular vectors. We adopt 1 for the number of similar tasks K. For dynamic gradient scaling, we set 100 for the number (g) of examples selected to calculate the gradient.
Table 11: The comparison of the average number of learnable parameters (Avg Para Num) and average ning time (Avg Time) between ACM and DMEA.
optimal model architecture, we analyze the model expansion stage of ACM and DMEA using sequence #4 in random scenario. For the final task nv, ACM decides to reuse modules from the first (e2e) and the third task (laptop) while DMEA reuses all mod- ules from laptop which is consistent with the ob- servation that the similarity between rv and laptop is much higher than that between and e2e.
A.7 Performance of Each Task
Table 10 shows the performance of each task for every task sequence after learning all tasks.
A.8
Number of Learnable Parameters and Computational Resources
We present the average number of learnable param- eters and average running time for ACM and DMFA in Table 1. From the comparison, we can observe that DMEA can outperform ACM with a negligible increase learnable parameters and computational resources.
A.10 Case Study of Model Output
We select RNNLGG.hotel (sequence #1 in similar scenario) and WikiSQL (sequence #4 in random scenario)
representative tasks and show sev- eral examples of output in Table 12. Compared with ACM, DMFA possesses the capability to con- vey more precise and relevant information from the input without introducing superfluous details.
A.9 Learned Model Architecture
To further demonstrate that dynamically initializ- ing learnable coefficients can facilitate finding the

---

RNNLG.hotel: inform(name-='nnn mannarmnmmmmmaiii oriental san francisco 'dogsallowed= yes' :pricerange= ppriipereeeesspipprrreeeeeeee ultra high end")
Reference the mandarin oriental san francisco in the ultra high end price range and allows dogs.
ACM the mandarin oriental san francisco is a hotel in the ultra high end range (missing: and allows dogs).
DMEA the mandarin oriental san francisco offers ultra high end accommodations and allows dogs.
WikiSQL:
on which date was the winning driver alain prost and had damon hill in the pole position ?
Reference select date from table where winning driver alain prost and pole position damon hill
ACM select date from table where winning driver = alain prost and pole position = damon (misssigg: hill)
select date from table where pole position damon kill and winning driver alain prost
Table 12: Output examples of different methods after learning the whole sequence. We color missing/wrong
information in red and redundant information in blue.
Method
Similar Tasks
Random Tasks
A.14 Real World Application
ACM
64.5
66.2
Apart from the aforementioned sequence gener- ation tasks, DMEA demonstrates the potential to be applied to various real-world lifelong learning scenarios. For example, it can continually train model to perform summarization and question- answering based on news articles from different domains during the onsct of an emerging event like Covid-19.
ACM w DI
64.7
66.5
Table The performance comparison between ACM and ACM with dynamic initialization (ACM DI).
A.11 Case Study of Dynamic Gradient Scaling
The ablation study in $6.2 demonstrates the im- portance of dynamic gradient scaling. We further conduct a case study using sequence #1 in random scenario. During the learning of this sequence, the fourth task res reuses several modules from the third task e2e. After applying dynamic gradient scaling, the performance of e2e is improved by 0.3 without compromising res, indicating that it does mitigate the bias towards the new task.
A.15 Hyperparameter Search
We select the number of training epochs before modules selection from (6. 9, 12), the number (n) of samples picked to obtain the input subspace from {50, 100, 200, 500} and the threshold for select- ing left-singular vectors from {0.90, 0.95, 0.99). The number of similar previous tasks K selected from {1, 2. The number (g) of examples for cal- culating the gradient in dynamic gradient scaling is selected from f20, 50, 100, 2003.
A.12 Case Study of Task Selection
To verify that the previous task chosen the se- lection stage is indeed the most similar to the new task, we analyze several cases using sequence #2 in random scenario. For the third task hotel, the selected first task e2e has the highest similarity score they share the same task type. In addition, the third task hotel shares a similar semantic space with the final task res. Therefore, it is selected for forward knowledge transfer when learning res.
A.13 Generalization of Dynamic Initialization
To demonstrate the generalization ability of dy- namic initialization, we apply it the expansion stage of ACM. For each scenario, pick one sequence for experiments. As reported in Table 13, dynamic initialization does benefit ACM, verifying its generalization capability.
randomly


Reviews:
[{"review_id": "C0RvZxH8NS", "reviewer": "Reviewer_5YgP", "summary": "", "questions": "1. The proposed method doesn't need to fine-tune the base model. Although the experiments are done on GPT-2, I'm wondering whether it's possible to apply DMEA to LLM, e.g., GPT-3.\n\n**Update**: After the rebuttal period, I raised my excitement score to 4.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer:\n\nThank you for the positive assessment and insightful comments.\n\n**Q1. The paper is a good introduction of new method but further insights on LSG or continual learning with PLMs are lacking. The proposed method is composed of several tricks, i.e., forward knowledge transfer module, dynamic gradient scaling, dynamic initialization, and the ablation study shows that each of them can contribute a small improvement to the system. However, according to Table 1, there still exists a gap between DMEA and MTL. It's unclear whether DMEA can be further extended to get close to MTL performance since DMEA is a nicely crafted system instead of a principle idea.**\n\nApart from the main results in Section 6.1, we do provide many insights on LSG in Section 6.3 and the Appendix. For example, we analyze the influence of different similarity metrics for selecting similar learned tasks on LSG and the robustness of DMEA to module types (can also indicate which module is more suitable for LSG) in Section 6.3. Besides, we show how different pseudo-data sampling rates affect LSG performance in Appendix A.10.\n\nSince MTL serves as the upper bound for LSG, it’s normal for DMEA to perform worse than MTL. Actually, DMEA is easy to extend. For example, it can be combined with a more advanced pseudo-sample generation method (AQF-RQ) [3]. We randomly select four task sequences for pilot experiments. From the average results shown below, we can observe that this simple extension can indeed improve the performance. We will conduct more extensive experiments in the final version. \n\n| DMEA  | DMEA + AQF-RQ |\n|  ----  | ----  | \n| 62.9 | 63.3  |\n\nIn addition, although DMEA is a nicely crafted system, its component (dynamic initialization) can be easily applied to the baseline ACM to improve the performance as shown in Appendix A.16.\n\n**Q2. The related literature is not treated seriously. Although the \"Related Work\" section surveys Lifelong Learning and Lifelong Sequence Generation, I think more recent works which study lifelong learning / continual learning with PLM should be considered. For example, [1] studies the influence of generation objective on PLM in the continual learning process; [2] studies continual instruction tuning which also involves generation. Specifically, since the input data used in this work also includes the task description, it's similar to the continual instruction tuning setup and it's recommended to compare DMEA with method in [2] if possible.**\n\nThanks for your suggestions! \n\nAs [1] was submitted to Arxiv on 22 June 2023, we did not know about it during the EMNLP submission. We will cite it in the final version.\n\nFor ConTinTin [2], we will cite it as related work in the final version. However, it’s difficult for us to compare DMEA with ConTinTin as (1) there is no open-source code for ConTinTin, and (2) ConTinTin requires instructions to contain specific content, e.g., “Things to avoid” and “negative examples”, which is not satisfied in our case.\n\n**Q3. The proposed method doesn't need to fine-tune the base model. Although the experiments are done on GPT-2, I'm wondering whether it's possible to apply DMEA to LLM, e.g., GPT-3.**\n\nWe cannot apply DMEA to GPT-3 as it’s not possible (at the moment) to add learnable modules to GPT-3. However, we can apply DMEA to open-source LLMs. We choose GPT-J-6B as an example because its corresponding code is relatively easy to implement. We randomly select four task sequences for pilot experiments. The comparison of average results between DMEA and ACM is shown below. We can see that DMEA still outperforms ACM when using a larger language model as the backbone. We will conduct more extensive experiments in the final version. \n\n| DMEA  | ACM |\n|  ----  | ----  | \n| 66.3 | 64.9  |\n\nThanks again for your feedback. We hope that this addresses your questions.\n\n\n[1] Class-Incremental Learning based on Label Generation, Shao et al., ACL 2023.\n\n[2] ConTinTin: Continual Learning from Task Instructions, Yin et al., ACL 2022.\n\n[3] Ask Question First for Enhancing Lifelong Language Learning. Wang H, Fu R, Zhang X, et al. COLING 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method is well motivated, i.e., having a link with human learning paradigm. DMEA divides the learning process of a new task into expansion, selection and adaptation which not only eases the forgetting problem but also enables knowledge transfer.\n2. DMEA gives new SoTA results by exceeding the performance of ACM. The experiment and analysis in the work are extensive.", "reasons_to_reject": "1. The paper is a good introduction of new method but further insights on LSG or continual learning with PLMs are lacking. The proposed method is composed of several tricks, i.e., forward knowledge transfer module, dynamic gradient scaling, dynamic initialization, and the ablation study shows that each of them can contribute a small improvement to the system. However, according to Table 1, there still exists a gap between DMEA and MTL. It's unclear whether DMEA can be further extended to get close to MTL performance since DMEA is a nicely crafted system instead of a principle idea.\n2. The related literature is not treated seriously. Although the \"Related Work\" section surveys Lifelong Learning and Lifelong Sequence Generation, I think more recent works which study lifelong learning / continual learning with PLM should be considered. For example, [1] studies the influence of generation objective on PLM in the continual learning process; [2] studies continual instruction tuning which also involves generation. Specifically, since the input data used in this work also includes the task description, it's similar to the continual instruction tuning setup and it's recommended to compare DMEA with method in [2] if possible.\n\n[1] Class-Incremental Learning based on Label Generation, Shao et al., ACL 2023.\n\n[2] ConTinTin: Continual Learning from Task Instructions, Yin et al., ACL 2022.", "questions_to_authors": "1. The proposed method doesn't need to fine-tune the base model. Although the experiments are done on GPT-2, I'm wondering whether it's possible to apply DMEA to LLM, e.g., GPT-3.\n\n**Update**: After the rebuttal period, I raised my excitement score to 4.", "paper_topic_and_main_contributions": "This paper focuses on lifelong sequence generation (LSG) which aims to continually train a model on a sequence of generation tasks to learn constantly emerging new generation patterns. The authors extend the previous SoTA method ACM to propose Dynamic Module Expansion and Adaptation (DMEA) for LSG. DMEA follows the learning paradigm of humans by dividing the learning process of a new task into expansion, selection and adaptation. Such a design eases forgetting while enables knowledge transfer. Experimental results demonstrate the effectiveness of DMEA.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Please refer to my second point in \"Reason To Reject\".", "Typos_Grammar_Style_and_Presentation_Improvements": "None."}, "extra_text": {}}, {"review_id": "wOBgMw35Fw", "reviewer": "Reviewer_9rh6", "summary": "", "questions": "See above", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer:\n\nWe thank you for your thoughtful comments and feedback. We address your questions here.\n\n**Q1. When talking about task similarity in continual learning, a popular paper is CAT [1], which considers not only similar tasks and dissimilar tasks, but also the mixture of them. Have you ever tried this challenging scenario and any interesting observations?**\n\nWe do consider this challenging scenario:\n\n(1) As shown in Table 6, the ‘random’ scenario already includes both similar tasks and dissimilar tasks.\n\n(2) In Section 6.3, Longer Sequence (line 588-599), we further fuse all 8 tasks investigated in this work together for experiments. DMEA still outperforms ACM when learning from longer sequences.\n\n\n**Q2. I think there are more systems to be compared. While you mentioned some systems are focused on classification, I do not see why they cannot be a baseline in generation setting by simply changing the underlying LM and classification head (in particular for those baselines based on adapter). I think the author should at least compare those NLP methods that are explicitly dealing with transfer (as cited by the author in the related work, like [2] or even more in the survey [4]).**\n\nThanks for your suggestions! \n\nDue to the time limit, we do not manage to change the code of [2] to GPT-2. We will try to include it in the final version.\n\nTo better demonstrate the superiority of DMEA over other methods, we further compare DMEA with two methods: (1) an adapter-based method AdapterFusion [5], and (2) TRGP [6] which explicitly encourages forward knowledge transfer (originally designed for classification). We randomly select four task sequences for pilot experiments. The average performance score comparison between different methods is shown below. We can observe that DMEA achieves better performance than AdapterFusion and TRGP, demonstrating its effectiveness. We will conduct more extensive experiments in the final version. \n\n| DMEA  | AdapterFusion | TRGP |\n|  ----  | ----  | ----  | \n| 62.9 | 61.1  | 59.6 |\n\n**Q3. Similarity detection has been used in different systems [1,3], what are the main advantages for DMEA compared to them? Or is it just another heuristic method?**\n\nCompared to [1], the main advantage of DMEA is that [1] requires training another transfer model and reference model while DMEA only needs to calculate the subspace using the original model (more computation-efficient).\n\nCompared to [3], the main advantages of DMEA are (1) we directly use subspace projection instead of its proxy (gradient projection), which could better reflect the correlation between tasks. (2) [3] calculated gradient projection before learning the new task while DMEA trains the model on the new task for several epochs in the first expansion stage, which provides more task-specific knowledge for subsequent subspace calculation (Section 4.2).\n\nTo support our claim, we replace the similarity detection method of DMEA with the method in [3] as it is relatively easy to implement (denoted as DMEA-G). We conduct pilot experiments using the same four task sequences as in **Q2**.  The average results are reported below, showing that our designed similarity detection method is superior to that in [3]. We will conduct more extensive experiments in the final version.\n\n| DMEA  | DMEA-G |\n|  ----  | ----  | \n| 62.9 | 62.5  |\n\nThanks again for your feedback. We hope that this addresses your questions.\n\n[1]: Continual Learning of A Mixed Sequence of Similar and Dissimilar Tasks, NeurIPS2020\n\n[2]: Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning, NeurIPS2021\n\n[3]: Beyond not-forgetting: Continual learning with backward knowledge transfer, NeurIPS2022\n\n[4]: Continual learning of natural language processing tasks: A survey, arXiv\n\n[5] AdapterFusion: Non-destructive task composition for transfer learning. Pfeiffer, Jonas, et al. EACL 2021\n\n[6] TRGP: Trust Region Gradient Projection for Continual Learning. Lin, Sen, et al. ICLR 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The problem concerned is important and valid. I believe the NLP community should pay more attention on it.", "reasons_to_reject": "1. When talking about task similarity in continual learning, a popular paper is CAT [1], which considers not only similar tasks and dissimilar tasks, but also the mixture of them. Have you ever tried this challenging scenario and any interesting observations?    \n2. I think there are more systems to be compared. While you mentioned some systems are focused on classification, I do not see why they cannot be a baseline in generation setting by simply changing the underlying LM and classification head (in particular for those baselines based on adapter). I think the author should at least compare those NLP methods that are explicitly dealing with transfer (as cited by the author in the related work, like [2] or even more in the survey [4]).    \n3. Similarity detection has been used in different systems [1,3], what are the main advantages for DMEA compared to them? Or is it just another heuristic method?\n\n[1]: Continual Learning of A Mixed Sequence of Similar and Dissimilar Tasks, NeurIPS2020  \n[2]: Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning, NeurIPS2021  \n[3]: Beyond not-forgetting: Continual learning with backward knowledge transfer, NeurIPS2022  \n[4]: Continual learning of natural language processing tasks: A survey https://arxiv.org/pdf/2211.12701.pdf", "questions_to_authors": "See above", "paper_topic_and_main_contributions": "This paper concerns continual learning in NLP, in particular the knowledge transfer issues. ", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "See above", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "vV2ttE7m1A", "reviewer": "Reviewer_X4NL", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer:\n\nThank you for the positive assessment and insightful comments.\n\n**Q1. From my perspective, the experimental section of the paper could benefit from conducting some ablation experiments. For example, if the paper proposes three stages to handle multi-task continual learning, it would be helpful to determine which location has the most significant impact on the final results and analyze the reasons for this. This would help us to understand the work in this area better.**\n\nWe do provide several ablations to analyze our method. For example, we investigate three variants of our method in Section 6.2 (line 519-545):\n\n(1) without selecting similar previous tasks for forward knowledge transfer (the whole selection stage)\n\n(2) removing dynamic gradient scaling (in the adaptation stage)\n\n(3) without dynamically initializing learnable coefficients (in the expansion stage)\n\nAccording to the results in Table 2, forward knowledge transfer plays the most important role in the final performance, demonstrating the necessity of discovering and transferring useful learned knowledge to facilitate adaptation to the new task in lifelong sequence generation (the effectiveness of forward knowledge transfer is also verified in Table 3).\n\nIn addition, we further analyze different aspects of our method in Section 6.3 and Appendix, e.g., similarity metrics, module types, and pseudo-data sampling ratios.\n\nThanks again for your feedback. We hope that this addresses your questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This article provides a new approach to solving the forgetting problem in continual learning. Unlike previous works, this work takes a novel perspective from human thinking and can somewhat mitigate the forgetting problem. Therefore, it has a degree of innovation. Additionally, the article provides a detailed explanation of the formulas, making it easier for people to understand. Finally, this article also conducts complex experiments and comparisons with different models, showing some improvements in the experimental results.", "reasons_to_reject": "From my perspective, the experimental section of the paper could benefit from conducting some ablation experiments. For example, if the paper proposes three stages to handle multi-task continual learning, it would be helpful to determine which location has the most significant impact on the final results and analyze the reasons for this. This would help us to understand the work in this area better.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Lifelong sequence generation (LSG) is a research area in continual learning that aims to train a model on a sequence of generation tasks while avoiding forgetting previously learned knowledge and continuously acquiring new generation patterns. As mentioned earlier, the biggest problem with continual learning is forgetting formerly known knowledge. This article proposes Dynamic Module Expansion and Adaptation, which allows the model to dynamically determine its architecture for acquiring new knowledge based on the relationships between tasks and select the most similar previous study to adapt to the new job or add new modules. Finally, dynamic gradient scaling is applied to balance the weight of the current task and previously learned lessons, mitigating the forgetting problem.\nI think the main contribution of this article is that it opens up a new approach to exploring the forgetting problem in continual learning. Through this approach, it is possible to mitigate the model's bloating that comes with continuously adding modules, i.e., keeping the model's parameters from becoming too large while maintaining good performance. Compared to the trend of increasing the size of models, incorporating more thinking can reduce parameters and deepen our understanding of the problem in this field.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JhtSnAikQL", "reviewer": "Reviewer_kDLs", "summary": "", "questions": "1. What is the model performance with different $K$ and number of samples picked to obtain the input subspace $n$?\n\n2. As TRGP is already a powerful continual learning method, why did you not include it as a baseline? Also, I would suggest you to compare with some other continual learning baselines, such as gradient projection memory [1]. \n\nI would like to increase my score if the questions are well answered.\n\n[1] Saha, Gobinda, Isha Garg, and Kaushik Roy. \"Gradient projection memory for continual learning.\" arXiv preprint arXiv:2103.09762 (2021).", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer:\n\nThank you for the positive assessment and insightful comments.\n\n**Q1. The claim that this is the first work to address LSG from a human learning perspective is aggressive. I suggest highlighting more on the methodology part.**\n\nThanks for your suggestion! We will highlight more on the methodology part in the final version.\n\n\n**Q2. More hyper-parameter analysis is needed. The authors mention the search space of hyper-parameters in Appendix A.18, but I would like to see more analysis about how the hyper-parameters affect the performance, for example, K. / What is the model performance with different K and number of samples picked to obtain the input subspace n?**\n\nWe select hyperparameters using a sequence that is not used for final evaluation (Table 1). The model performance with different K and n is shown below. For the number of similar tasks K, the model achieves the best performance with K=1, which should be because selecting too many previous tasks introduces unnecessary noise. For the number (n) of samples selected to obtain the input subspace, increasing n does not always improve the performance. When n reaches 100, the performance tends to be stable. Therefore, we use n=100 to reduce computational cost.\n\n| K  | 1 | 2 | 3 |\n|  ----  | ----  | ----  | ----  |\n| Performance | 59.4  | 59.0 | 58.8 |\n\n| n  | 50 | 100 | 200 | 500 |\n|  ----  | ----  | ----  | ----  | ----  |\n| Performance | 58.9  | 59.4 | 59.4 | 59.4 |\n\n**Q3. As TRGP is already a powerful continual learning method, why did you not include it as a baseline? Also, I would suggest you to compare with some other continual learning baselines, such as gradient projection memory [1].**\n\nTo better demonstrate the effectiveness of DMEA, we further compare DMEA with two methods originally designed for classification: GPM[1], and TRGP [2]. We randomly select four task sequences for pilot experiments. From the average results shown below, we can see that DMEA outperforms both methods by a large margin, demonstrating the effectiveness of pseudo-sample replay and forward knowledge transfer. We will conduct more extensive experiments in the final version.\n\n| DMEA  | GPM | TRGP |\n|  ----  | ----  | ----  |\n| 62.9 | 59.1  | 59.6 |\n\nThanks again for your feedback. We hope that this addresses your questions.\n\n[1] Gradient projection memory for continual learning. Saha, Gobinda, Isha Garg, and Kaushik Roy. ICLR 2021\n\n[2] TRGP: Trust Region Gradient Projection for Continual Learning. Lin, Sen, et al. ICLR 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is clearly written and easy to follow. \n\n2. The idea of using knowledge of previous tasks is novel in lifelong sequence generation tasks. \n\n3. The experiments are comprehensive and support the claim.\n", "reasons_to_reject": "1. The claim that this is the first work to address LSG from a human learning perspective is aggressive. I suggest highlighting more on the methodology part.\n\n2. More hyper-parameter analysis is needed. The authors mention the search space of hyper-parameters in Appendix A.18, but I would like to see more analysis about how the hyper-parameters affect the performance, for example, K.\n", "questions_to_authors": "1. What is the model performance with different $K$ and number of samples picked to obtain the input subspace $n$?\n\n2. As TRGP is already a powerful continual learning method, why did you not include it as a baseline? Also, I would suggest you to compare with some other continual learning baselines, such as gradient projection memory [1]. \n\nI would like to increase my score if the questions are well answered.\n\n[1] Saha, Gobinda, Isha Garg, and Kaushik Roy. \"Gradient projection memory for continual learning.\" arXiv preprint arXiv:2103.09762 (2021).", "paper_topic_and_main_contributions": "This paper focuses on lifelong learning for sequence generation. The authors introduce a novel framework that can dynamically determine whether to add new modules for acquiring novel knowledge from new tasks. Additionally, they suggest retrieving knowledge from prior tasks to enhance adaptation. The authors have carried out comprehensive experiments in two lifelong learning settings: (1) identical tasks across different domains, and (2) distinct, randomly ordered tasks. In most instances, the proposed methods outperform the baselines. Overall, I find the proposed method reasonable, and the experimental results support it well. However, the authors claim this is the first work to address LSG from a human learning standpoint. To my knowledge, most lifelong learning studies purport to be inspired by human learning. This claim might be overstated, but I don't believe it diminishes the contributions made by the authors.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers mostly agree that the paper is well-written and easy to follow. Also, most reviewers reach a consensus that the proposed method is novel and acknowledge that the method achieves new SoTA results. The reviewers also asked more analysis on the hyper-parameters, additional baselines, ablation studies on each parts and missing references. The authors mostly addressed the concerns raised by the reviewers except for the reviewer 9rh6. Reviewer 9rh6 could not provide further feedback or comments on the authors' rebuttal. Given the discussion, we weight more on the other reviewers' scores.

As a result, the reviewers mostly agree that the paper is 'strong' on the soundness. ([4,4,4], *3 - 9rh6*). In terms of excitement, two reviewers give 'Strong' rating, one 'Ambivalent' and one 'Mediocre'. Therefore, the excitement score is more towards 'strong'.
Decision: Accept-Main