Domain Incremental Lifelong Learning in an Open World
Year: 2023

Abstract:
Lifelong learning (LL) is an important ability for NLP models to learn new tasks continuously. Architecture-based approaches are reported to be effective implementations for LL models. However, it is non-trivial to extend previous approaches to domain incremental LL scenarios since they either require access to task identities in the testing phase or cannot handle samples from unseen tasks. In this paper, we propose Diana: a dynamic architecture-based lifelong learning model that tries to learn a sequence of tasks with a promptenhanced language model. Four types of hierarchically organized prompts are used in Diana to capture knowledge from different granularities. Specifically, we dedicate tasklevel prompts to capture task-specific knowledge to retain high LL performances and maintain instance-level prompts to learn knowledge shared across input samples to improve the model’s generalization performance. Moreover, we dedicate separate prompts to explicitly model unseen tasks and introduce a set of prompt key vectors to facilitate knowledge sharing between tasks. Extensive experiments demonstrate that Diana outperforms state-ofthe-art LL models, especially in handling unseen tasks. We release the code and data at https://github.com/AlibabaResearch/ DAMO-ConvAI/tree/main/diana.

heading: 1 Introduction
text: An essential ability of humans is to learn new tasks continuously in their lifetime since our surrounding world is ever involving (Thrun and Mitchell, 1995). Humans need to learn inputs from unseen new tasks everyday. However, neural network based NLP models tend to rapidly lose previously acquired knowledge when trained on new tasks. This phenomenon is referred to as catastrophic forgetting
∗ Work done while the author was interning at Alibaba. † Equal contribution. ‡ Corresponding author.
PLM
Question 𝑄: What is the first name of the person who purchases a revolver? Context 𝐶: The Frankie Bono, a mentally …
Question 𝑄: How is Paul's problem fixed? Context 𝐶: In a Louisiana assisted-living ... Answer 𝐴:
Frankie
Answer 𝐴: Jessie
Question 𝑄: Who was having a birthday? A. Jessie B. no one C. Lion D. Tiger Context 𝐶: It was Jessie's birthday ...
Answer 𝐴: John uses magic
Format 𝐹 : Extractive
Format 𝐹 : Abstractive
Format 𝐹 : Multiple-Choice
𝑃
Task 𝑇
Task 𝑇
Task 𝑇
General Prompt
Format Prompt
Task Prompt Meta Prompt
 𝑃 𝐹𝑃 𝐹
𝑃 𝑇 𝑃 𝑇 𝑃 𝑇 𝑃 𝑃 𝑃 𝑃𝑃
Figure 1: An overview of Diana. A pre-trained language model is used to learn tasks in different formats with hierarchically organized prompts.
(French, 1999), and it’s important to equip NLP models with the lifelong learning (LL) ability to alleviate this issue in advanced AI applications.
An effective method to build LL models is the architecture-based approach (Chen et al., 2016; Rusu et al., 2016; Fernando et al., 2017; Wiwatcharakoses and Berrar, 2020), in which task-specific components are used to isolate knowledge for each separate task (Mancini et al., 2018). Recently, to leverage the power of pre-trained language model (PLM), some architecture-based LL models convert NLP tasks into a unified language modeling (LM) format (Sanh et al., 2021; Xie et al., 2022) and learn these tasks using a PLM. Separate prompts (Qin and Joty, 2022) or adapters (Madotto et al., 2021b) are allocated for different tasks to avoid the catastrophic forgetting issue.
However, despite the reported effectiveness, most above models are designed for the task incremental learning scenario, in which we assume task IDs for testing samples are available (Wang et al., 2022a,b). This setting limits the application of LL models because practical applications usually follow a more general domain incremental learning scenario (van de Ven et al., 2022), i.e., we cannot access the task IDs of most input samples.
There are generally two approaches to building LL models for domain incremental learning. One is to predict the task ID of each testing sample (Worts-
5844
man et al., 2020), and activate specified components based on the prediction (Figure 2a). This scheme achieves high LL performances if the predicted ID is correct (Madotto et al., 2021a). However, these models cannot handle samples from unseen tasks since there are no components designated for these samples and thus no task IDs to be predicted. This hinders the application of LL models because we often encounter samples from unseen tasks in practical situations (Dietterich, 2017).
Another approach to building domain incremental LL models is to organize model components at the instance-level, i.e., a pool of fine-grained components are dynamically combined in the forward pass for each input instance (Figure 2b). This approach avoids the trouble of explicitly determining task IDs. However, it usually yields low LL performance because there are no dedicated components for each task to capture task-specific knowledge (Wang et al., 2022a).
In this study, we combine the advantages of the above two approaches and propose Diana: a dynamic architecture-based lifelong learning model. We convert different NLP tasks into a unified LM format and propose to learn these tasks using a prompt-enhanced PLM (Figure 1). Specifically, Diana maintains four types of prompts to capture task knowledge from different granularities: 1. A general prompt Pg is used for all tasks; 2. The format prompts Pf are shared between tasks in a similar format; 3. A task prompt Pt is assigned for each incoming task; 4. A pool of meta prompts Pm are dynamically combined for each input instance. These four types of prompts present a hierarchical structure with a decreasing knowledge granularity, i.e., Pg captures global knowledge between all tasks, while Pm captures local knowledge that is shared between instances.
Diana can better generalize to unseen tasks while achieving high LL performances since its components are organized at both task and instance level. Moreover, we also maintain key vectors for Pt and Pm to better share task knowledge, and allocate separate task prompts to explicitly model samples for unseen tasks. Extensive experiments on benchmark NLP tasks indicate that Diana outperforms state-of-the-art (SOTA) baselines, especially in handling unseen tasks. Our main contributions are:
1. We propose Diana: a novel architecture-based domain incremental LL model that uses hierarchically organized prompts to capture knowledge in
different granularities. 2. We are the first to consider unseen tasks in the testing phase of LL models. Specific prompts are designated in Diana to handle unseen tasks, and prompt keys are built to facilitate sharing of task knowledge.
3. Extensive experiments show that Diana outperformed SOTA baselines.

heading: 2 Related Work
text: Lifelong Learning aims at incrementally acquiring new knowledge without catastrophically forgetting previously learned ones. Generally, three categories of LL methods are proposed: 1. Rehearsal-based methods (Rebuffi et al., 2017; Shin et al., 2017; Sun et al., 2019a; Chaudhry et al., 2019a; Buzzega et al., 2020) preserve past knowledge by replaying data from learned tasks; 2. Regularization-based methods (Kirkpatrick et al., 2017; Zenke et al., 2017; Li and Hoiem, 2017; Ritter et al., 2018; Farajtabar et al., 2020) consolidate model parameters that are important to previous tasks by introducing additional regularization terms; 3. Architecture-based methods (Chen et al., 2016; Rusu et al., 2016; Fernando et al., 2017; Maltoni and Lomonaco, 2019) add task-specific parameters to an existing base model for each task to prevent forgetting.
Experiment settings of LL methods can be generally classified into three scenarios based on whether the task ID is provided for testing samples and whether it must be inferred (van de Ven and Tolias, 2019), i.e., task-incremental learning (Mallya and Lazebnik, 2018; Ebrahimi et al., 2020), domain-incremental learning (Pu et al., 2021; Gao et al., 2022), and class-incremental learning (Zhang et al., 2020). In this work, we focus on the domain-incremental learning setting, where task ID is not provided for each testing sample. One line of methods in this category attempt to detect the task ID for each input sample (Madotto et al., 2021a). However, these methods fail to generalize to unseen tasks (Wang et al., 2022a). Another line of methods try to build a dynamic architecture for each input sample, for example, maintaining a pool of prompts that can be dynamically combined (Wang et al., 2022b). However, these methods yield sub-optimal performance since no task-specific parameters are used. Our model Diana is the first attempt to take advantage of the two aforementioned types of methods.
Pre-trained LM is becoming the de facto standard component for NLP models. To encourage knowledge sharing, existing approaches attempt to cast all NLP tasks into a unified text-to-text format (McCann et al., 2019) and learn these tasks by finetuning a PLM. A similar work compared to ours is ProQA (Zhong et al., 2022a), in which different QA tasks are unified and a set of structured prompts are used. However, ProQA only considers two QA tasks and is limited to the task incremental learning scenario, while our model is designed to tackle more general NLP tasks in a more general domain incremental learning scenario.

heading: 3 Method

heading: 3.1 Task Formulation
text: In this study, we aim to sequentially learn N tasks T1, · · · , TN that are presented in L different formats F1, · · · , FL, (L ≤ N). Each task Ti is presented in a specific format Fj (such as “Classification” or “Summarization”), and each training sample of Ti is a tuple of a context C, a question Q, and an answer A: (C,Q,A). Note that the format of each task can be easily inferred from the context-question pair (C,Q). Our model gθ is built to predict A based on C and Q. We also consider a more challenging open domain lifelong learning setting, i.e., the model needs to predict answers for unseen tasks. Therefore, we collect another N ′ unseen tasks TN+1, · · · , TN+N ′ that are only used for testing. We assume that all task identities of inputs are not available in the testing phase.

heading: 3.2 Framework of Hierarchical Prompts
text: We follow previous approaches to serialize the context C, question Q, and answer A into text sequences (Khashabi et al., 2020; Zhong et al., 2022a) and use a prompt-enhanced encoder-decoder model gθ to learn each task Ti in Diana. We use soft prompts (Liu et al., 2021; Lester et al., 2021; Vu et al., 2022) in our study, i.e., each prompt is a sequence of trainable embeddings that are randomly initialized and learned in the training process. For each training sample (C,Q,A) from task Ti, we first construct a prompt P (C,Q) based on (C,Q). Then the encoder takes in the concatenation of P (C,Q), C, and Q and the decoder predicts A, i.e., A = gθ([P (C,Q);C;Q]), in which “[; ]” denotes the sequence concatenation operation.
Four types of prompts are contained in P (C,Q), i.e., P (C,Q) = [Pg;Pf (Fj);Pt(Ti);Pm(C,Q)] (Figure 2c). Specifically, Pg is a general prompt, Pf (Fj) is a format prompt (where Fj is the format of task Ti), Pt(Ti) is a task prompt and Pm(C,Q) is a combined meta prompt. These four types of prompts are organized hierarchically so that they are shared by samples in different granularities:
1. General Prompt Pg is shared for all training tasks so that it encodes global task knowledge.
2. Format Prompt Pf (Fj) is shared between tasks in the same format Fj so that it captures format-related knowledge, i.e., knowledge that is shared between tasks in the format Fj .
3. Task Prompt Pt(Ti) is specifically allocated for the task Ti and it is only shared for samples from Ti. We use Pt(Ti) to learn task-specific knowledge. Moreover, to explicitly model samples from unseen tasks, we enlarge the set of task prompts with L extra prompts P̂t(F1), · · · , P̂t(FL), in which each prompt P̂t(Fj) models the unseen task for a particular format Fj .
4. Meta Prompt Pm(C,Q) is a dynamic combination of various instance-level prompts. Specifically, we maintain M instance-level meta prompts {P im}Mi=1 and dynamically combine these prompts based on the (C,Q) to obtain Pm(C,Q). Pm(C,Q) captures the knowledge shared between similar training instances.
We expect these four types of prompts can capture knowledge from different granularities since they are shared in different scopes. Moreover, to facilitate knowledge sharing, we allocate a key vector kt(Ti) and k j m to each task prompt Pt(Ti) and meta prompt P jm, respectively, and build a fixed text en-
coder h to map a context-question pair (C,Q) to a query vector q = h(C,Q). A two-stage learning process is introduced in Diana to learn these keys and P (C,Q). Specifically, the first stage focuses on learning a representation space for prompt keys so that we can determine proper prompts to construct P (C,Q). The second stage optimizes the constructed prompt P (C,Q) and the backbone language model. These two stages are detailed in the following sections.

heading: 3.3 Key Vector Space Learning
text: We first optimize key vectors assigned to each task prompt and meta prompt to construct the prompt P (C,Q) for each input (C,Q). Note that these key vectors are only used to determine the task prompt and meta prompt in P (C,Q) because the general prompt Pg is shared by all tasks in Diana, and the format prompt Pf (Fj) can be determined based on the format of C and Q directly.
Task Prompt Keys help to determine the task prompt in P (C,Q). Specifically, for a given input (C,Q), we first calculate its query vector q and then determine the most similar task prompt key kt(Ti) to q. The task prompt Pt(Ti) associated with kt(Ti) is used to construct P (C,Q).
Ideally, the key vector kt(Ti) for a task prompt Pt(Ti) should be located near samples from task Ti and distant to samples from other tasks Tj (j ̸= i). Therefore, when learning each task Ti, we maintain a small memory bufferM for samples from previously learned tasks Tj , (j < i), and design the following exponential angular triplet loss (Ye et al., 2021) to enforce the above property:
Lt =exp(||h(C,Q),kt(Ti)||+ max(1− ||h(Cn, Qn),kt(Ti)||, 0)),
(1)
in which the operator ||·, ·|| determines the distance between two input vectors (here we use cosine distance), (Cn, Qn) is a negative sample extracted from the memory bufferM:
(Cn, Qn) = argmin (C′,Q′)∈M
||h(C′, Q′),kt(Ti)||. (2)
Meta Prompt Keys help to combine these instance-level meta prompts {P im}Mi=1 to produce Pm(C,Q). Specifically, for each input (C,Q), we select M ′ meta prompt keys that are closest to its query vector q = h(C,Q). Then Pm(C,Q) is obtained by concatenating these M ′ meta prompts. Intuitively, the knowledge associated with (C,Q,A) is distributed in these M ′ meta prompts.
When learning meta prompt keys, we expect the distribution of these keys to balance two properties: diversity and locality (Figure 3). Specifically, the diversity property aims to distribute these keys to the whole vector space so that every meta prompt can be involved in the training process. The locality property aims to cluster similar meta prompts keys so that the knowledge of each sample can be better shared. For each input C and Q, we propose the following loss to enforce the above two properties:
Lm = ∑
i∈S(C,Q) max(0, ||kim, h(C,Q)|| − η)+
∑
i,j∈S(C,Q) max(0, γ − ||kim,kjm||)/M ′2,
(3)
where S(C,Q) is the index set of these M ′ meta prompt keys that are closest to h(C,Q), η and γ are scalar hyper-parameters for the distance margin. Specifically, the first term in Eq. 3 enforces the locality property by pulling these M ′ meta prompt keys around the query vector. The second term enforces the diversity property by pushing these meta prompt keys away from each other to occupy the whole vector space.
Note that Eq. 3 only involves a single query h(C,Q) from the current task. This may limit the learned meta prompt keys since samples from previously learned tasks are not considered. In this study, we extend Eq. 3 to better shape the distributions of meta prompt keys with the help of the memory buffer M, in which samples from previously learned tasks are contained. Specifically, when learning the task Ti, we first calculate query vectors for samples in M and then group these query vectors into B clusters (we set B = 5× i in our experiments, where i is the number of received tasks). Centroids of these B clusters are denoted as c1, · · · , cB . For each sample (C,Q) fromM, the subsequent loss is optimized:
L′m = ∑
i∈S(C,Q) max(0, ||kim, ck|| − η), (4)
where ck is the centroid to which (C,Q) belong. The above loss enforces the global diversity by scattering meta prompt keys to each centroid.

heading: 3.4 Model Training
text: Scheduled Sampling of Task Prompts When training Diana, the task ID of each sample (C,Q) is given so that we can directly get the task prompt Pt(Ti). However, naively using golden truth task IDs leads to an exposure bias issue, i.e., task IDs inferred in testing may not always be correct.
In this study, we introduce a scheduled sampling process to tackle the exposure bias issue. Specifically, for a given sample (C,Q,A) in the k-th training step, we toss a coin and use the golden truth task ID with probability ϵk, or use the task ID inferred based on task prompt keys with probability 1 − ϵk (Bengio et al., 2015). Note that when starting to learn each task, prompt keys are not well optimized, and thus the selected task ID is not accurate. Therefore, we set the value of ϵk to favor the golden truth task ID at the beginning (i.e., when k is small) and gradually switch to the inferred task ID as the training proceeds (i.e., when k is large), i.e., a linear decrement of ϵk is scheduled:
ϵk = max(0, α− kβ), (5)
in which α and β are scalar hyper-parameters. Note that LL models may encounter another source of exposure bias since we may receive inputs from unseen tasks in the testing phase. In this study, we use these L extra prompts P̂t(F1), · · · , P̂t(FL) to explicitly model unseen tasks. Specifically, for each training sample (C,Q,A), we first determine its task format Fj based on (C,Q), and allocate a small probability to use P̂t(Fj) as its task prompt in P (C,Q). In this way, we can capture general knowledge about all tasks for a given format in P̂t(Fj) and expect the knowledge to facilitate handling unseen tasks.
Train with LM Loss For each training sample (C,Q,A), we first construct the prompt P (C,Q) using approaches introduced above, and then optimize P (C,Q) together with the encoder-decoder model gθ using the following LM loss:
LLM = −log gθ(A|[P (C,Q);C;Q]). (6)
The overall loss that we optimize for Diana is:
L = Lm + L′m + Lt + LLM . (7)
After learning each task Ti, we select a small number of samples from Ti based on the query vector of each sample to update the memoryM. This selection process aims to maintain diverse samples inM. More details are in Appendix B.
See summarized training process in Algorithm 1.

heading: 3.5 Model Inference
text: When testing, we determine the prompt P (C,Q) for each input context C and question Q, and use the learned model gθ to predict the answer A.
Adaptive Decision Boundaries (ADB) are used to select proper task prompts in the testing phase. Specifically, for each task Ti, a scalar boundary δi is constructed following the approach proposed by Zhang et al. (2021). An input (C,Q) is regarded as a sample from unseen tasks if its query vector h(C,Q) falls outside the boundary of every task:
||h(C,Q),kt(Ti)|| > δi, ∀i ∈ [1, N ]. (8)
For samples from unseen tasks, we use the prompt P̂t(Fj) as its task prompt in P (C,Q), where Fj is the format of (C,Q).
Answer Prediction is performed with a greedy decoding process:
A = argmax A′
gθ(A ′|[P (C,Q);C,Q]). (9)

heading: 4 Experiments

heading: 4.1 Datasets
text: We use two sets of tasks to evaluate Diana:
1. decaNLP tasks: We follow Sun et al. (2019a) to select 5 tasks from the decaNLP (McCann et al., 2018) to train Diana. These tasks cover 3 different formats: Span Extraction, Sequence Generation, and Text Classification. We also collect N ′ = 3 additional tasks for each of these 3 format from decaNLP to serve as unseen tasks in the testing phase, i.e., our model is trained on N = 5 seen tasks while tested on 8 tasks;
2. QA tasks: The second set focuses on question answering (QA) benchmarks. Specifically, we use 8 QA datasets over 3 QA formats, i.e., Extractive QA, Abstractive QA and Multiple-Choice QA to train Diana. We also collect N ′ = 3 additional QA datasets for each of these three formats as unseen tasks, i.e., our model is trained on N = 8 seen tasks while tested on 11 tasks.
Note that task IDs for all testing samples are not available in our experiments. See Appendix C,J for more details of our dataset settings.

heading: 4.2 Evaluation Metrics
text: Individual tasks from above two task sets are evaluated following McCann et al. (2018) and Zhong et al. (2022a), respectively (see Appendix C). To evaluate the LL performance of Diana, we build a performance matrix R ∈ RN×(N+N ′), where Ri,j is the model performance on task Tj after learning task Ti. The following LL metrics are computed:
1. Average Performance AN and AN ′ is defined as the average performance of the final model on N seen tasks and N ′ unseen tasks, respectively:
AN = 1
N
N∑
j=1
RN,j , AN′ = 1
N ′
N+N′∑
j=N+1
RN,j . (10)
2. Average Forget FN is defined as the average performance decrease of each task after it is learned:
FN = 1 N − 1 N−1∑
j=1
max i∈{1,··· ,N−1}
(Ri,j −RN,j). (11)
In our experiments, we perform five runs with different random seeds and task orders. All reported metric scores are averages of these five runs. Ideally, we expect a strong LL model to yield high AN and AN ′ scores, and low FN scores.

heading: 4.3 Implementation Details
text: We use T5-base (Raffel et al., 2020) to initialize our encoder-decoder model, and set the lengths of soft prompts Pg, Pf , Pt, Pm to 20, 40, 40, 20, respectively. We maintain totally M = 30 meta prompts, and for each sample (C,Q) we choose M ′ = 5 meta prompts to construct Pm(C,Q). We use the AdamW (Loshchilov and Hutter, 2017) optimizer with a learning rate of 1e-4 and batch size of 64. Each task is trained for five epochs. We set η = 0.15 and γ = 0.3 in Eq. 3 and α = 0.9 and β = 3e− 4 in Eq. 5. We maintain 50 samples from each learned task in the memory M. All experiments are performed on 4 V100 GPUs, and the computational cost of our model is analyzed in Appendix G. See more details in Appendix A.

heading: 4.4 Baselines
text: We use the following competitive baselines covering all three types of LL models:
1. Regularization-based methods: EWC (Kirkpatrick et al., 2017) adopts the elastic weight consolidation approach to add regularization on parameter changes; FLCB (Gao et al., 2022)
uses knowledge learned from previous tasks to guide future task learning; 2. Rehearsal-based methods: ER (Chaudhry et al., 2019b) replays memory samples from previous tasks to consolidate learned knowledge; DER++ (Buzzega et al., 2020) augments ER with a L2 loss on the soft labels; AFPER (Mi et al., 2020) combines ER with an adaptive elastic weight consolidation mechanism; 3. Architecture-based methods: AdapterCL (Madotto et al., 2021a) allocates separate adapters for different tasks; L2P (Wang et al., 2022b) attaches a group of prompts on a pre-trained model to share fine-grained knowledge; DualPrompt (Wang et al., 2022a) uses different prompts to encode task-invariant and task-specific knowledge; ProQA (Zhong et al., 2022a) uses a unified structural prompt to implement LL models. Note that ProQA is designed for task incremental learning that requires accessing task IDs in the testing phase.
We combine ProQA and ER to implement a stronger baseline ProQA+ER, in which samples from previous tasks are replayed for the ProQA model, and we also implement a variant of Diana by removing the memory buffer Diana w/o M. We further report the performance for sequentially fine-tuning the LL model on all tasks (Finetune) and multi-task learning (Multitask). Note that the performance of Multitask is generally regarded as the upper bound of LL models when only seen tasks are considered.
All the above baselines are implemented following the same settings of our model, including using
the same backbone PLM, prompt size, and memory size used for replay. Note that for the ProQA baseline, we follow its original setting to provide task IDs for testing samples when evaluating.

heading: 4.5 Experiment Results
text: Results on Seen Tasks Table 1 shows the result on seen tasks from our two task sets. It can be seen that Diana outperforms all competitive baselines. Specifically, in the more general domain incremental learning scenario, i.e., when task IDs are unavailable in testing, Diana outperforms the best-performing baseline AFPER by a large margin. On QA tasks, Diana achieves 6.15% relative improvement on the AN score and 27.26% relative decrease on the FN score. Similar trend is also observed on decaNLP tasks. This means that Diana obtains higher performance with less forgetting in the LL process compared with other baselines.
We can also observe that: (1) Diana even outperforms the ProQA+ER baseline, which leaks task IDs in testing. This proves the superiority of our model design. (2) When task IDs are unavailable, Diana w/oM outperforms all baselines that do not use the memory buffer. This demonstrates that Diana’s hierarchical prompts help to improve the LL performance even without the memory buffer.
Results on Unseen Tasks Table 2 shows the result on unseen tasks from our two task sets. Note that we cannot compute the average forget score for unseen tasks since these tasks are never learned. Diana yields the best performances on all settings. It also achieves a relative improvement of 9.49% and 11.04% on the AN ′ score compared with the best baseline DER++ on these two task sets.
We can also observe that: (1) WhenM is unavailable, models that share knowledge through fine-grained components (i.e., Diana and L2P) generally obtain high performance, and our model that allocates extra prompts for unseen tasks achieves the best performance. This validates our approach of using hierarchical prompts to explicitly model unseen tasks. (2) It is interesting to see that Diana even outperforms Multitask, which is usually regarded as the upper bound of traditional LL models when only seen tasks are considered. This indicates that traditional LL models have limited generalization ability to unseen tasks and it also proves that our model is effective in modeling unseen tasks.
See Appendix D for detailed experimental results of all tasks.

heading: 4.6 Ablation Studies
text: We conduct ablation studies on different components of Diana. Specifically, three types of variants are implemented:
1. Each of these four prompt types is ablated: w/o general prompt, w/o format prompt, w/o task prompt, w/o meta prompt.
2. Schemes to enhance task prompts are ablated: w/o Sched. Sampling removes the scheduled sampling scheme and only uses the ground truth task IDs in training; w/o G.T. Identity is similar to the above variant. Instead, it only uses predicted task IDs in training; w/o Neg. Samples only uses positive samples to train task prompt keys, i.e., the second term in Eq. 1 is removed; w/o ADB uses fixed decision boundaries instead of ADBs to detect unseen tasks.
3. Schemes to enhance meta prompts are ablated: w/o Sample Dive. does not enforce the diversity property of the meta prompt keys, i.e., the second term in Eq. 3 is removed; w/o Memory Dive. does not use samples from previous tasks to enhance the diversity property, i.e., the loss L′m (Eq. 4) is removed; w/o Loc. does not enforce the locality property of the meta prompt keys, i.e., the first term in Eq. 3 is removed; w/o Cluster does not cluster samples inM, i.e., ck in Eq. 4 is replaced with the query vector of each sample fromM.
Table 3 shows the performance of the above variants on QA tasks. It can be observed that Diana outperforms all the above variants. We can also see that: (1) “w/o Meta Prompt” lowers the LL performance by a large margin. This indicates that these
fine-grained meta prompts are important in building lifelong models. (2) The scheduled sampling scheme helps to learn better task prompts and thus improves the LL performance. (3) ADB improves model performance on unseen tasks (i.e., AN ′) by a large margin. (4) Enforcing the diversity property of meta prompt keys is important to obtain good key representations and facilitates the learning of each task.

heading: 4.7 More Analysis

heading: 4.7.1 Task ID Detection Performance
text: Diana needs to detect task IDs of input samples when determining the task prompt to be used. To verify the performance of the task ID detector implemented in Diana (Section 3.3 and 3.5), we compare the approach used in Diana with other task ID detectors: (1) Perplexity-based detector implemented in baseline “AdapterCL” determines the task IDs based on the perplexity of the PLM when different adapter modules are activated. (2) Distance-based detector implemented in our variant “w/o Neg. Samples” determines the task identity based on the distance between each key and query vectors. (3) Advanced distance-based detector implemented in our variant “w/o ADB” utilizes negative samples based on the above detector. Note that we do not apply ADB in the above two distancebased detectors. On our testing data, the above three approaches achieve a task ID detection accuracy of 59.84%, 52.72%, and 63.43%, respectively, while Diana reaches a task ID detection accuracy of 66.97%. This verifies the effectiveness of our approaches to optimize task prompt keys in detecting task IDs. More detailed comparisons of these
task ID detectors can be found in Appendix E.

heading: 4.7.2 Distribution of Meta Prompt Keys
text: We also analyze the distribution of meta prompt keys K = {kjm}Mj=1 constructed in Diana, which are expected to balance the locality and diversity property. Specifically, we introduce two metrics to quantify these two properties. For the diversity property, we follow Mansoury et al. (2020) to measure whether these meta prompt keys cover the whole vector space:
Diversity = | M∪ j=1 NZ(kjm,M)|/(Z ·M), (12)
where NZ(kjm,M) represents the set of top-Z nearest samples inM around kjm, and | · | returns the sample count of a set. High diversity scores are received if we can scatter meta prompt keys near every query vector fromM. For the locality property, we follow Scellato et al. (2010) to measure whether there are keys clustered around each query vector q inM:
Locality = ∑
q∈M
∑
k∈NZ(q,K) (1− ||q,k||)/(Z ·|M|). (13)
High locality scores are received if meta prompt keys in K are tightly clustered.
On the QA tasks, we compare the above two metrics between Diana and our ablation variants for meta prompts under different values of Z. As can be seen from Table 4, the strategies we introduced in Diana (Section 3.3) help to enforce the locality and diversity properties of meta prompt keys.

heading: 5 Conclusion
text: We propose Diana, a novel LL model for the domain incremental learning scenario. Diana converts different NLP tasks into a unified sequence generation format and uses a prompt-enhanced PLM to learn these tasks. We introduce four types of hierarchically organized prompts in Diana to capture knowledge in different granularities. These
prompts are shared between different scopes of samples and are dynamically combined based on a set of key vectors. The space of key vectors is learned with several distance-based regularization terms. Dedicated components are also allocated in Diana to model samples from unseen tasks. Experiments and empirical analysis on two sets of tasks show that Diana outperforms SOTA LL models, especially in handling samples from unseen tasks.
Limitations
One major limitation of this study is its input modality. Specifically, our model is limited to textual inputs and ignores other modalities (e.g., vision and audio). Open and domain incremental lifelong learning across modalities is more realistic and challenging. Fortunately, we can obtain robust features of different modalities via multi-modal pre-training models (Xu et al., 2021; Huo et al., 2021). For future work, we will try to tackle multimodal tasks in an open (including out of distribution data (Lang et al., 2022, 2023a,b)) and domain incremental lifelong learning scenario with better approaches.
Ethics Statement
This work does not raise any direct ethical issues. In the proposed work, we seek to develop a model for domain incremental lifelong learning in an open world, and we believe this work leads to intellectual merits that benefit from a realistic and efficient lifelong learning model. All experiments are conducted on open datasets.

heading: A More Implementation Details
text: We use T5-base (Raffel et al., 2020) to initialize our encoder-decoder model (12 layers, 768 dimensional hidden size, and 12 attention heads), and set the lengths of soft prompts Pg,Pf ,Pt,Pm to 20, 40, 40, 20, respectively. We use a fixed T5-base encoder with an average pooling layer to obtain the query vector. We maintain a pool of M = 30 meta prompts, and for each sample (C,Q) we choose M ′ = 5 meta prompts to construct Pm(C,Q). We use the AdamW (Loshchilov and Hutter, 2017) optimizer for training. All hyperparameters are tuned according to the average score on validation datasets of NarQA, RACE, OBQA, SIQA and Dream. We tried epoch number of {2, 3, 4, 5, 6, 7, 8} and learning rate of {1e−5, 5e− 5, 1e− 4, 5e− 4, 1e− 3}. We finally set the learning rate to 1e-4 and the number of training epochs to 5. We set η = 0.15 and γ = 0.3 in Eq. 3 and α = 0.9 and β = 3e − 4 in Eq. 5. For η and γ, we have a grid search between 0 and 0.5 with an interval of 0.05. For α and β, α is searched among {0.9, 0.7, 0.5}, while β is searched among {1e − 5, 3e − 5, 1e − 4, 3e − 4, 1e − 3}. All experiments are performed on 4 V100 GPUs (32GB). The batch size is set to 64. In each set of tasks, We perform 5 runs with different task orders by setting the random seed to {42, 43, 44, 45, 46} respectively. In this way, we report the average score of each method. Note that we only use the random seed 42 for tuning hyper-parameters.
In order to train extra task prompts {P̂t(F1), · · · , P̂t(FL)} for unseen tasks, we allocate a small probability ω = 5% for each training sample (C,Q,A) to use P̂t(Fj) as its task prompt in P (C,Q), where Fj is the task format of (C,Q,A). To implement variant “w/o ADB” for ablation study, we use a fixed decision boundary instead of ADB. If for any task Ti, the distance ||h(C,Q),kt(Ti)|| > 0.35, we regard the sample is from unseen tasks.
The adaptive decision boundary for each task is determined following the approach proposed by Zhang et al. (2021). We use AdamW optimizer with a learning rate of 0.02 to learn each decision boundary. To obtain the ROUGE-L score, we use the NLTK package for sentence tokenization, and python rouge-score package for evaluation.

heading: B Memory Update
text: After learning task Ti, we select E diverse samples (we set E = 50 in our experiments) from Ti to update the memoryM based on the query vector of each sample. Specifically, our selection criteria are built based on the distance of these prompt keys and query vectors. For each meta prompt key kjm (j = 1, · · · ,M ), we select top-⌈ EM ⌉ samples (⌈·⌉ is the ceiling function), whose query vectors are closest to kjm. After accumulating M⌈ EM ⌉memory samples selected by M meta prompt keys, we rank these samples based on their distance to the corresponding meta prompt keys, and choose top-E samples with the smallest distance to be fed into M. In this way, the memoryM we constructed can expand to the whole space of prompt keys.
Note that, the memory bufferM is optional in Diana. WithoutM, the loss in Eq. 4 is not optimized, and the second term in Eq. 1 is removed.

heading: C Detailed Dataset Setting and Evaluation Metrics
text: For the decaNLP task set, 8 benchmarks over 3 formats are covered, i.e., (1) Span Extraction, including SQuAD (Rajpurkar et al., 2016), QA-ZRE (Levy et al., 2017), QA-SRL (He et al., 2015); (2) Sequence Generation, including WOZ (Wen et al., 2017), WikiSQL (Zhong et al., 2017), CNN/DM (Hermann et al., 2015); (3) Text Classification, including SST (Socher et al., 2013) and MNLI (Williams et al., 2018). For the QA task set, 11 QA benchmarks over 3 QA formats are covered, i.e., : (1) Extractive QA, including SQuAD (Rajpurkar et al., 2016), NewsQA (Trischler et al., 2017), and Quoref (Dasigi et al., 2019); (2) Abstractive QA, including NarQA (Kocisky et al., 2018), NQOpen (Kwiatkowski et al., 2019), and Drop (Dua et al., 2019); (3) Multiple-Choice QA, including RACE (Lai et al., 2017), OBQA (Mihaylov et al., 2018), MCTest (Richardson et al., 2013), SIQA (Sap et al., 2019), and Dream (Sun et al., 2019b). The statistics of the above datasets are summarized in Table 5. We follow the preprocess scheme released by Khashabi et al. (2020) to tackle these datasets. Some of these datasets do not contain a validation set, thus we only use the validation sets of NarQA, RACE, OBQA, SIQA and Dream in the QA task set to search hyperparameters.
The evaluation for each single task follows Mc-
Cann et al. (2018); Zhong et al. (2022b). Among the decaNLP tasks, we compute F1 score for QASRL and QA-ZRE, Exact Match (EM) score for SQuAD, MNLI and SST, ROUGE-L for CNN/DM. For WOZ, we adopt turn-based dialogue state exact match (dsEM). For WikiSQL, we use exact match of logical forms (lfEM). For the QA task set, we compute the accuracy of option selection for all Multi-Choice QA tasks and use EM score for all Extractive QA tasks. Among Abstractive QA tasks, we use F1 score for Drop and NQOpen, and ROUGE-L (Lin, 2004) for NarQA.

heading: D Detailed Experimental Results
text: We provide the detailed performance of Diana under each single task compared with competitive baselines. The results under five seen tasks of the decaNLP task set, and eight seen tasks of the QA task set are shown in Table 6 and Table 7. The results of unseen tasks for the decaNLP task set and the QA task set are shown in Table 8 and Table 9.

heading: E More Analysis of Task Identity Detection Performance
text: Architecture-based LL models need to detect task identities of input samples when these identities are unavailable in the testing phase. To verify the performance of the task identity detector implemented in Diana, we compare our approach with other task identity detectors: (1) Perplexity-based detector implemented in baseline “AdapterCL” determines the task identities based on the perplexity of the PLM when different adapter modules are activated. (2) Distance-based detector implemented in our
variant “w/o Neg. Samples” determines the task identity based on the distance between each key and query vectors. (3) Advanced distance-based detector implemented in our variant “w/o ADB” utilizes negative samples based on the above detector. Note that we do not apply ADB in the above two distance-based detectors.
The above approaches are trained and evaluated with the QA tasks under two scenarios: (1) In Closed-world: detectors are only required to detect samples from seen tasks. Note that in this setting, the Advanced distance-based detector used in “w/o ADB” is the same as the task identity detector implemented in Diana. (2) In Open-world: detectors are required to handle unseen task samples as well. When tested in the open-world scenario, these two distance-based detectors adopt a fixed decision boundary of 0.35 (see Appendix A). The perplexity-based detector adopts a perplexity threshold of 4, i.e., samples with a perplexity score above 4 are regarded as unseen task samples. This perplexity threshold is selected based on the model performance on the validation set.
We report the task identity detection accuracy and Marco F1 scores for seen samples and unseen samples separately in Table 10. we can observe that: (1) The task identity detector used in Diana achieves the best performance in both scenarios. This proves the effectiveness of our task prompt keys in detecting task identities. (2) Negative samples used in Advanced distance-based detector significantly improve the task identity detection performance on seen tasks. (3) ADB is effective in improving the task identity detection performance on unseen tasks.

heading: F More Analysis of Scheduled Sampling
text: We perform a more detailed analysis of the scheduled sampling scheme introduced in Diana. Specifically, in the ablation variant “w/o G.T. Identity”, the model only uses predicted task identities in training. This scheme helps to alleviate the discrepancy between training and testing with the cost of the model coverage speed. In the ablation variant “w/o Sched. Sampling”, the model only uses golden truth task identities in the training process. This scheme leads to the discrepancy between training and testing. The above two schemes under-perform our model Diana.
In this section, we analyze the task identity detection accuracy yield by the above schemes in
ProQA 0 13.25 37.58 39.42 30.08 ProQA+ER 50 14.18 38.42 40.17 30.92 Finetune 0 10.61 36.50 37.12 28.08 EWC 0 11.78 37.62 39.88 29.76 FLCB 0 12.98 40.02 40.52 31.17 AdapterCL 0 13.23 37.88 39.84 30.32 L2P 0 13.09 40.16 40.31 31.19 DualPrompt 0 12.92 37.04 39.18 29.71 ER 50 13.04 38.06 39.04 30.05 DER++ 50 14.67 39.74 39.32 31.24 AFPER 50 12.14 38.66 39.85 30.22
Diana w/o M 0 14.94 43.95 40.69 33.19 Diana 50 15.80 44.74 43.53 34.69
Multitask - 15.98 42.12 40.07 32.72
Figure 4 when learning the last task TN in the in-
put task sequence of QA task set. We can observe
that the task identity detection accuracy achieved by “w/o G.T. Identity” is extremely low in earlier iterations, which hinders task prompts from sharing task-specific knowledge in the early training stage. The scheduled sampling process introduced in Diana effectively compromises between detecting correct task identities and alleviating the train-test discrepancy, and thus it results in the best LL performance among these variants. Note that the task identity detection accuracy in “w/o Sched. Sampling” is almost zero in the first 1,000 iterations when learning task TN . This is because the task prompt keys for previous N − 1 tasks are already well learned. The randomly initialized prompt key for task TN needs to be pulled to the query vector space before starting to be functional.

heading: G More Analysis of Computational Cost
text: We analyze the computational cost of Diana when learning the QA tasks, including the number of tunable parameters, time used for training and testing, and size of required memories retained from previous tasks. As indicated in Table 11, Diana does not introduce too much computation overhead.

heading: H Effect of PLM Size
text: We evaluate Diana and the best-performing baseline DER++ on different sized PLM using QA datasets. As shown in Table 12, Diana obtains better performance with larger PLM size, and consistently outperforms the baseline.

heading: I Analysis of Training Method
text: During training, we follow a full tuning scheme that updates parameters of the backbone language models (T5) along with prompts. We also investigate the performance of prompt tuning, which fixes the backbone language model and only updates the prompts. As indicated in Table 13, prompt tuning dramatically degenerates the performance of Diana.

heading: J Cases
text: We list some samples for tasks we modeled from the decaNLP task set and the QA task set respectively, shown in Table 14 and Table 15.

heading: K Training Process
text: Details about the training process of Diana are shown in Algorithm 1.
Algorithm 1 Training process of Diana Input: prompt-enhanced model gθ, datasets {(Cj , Qj , Aj)}nij=1 for each task Ti (i=1, · · · , N ), memory buffer M, general prompt Pg, format prompts {Pf (Fj)}Fj=1, task prompts {Pt(Ti)}Ni=1 ∪ {P̂t(Fj)}Fj=1, meta prompts {P im}Mi=1, task prompt keys {kt(Ti)}Ni=1, meta prompt keys {kim}Mi=1
1: Initialize:M← ∅ 2: for Each task Ti, i = 1, · · · , N do 3: ifM ≠ ∅ then 4: Calculate cluster centroids c1, · · · , cB ofM 5: end if 6: for number of training epochs do 7: for Each mini-batch I ∈ {(Cj , Qj , Aj)}nij=1 ∪M do 8: Obtain ϵk by Eq. 5 9: for (C,Q,A) ∈ I do
10: Obtain format Fj of (C,Q,A) 11: Sample ϵ, ζ from U(0, 1) 12: if ζ < ω then 13: Pt(C,Q)← P̂t(Fj) {Use task prompt P̂t(Fj) for unseen tasks} 14: else if ϵ < ϵk then 15: Pt(C,Q)← Pt(Ti) {Use the golden truth task identity to select task prompt} 16: else 17: Pt(C,Q)← Pt( argmin
Tτ∈{T1,··· ,Ti} (||q,kt(Tτ )||)) {Use the inferred task identity to select task
prompt} 18: end if 19: S(C,Q)← indexes of M ′ meta prompt keys that are closest to q 20: Pm(C,Q)← {P jm} j∈S(C,Q) 21: P (C,Q)←[Pg;Pf (Fj);Pt(C,Q);Pm(C,Q)] 22: Calculate per sample loss LLM on gθ and P (C,Q) by Eq. 6 23: Obtain negative sample (Cn, Qn) fromM by Eq. 2 24: Calculate per sample loss Lt on kt(Ti) by Eq. 1 25: Calculate per sample loss Lm on {ksjm}(sj ∈ S(C,Q)) by Eq. 3 26: if (C,Q,A) ∈M then 27: Calculate per sample loss L′m on {ksjm}(sj ∈ S(C,Q)) by Eq. 4 28: end if 29: end for 30: Update gθ and prompts with accumulated LLM 31: Update task prompt keys {kt(Ti)}Ni=1 with accumulated Lt 32: Update meta prompt keys {kim}Mi=1 with accumulated Lm and L′m 33: end for 34: end for 35: UpdateM with {(Cj , Qj , Aj)}nij=1 according to details in Appendix B 36: end for
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section Limitations
3 A2. Did you discuss any potential risks of your work? Section Ethics Statement
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4
3 B1. Did you cite the creators of artifacts you used? Section 4, Appendix C
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix C
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section4, Appendix A, Appendix G
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4, Appendix A
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4, Appendix A
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: For future work, we will try to tackle multimodal tasks in an open (including out of distribution data (Lang et al. , 2022, 2023a,b)) and domain incremental lifelong learning scenario with better approaches