Prompts Can Play Lottery Tickets Well: Achieving Lifelong Information Extraction via Lottery Prompt Tuning
Year: 2023

Abstract:
Thanks to the recent success of Pre-trained Language Models (PLMs), it has become a promising research direction to develop a universal model (UIE) that can solve all typical information extraction tasks within one generative framework. Nonetheless, in real-world scenarios of UIE applications, new data of different IE tasks and domains usually come in a stream over time. A desirable UIE system should be capable of continually learning new tasks without forgetting old ones, thereby allowing knowledge and functionalities expansion without retraining the whole system. In this paper, we study the UIE system under a more challenging yet practical scenario, i.e., “lifelong learning” settings, to evaluate its abilities in three aspects, including knowledge sharing and expansion, catastrophic forgetting prevention, and rapid generalization on few-shot and unseen tasks. To achieve these three goals, we present a novel parameterand deployment-efficient prompt tuning method namely Lottery Prompt Tuning (LPT). LPT freezes the PLM’s parameters and sequentially learns compact pruned prompt vectors for each task leveraging a binary prompt mask, while keeping the prompt parameters selected by the previous tasks insusceptible. Furthermore, we use a simple yet effective method to perform mask selection and show the powerful transferability of Lottery Prompts to novel tasks. Extensive experiments demonstrate that LPT consistently sets state-ofthe-art performance on multiple lifelong learning settings of UIE, including task-incremental setting on seen tasks, few-shot adaptation, and zero-shot generalization on novel tasks1.

heading: 1 Introduction
text: Information Extraction (IE) is one of the fundamental tasks in Natural Language Processing (NLP), which aims to extract the desired structural information from unstructured texts (Andersen et al.,
1The code is available at https://github.com/ jokieleung/Lottery_Prompt.
1992; Surdeanu et al., 2003; Ma and Hovy, 2016; Kolluru et al., 2020). Previous IE research mostly focuses on one specific IE task (Miwa and Bansal, 2016; Wang et al., 2020; Lin et al., 2020; Zheng et al., 2021) and designs different model architectures (Lample et al., 2016; Sohrab and Miwa, 2018; Li et al., 2020; Hsu et al., 2022) to tackle different tasks. To facilitate knowledge sharing between different tasks, various efforts have been paid for unifying all IE tasks with one model structure (Wadden et al., 2019; Nguyen et al., 2021; Paolini et al., 2021). Most recently, Lu et al. (2022); Fei et al. (2022) unify general IE tasks in a generative way with a text-to-structure framework (UIE), which proves that universally modeling various IE tasks can better learn general knowledge from varying data sources.
Nonetheless, current work usually assumes the accessibility of training data for every task. In many real-world scenarios, as shown in Figure 1, the training data are often streamed, and the IE
277
systems are required to identify new mention spans or semantic relations to support new domains and functionalities, which can be formulated as the paradigm of lifelong learning. The ability to accumulate knowledge continually is crucial for the quick deployment of UIE systems based on PLMs, which allows the system to add new domains and functionalities over time without incurring the high cost of re-training the whole system each time. In addition, considering that humans can acquire new knowledge from a few examples (Montague, 1974), it is expected for the models to generalize well on novel tasks with few-shot data or even no data.
Motivated by this, our work aims to address these more challenging yet practical issues by proposing a lifelong learning setup for UIE. In this setup, the system sequentially learns over multiple IE tasks (potentially of different task types and varying domains) one by one. Then it will be evaluated to preserve its performance on solving previously seen tasks, and generalize well to novel tasks with few examples or even no examples. We cover two conventional properties of lifelong learning (Ke and Liu, 2022), i.e., catastrophic forgetting prevention (CF) and knowledge transfer (KT), while in our setup, the evaluation of KT extends to the novel tasks. In NLP community, large Pre-trained Language Models (PLMs) have been widely applied in many downstream tasks. In order to lower computation and storage costs, recent popular lifelong learning techniques (Madotto et al., 2021; Ke et al., 2021a; Zhu et al., 2022; Wang et al., 2022c) try to solve the CF and KT leveraging parameter-efficient fine-tuning (PEFT) methods (He et al., 2022a).
In this work, we inherit this wisdom and also focus on parameter-efficient methods for lifelong learning. Inspired by the lottery ticket hypothesis and the efficiency of prompt tuning, we propose a novel framework for lifelong UIE, named Lottery Prompt Tuning (LPT). Specifically, we adopt an encoder-decoder model architecture (Raffel et al., 2020) and re-frame all types of IE tasks into a text-to-structure format (Lu et al., 2022). First, we prepend a sequence of continuous prompt vectors to the input, which is shared across tasks. To continually learn a new IE task, we simultaneously learn the prompt vectors together with a task-aware binary prompt mask. The task-aware mask is devoted to pruning the shared prompt vectors and producing an optimal task-specific pruned prompt,
i.e., lottery prompt. To provide a pruning criterion for finding the lottery prompt online, we introduce a separate set of learnable parameters serving as the importance scores, which have the same shapes as the soft prompts. Hence, the lottery prompt can be easily found by selecting the parameters with the Top-k% importance scores online, without iterative retraining and pruning procedure. To facilitate the forward knowledge transfer when learning a new task, the lottery prompt is permitted to selectively reuse the learned prompt parameters for the former tasks. Besides, the proposed LPT eliminates catastrophic forgetting and negative transfer by freezing the prompt parameters for the previous tasks during back-propagation. In the whole learning process, the PLM is kept frozen to maintain general knowledge. During inference, the same model can handle different tasks by inputting different lottery prompts, which is friendly for deployment.
We show that our proposed framework effectively outperforms state-of-the-art baselines on lifelong learning for UIE in terms of catastrophic forgetting prevention and knowledge transfer. Moreover, LPT closes the gap between continual learning and multi-task learning. The efficacy of the proposed modules is thoroughly studied both empirically and analytically. In summary, this work makes three key contributions:
• A challenging yet practical benchmark is proposed for lifelong UIE, where one UIE system should not only keep its performance on solving seen IE tasks, but also generalize well on novel IE tasks with few or even no examples.
• We proposed Lottery Prompt Tuning (LPT), an extremely efficient prompt tuning framework for lifelong UIE that directly learns pruned prompts sequentially without an extra pruning stage.
• Extensive experiments on the benchmark show that our approach outperformed baselines with higher parameter efficiency.

heading: 2 Related Work
text: Lifelong Learning Lifelong Learning, also known as Continual Learning, aims to learn a sequence of tasks with one single model. Two main goals are demanded: catastrophic forgetting (CF) prevention and positive knowledge transfer (KT). The research in this area can be categorized into three folds: Regularization, Rehearsal, and Architecture based methods. (a) Regularization-based
methods (Li and Hoiem, 2017; Kirkpatrick et al., 2017; Ritter et al., 2018) ease the catastrophic forgetting issue by regularizing important parameters for learned tasks. These approaches usually need a trade-off between learning new tasks and forgetting the old tasks. In NLP, it is studied (Han et al., 2020) to constrain the useful information from the huge amount of knowledge inside the PLMs. (b) Rehearsal-based methods methods reuse old examples from the previously learned tasks while learning new tasks. These examples are either derived from real training data of previous tasks (Rebuffi et al., 2017; Lopez-Paz and Ranzato, 2017; Mi et al., 2020), or generated by a pseudo-data generator (Sun et al., 2019; Qin and Joty, 2021; Zhao et al., 2022). Although these methods work well, they are limited by data privacy or the quality of generated data. (c) Architecturebased methods tackle the continual learning problem by expanding new modules to the network over time (Veniat et al., 2020; Douillard et al., 2022) or isolating the network’s parameters for different tasks (Serra et al., 2018; Mallya and Lazebnik, 2018; Mallya et al., 2018; Wortsman et al., 2020; Geng et al., 2021; Kang et al., 2022). In NLP, in order to better take advantage of the PLMs, these methods usually are in conjunction with parameterefficient fine-tuning approaches, including adapter tuning (Houlsby et al., 2019) and prompt tuning (Lester et al., 2021a; Li and Liang, 2021; Liu et al., 2022b). AdapterCL (Madotto et al., 2021) trains a separate adapter for each task, leaving knowledge transfer out of consideration. Ke et al. (2021b,a); Ermis et al. (2022); Zhang et al. (2022) overcome this drawback by introducing capsule network (Sabour et al., 2017), distillation mechanism and adaptive compositional modules, respectively. For the latter, CPT (Zhu et al., 2022) learns a separate prompt with continual prompt initialization for each task. Wang et al. (2022c,b) propose to learn a prompt pool and then select the useful prompts to alleviate forgetting and potentially share knowledge across tasks. Dai et al. (2022) extend the idea to organize the prompt pools in a hierarchical way to guide the pre-trained models in different granularities. In contrast, we here share a single copy of prompt parameters to instruct the PLMs, yet incrementally learn a task-aware prompt mask for each task whilst keeping the prompt parameters used by the previous tasks unchanged. This not only isolates the harmful prompt parameters that lead to
forgetting but also shares useful prompt parameters for knowledge transfer.
Lifelong learning in Information Extraction In IE areas, some efforts are paid for building IE systems to handle continual learning scenarios, including continual NER (Monaikul et al., 2021; Zheng et al., 2022), relation extraction (Cui et al., 2021; Qin and Joty, 2022; Wang et al., 2022a), and event detection (Yu et al., 2021; Liu et al., 2022a). However, they merely study continual learning on one single IE task. Very recently, UIE (Lu et al., 2022; Fei et al., 2022) regards general IE tasks as a text-to-structure generation task, thus unifies all IE tasks with one model framwork. To a step further, our work studies a more challenging yet practical continual learning paradigm for UIE, where one universal IE system needs to solve different types of IE tasks across different domains incrementally.
Lottery Ticket Hypothesis Frankle and Carbin (2018) propose the The Lottery Ticket Hypothesis (LTH) that an over-parameterized network contains a sub-network (lottery ticket) that, when initialized and trained in isolation, can match or exceed the test accuracy of the original network after training for at most the same number of iterations. The LTH has been widely explored in many fields of deep learning (Liu et al., 2018; Frankle et al., 2019; Gong et al., 2022; Yu et al., 2019) In NLP, researchers also explore the existence of winning tickets under transfer learning regimes for over-parametrized pre-trained language models across various tasks (Morcos et al., 2019; Desai et al., 2019). Chen et al. (2020); Prasanna et al. (2020) show the existence of winning tickets when fine-tuning BERT on downstream tasks. Liang et al. (2021) shows the existence of super tickets inside PLMs that can improve generalization. Xprompt (Ma et al., 2022) is the pioneer to explore the LTH in the context of prompt tuning by hierarchical structure pruning. However, Xprompt needs iterative retraining, pruning and rewinding to get the pruned prompts, which is impractical to perform during continual learning settings since it needs excessive computational time and costs. By contrast, our LPT does not require an explicit pruning stage and jointly learns prompt and task-related masks together, which accelerates convergence during continual learning. Moreover, our pruning is performed at the parameter level while Xprompt’s pruning is performed at the token and piece level.

heading: 3 Preliminary

heading: 3.1 Lifelong Learning Protocols
text: Conventional continual learning is defined as training machine learning models on a continuum of data from a sequence of tasks. Here in our lifelong learning protocols for UIE, the incoming task on the task sequence can be of different types (e.g., entity extraction, relation extraction, event extraction, and aspect-based sentiment analysis.), or of the same type but potentially of different domains. An intuitive demonstration can be found in Figure 1. Formally, we define a sequence of tasks D = {D1, · · · ,DT }, where the k-th task Dk = {( xki ,y k i )}Nk i=1
contains a set of data samples. For each data sample, the input xki is constructed by the raw text tki and a specific predefined schema ski , while the desirable output y k i is structural information contained in the text xki indicated by the schema ski . Note that our approach is Rehearsal-free, meaning that data from the previous tasks can not be used anymore when training future tasks. The goal of a lifelong UIE model should perform well on all T tasks after being trained with the samples of these tasks sequentially. Further, in the realistic scenario, it is usually expensive and impractical to acquire plenty of labeled data for a newly emerged task. To simulate this circumstance, we adapt the sequentially trained model on a set of nnovel novel tasks individually {Di}Nnoveli=1 . Hence, we can access the model’s ability to accumulate previously learned knowledge for generalization to new tasks by evaluating the few-shot/zero-shot transferability of the lifelong model.

heading: 3.2 Generative UIE Framework
text: In this section, we cast all IE tasks as text generation and model the UIE system in a text-to-structure framework (Lu et al., 2022). In this generative framework, different IE structure generation is decomposed into two atomic operations, i.e., spotting and associating. Spotting indicates locating target information pieces from the sentence, e.g., the entity and the trigger word in the event. Associating means connecting spans by assigning them with specific semantic roles based on pre-defined schemas, such as the relation between entity pairs or the role between an event and its argument.
Input the input x for the UIE model is formulated as the concatenation of the raw sentence and
a schema-based prompt in the form of:
x = [s; t] = [ s1, s2, . . . , s|s|, t1, t2, . . . , t|t| ]
=[[spot],SPOT1, [spot],SPOT2 . . . ,
[asso],ASSO1, [asso],ASSO2 . . . , [ text ], t1, t2, . . . , t|t| ]
(1) SPOTi represents the targeted spotting name in the IE tasks, e.g., “organization" in the NER task; and ASSOi represents the targeted association name, e.g., “work for” in the relation extraction task.
Output the output text y is a unified Structured Extraction Language (SEL) that describes how the structural elements organize into the target structure, which can be represented as “{Spot Name: Info Span, (Asso Name: Info Span) (Asso Name: Info Span)}". The Spot Name and Asso Name are the target structure from the pre-defined schemas, while the Info Span refer to the text span mentioned in the raw text.
Model We employ a Transformer-based encoderdecoder language model i.e., T5 (Raffel et al., 2020), as the model architecture for UIE. Given the schema and the raw sentence as input sequences x and the SEL as output sequences y, the model computes the conditional language model distribution of each token yi using the chain rule of probability as p (yi | y<i, x). It finishes prediction when outputting the end signal [EOS]. The predicted SEL expression will be converted back into the extracted information record for evaluation.

heading: 4 Method

heading: 4.1 Overview
text: In this section, we present a novel pruning-based parameter-efficient tuning method for lifelong learning, called Lottery Prompt Tuning (LPT). The overall process of LPT is illustrated in Figure 2. To continually learn a new IE task, we simultaneously learn the prompt vectors together with a paired task-aware binary prompt mask, while the mask is devoted to producing a pruned prompt, i.e., Lottery Prompt. During training for each incoming task, LPT can selectively re-use the previously learned prompt parameters to encourage knowledge transfer, while the parameter updates only happen on those soft prompt parameters that have not been selected by the previous tasks. Finally, the model shares the same set of soft prompts for all tasks however uses the binary masks to isolate the shared
parameters and get the lottery prompt for each task, which solves catastrophic forgetting.

heading: 4.2 Lottery Prompt Tuning
text: Prompt tuning (Li and Liang, 2021; Liu et al., 2022b) learns a set of continuous prompts and only tunes the prompts while fixing the whole parameters in PLM, which has been proven to be effective in various downstream tasks. In this work, we combine prompt tuning and the aforementioned generative UIE into one unified framework, where the PLM takes the concatenation of continuous learnable soft prompts p, schema instruction s and the raw text t, i.e., x = [p; s; t]. The training objective is formalized as
L = ∑
(x,y)∈Dk − log p (y | x; θp) (2)
Note that only the soft prompt parameters θp are trainable. Recently, Ma et al. (2022) show that pruning prompts at token and piece level yields a more parameter-efficient prompt yet with competitive performance. Inspired by this, we propose a novel Lottery Prompt Tuning (LPT) which acquires high-performing pruned prompts for continual learning by assigning the prompt vectors θk together with a task-aware binary mask mk. The mask selects the top-c% of soft prompts that lead to good performance on the current task. To achieve this, we introduce a set of learnable parameters sk that have the same shape as the soft prompts, which indicates the importance scores of the prompt parameters. Once trained, these scores are thresh-
olded to obtain the prompt mask, i.e., mk = h(sk), where h(.) is an indicator function that outputs "1" for top-c% of the scores in the prompt parameters or "0" otherwise. Therefore, the pruned prompt parameters θ̂ k
p for task k, i.e., lottery prompt, is
obtained by θ̂ k
p = θp ⊙mk. To get rid of the need for iterative retraining, pruning and rewinding procedures during continual learning, we perform online pruning by simultaneously optimizing the prompt parameters and the importance scores together. To achieve this, we use a straight-through gradient estimator (Bengio et al., 2013) to ignore the derivatives of the indicator function h(.) and directly update the scores as follows:
minimize θp,sk
L (θp ⊙mk;Dk) ;
sk ← sk − η ( ∂L ∂sk ) (3)
where the η is learning rate. While training on a newly emerge task k, we use an extra binary mask Mk−1 = ∨k−1i=1mi to prevent updating the prompt parameters allocated by previous tasks. Hence, the prompt parameters θp are updated as follows:
θp ← θp − η (
∂L ∂θp ⊙ (1−Mk−1)
) (4)
To summarize, LPT circumvents the forgetting issue by isolating the prompt parameters for each task. Meanwhile, taking the separate scores as the pruning criterion allows sharing some of the parameters from previously chosen parameters θp ⊙mk
in solving the current task k, which contributes to knowledge transfer.

heading: 4.3 Mask Selection for Novel Tasks
text: When generalizing to the use-case on novel tasks where few or no labeled data for training, it is a desired property to transfer knowledge learned by the previous tasks to achieve better performance. Hence, we provide two simple solutions to select the binary masks in hands for initializing the lottery prompt. The first way is to utilize the perplexity (PPL) of each mask mk over the input X as a measurement (Madotto et al., 2021), i.e., PPLθkp (X). The mask with the lowest PPL will be chosen for initialization. Another solution is to select the mask by the gradient-based one-shot algorithm (Wortsman et al., 2020). It first associates each of the T learned masks mk with a proxy coefficient αi, initially set to 1/T . Then, infer the novel example with the weighted mask m̂ = ∑T k=1 αimk to get the entropy. Further, the one-shot gradient calculated by the entropy for each αi indicates the transferability of each mask. The mask with the highest gradient will be chosen for initialization.

heading: 5 Experimental Settings

heading: 5.1 Datasets
text: To cover all four typical IE task types (including NER, relation extraction, event extraction, and sentiment extraction), we formalize the lifelong UIE benchmark by leveraging 13 IE datasets to construct the task sequence. Specifically, NER tasks include ACE04 (Mitchell et al., 2005), ACE05-Ent (Walker et al., 2006), CoNLL03 (Tjong Kim Sang and De Meulder, 2003); Relation extraction tasks include CoNLL04 (Roth and Yih, 2004), ACE05Rel, SciERC (Luan et al., 2018), NYT (Riedel et al., 2010); Event extraction tasks include CASIE (Satyapanich et al., 2020), ACE05-Evt; AspectBased Sentiment Analysis (ABSA) tasks include SemEval-14 (Pontiki et al., 2014), SemEval-15 (Pontiki et al., 2015), SemEval-16 (Pontiki et al., 2016). Refer to Appendix A for more detail about the datset statistics. For dataset split, we follow the same practice of the relevant prior works (Lu et al., 2022) when using it. As the task order could influence the performance, we create 5 different task orders by random permutation, which are listed in
Table 4.

heading: 5.2 Evaluation Metrics
text: For the evaluation of IE performance, we use the widely adopted span-based offset Micro-F1 as the primary metric following previous work (Lu et al., 2022). Given the generated text spans by our model, we map spans to offsets by finding the first matched offsets that are not already matched in the same SEL hierarchical level. For the evaluation of lifelong learning ability, we denote aT,i as the F1 on the test set of task i after training on task T . The average F1 on all tasks after training on the final task is reported following the common protocol (LopezPaz and Ranzato, 2017; Madotto et al., 2021):
Average = 1
T
T∑
i=1
aT,i (5)
To measure the forgetting during lifelong learning, we use the BWT, which assesses the impact that learning on subsequent tasks has on a previous task. Negative BWT indicates that the model has forgotten some previously acquired knowledge.
BWT = 1 T − 1 T−1∑
i=1
aT,i − ai,i (6)
Another metric is FWT (Ke et al., 2020), which measures how much performance boost has happened to a new task after learning the task, representing the forward knowledge transfer.
FWT = 1
T
T∑
i=1
ai,i − a0,i (7)
where a0,i refers to the performance of training task i individually.

heading: 5.3 Baselines and Training Details
text: We adopt the following methods including recent SOTA as our baselines, which covers both continual learning (CL) and Non-CL methods. (1) continual learning methods: Naive Fine-tuning: fine-tunes the whole model on new task data continually. EWC (Kirkpatrick et al., 2017) is a Regularization-based method that regularizes the change of important model parameters during training. ER (Chaudhry et al., 2019) is a Rehearsalbased method that saves |M | (50 here) samples randomly sampled from the training set of each task i to memory Mi and jointly trains the model on new task data Dk and memory M < k. Individual saves a separate model for each task by fine-tuning
the whole PLM, which clearly has neither forgetting nor knowledge transfer. AdapterCL (Madotto et al., 2021) trains an adapter for each task separately. Similarly, C-PT (Zhu et al., 2022) trains a prompt for each task. L2P (Wang et al., 2022c) trains a prompt pool to transfer task knowledge and a distance-based prompt selection strategy to select the task-specific prompt. (2) Non-CL methods: Multi-task Learning: Fine-tuning the whole model in a multi-task manner using all tasks’ data concurrently. Multi-task Prompt/Adapter Tuning: Prompt/Adapter Tuning in a multi-task manner instead of CL. These multi-task setups are widely accepted as the upper bound of continual learning. As for the LPT, we set the pruning ratio top-c% of LPT as 0.7 in our experiments. For all the prompt tuning methods mentioned above, the
prompt length is set to 20. The parameters of PLM are initialized from UIE-large checkpoints (Lu et al., 2022). We keep all the same hyperparameters for the UIE model reported in their paper. We train the model for 30 epochs per task with batch size 24 on 8 NVIDIA A100 GPUs. All the CL and Non-CL baselines are implemented under the same UIE framework. For the prompt tuning methods, we adopt the deep prompt tuning version (Li and Liang, 2021; Liu et al., 2022b) to allow more per-task capacity.

heading: 6 Results & Analysis

heading: 6.1 Results on Seen Tasks
text: The proposed LPT’s performance is compared with current SOTAs w.r.t six measurements on the aforementioned 13 IE tasks as shown in Table 1. Among
all the continual learning methods, we highlight that our method achieves the highest average F1 (improvements of up to 3% compared with L2P), BWT and FWT with the lowest computation resource usage, which verifies the effectiveness of LPT. While compared with the non-CL methods, we can see the results of LPT are even comparable with Multi-task prompt tuning, which is deemed as the upper bound of prompt tuning methods for continual learning. That could be due to some negative interference among tasks during multitask learning, however in our case, the parameter-isolation mechanism solves that. Note that w.r.t computation resource usage, the parameter-efficient-based methods generally require no memory and only add a small number (around 0.29% to 5.6% ) of additional parameters for each task, largely decreasing the computational and storage overhead. Even so, the LPT shows a remarkable superiority over other methods (only 0.097% and 0.302% on "Tune Param." and "+ Param." respectively). That’s because the saved binary masks for lottery prompts only introduces an approximate overhead of 1/32 of the prompt vectors, which are usually represented by 32-bit float values. Detailed results on each IE task refer to Table 5.

heading: 6.2 Results on Novel Tasks
text: We exclude 4 datasets in the task sequences (with different IE task types) as novel tasks and conduct experiments on them in the few-shot/zero-shot adaptation settings respectively. For the few-shot setting, we conduct 10-shot learning where 10 samples per class are used for the training. While in the zero-shot setting, the sequentially trained model is directly used for testing. We perform the aforementioned PPL-based mask selection method due to its simplicity and effectiveness. Performances are reported in Table 2 for the four evaluation tasks individually and on average. We see LPT could outperform all the CL baselines in few-shot and zero-shot settings, which implies that the mask selection module can make good use of upstream
tasks for novel task generalization. This points to the fact that explicitly transferring knowledge learned from a similar task is critical for systematic adaptation to novel tasks.

heading: 6.3 Ablation Studies

heading: 6.3.1 Sparsity & capacity
text: We choose task order #1 to visualize the model performance and the capacity of total prompts varing with the prompt pruned ratio. As shown in Figure 4, with the decrease of sparsity, the performance of the model (blue bar) presents a trend of first rising and then declining, while the prompt parameter usage (orange line) keeps rising with the decrease of sparsity. It is noteworthy that when the model is trained on a very long sequence of tasks, the prompt capacity could approach full. In this case, our LPT framework is capable of expanding the parameters by introducing new prompt tokens, which shows great flexible.

heading: 6.3.2 Mask Correlations
text: To investigate how LPT reuses parameters over sequential tasks, we visualize all the task-wise binary mask correlations trained from 5 different task sequences in Figure 3. We see LPT shares parameters used for prior tasks with new ones, and is capable
of self-adaptively exploring not-yet-chosen parameters. This demonstrates the effectiveness of LPT in both transferring positive knowledge from similar tasks and automatically exploring new patterns for dissimilar tasks.

heading: 7 Conclusions
text: In this paper, we study a lifelong learning paradigm for UIE systems, which we regard as an important step towards general IE intelligence. We propose a novel parameter-efficient framework, i.e., Lottery Prompt Tuning (LPT), to achieve positive knowledge transfer, catastrophic forgetting prevention, and rapid generalization. Experimental results validate the capability of our method on three settings.
Limitations
Though our method does not require iterative retraining, pruning, and rewinding process, one question still remains under-explored: how to selfadaptively find the optimal sparsity instead of trial training, which can boost the training efficiency. Also, we plan to further investigate the effectiveness of Lottery Prompt Tuning in other scenarios, including the multi-task learning (He et al., 2022b), prompt ensembling (Lester et al., 2021b), etc. Furthermore, the proposed learning method should be compatible with other parameter-efficient finetuning methods, such as Adapter tuning (Houlsby et al., 2019) and LoRA (Hu et al., 2021). We leave these for future research.

heading: A Dataset Statistics

heading: B Detailed results of task-incremental setting
text: Here we present detailed experimental results on all 13 IE tasks across different task types including NER, relation extraction, event extraction and sentiment extraction. As shown in Table 5, the proposed LPT outperforms all competitive baselines.
Ta sk
or de
r Ta
sk in
or de r 1 ab sa :1 4r es en tit y: m rc _a ce 05 ev en t:o ne ie _a ce 05 _e n_ ev en t ab sa :1 6r es ab sa :1 4l ap ab sa :1 5r
es re
la tio
n: ac
e0 5-
re l
en tit
y: m
rc _a
ce 04
re la
tio n:
N Y
T ev
en t:c
as ie
en tit
y: co
nl l0
3 re
la tio
n: sc
ie rc
re la
tio n:
co nl
l0 4
2 ev
en t:o
ne ie
_a ce
05 _e
n_ ev
en t
en tit
y: m
rc _a
ce 05
re la
tio n:
N Y
T ab
sa :1
5r es
ev en
t:c as
ie re
la tio
n: co
nl l0
4 ab
sa :1
4r es
ab sa
:1 6r
es en
tit y:
co nl
l0 3
re la
tio n:
sc ie
rc ab
sa :1
4l ap
en tit
y: m
rc _a
ce 04
re la
tio n:
ac e0
5- re l 3 re la tio n: co nl l0 4 re la tio n: sc ie rc en tit y: m rc _a ce 05 en tit y: m rc _a ce 04 ev en t:o ne ie _a ce 05 _e n_ ev en t ab sa :1 5r es re la tio n: ac e0 5- re l ab sa :1 4r es ev en t:c as ie re la tio n: N Y T en tit y: co nl l0 3 ab sa :1 6r es ab sa :1 4l ap 4 ab sa :1 5r es ev en t:o ne ie _a ce 05 _e n_ ev en t re la tio n: sc ie rc en tit y: m rc _a ce 05 ab sa :1 4l ap en tit y: co nl l0 3 re la tio n: ac e0 5- re l re la tio n: co nl l0 4 ev en t:c as ie en tit y: m rc _a ce 04 ab sa :1 4r es re la tio n: N Y T ab sa :1 6r es 5 en tit y: m rc _a ce 05 ev en t:o ne ie _a ce 05 _e n_ ev en t ab sa :1 5r es re la tio n: co nl l0 4 ab sa :1 4r es ev en t:c as ie re la tio n: N Y T re la tio n: sc ie rc ab sa :1 6r es en tit y: co nl l0 3 re la tio n: ac e0 5- re l ab sa :1 4l ap en tit y: m rc _a ce 04
Ta bl
e 4:
Ta sk
O rd
er ac
ro ss
13 IE
ta sk
s.
M od
el /T
as ks
C L
-m et
ho ds
N on
-C L
m et
ho ds
Fi ne
-t un
in g
E W
C E
R A
da pt
er C
L C
-P T
L 2P
L PT
(o ur
s) In
di vi
du al
M T-
PT M
TA
T M
TFT
ab sa
/1 4r
es 63
.7 8
61 .8
6 75
.2 7
60 .0
7 71
.3 3
69 .6
4 74
.9 1
74 .3
3 76
.1 1
77 .2
8 76
.7 6
en tit
y/ m
rc _a
ce 05
52 .2
9 45
.7 5
76 .1
4 85
.3 8
82 .6
5 86
.3 0
87 .3
6 85
.1 7
85 .8
9 88
.9 7
87 .5 9 ev en t/o ne ie _a ce 05 _e n_ ev en t 12 .9 7 5. 10 51 .8 2 70 .3 6 51 .7 0 72 .1 8 73 .5 2 53 .6 2 67 .8 9 70 .4 3 72 .8 7 ev en t/o ne ie _a ce 05 _e n_ ev en t( tr ig ge r) 26 .6 4 19 .2 5 62 .1 1 90 .5 2 70 .7 3 89 .6 0 89 .4 5 71 .5 9 88 .3 1 90 .2 9 90 .6 1 ab sa /1 6r es 65 .6 0 63 .2 9 78 .5 8 55 .6 7 69 .9 0 74 .3 8 78 .6 8 73 .9 1 76 .8 9 77 .8 1 79 .9 6 ab sa /1 4l ap 52 .2 7 50 .0 1 69 .1 8 43 .2 2 62 .1 8 63 .4 8 70 .1 8 62 .4 6 68 .8 1 68 .2 8 73 .5 2 ab sa /1 5r es 76 .2 0 72 .0 2 90 .2 1 38 .0 7 60 .1 5 63 .3 0 66 .7 1 64 .2 6 84 .1 8 84 .6 0 93 .2 9 re la tio n/ ac e0 5- re l 24 .2 3 15 .2 1 46 .8 0 66 .1 8 61 .5 4 71 .2 5 73 .8 7 66 .3 1 74 .0 5 76 .6 1 77 .9 1 en tit y/ m rc _a ce 04 49 .2 3 42 .1 3 71 .1 6 84 .2 9 83 .7 8 87 .4 4 88 .8 8 85 .7 8 86 .5 4 88 .3 8 89 .3 5 re la tio n/ N Y T 50 .4 6 43 .4 9 79 .1 6 87 .0 6 88 .8 8 85 .9 0 86 .7 2 85 .4 1 86 .0 1 86 .8 6 88 .0 3 ev en t/c as ie 23 .2 2 17 .1 8 57 .2 9 68 .4 5 56 .0 3 69 .4 2 72 .7 6 58 .9 7 70 .3 7 73 .4 7 74 .5 1 ev en t/c as ie (t ri gg er ) 36 .5 2 29 .6 0 65 .4 6 69 .6 1 61 .9 3 69 .3 2 74 .7 7 67 .3 2 70 .2 5 75 .9 7 78 .1 2 en tit y/ co nl l0 3 73 .0 0 69 .0 9 86 .3 4 95 .2 0 91 .9 4 95 .4 4 95 .7 0 92 .3 4 93 .8 1 94 .4 0 94 .9 2 re la tio n/ sc ie rc 12 .4 3 9. 15 43 .6 5 27 .8 0 30 .7 8 39 .9 0 46 .0 9 34 .6 9 47 .0 8 46 .5 1 52 .0 5 re la tio n/ co nl l0 4 25 .1 3 18 .1 3 68 .1 6 41 .7 2 68 .9 8 66 .6 2 74 .1 0 72 .2 6 75 .4 2 75 .2 4 77 .7 8 A ve ra ge 42 .9 3 37 .4 2 68 .0 9 65 .5 7 67 .5 0 73 .6 1 76 .9 1 69 .9 0 76 .7 7 78 .3 4 80 .4 8
Ta bl
e 5:
T he
fin al
m od
el pe
rf or
m an
ce on
al l1
3 IE
ta sk
s af
te rb
ei ng
se qu
en tia
lly tr
ai ne
d. O
ur m
od el
L PT
si gn
ifi ca
nt ly
ou tp
er fo
rm s
ot he
rb as
el in
es ."
M T-
PT "
m ea
ns M
ul ti-
Ta sk
Pr om
pt Tu
ni ng
." M
TA
T "
m ea
ns M
ul ti-
Ta sk
A da
pt er
Tu ni
ng ."
M T-
FT "
m ea
ns M
ul ti-
Ta sk
Fi ne
-T un
in g.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section Limitations
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section Abstract; Section1 Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Not applicable. Left blank.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? Section

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section Implementation Details
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Left blank.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave these for future research