Venue: NAACL 2024

# Lifelong Event Detection with

Embedding Space Separation and Compaction

 Chengwei Qin\({}^{\dagger}\), Ruirui Chen\({}^{\star}\), Ruochen Zhao\({}^{\dagger}\), Wenhan Xia\({}^{\bullet}\), Shafiq Joty\({}^{\dagger}\)\({}^{\star}\)

\({}^{\dagger}\)Nanyang Technological University \({}^{\bullet}\)Princeton University \({}^{\bullet}\)Salesforce Research

\({}^{\bullet}\)Institute of High Performance Computing (IHPC),

Agency for Science, Technology and Research (A*STAR)

1 Fusionopolis Way, #16-16 Connexis, Singapore 138632, Republic of Singapore

{chengwei003@e.ntu, ruochen002@ee.ntu, srjoty@ntu }.edu.sg

chen_ruirui@ihpc.a-star.edu.sg, wxia@princeton.edu

###### Abstract

To mitigate forgetting, existing lifelong event detection methods typically maintain a memory module and replay the stored memory data during the learning of a new task. However, the simple combination of memory data and new-task samples can still result in substantial forgetting of previously acquired knowledge, which may occur due to the potential overlap between the feature distribution of new data and the previously learned embedding space. Moreover, the model suffers from overfitting on the few memory samples rather than effectively remembering learned patterns. To address the challenges of forgetting and overfitting, we propose a novel method based on embedding space separation and compaction. Our method alleviates forgetting of previously learned tasks by forcing the feature distribution of new data away from the previous embedding space. It also mitigates overfitting by a memory calibration mechanism that encourages memory data to be close to its prototype to enhance intra-class compactness. In addition, the learnable parameters of the new task are initialized by drawing upon acquired knowledge from the previously learned task to facilitate forward knowledge transfer. With extensive experiments, we demonstrate that our method can significantly outperform previous state-of-the-art approaches.

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)_, pages 594–602

June 16-21, 2024 02024 Association for Computational Linguistics

## 1 Introduction

Event detection (ED) aims to detect the event type of trigger words in a given sentence, _e.g.,_ extracting the event type _injure_ from the trigger word _scaled_ in text "_He was scadded by hot water_". Traditional ED methods typically consider a fixed pre-defined set of event types [Chen et al., 2015; Nguyen et al., 2016; Huang and Ji, 2020; Chen et al., 2024]. However, as the environment and data distributions change in real scenarios, the model might face challenges in handling rapidly emerging new types [Lu et al., 2022].

A more practical setting is lifelong event detection or led[Cao et al., 2020], where the model learns event knowledge from a sequence of tasks with different sets of event types. In led, the model is expected to retain and accumulate knowledge when learning new tasks, which is challenging due to _catastrophic forgetting_[McCloskey and Cohen, 1989] of previously acquired knowledge. Existing methods [Cao et al., 2020; Yu et al., 2021] for mitigating forgetting in led typically maintain a memory that saves a few key samples of previous tasks, which are then combined with new data for training. Recently, Liu et al. liu2022 introduce Episodic Memory Prompts (EMP) that leverages soft prompts to remember learned event types, achieving state-of-the-art performance on led.

Despite its effectiveness, EMP has two key limitations. First, simply combining new data and memory samples for training can still result in forgetting as the feature distribution of new data might overlap with the previously learned embedding space (see Appendix A.1). Second, it may overfit on a few memory samples after frequent replays rather than effectively retaining learned patterns.

To address the above limitations of EMP, in this paper, we introduce a novel method based on Embedding space Separation and COmpaction (ESCO) for led. In particular, we propose a margin-based loss that forces the feature distribution of new event types away from the learned embedding space to alleviate _forgetting_. Inspired by Han et al. han2020, we introduce a memory calibration mechanism to encourage memory data to be close to its prototype to avoid _overfitting_ on the few memory samples. In addition, the learnable parameters of the new task are initialized using those of the previously learned task to facilitate _forward knowledge transfer_, which is as important for lifelong learning as preventing forgetting [Ke et al., 2020; Qin et al., 2023a,b]. The empirical results show that our method significantly outperforms

previous state-of-the-art approaches. In summary, our main contributions are:

* We propose ESCO, a novel method based on embedding space separation and compaction to mitigate forgetting and overfitting in led.
* With extensive experiments and analysis, we demonstrate the effectiveness of our method compared to existing ones.

## 2 Problem Formulation

led involves learning from a stream of event detection tasks \(\mathbb{T}=(\mathcal{T}^{1},\ldots,\mathcal{T}^{n})\), where each task \(\mathcal{T}^{k}\) has its own training set \(\mathcal{D}_{\text{train}}^{k}\), validation set \(\mathcal{D}_{\text{valid}}^{k}\), and test set \(\mathcal{D}_{\text{test}}^{k}\). For every input text \(x^{i}\) in \(\mathcal{D}^{k}\), it contains a set of target spans \(\{\overline{x}_{t}^{j}\}\) and their corresponding labels \(y_{t}^{i}\) which belong to the event type set \(\mathcal{C}^{k}\) of task \(\mathcal{T}^{k}\). Note that the event type sets of different tasks are non-overlapping.

After the training on \(\mathcal{D}_{\text{train}}^{k}\), the model is expected to perform well on all the \(k\) tasks that it has learned and will be evaluated on the combined test set \(\hat{\mathcal{D}}_{\text{test}}^{k}=\cup_{i=1}^{k}\mathcal{D}_{\text{test}}^{ k}\) consisting of all known event types \(\hat{\mathcal{C}}^{k}=\cup_{i=1}^{k}\mathcal{C}^{i}\). During the learning, a memory module \(\mathcal{M}\) which stores a few key samples of previous tasks is maintained to overcome the forgetting problem.

## 3 Embedding Space Separation and Compaction

When learning a new task \(\mathcal{T}^{k}\), following Liu et al. (2022), we first initialize a set of soft prompts \(\mathcal{P}^{k}=\{p_{1}^{k},...,p_{|\mathcal{C}^{k}|}^{k}\}\) where \(\mathcal{C}^{k}\) is the event type set of \(\mathcal{T}^{k}\). The accumulated prompts \(\mathcal{Q}^{k}=[\mathcal{P}^{1},...,\mathcal{P}^{k}]\) until time step \(k\) are then combined with the input text \(x^{i}\) to obtain the contextual representations using a frozen BERT Devlin et al. (2019):

\[[\mathbf{x}^{i},\mathbf{Q}^{k}]=\text{BERT}([x^{i},\mathcal{Q}^{k}]) \tag{1}\]

where \(\mathbf{x}^{i}\) and \(\mathbf{Q}^{k}\) are the representations of \(x^{i}\) and \(\mathcal{Q}^{k}\), respectively. To facilitate _forward knowledge transfer_ Qin and Joty (2022), we initialize soft prompts \(\mathcal{P}^{k}\) of the new task using learned prompts \(\mathcal{P}^{k-1}\) of the previous task. For the first task \(\mathcal{T}^{1}\), we initialize each event type prompt \(p_{i}^{1}\) in \(\mathcal{P}^{1}\) using its corresponding name.

To predict the event type of the span \(\overline{x}_{t}^{i}\), we concatenate the representations corresponding to the start and end token and obtain the logits over all learned types through a feed-forward network (FFN) as well as a linear layer:

\[Z_{t}^{i}=\text{Linear}(\text{FFN}([\mathbf{s}_{m}^{i},\mathbf{x}_{n}^{i}])) \tag{2}\]

where \(\overline{\mathbf{x}}_{t}^{i}=\text{FFN}([\mathbf{s}_{m}^{i},\mathbf{x}_{n}^{ i}])\) is the span representation, \(m\) and \(n\) denote the start and end index of the span, respectively. Following Liu et al. (2022), to entangle span representations with soft prompts, the probability distribution over all prompts is calculated as \(Z_{q}=\text{FFN}(\mathbf{Q}^{k})\cdot\overline{\mathbf{x}}_{t}^{i}\), where \(\cdot\) is the inner product. \(Z_{q}\) is then combined with \(Z_{t}^{i}\) to optimize the cross entropy loss:

\[\mathcal{L}_{\text{new}}=-\sum_{(\overline{x}_{t}^{i},y_{t}^{i})\in\mathcal{ D}_{\text{train}}^{k}}\text{CE}(Z_{t}^{i}+Z_{q},y_{t}^{i}) \tag{3}\]

After learning the previous task \(\mathcal{T}^{k-1}\), we select the top-\(l\) most informative training examples for each event type in \(\mathcal{C}^{k-1}\) using the herding algorithm Welling (2009), which are then saved in the memory module \(\mathcal{M}\) for replay to mitigate forgetting. Similar as Eq. 3, the training objective for memory replay when learning \(\mathcal{T}^{k}\) is:

\[\mathcal{L}_{\text{mem}}=-\sum_{(\overline{x}_{t}^{i},y_{t}^{i})\in\mathcal{ M}}\text{CE}(Z_{t}^{i}+Z_{q},y_{t}^{i}) \tag{4}\]

However, the simple combination of \(\mathcal{L}_{\text{new}}\) and \(\mathcal{L}_{\text{mem}}\) can still result in substantial forgetting of acquired knowledge due to the potential overlap between the feature distribution of new event types and the previously learned embedding space (see Appendix A.1). To ensure that the new feature distribution is away from the learned embedding space, we design a _margin-based_ loss, which decreases the similarity scores between new samples and prototypes (see Eq. 8 for the calculation of prototypes) of learned event types:

\[\mathcal{L}_{\text{sim}}=\sum_{(\overline{x}_{t}^{i},y_{t}^{i})\in\mathcal{D}_ {\text{train}}^{k}}\sum_{\mathbf{e}_{t}\in\mathcal{E}^{k-1}}\max(0,g(\overline {\mathbf{x}}_{t}^{i},\mathbf{e}_{t})-m_{1}) \tag{5}\]

where \(\mathcal{E}^{k-1}\) is the prototype set of previous \(k-1\) tasks, \(g(,,)\) is the similarity function (cosine similarity) and \(m_{1}\) is the margin for \(\mathcal{L}_{\text{sim}}\). Note that \(\mathcal{L}_{\text{sim}}\) is different from metric learning or contrastive learning Qin and Joty (2022) which typically considers both positive and negative pairs. \(\mathcal{L}_{\text{sim}}\) only includes negative pairs while ignoring positive ones as our goal in designing \(\mathcal{L}_{\text{sim}}\) is to _separate_ the new feature distribution and the learned embedding space 

As the size of memory \({\cal M}\) is typically small, the model is prone to overfit on the few memory samples after frequent replays, making learned distributions distorted. To effectively recover from distorted learned distributions, we introduce a _memory calibration_ mechanism inspired by Han et al. (2020). Specifically, for each memory sample in \({\cal M}\), we encourage it to be close to its corresponding prototype to improve the intra-class compactness of learned distributions. More formally,

\[\mathcal{L}_{\text{cal}}=-\sum_{(\overline{x}^{i}_{t},y^{i}_{t})\in{\cal M}} \log\frac{\exp g(\overline{\mathbf{x}}^{i}_{t},\mathbf{e}_{t})}{\sum_{j=1}^{| E^{k-1}|}\exp g(\overline{\mathbf{x}}^{i}_{t},\mathbf{e}_{j})} \tag{6}\]

where \(\mathbf{e}_{l}\) is the prototype of \(y^{i}_{t}\). The _total_ loss for learning on \({\cal T}^{k}\) is defined as:

\[\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{new}}+\lambda_{1}\mathcal{L}_{ \text{sim}}+\lambda_{2}(\mathcal{L}_{\text{mem}}+\mathcal{L}_{\text{cal}}) \tag{7}\]

where \(\lambda_{1}\) and \(\lambda_{2}\) are loss weights.

After learning \({\cal T}^{k}\) and selecting memory data for \({\cal T}^{k}\), we use the memory \({\cal M}\) to calculate prototypes of all learned event types in \({\cal C}^{k}\). Specifically, for each event type \(e_{j}\) in \({\cal C}^{k}\), we obtain its prototype \(\mathbf{e}_{j}\) by averaging the span representations of all samples labeled as \(e_{j}\) in \({\cal M}\) as follows:

\[\mathbf{e}_{j}=\frac{1}{|\mathcal{M}_{e_{j}}|}\sum_{(\overline{x}^{i}_{t},y^{ i}_{t})\in\mathcal{M}_{e_{j}}}\overline{\mathbf{x}}^{i}_{t} \tag{8}\]

where \(\mathcal{M}_{e_{j}}=\{(\overline{x}^{i}_{t},y^{i}_{t})|(\overline{x}^{i}_{t}, y^{i}_{t})\in{\cal M},y^{i}_{t}=e_{j}\}\).

## 4 Experiment

### Experimental Setup

We conduct experiments on two representative event detection datasets in our work: ACE05 Doddington et al. (2004) and MAVEN Wang et al. (2020). Following Liu et al. (2022), we divide each dataset into a sequence of 5 tasks with non-overlapping event type sets to form a _class-incremental_ setting. After learning \({\cal T}^{k}\), the model is evaluated on the combined test set \(\hat{\mathcal{D}}^{k}_{\text{test}}=\cup_{i=1}^{k}\mathcal{D}^{i}_{\text{test}}\) of all seen tasks. As the task order might influence the model performance, we run experiments for each dataset 5 times with different task order permutations and report the average results. More details of the experimental setup are in Appendix A.2.

### Methods Compared

We compare our approach with the following methods: (1) **Fine-tuning** tunes the model only on new data without memory; (2) **BiC**Wu et al. (2019) introduces a bias correction layer to improve lifelong learning performance; (3) **KCN**Cao et al. (2020) designs prototype enhanced retrospection and hierarchical distillation to alleviate semantic ambiguity and class imbalance; (4) **KT**Yu et al. (2021) proposes to transfer knowledge between related types; (5) **EMP**Liu et al. (2022) leverages typecific soft prompts to remember learned event knowledge; and (6) **Multi-task learning (MTL)** simultaneously trains the model on all data, serving as the _upper bound_ in led.

### Main Results

We report the F1 score of different methods at each time step in Table 1. From the results, we can observe that ESCO significantly outperforms previous baselines on both datasets, demonstrating its superiority. Simply fine-tuning the model on new data without memory replay results in poor performance due to severe forgetting of learned knowledge. Although BiC, KCN and KT could alleviate forgetting to some extent, there is still a large performance drop after learning all tasks. EMP achieves better performance because the type-specific soft prompts help retain previously acquired knowledge. However, it does not necessarily ensure large distances among feature distributions of different event types, and easily overfits on the memory samples. Our proposed ESCO outperforms EMP by a large margin through embedding space separation and compaction. To verify its effectiveness, we visualize the embedding spaces of EMP and ESCO on ACE05 in Fig. 1. Specifically, we randomly select 6 event types from different learning stages and visualize their test data using t-SNE Van der Maaten and Hinton (2008). The comparison demonstrates that ESCO could achieve larger inter-class distances and better intra-class compactness in the embedding space.

### Ablation Study

To analyze the contribution of different components of ESCO, we conduct several ablations. Specifically, we investigate three variants of ESCO: (_a_) without the margin-based loss (_w.o._ margin), (_b_) removing the memory calibration mechanism (_w.o._ calibration), and (_c_) without forward knowl 

edge transfer (FKT) (_w.o._ FKT). The results of different ablations after learning all tasks are reported in Table 2. We can see that all components contribute to overall performance. The margin-based loss yields about **1.25%** performance boost as it can bring feature distribution of new data away from the learned embedding space. The memory calibration mechanism improves the F1 score by **1.47**%, which demonstrates the necessity of improving intra-class compactness of learned distributions. The adoption of forward knowledge transfer leads to **0.80**% improvement, indicating that it can indeed transfer useful learned knowledge to facilitate the learning of new tasks.

### Further Analysis

Quantify Knowledge TransferFollowing Lopez-Paz and Ranzato (2017), we report the backward transfer (BWT) and forward transfer (FWT) of EMP and ESCO after learning all tasks in Table 3. From the comparison, we can observe that ESCO outperforms EMP by a large margin in terms of BWT and FWT on both datasets, demonstrating its effectiveness.

In addition, we show results of a different backbone model (RoBERTa Liu et al. (2019)), the effect of memory size, results of different memory sample selection methods, the comparison with the contrastive loss in Qin and Joty (2022), and a case study of the model output in Appendix A.5\(\sim\) A.9, respectively.

## 5 Related Work

Lifelong event detection(led) aims to continually learn from a sequence of event detection tasks

Figure 1: Comparison between the embedding spaces of EMP (left) and ESCO (right). Colors represent different event types with numbers being the event indexes. Compared with EMP, ESCO shows larger inter-class distances, _e.g.,_ the distance between 13 and 18, and better intra-class compactness (circled regions).

 

with different sets of event types. Cao et al. (2020) propose KCN which addresses the semantic ambiguity and data imbalance problems in led by prototype enhanced retrospection and hierarchical distillation. KT Yu et al. (2021) encourages bi-directional knowledge transfer between old and new event types. Liu et al. (2022) introduce EMP to retain previously learned task-specific event knowledge through soft prompts. In contrast to previous works, we innovate on the methodology by imposing further constraints in the embedding space to mitigate forgetting and overfitting.

## 6 Conclusion

In this work, we have introduced embedding space separation and compaction (ESCO) for lifelong event detection (led). ESCO imposes novel feature constraints in the embedding space to alleviate forgetting and overfitting problems. It initializes the learnable parameters for the new task by inheriting those from the previously learned task to facilitate forward knowledge transfer. With extensive experiments and analysis, we have demonstrated that ESCO significantly outperforms previous methods. For future work, we are interested in exploring ESCO in a meta-learning paradigm for led.

## Limitations

Although effective, ESCO also has some limitations. For example, ESCO mainly focuses on the setting where each task has enough training data. Few-shot learning and in-context learning have been receiving increasing interest within the community. We leave how to explore lifelong event detection in few-shot settings as future work. Besides, as large language models (LLMs) have shown impressive performance on a variety of tasks Brown et al. (2020); Ouyang et al. (2022); Touvron et al. (2023); Qin et al. (2023c,d,e); Achiam et al. (2023); Xia et al. (2024); Ding et al. (2023), a further improvement could be to explore lifelong event detection with LLMs.

## References

* Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
* Brown et al. (2019) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _arXiv preprint arXiv:2005.14165_.
* Cao et al. (2020) Pengfei Cao, Yubo Chen, Jun Zhao, and Taifeng Wang. 2020. Incremental event detection via knowledge consolidation networks. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 707-717, Online. Association for Computational Linguistics.
* Chen et al. (2024) Ruirui Chen, Chengwei Qin, Weifeng Jiang, and Dongkyu Choi. 2024. Is a large language model a good annotator for event extraction? In _Proceedings of the AAAI Conference on Artificial Intelligence_, 16, pages 17772-17780.
* Chen et al. (2015) Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multi-pooling convolutional neural networks. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 167-176, Beijing, China. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Ding et al. (2023) Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. 2023. Is GPT-3 a good data annotator? In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 11173-11195, Toronto, Canada. Association for Computational Linguistics.
* tasks, data, and evaluation. In _Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04)_, Lisbon, Portugal. European Language Resources Association (ELRA).
* Han et al. (2020) Xu Han, Yi Dai, Tianyu Gao, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2020. Continual relation learning via episodic memory activation and reconsolidation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6429-6440, Online. Association for Computational Linguistics.
* Huang and Ji (2020) Liffu Huang and Heng Ji. 2020. Semi-supervised new event type induction and event detection. In _Proceedings of the 2020 Conference on Empirical Methods 

in Natural Language Processing (EMNLP)_, pages 718-724, Online. Association for Computational Linguistics.
* Ke et al. (2020) Zixuan Ke, Bing Liu, and Xingchang Huang. 2020. Continual learning of a mixed sequence of similar and dissimilar tasks. In _Advances in Neural Information Processing Systems_, volume 33, pages 18493-18504. Curran Associates, Inc.
* Liu et al. (2022) Minqian Liu, Shiyu Chang, and Lifu Huang. 2022. Incremental prompting: Episodic memory prompt for lifelong event detection. In _Proceedings of the 29th International Conference on Computational Linguistics_, pages 2157-2165, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Lopez-Paz and Ranzato (2017) David Lopez-Paz and Marc'Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. _Advances in neural information processing systems_, 30.
* Lu et al. (2022) Zhou Lu, Wenhan Xia, Sanjeev Arora, and Elad Hazan. 2022. Adaptive gradient methods with local guarantees. _arXiv preprint arXiv:2203.01400_.
* McCloskey and Cohen (1989) Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In _Psychology of learning and motivation_, volume 24, pages 109-165. Elsevier.
* Nguyen et al. (2016) Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. 2016. Joint event extraction via recurrent neural networks. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 300-309, San Diego, California. Association for Computational Linguistics.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in Neural Information Processing Systems_, 35:27730-27744.
* Qin and Joty (2022a) Chengwei Qin and Shafiq Joty. 2022a. Continual few-shot relation learning via embedding space regularization and data augmentation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2776-2789, Dublin, Ireland. Association for Computational Linguistics.
* Qin and Joy (2022b) Chengwei Qin and Shafiq Joty. 2022b. LFPT5: A unified framework for lifelong few-shot language learning based on prompt tuning of t5. In _International Conference on Learning Representations_.
* Qin et al. (2023a) Chengwei Qin, Shafiq Joty, and Chen Chen. 2023a. Lifelong sequence generation with dynamic module expansion and adaptation. _arXiv preprint arXiv:2310.09886_.
* Qin et al. (2023b) Chengwei Qin, Shafiq Joty, Qian Li, and Ruochen Zhao. 2023b. Learning to initialize: Can meta learning improve cross-task generalization in prompt tuning? In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 11802-11832, Toronto, Canada. Association for Computational Linguistics.
* Qin et al. (2023c) Chengwei Qin, Wenhan Xia, Fangkai Jiao, and Shafiq Joty. 2023c. Improving in-context learning via bidirectional alignment. _arXiv preprint arXiv:2312.17055_.
* Qin et al. (2023d) Chengwei Qin, Aston Zhang, Anirudh Dagar, and Wenming Ye. 2023d. In-context learning with iterative demonstration selection. _arXiv preprint arXiv:2310.09881_.
* Qin et al. (2023e) Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michiihiro Yasunaga, and Diyi Yang. 2023e. Is chatgpt a general-purpose natural language processing task solver? _arXiv preprint arXiv:2302.06476_.
* Sun et al. (2022) Qing Sun, Fan Lyu, Fanhua Shang, Wei Feng, and Liang Wan. 2022. Exploring example influence in continual learning. In _Advances in Neural Information Processing Systems_.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. LLama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. _Journal of machine learning research_, 9(11).
* Wang et al. (2020) Xiaozhi Wang, Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li, Yankei Lin, and Jie Zhou. 2020. MAVEN: A Massive General Domain Event Detection Dataset. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1652-1671, Online. Association for Computational Linguistics.
* Welling (2009) Max Welling. 2009. Herding dynamical weights to learn. In _Proceedings of the 26th Annual International Conference on Machine Learning_, pages 1121-1128.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delagnue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Sheifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing.

 

In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Wu et al. (2019) Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. 2019. Large scale incremental learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 374-382.
* Xia et al. (2024) Wenhan Xia, Chengwei Qin, and Elad Hazan. 2024. Chain of lora: Efficient fine-tuning of language models via residual learning. _arXiv preprint arXiv:2401.04151_.
* Yu et al. (2021) Pengfei Yu, Heng Ji, and Prem Natarajan. 2021. Lifelong event detection with knowledge transfer. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 5278-5290, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

## Appendix A Appendix

### Overlap of Feature Distributions

Fig. 2 shows an example of the overlap between feature distributions of event types at different learning stages.

### Implementation Details

All methods are implemented with PyTorch/Transformers library Wolf et al. (2020). For hyperparameters, we mainly follow the settings in Liu et al. (2022) to have a fair comparison. We adopt \(-0.1\) for the margin value \(m_{1}\) so that the similarity score between a new sample and the prototype of a previously learned event type could be optimized to a negative number, _i.e.,_ large inter-class distance. We set the weight \(\lambda_{1}\) to \(0.1\) so that its corresponding loss \(\mathcal{L}_{\text{sim}}\) has roughly the same order of magnitude as other losses. The loss weight \(\lambda_{2}\) is set to \(\frac{s}{k+s}\), where \(k\) is the number of target spans in the current batch and \(s\) is equal to 50 following Liu et al. (2022). For each task, we train the model for 20 epochs with early stopping.

For the state-of-the-art EMP Liu et al. (2022), we reproduce the results using its open-source code and the same environment. For our method, we use the same environment and shared hyperparameters as EMP. For other baselines, we reuse the results in Liu et al. (2022). There are mainly two reasons: (_a_) They perform much worse than EMP, _i.e.,_ they are not primary comparison approaches in our work; and (_b_) EMP reports different baseline results from Yu et al. (2021), indicating different settings. However, EMP does not provide details on how to obtain baseline results. As we use the same setting as EMP, we decide to reuse its results for other baselines.

### Results with Variance

We show results with variance for EMP and ESCO in Table 5.

### Detailed Results for Different Task Orders

Table 6 reports detailed results for different task orders.

Figure 3: The performance of ESCO and EMP with different memory sizes.

\begin{table}
\begin{tabular}{l|c c c c c} \hline \hline  & \multicolumn{5}{c}{MAVEN} \\ \hline Task index & 1 & 2 & 3 & 4 & 5 \\ \hline EMP & **67.1** & 58.3 & 55.7 & 53.2 & 52.9 \\ ESCO & **67.1** & **60.8** & **59.0** & **55.3** & **55.1** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance comparison between EMP and ESCO on MAVEN using RoBERTa as the backbone.

Figure 2: Overlap between feature distributions of event types at different learning stages, _e.g.,_ circled regions. Colors represent different event types with numbers being the event indexes.

 



### Different Backbone Model

To investigate the generalization ability of ESCO, we further conduct experiments on MAVEN using RoBERTa Liu et al. (2019) backbone. The F1 scores of EMP and ESCO at each time step are reported in Table 4, which verify that ESCO can indeed generalize to different models.

### Effect of Memory Size

Following Liu et al. (2022), we select 20 samples as memory data for each event type. To investigate whether different memory sizes influence the performance gain of ESCO, we conduct controlled experiments on ACE05 with memory size \(\{5,10,15,25,30,35\}\). The performance comparison between ESCO and EMP is shown in Fig. 3. We can observe that ESCO consistently outperforms EMP by a large margin with different memory sizes, demonstrating its robustness.

### Different Memory Sample Selection Approaches

Following EMP Liu et al. (2022), we use the herding algorithm Welling (2009) to select memory samples. To validate whether different memory sample selection approaches influence the performance gain of ESCO, we replace the herding algorithm of EMP and ESCO with _example influence_Sun et al. (2022) for memory selection. We randomly select three sequences for experiments and report the performance comparison between EMP and ESCO in Table 7. We can see that: (_a_) ESCO consistently outperforms EMP in different cases, demonstrating its effectiveness; and (_b_) herding algorithm performs better than example influence, justifying our choice.

### Comparison with the Contrastive Loss

As mentioned in SS3, our designed margin-based loss \(\mathcal{L}_{\text{sim}}\) is different from the contrastive loss as \(\mathcal{L}_{\text{sim}}\) only includes negative pairs while ignoring positive ones. To further demonstrate its superiority, we replace it with the contrastive loss in Qin

\begin{table} max width= 

\end{table}
Table 6: F1 score (%) of 5 runs with different task orders on two datasets. For every order, the upper row shows the performance of EMP and the lower row is the result of ESCO.

\begin{table} max width= 

\end{table}
Table 8: Performance comparison between ESCO and ESCO\({}_{\text{con}}\).

\begin{table} max width= 

\end{table}
Table 5: F1 score (%) and variance of EMP and ESCO at every time step on two datasets.



and Joty, 2022a), namely ESCO\({}_{\text{con}}\). We use the same sequences as Appendix A.7 for experiments and report the results of ESCO and ESCO\({}_{\text{con}}\) in Table 8, which verify the effectiveness of \(\mathcal{L}_{\text{sim}}\).

### Case Study

We select MAVEN as a representative task and show several example outputs in Table 9. Compared with EMP, ESCO is able to retain more precise and fine-grained event knowledge, _e.g.,_ ESCO can successfully detect the event type _Escaping_ from the target span _evacuated_ while EMP is confused by another semantically similar type _Removing_.