Continual Event Extraction with Semantic Confusion Rectification
Authors: Zitao Wang
Xinyi Wang
Wei Hu
Venue: EMNLP 2023 Main (2023)
Keywords: Event extraction
Continual learning
Semantic confusion
Knowledge transfer

Abstract: We study continual event extraction, which aims to extract incessantly emerging event information while avoiding forgetting. We observe that the semantic confusion on event types stems from the annotations of the same text being updated over time. The imbalance between event types even aggravates this issue. This paper proposes a novel continual event extraction model with semantic confusion rectification. We mark pseudo labels for each sentence to alleviate semantic confusion. We transfer pivotal knowledge between current and previous models to enhance the understanding of event types. Moreover, we encourage the model to focus on the semantics of long-tailed event types by leveraging other associated types. Experimental results show that our model outperforms state-of-the-art baselines and is proficient in imbalanced datasets.

Continual Event Extraction with Semantic Confusion Rectification
Zitao Wang Xinyi Wangi Wei Hut.
t State Key Laboratory for Novel Software Technology, Nanjing University, China 3 National Institute of Healtheare Data Science, Nanjing University, China Z zwwagg. njuegmall....m com xywang. nongg.jjuue nju@gmail com, whu@nju.edee edu. cn
Abstract
"Die"
Emerges
We study continual event extraction, which aims to extract incessantly emerging event in- formation while avoiding forgetting. We ob- serve that the semantic confusion event types stems from the annotations of the same text being updated over time. The imbalance between event types even aggravates this is- sue. This paper proposes a novel continual event extraction model with semantic confu- sion rectification. We mark pseudo labels for each sentence to alleviate semantic confusion. We transfer pivotal knowledge between current and previous models to enhance the understand- ing of event types. Moreover, we encourage the model to focus the semantics of long- tailed event types by leveraging other associ- ated types. Experimental results show that our model outperforms state-of-the-ar baselines and is proficient in imbalanced datasets.
Previous
My uncle died a year before I got married.
"Die'
Die
NA
Learn semantics of
Marry'
"died":
Emerges
"died": NA Die
Prev ious
Model
New
My uncle died a year before I got married. Manry
"Marry Data
NA
"Altack"
Semantic
Emerges
confusion:
Forgetting
"died": Die?
Current Model
(a) Example of semantic confusion when new types emerge.
[0, 0.2)
[0.2, 0.4) m [0.4, 0.6) [0.6, 0.8) [0.8, 1.0)
100
Proportion (%)
80
60
ZB422
40
20
30:95
SU47
5.5
3632
2341
10:3
ACE05-EN+
ERE-EN
MAVEN
1 Introduction
(b) Imbalanced number distribution of types.
Event extraction (Grishman, 1997; Ahn, 2006) aims to detect event types and identify their event arguments and roles from natural language text. Given sentence "The Oklahoma City bombing conspirator is already serving life term in federal prison", an event extraction model is expected to identify "bombing" and "serving", which are the event triggers of the "Attack" and "Sentence" types, respectively. Also, the model should identify argu- ments and roles of corresponding event types such as "conspirator" and "Oklahoma City" are two ar- guments involved in "Attack" and as argument roles of "Attacker" and "Place", respectively.
Figure 1: Problems in continual event extraction.
continually. We define a new problem called con- tinual event extraction for this scenario. Compared to conventional studies, continual event extraction expects the model not only to detect new types and identify corresponding event arguments and roles but also remember the learned types and roles. This scenario belongs continual learning (Ring, 1994), which learns from data streams with new emerging data.
To alleviate the so-called catastrophic forgetting problem (Thrun and Mitchell, 1995; French, 1999), existing works focus event detection and apply knowledge transfer prompt engineering (Cao et al., 2020; Yu al., 2021: Liu et al., 2022). On one hand, they do not consider the task of argu- ment extraction, making them incomplete in event extraction. On the other hand, they ignore that the
Conventional studies (McClosky et al., 2011; Nguyen et al., 2016; Du and Cardie, 2020; Lin et al., 2020; Nguyen et al., 2021; Wang et al., 2022) model event extraction as a task to extract from the pre-defined event types and argument roles. In prac- tice, new event types and argument roles emerge
"Correspondigg author

---

semantic understanding by the model deviates from correct semantics when new types emerge, which we call semantic confusion.
provement and obtains better performance on long-tailed types.
First, semantic confusion caused by the an- notations of previous types and new types do not generate at the same time. As shown in Figure (a)), a sentence may have multi-type annotations. How- ever, current training data only contains new an- notations, and the model misunderstands the text "died" with the previous annotation "Die" neg- ative label "NA". Similarly, the model that is only trained on the previous data would identify the new types as negative labels. Existing works (Cao et al., 2020; Yu et al., 2021; Liu et al., 2022) sim- ply transfer all learned knowledge to the current model, which would disturb new learning. The second problem is the imbalanced distribution of event types in natural language text. Figure ((b) shows the number distribution of event types in three widely used event extraction datasets. The model confused with the semantics of the long- tailed event types in two aspects. On one hand, it suffers from the lack of training on long-tailed event types due to their few instances. On the other hand, the semantics of long-tailed types would be disturbed by popular types during training.
Related Work
2.1 Event Extraction
Conventional event extraction models (McClosky et al., 2011; Li et al., 2013; Nguyen et al., 2016; Lin et al., 2020; Wang et al., 2021) regard the event extraction as a multi-class classification task. recent years, several new paradigms have been proposed to model event extraction. The works (Du and Cardie, 2020; Liu et al., 2020; Li et al.. 2020; Lyu al., 2021) treat event extraction as question-answering task. They take advantage of the pre-defined question templates and have spe- cific knowledge transfer abilities on event types. The work (Wang et al., 2022) refines event extrac- tion as a query-and-extre and-extract process by leveraging rich semantics of event types and argument roles. These models cannot apply to continual event ex- traction as they learn all event types at once.
2.2 Continual Learning
Mainstream continual learning methods be dis- tinguished into three families: regularization baasdd methods (Li and Hoiem, 2017; Kirkpatrick et al., 2017), dynamic architecture methods (Aljundi et al., 2017; Rosenfeld and Tsotsos, 2018; Qin et al., 2021), and memory- based methods (Lopez-Pat and Ranzato, 2017; Rebuffi et al., 2017; Castro et al., 2018; Wu et al., 2019).
This paper proposes novel continual event extraction method to rectify semantic confusion and address the imbalance issuc. Specifically, we propose data augmentation strategy that marks pseudo labels of each sentence to avoid the dis- turbance of semantic confusion. We apply a piv- otal knowledge distillation to further encourage the model to focus on vital knowledge during train- ing the feature and prediction levels. Moreover, we propose prototype knowledge transfer, which leverages the semantics of other associated types enrich the semanties of long-tailed types.
For many NLP tasks, the memory-base meth- ods (Wang et al., 2019; de Masson d' Autume et al., 2019; Cao et al., 2020) show superior performance than other methods. Existing works (Cao et al., 2020; Yu et al., 2021; Liu et al., 2022) make use of knowledge transfer to alleviate catastrophic for- getting in event detection. KCN (Cao al., 2020) employs memory reply and hierarchical distilla- tion to preserve old knowledge. KT (Yu et al.. 2021) transfers knowledge between related types to enhance the learning of old and new event types. EMP (Liu et al., 2022) leverages soft prompts to preserve the knowledge learned from each task and transfer it to new tasks. All the above models are unable to identify event arguments and roles, they are incomplete in continual event extraction. Furthermore, they ignore semantic confusion on event types while training. We address these prob- lems and propose a new model.
Our main contributions are outlined as follows:
Unlike existing works, we extend continual learning to event extraction and propose a new continual event extraction model.
We explicitly consider semantic confusion on event types. We propose data augmentation with pseudo labels, pivotal knowledge distil- lation, and prototype knowledge transfer to rectify semantic confusion.
We conduct extensive experiments on three benchmark datasets. The experimental results demonstrate that our model establishes new state-of-the of-e-o--the-rr baseline with significant im-

---

Data Augmentation
New Task Data
&
Memory Data
Melony Morshall was married before left for Iraq.
the side that's moving
going take the casualties.
pscudo label
pold
pscudo
gold
training
After training
New event types: Acquit(Long-tainne Transpor(Teff. moving)
Truin Init. Modill
Previous Model
Current Model
Previous types: Marry(married) Dio(casunlties)
Module
Module
Augini Tradoncer
Auyui Instance
New Task Duta Memory Data
New Task Data Memory Dara
Encoder
Encoder
Encoder
Evast Prototype
Only loigetailed types
Marry
Die(Pg. (fg)
Attention Festure Distiliation
Acquit(R.
Transpanlep;
logit
logit
Classifier
Classifier
d/ di cos(()n.
Selective Prediction Distillation
CO,N(O,0))
Previous Model
Current Model
Pivotal Knowledge Distillation
Prototype Knowledge Transfer
Figure Framework our proposed continued event detection model.
3
Task Definition
current training data with pseudo labels. Then, train the current model on augmented data and memory data with pivotal knowledge distillation. For long-tailed event types, enhance their mantics with event prototypes. At last, we pick and store a few instances for new types and augment them with pscudo labels for the next task T.+1- The parameters are updated during training. After pre- dicting each candidate event type, we train Gi to obtain corresponding event arguments and roles. Similar to event detection, also pick and store few instances. The accuracy of argument extraction highly depends on correct event types.
Given sentence, the event extraction task aims to detect the event types in this sentence (a.k.a, event detection) and identify the event arguments and roles (a.k.a. argument extraction).
In a continual event extraction task, there is a sequence of K tasks T2, Tk}. Each in- dividual task Ti is conventional event extraction task that contains its own event type set E,, role type set R2 and respective training set Dirain, de- velopment set Diee and test set D.est F; of each task T; is disjoint with other tasks. Dirain contains instances for E; and negative instances for "NA". Dolev and Deet only contain sentences for E,. At the i-th stage, the continual event extraction model is trained on Dirain and evaluated on all seen test sets Diest Ui_ Diest to detect all seen event types E, Ul_1 E, and identify all event argu- ments and roles of corresponding event types.
4.2 Event Detection
4.2.1 Base Model and Experience Replay
Base model. Following (Cao et al., 2020; Yu et al., 2021; I; Liu al., 2022; Du and Cardie, 2020), we use the pre-trained language model BERT (De- vlin et al., 2019) as the encoder to extract the hidden representation of text.
4 Methodology
Given sentence we first use the BERT en- coder to get the hidden representation ha, for each token æ; in Then, obtain the feature repre- sentation Cey of xt by
4.1
Overall Framework
Our framework for continual event extraction con- sists of two models: event detection model F; and argument extraction model Gi When new task T, comes, we detect the candidate event types for each sentence by F, The framework of our proposed model F, is shown in Figure 2. We first augment
fgy = LayerNorm( W Dropout(hz,) b), (1)
where W E Rlxd and b E I are trainable pa- rameters. h. d are the dimensions of feature repre-

---

sentations and hidden layers in BERT, respectively. LayerNorm()) is the normalization operation. We use linear softmax classifier to get zy 's output probability distribution on the basis of f.y. The cross-entropy classification loss of the current dataset is defined as follows:
To address this issue, we propose data aug- mentation strategy with pseudo labels excavate potential semantics and rectify semantic confusion. Before training the current model, we first augment the training data with the previous model. For each instance in the training set Dirain, it is just annotated by the new event types E, We regard this sentence as a test instance and leverage the previous model to predict the event types for each token. Once the prediction confidence exceeds threshold т, we mark this token as the predicted type, serving as a pseudo label. Then, the augmented data can be used to train the current model. After training, we also leverage the trained model to obtain pseudo labels for the memory data. Note that we just use augmented task data and memory data for training, rather than for prototype generation in prototype knowledge transfer, since the pseudo labels are not completely reliable.
Lela
1
ΣΣ OCE
Mr.c log P(e 2;; Fi), (2)
W
where N is the token set from the current training data. the seen event types set E, and "NA". Ma.e: indicates whether the reference type of x is e. P(el x; Fi) is the probability of classified as by the event detection model Fü
Experience replay. Inspired by the previous works on continual learning (Wang et al., 2019;
de Masson Autume et al., 2019; Yu et al., 2021;
Liu et al., 2022), we pick and store a small number m of instances for each event type. At i-th stage, the memory space to store the current training data is denoted by 6,, the accumulated memory space is 8; - 5g. Note that we do not store nega- tive instances in o, owing to the fact that negative instances are prevalent at each stage. At the i-th stage, we train the model with current training data Dimin and memory space 5; We leverage the k-means algorithm to cluster the feature represen- tations of each event type's 's instances, where the number of clusters equals the memory size m. We select the instances closest to the centroid of each cluster and store them.
4.2.3 Pivotal Knowledge Distillation
Knowledge distillation (Hinton et al., 2015) aims to remind the current model about learned knowledge by leveraging the knowledge from the previous model. It is important to leverage precise learned knowledge, otherwise, it would lead to semantic confusion like in previous works (Cao et al., 2020; Yu al., 2021; Liu al., 2022). In this paper, we propose pivotal knowledge distillation, which enables the model to focus on critical knowledge and transfers precise knowledge between the previ- ous model and the current model at the feature and prediction levels.
4.2.2 Data Augmentation
Attention feature distillation. At the feature level, we expect the features extracted by the cur- rent model similar to those by the previous model. Unlike existing works (Lin et al., 2020; Cao et al., 2020; Liu et al., 2022), we consider that each to- ken in a sentence should not have an equal fea- ture weight toward an event trigger. The tokens associated closely with the event trigger are more important than others. To capture such context in- formation, we propose attention feature distillation. We first apply in-context attention obtain atten- tive feature Ar, for each token Tj in a sentence:
the event detection task, a sentence may have several annotations of event types. For example, in the sentence "Melony Marshall was married before she left for Iraq", "married" and "left" indicate the event types "Marry" and "Transporr", respectively. Let us assume that "Marry" is the previously seen type and "Transport" is the newly emerging type. the current memory space does not include this sentence, the annotation corresponding to "mar- ried" would be "NA". Thus, the model would treat the text "married" as a negative label "NA" at the current stage. However, "married" has been consid- ered as the event trigger of "Marry" at the previous stage. It is noticeable that the semantics of "mar- ried" are confused at these two different stages. Previous works (Cao al., 2020; Yu et al., 2021; Liu et al., 2022) simply ignore this problem and suffer from semantic confusion.
1 W
Ar
Σ JEW
(f,,,,
(3)
where W denotes all tokens in this sentence. (()) is an attention function, which is calculated as the average of the self aelfnattenton weights from the last

---

L layers of BERT, where L a hyperparameter. f is the feature representation of r. After capturing attentive feature Azy. we pre- serve previous features through an attention feature distillation loss:
task learning. The final loss is
c-(1- [B--1] Cak +
[E [Ad
(a
Lasa + 3 Lypa);
(6)
where and [ are hyperparameters.
Cafd
Σ1-
(A 1 cos
A)
(4)
M
4.2.4 Prototype Knowledge Transfer
DEN
The distribution of event types is naturally imbal- anced in the real world, where the majority of in- stances belong to few types. Due to the lack of instances, the semantics of long-tailed event types are difficult to capture by the model. Moreover, if a long-tailed event type is analogous to a popu- lar event type with many instances, its semantics are likely to be biased toward that of the popular one. Consequently, the model obtains confused semantics on long-tailed event types.
where cos(: (.) is the cosine similarity between two features. A and A are two attentive features computed by F; and -1, respectively.
During feature distillation, the current model would pay more attention to the associated tokens and obtain the critical and precise knowledge of these tokens from the previous model to remember the seen event types. Morcover, with the lower attention to irrelevant tokens, the current model avoids being confused by irrelevant semantics.
To address this issue, we propose a prototype knowledge transfer method to enhance the seman- tics long-tailed types with associated event pro- totypes. In our viewpoint, the prototypes imply the semantics of their event types. To get the exact prototypes of emerging event types, we first train the base model with the current training data Dirain For each event type in the seen event types És, we calculate the average Pe. and the standard devia- tion oe of feature representations of corresponding tokens in the current training data Dimain or memory space 04-1- If the event type is newly emerging, we calculate its prototype by
Selective prediction distillation. At the predic- tion level, we enforce that the probability distribu- tion predicted by the current model F; does not deviate from that of the previous model F; 1. The previous methods (Yu al., 2021; Song et al., 2021; Liu al., 2022) transfer the probability distribution of each token in a sentence. However, we argue that this brings semantic confusion from the previous model. The tokens of emerging event types should not be transferred. The previous model F; gives an inaccurate probability distribution of these to- kens due to that it has not been trained on emerging event types. Therefore, if we transfer the current model F, with wrong probability distribution, it would be confused with the semantics of new event types. To overcome this problem, we directly lever- age the tokens of the previously seen types and the "NA" type to transfer knowledge. Furthermore, we do not transfer the probability distribution of "NA" owing to the availability of negative training data every task.
Me =
Σ f.. ICNe
(7)
[Wo.
Σ (fz (E2-Me)2
(8)
We
ZEN.
where Ne is the tokens of event type in Dirain For the previous event types, we compute their prototype by tokens in memory space above. For each token r of long-tailed type e, we clarify its semantics through the associated event proto- types. We first measure the similarity of e and another event type e' in the seen event types E, by the cosine distance. Then, we calculate the sociated standard deviation with associated event prototypes by
Based the above observation, propose selective prediction distillation to avoid semantic confusion in knowledge distillation:
Cap
x: ((e,
MT
TEN cEE
(5)
where A is the token set excluding the tokens of new types. E is the previously seen type set. Inspired by (Yu et al., 2021), we optimize the classification loss and distillation loss with multi-
8_ Edo' de = cos (Pe: 11));
(9)
e'EP
where P is the prototypes of all seen event types.

---

We assume that the hidden representations of event types follow the Gaussian distribution and generate the intensive vector fe by
5 Experiments and Results
In this section, evaluate model and report the results. The source code is accessible online.'
f. 6.)
(10)
5.1 Experiment Setup
Datasets. We carry out our experiments on 3 widely-used benchmark datasets. (1) ACE05-EN+ (Doddington et al., 2004) is a classic event extrac- tion dataset containing 33 event types and 22 ar- gument roles. We follow (Lin et al., 2020) to pre- process the dataset. Since several event types are missing in the development and test sets, split the dataset for better distribution of the event types. (2) ERE-EN (Song al., 2015) is another popular event extraction dataset, which contains 38 event types and 21 argument roles. We pre-proccss and re-split the data like ACE0S-EN+. (3) MAVEN (Wang et al., 2020) is large-scale dataset with 168 event types. Due to the lack of argument an- notations, we can only evaluate the event detection task. Since its original dataset does not provide the annotations of the test set, we re-split the dataset well. More details about the dataset statistics and splits can seen in Appendix A.
We add the intensive vector fe with the feature vector fg as the final representation Ez of this long- tailed token by f f. + fa- We leverage the final representation for further learning like the feature representation of popular types. Note that we do not apply the average to generate the intensive vec- tor. We think that the use of average would align the semantics of long-tailed event types and their associated event lypes, causing semantic confusion certain extent. In this paper, categorize the last 80% of types in terms of the number of instances long-tail types.
4.3 Argument Extraction
After obtaining the event types of sentences, we further identify their arguments and roles based on the pre-defined argument roles of cach event type. The arguments are usually identified from the entities in the sentence. Here, we first recog- nize the entities in each sentence with the BERT- BiLSTM-CRE model, which is optimized the same current training data. We treat these entities as candidate arguments. Then, we leverage another BERT encoder to encode each candidate argument and concatenate its starting and ending token rep- resentations as the final feature vector. Finally, for each candidate event type, we classify each entity into argument roles by linear softmax classifier. The training objective is to minimize the following cross-entropy loss function:
Implementation. As defined in Section 3, we conduct a sequence of event extraction tasks. Fol- lowing (Yu et al., 2021; Liu et al., 2022), we par- tition each dataset into 5 subsets to simulate 5 dis- joint tasks, denoted by (Ti...., 75}. jority of event types have more than 10 training instances, we the memory size to 10 for all competing models. To reduce randomness, we run every experiment 6 times with different subset per- mutations. See Appendix B for hyperparameters.
As the ma-
Competing models. We compare our model with 7 competitors. Fine-tuning simply trains on the current training set without any memory. It for- gets the previously learned knowledge seriously. Joint-training stores all previous training data memory and trains the model on all training data for each new task. It simulates the behavior of re- training. It viewed as an upper bound in continual event extraction. To reduce catastrophic forgetting. KCN (Cao et al., 2020) combines experience reply with hierarchical knowledge distillation. KT' (Yu et al., 2021) transfers knowledge between learned event types and new event types. EMP (Liu et al., 2022) uses prompting methods to instruct model
Luc
1
ΣΣ eEQrER
Herr log P(rle),
(11)
O
where Q is the set of candidate entities. R. is the pre-defined argument roles of the corresponding event type. We.r denotes whether the reference role of is r. P(r| denotes the probability of e clas- sified as r.
For argument extraction, we also apply another memory space store few instances to alleviate catastrophic forgetting. We pick and store instances in the same way as described in Section 1.2.1.
hhttps://github. htttps://gituub.commjjjwuwwee

---

Datasets
ACEOS-EE++
BRE-EN
MAVEN
Models
T;
T3
T
T4
T5
7j
72
T3
T:
73
Ti
T3
T4
73
Fine-tuning
84.35
54.37
36.82
31.15
24.75
80.73
48.6 0
36.08
27.28
24.98
78.88
47.47
35.16
28.79
22.67
detection
Joint-training
84.35
81.24
80.02
77.03
75.51
80.73
73.61
69.75
65.78
63.60 63.66
78.88
71.78
68.52
67.74
65.88
KCN
82.15
72.69
65.38
60.17
55.68
77.38
63.37
53.05
50.09
45,47
75.32
49.85
43.49
40.61
37.98
KT
82.15
73.02
66.36
61.56
57.23
77.38
65.89
60.06
56.68
53.40
75.32
56.95
51.68
47.59
44.08
FMP
82.52
71.85
65.48
60.95
58.19
79.66
63.78
53.79
51.02
50.17
77.96
55.86
49.41
47.19
44.98
BERT QA_ Ang
85.00
54.65
39,26
31.73
27.95
78.63
48.14
36.72
28.14
23.81
77.36
47.17
34.79
28.16
21.98
ChatGPT
16.43
16.03
15.43
17.41
17.11
11.52
10.79
7.81
10.69
9.97
10.36
7.99
7.49
7.72
Our model
84.35
80.75
76.39
76.10
73.08
80.73
68.69
63.82
58.18
78.88
68.03
61.74
57.31
55.87
Arg. extruct.
tuning
48.45
35.00
17.51
50.39
30.57
14.27
Joint-training
48.45
51.73
48.85
48.84
47.89
50.39
45.54
40.11
BERT QA_
50.63
35.63
25.81
20.31
19.13
48.41
29.34
22.12
17.59
13.91
ChatGPT
4.59
2.53
5.6]
5.43
5.59
3,36
4.59
3,71
3.19
3.84
Our model
48.45
47.24
42.M0
40.36
37.90
50.39
39.49
35.94
34.19
31.25
Table I: Fl scores of event detection and argument extraction on the ACBOS-EN+, BRE-EN, and MAVEN datasets. The best and second-bes scores except for joint-training are marked bold and with underline, respectively.
prediction. KCN, KT, and EMP are only feasi- ble for event detection we cannot apply them to argument extraction. BERT Arg (Du and Cardie, 2020) casts event extraction as a question- answering task. is capable of transferring specific knowledge based on its question templates. Be- sides, we consider ChatGPT2 as zero-shot event extraction model. Following (Li et al., 2023), we use the same prompt in the experiments.
of our model are even close to joint-training, espe- cially on ACE-05EN+. Differently, our model is cost-effective to handle newly emerging event types without retraining the model on all seen types.
For continual argument extraction, our model also achieves superior performance. This indicates that based on accurately detected event types, our model can extract corresponding arguments well with few memorized instances. Thus, the perfor- mance of continual event extraction highly relies on the effectiveness of continual event detection.
Evaluation metries. We assess the performance models with Fl scores and report the average Fl scores over all permutations for each model. For event detection, we follow (Cao et al., 2020; Yu et al., 2021; ; Liu et al., 2022) to calculate F1 scores. For argument extraction, an argument is correctly classified if its event type, offsets, and role label all match the ground truth.
Regarding BERT QA Arrg it performs poorly under our setting. This shows that conventional event extraction models may not handle continual event extraction well, although it has transferability on event types. The results of ChatGPT indicate that large language models, which can be viewed zero-shot event extraction methods, also struggle with continual event extraction.
5.2 Results and Analyses
5.2.1 Main Results
Table shows the main results of event detection and argument extraction. Note that the results on Ti are based on each model's own baseline.
5.2.2 Ablation Study
To validate the effectiveness of each module in our model, we conduct an ablation study on continual event detection, and the results are listed in Table 2. Specifically, for "w/o DA", we use the original dataset rather than the augmented data with pseudo labels; for "w/o AFD", we disable the attention feature distillation module; for "w/o SPD", we dis- able the selective prediction distillation module; for "w/o PKD", remove the pivotal knowledge distillation module; for "w/o PKT", we directly train the model without transferring the knowledge of event prototypes to long-tailed tailed event types. We observe that all modules are effective.
For continual event detection, our model outper- forms all competitors on three datasets by a large margin. After training on all tasks, compared with the statc-of-the-arn models KT and EMP, our model gains 14.89% 4.78%, and 10.89% improvement of Fl scores on ACE05-EN++ ERE-EN, and MAVEN, respectively. The significant gap demonstrates that semantic confusion on event types is the key to continual event extraction. We can also conclude that our confusion rectification is effectively adapt- able to different datasets. Furthermore, the results #htnps//chat.opeeaa.com/

---

event types from semantic confusion and classify- Event detection
T2
Ts
T4
Tg
ACEOS-EN+
ERE-EN
MAVEN
Intact model
84.35
80.75
76.39
76.10
73.08
KCN
-28.02
-36.75
-33,43
ACE205- EN++
w/o DA
84.35
77.58
75.18
68.98
67.19
KT
-29.49
-29.49
-27.71
w/o AFD
84.35
78.83
76.22
74,46
71.89
EMP
-22.80
-29.76
-28.76
w/o SPD
84.35
78.09
73.28
68.82
65.85
Our model
-18.41
-22.64
-21.83
w/o PKD
84.35
79.16
71.01
65,70
65.06
w/o PKT
84.35
79.12
74.87
74.89
71.25
Table 4: BWT scores of event detection on ACE0S-EN+,
ERE-EN dataset, and MAVEN.
Intact model
80.73
68.69
63.82
59.96
58.18
ERE-EN
w/o DA
80.73
63.28
58.10
57.54
54.12
w/o AFD
80.73
68.30
63.61
59.69
57.69
the best performance on all datasets. Compared with the competing models, our model obtains the
w/o SPD
80.73
67.71
61.34
56.69
55.05
w/o PKD
80.73
66.83
62.04
58.50
53.83
w/o PKT
80.73
68.55
63.09
59.74
lowest performance decline on ACE05-EN+ and 57.72
Intact model
78.88
68.03
61.74
57.31
even gains improvement on ERE-EN and MAVEN. Thus, our model is effective in saving long-tailed tail-t 48.09
55.87
w/o DA
78,88
64.70
53.80
52.08
MAVEN
w/o AFD
78.88
67.29
60.78
56.77
54.96
w/o SPD
78.88
66.71
60.45
55.42
52.57
ing them correctly.
w/o PKD
78.88
65.69
59.92
54.57
51.86
w/o PKT
78.88
67.76
60.68
56.74
55.20
Analysis of Knowledge Transfer Ability We widely-used metric backward transfer (BWT) (Lopez-Paz and Ranzato, 2017) to measure the knowledge transfer ability and how well the model alleviates catastrophic forgetting. The BWT score is defined as follows:
5.2.4
Table Fl scores of ablation study on event detection. All models have the same results on Ti since continual learning has not been executed.
Event detection
Ti
T
T3
T4
Ts
Fine-tuning
80.09
42.65
34.28
31.97
20.76
K-
ACE205 +NN+
Joint-training
80.09
77.87
75.13
72.47
71.89
BWT
1
(FIK;-F1i.)) (12)
KCN
73.37
70.11
63.59
55.39
52.23
K-1
KT
73.37
71.71
64.94
58.66
54.26
EMP
78.35
70.57
64.58
59.50
54.72
where K is the number of tasks. Fli, is the Fl score the test set of task Tj after training the
BERT
81.39
41.88
32.
24.06
Our model
80.09
75.36
74,20
72.24
70.45
model task T. Note that BWT scores nega- tive due to catastrophic forgetting. A higher score indicates a better performance.
Fine-tuning
76.45
47.64
36.40
24.00
22.79
Joint-training
76.45
70.27
65.37
64.72
61.12
BRE-EN
KCN
69.34
58.27
51.87
47.02
41.73
KT
69.34
61.57
58.88
54.16
51.54
Table 4 shows the results. Our model performs best, indicating its superiority alleviating catas- trophic forgetting. Benefiting from semantic confu- sion rectification, our model has better transferabil- ity than the competing models.
EMP
76.43
60.22
57.51
52.62
49.20
BERT_ BERT_QA_An Qra
76.21
47.68
35.87
24.98
22.76
Our model
76.45
67.42
62.49
61.27
59.14
Fine-tuning
75.89
47.30
33.10
24.93
22.44
Unint-trainig
75.89
67.97
64.45
63.04
62.56
MAVEN
KCN
62.61
48.09
43.46
37.41
35.06
KT
62.61
54.71
49.55
45.63
42.21
5.2.5
Analysis of Memory Size Influence
EMP
74.40
53.13
48.44
45.72
42,64
BERT _A_Arg
74,23
46.21
33.72
25.58
24.27
The performance of memory- models is highly related memory size. We conduct an ex- periment with different memory sizes. Table 5 lists the results on ACE05-EN+, ERE-EN, and MAVEN. is observed that our model maintains state-of- the-art performance with different memory sizes. Compared to other models, the performance gap of our model between memory sizes 5 and 20 is the smallest, which demonstrates the robustness of our model to the change of memory size.
Our model
75.89
64.73
59.28
56.84
56.52
Table 3: Fl scores of event detection on long-tailed event types. All models are trained with event types but only evaluated long-tailed event types.
5.2.3 Analysis of Long-Tailed Event Types
We analyze the performance of long-tailed event types in continual event detection and Table presents the results. Note that we do involve ChatGPT in this setting since we regard it as a zero-shot model that does not suffer from imbal- anced data. From the results, our model achieves
Conclusion
In this paper, we observe the continual lcarning of event types suffering from semantic confusion

---

Memory size 5
Memory size 20
References
ACE
T5
David Ahn. 2006. The stages of event extraction. In Proceedings of the Workshop on Annotating and Rea- soning about Time and Events, pages 1-8.
KCN
71.29
63.14
58.57
52.71
72.36
65.92
63.57
59.08
KT
72.35
64.74
59.75
55.31
74.75
68.76
64.93
60.74
EMP
72.02
59.57
74.09
68.49
Ours
80.15
73.41
71.55
70.29
81.72
78.15
76.66
73.38
Memory size
Memory size
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuyte-
ERE
T3
T4
7;
2;
Z:
T1
2;
laars. 2017. Expert gate: Lifelong learning with a network of experts. In CVPR, pages 3366-3375.
KCN
56,60
49,70
43.11
36.64
64.65
55,10
52.32
47.14
KT
63.68
58.48
53.62
47.54
66.63
60.73
58.84
55.23
RMP
62,20
54,02
50.05
46,41
66.21
56.08
55.62
53.90
Pengfei Cao, Yubo Chen, Jun Zhao, and Taifeng Wang.
61.74
58.71
54.63
68.50
64.49
59.96
2020. Incremental event detection via knowledge consolidation networks. In EMNLP, pages 707-717.
Memory size 5
Memory size 20
MAVEN
T;
KCN
41.81
35.97
32.01
28.36
53.18
46.10
43.84
40.93
Francisco M Castro, Manuel Marin-Jiménez, Nicolás
KT
50.84
45.29
40.87
37.91
59.47
53.66
49.54
47.19
Guil, Cordelia Schmid, and Karteek Alahari. 2018, End-to-ens ood incremental learning. In ECCV, pages 233-248.
EMP
49.35
43.52
40.17
39.07
59.67
52.33
50.12
48.20
Ours
68.00
60.80
56.11
54.44
68.71
62.06
59.26
57.49
Table Fl scores of event detection w.r.t. memory size ACE05-EN+, ERE-EN, and MAVEN.
Cyprien de Masson d' Autume, Sebastian Ruder, Ling- peng Kong, and Dani Yogatama. 2019. Episodic memory lifelong language learning. arXiv preprint arXiv:1906.01076
and propose a novel continual event extraction model. Specifically, we mark pseudo labels in training data for previously seen types. For newly emerging types, we select accurate knowledge to transfer. For long-tailed types, we enhance their semantic representations by the semantics of associ- ated event types. Experiments on three benchmark datasets show that our model achieves superior per- formance, especially on long-tailed types. Also, the results verify the effectiveness of our model in alleviating catastrophic forgetting and rectifying semantic confusion. In future work, we plan to study continual few-shot event extraction or other classification-baat based continual learning tasks.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In NAACI, pages 4171-4186.
George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephamic Strassel, and Ralph Weischedel. 2004. The automatic content extrac- tion (ACE) program tasks, data, and evaluation. In LREC.
Xinya Du and Claire Cardie. 2020. Event extraction by answering (almost) natural questions. In EMNLP, pages 671 1-683.
Robert M. French. 1999. Catastrophic forgetting in con- nectionist networks. Trends in Cognitive Sciences. 3:128-135.
Limitations
Ralph Grishman. 1997. Information extraction: Tech- niques and challenges. SCIE, pages 10-27.
Our model may have two limitations: (1) It re-
quires an additional memory space store a few instances, which is sensitive to storage capacity. (2) It relies on the selection of instances in memory space. The prototype knowledge transfer may suf- fer from the low quality of selected instances in memory space, causing performance decline.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in neural network. arXiv preprint arXiv: 1503.0233.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz. Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan. Tiago Ramalho, Ag- nieszka Grabska- Barkikkkaa et al. 2017. Over- coming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114:3521-353 -3266.
Acknowledgments
This work was supported by the National Natu- ral Science Foundation of China (No. 62272219) and the Collaborative Innovation Center of Novel Software Technology Industrialization.
Bo Li, Gexiang Fang, Yang Yang. Quansen Wang, Wei Ye. Wen Zhao, and Shikun Zhang. 2023. Eval- uating ChatGPT's information extraction capabili- ties: An assessment of performance, explainabil- ity, calibration, and faithfulness. arXiv preprint arXiv:2304.1163:

---

Payuan Li, Weihua Peng, Yuguang Chen, Quan Wang. Lu Pan, Yajuan Lyu, and Yong Zhu. 2020. Event extraction multi-tum question answering. In Find- ings of EMNLP, pages 829-838.
Dandan Song, Jing Xu, Jinhui Pang, and Heyan Huang. 2021. alassitier-ddaptttion knowledge distillation framework for relation extraction and event detec- tion with imbalanced data. Information Sciences, 573:222-238.
Qi Li, Heng Ji, and Liang Huang. 2013. Joint event ex- traction via structured prediction with global features. InACL, pages 73 82.
Zhiyi Song, Ann Bies, Stephanie Strassel, Tom Riese, Justin Mott, Joe Ellis, Jonathan Wright, Seth Kulick, Neville Ryant, and Xiaoyi Ma. 2015. From light to rich ERE: Annotation of entities, relations, and events. In Proceedings of the 3rd Workshop EVENTS: Definition, Detection, Coreference, and Representation, pages 89-98.
Zhizhong Li and Derek Hoiem. 2017. Learning without forgetting. IEEE Transactions Pattern Analysis and Machine Intelligence, 40:2935-29477.
Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020. A joint neural model for information extraction with global features. In ACL, pages 7999-8009.
Sebastian Thrun and Tom M. Mitchell. 1995. Lifelong robot learning. Robotics and Autonomous Systems, 15:25 46.
Jian Liu, Yubo Chen, Kang Liu, Wei Bi, and Xiaojiang Liu. 2020. Event extraction as machine reading com- prehension. In EMNLP, pages 1641-1651
Hong Wang, Wenhan Xiong, Mo Yu, Xiaoxiao Guo, Shiyu Chang, and William Yang Wang. 2019. Sen- tence embedding alignment for lifelong relation traction. In NAACL, pages 796 806.
Minqian Liu, Shiyu Chang, and Lifu Huang. 2022, In- cremental prompting: Episodic memory prompt for lifelong event detection. In COLING, pages 2157- 2165.
Sijia Wang. Mo Yu, Shiyu Chang, Lichao Sun, and Lifu Huang. 2022. Query and extract: Refining event extraction type-oriented binary decoding. In Find- ings of ACL, pages 169-182.
David Lopez-Paz and Mare' Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. NeurlPS, 30.
Xiaozhi Wang. Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han. Zhiyuan Liu, Juanzi Li. Peng Li, Yankai Lin. and Jie Zhou. 2020. MAVEN: A massive general domain event detection dataset. In EMNLP, pages 1652-1671.
Qing Lyu, Hongming Zhang, Elior Sulem, and Dan Roth. 2021. Zero-shot event extraction via transfer learning: Challenges and insights. In ACI-IJCNIPP pages 322-332.
David McClosky, Mihai Surdeanu, and Christopher Manning. 2011. Event extraction dependency pars- ing. In ACL, pages 1635.
Ziqi Wang, Xiaozhi Wang, Xu Han, Yankai Lin, Lei Hou, Zhiyuan Liu, Peng Li, Juanzi Li, and Jie Zhou. 2021. CLEVE: Contrastive pre-training for event extraction. In ACL-IJCNLP, pages 6283-6297
Minh Van Nguyen, Viet Dae Lai, and Thien Huu
Nguyen. 2021. Cross-task instance representation interactions and label dependencies for joint infor- mation extraction with graph convolutional networks. In NAACL, pages 27-38.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye. Zicheng Liu, Yandong Guo, and Yun Fu. 2019. Large scale incremental learning. In CVPR, pages 374-382.
Pengfei Yu, Heng Ji, and Prem Natarajan. 2021. Life- long event detection with knowledge transfer. In EMNLP, pages 5278-5290.
Thien Huu Nguyen, Kyunghyun Cho, and Ralph Gr- ishman. 2016. Joint event extraction via recurrent neural networks. In NAACL, pages 300-309.
Qi Qin, Wenpeng Hu, Han Peng, Dongyan Zhao, and Bing Liu. 2021. Bus: Building network structures dy- namically for continual learning. NeurIPS, 34:20608- 20620.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017. iCaRL: In- cremental classifier and representation learning. In CVPR, pages 2001-2010:
Mark Bishop Ring. 1994. Continual learning in rein- forcement environments. Ph.D. thesis, University of Texas at Austin.
Amir Rosenfeld and John K Tsotsos. 2018. Incremental learning through deep adaptation. IEEE Transac- tions on Pattera Analysis and Machine Intelligence, 42:651-663

---

A Dataset Statisties and Splits
B Environment and Hyperparameters
In this section, we introduce how we split the datasets. For ACEOS-EN+ (Doddington et al., 2004), we first follow (Lin et al., 2020) to pre- process the dataset. The original development set and test set miss several event types and the num- ber of instances in the test set is much smaller than that in the development set. So we re-split the training set for the development set and the test. set following (Yu et al., 2021). We combine the original development set and test set as a new test set. Then, we randomly sample 10% of instances from the training set as a new development set. For the new test set, if the number of instances for a type is less than 10% of the total instances for this type, we randomly sample instances from the training set and remove them from the training set make up for the difference. We do include "Justice:Pardie
We run all the experiments on an X86 server with two Intel Xeon Gold 6326 CPUs, 512 GB memory, four NVIDIA RIX A6000 GPU cards, and Ubuntu 20.04 LTs. We use grid search to choose the hyperparameter values. The search space of key hyperparameters is as follows: (1) The search range for the dropout ratio is [0.1, 0.6] with a step size 0.1. (2) The search range for at 3 is [0.1, 2.0) with step size of 0.1. (3) The search range for the L is [1, 12] with step size of 1. (4) The search range for all learning rates is [1 X 10-5 1 10-4) with step size of 1 10 (5) The search range for the threshold is [0.65, 0.95) with step size of 0.05, The selected values are listed in Table 7.
Hyperparameters
Values
Batch size Dropout ratio a,
8
in both the development set and the test set, as only has instances the entire dataset. For ERE-EN (Song et al., 2015), we fol- low the above method to split the dataset as well. For MAVEN (Wang et al., 2020), we take the origi- nal development set as the test set and re-split the training set by the above method.
0.2
1, 1,3
Gradient accumulation steps Learning rate for event detection encoder
5 10-5
Learning rate for event detection classifier
5 x 10-5
Learning rate for argument extractor Learning rate for entity extractor
5 10-5
3 x 10
Dimension hidden representations Dimension of feature representations
768
We show the statisties of the new splits in Table 6.
512
Threshold r for pseudd--labeling
0.8
Following conventional event extraction (Du and Cardie, 2020; Lin et al., 2020; Wang et al., 2022), we use all unlabeled tokens as negative tokens for the three datasets.
Table 7: Hyperparameter setting in our model.
Datasets
Splits
Events
Arguments
Training 3,908
5,624
ACB0S-E+
Development
457
659
Test
946
1,466
Training 5,546
7.505
ERE-EN
Development
639
895
Test
1,095
1,501
Training 70,112
MAVEN
Development
7,881
Test
18,904
Table 6: Statistics of the re-split ACE05-EN+, ERE-EN, and MAVEN datasets.


Reviews:
[{"review_id": "s65K7jIdWv", "reviewer": "Reviewer_p7bH", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort in providing feedback on our paper. We appreciate your acknowledgment of the merits of our work. We are available to reply to any further questions that you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors explicitly consider semantic confusion on event types. They propose data augmentation with pseudo labels, pivotal knowledge distillation, and prototype knowledge transfer to rectify semantic confusion. Experiments on three benchmark datasets demonstrate that their model establishes a new state-of-the-art baseline with significant improvement and obtains better performance on long-tailed types.", "reasons_to_reject": "No specific risk to accept.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is about extending continual learning to event extraction. The authors propose a new continual event extraction model.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5ENHJnojbo", "reviewer": "Reviewer_4NER", "summary": "", "questions": "After addressing the previous issues of event extraction, why did the proposed method not able to outperform the joint training method?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort in providing feedback on our paper. We answer your question below.\n\n**Q1**: Though the authors claimed new SOTA performance but still it did not overcome the joint training approach. After addressing the previous issues of event extraction, why was the proposed method not able to outperform the joint training method?\n\n**A1**: In this paper, the joint training approach stores all previous training data in the memory and trains the model on all training data for each new task. It simulates the behavior of re-training. In contrast, our model and other memory-based models only store a small number of previous training data in the memory. For example, when the memory size is set to 10, our model only picks and stores 10 training instances for event type \"*Attack*\" on the ACE05-EN+ dataset. However, the joint training approach stores all 1,144 training instances for event type \"*Attack*\" and trains the model on them at the next tasks. Given that the joint training approach uses much more training data, it is difficult for our model (and other memory-based models) to outperform it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presented a new method for event extraction by addressing the previously existing issues. ", "reasons_to_reject": "Though the authors claimed new SOTA performance but still it did not overcome the joint training approach. ", "questions_to_authors": "After addressing the previous issues of event extraction, why did the proposed method not able to outperform the joint training method?", "paper_topic_and_main_contributions": "In this paper, the authors proposed a method for continual event extraction with semantic confusion rectification. Semantic confusion and imbalance distribution of events in natural language text are the hinderance towards proper and accurate continual event extraction. This paper addresses these issues of event extraction by introducing new method and achieve new SOTA performance on three dataset.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "ToyEF9L1Ao", "reviewer": "Reviewer_ssjz", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort in providing feedback on our paper. We answer your questions below.\n\n**Q1**: They include some comments about what is different, as in when a key aspect of their approach was not done by past researchers, but I wasn’t always able to follow how much of the authors’ specific method was borrowed from past work.\n\n**A1**: Thanks for your comments. In this paper, we focus on continual event extraction. Below we introduce the originality of our model.\n\nTo the best of our knowledge, the methods used in our data argumentation module and prototype knowledge transfer module are novel for continual learning. In the data argumentation module, we augment the training data with pseudo labels predicted by the previous model. In the prototype knowledge transfer module, we propose a new method to enrich the semantics of long-tailed types from associated types. We generate the representations of long-tailed types following the Gaussian distribution with the help of the associated event prototypes.  \n\nIn the pivotal knowledge distillation module, we follow previous works and use the knowledge distillation method as it is a widely used strategy in continual learning. However, we observe that the knowledge distillation method directly used in previous works leads to semantic confusion as we describe in Section 4.2.3. So, we ameliorate it and propose our pivotal knowledge distillation method. Unlike previous works, we expect the model to pay more attention to the tokens associated closely with the event triggers in the attention feature distillation module. In the selective prediction distillation module, we directly leverage the tokens of the previously seen types and \"NA\" type to transfer knowledge, avoiding confusing the current model with wrong semantics. Furthermore, we do not transfer the probability distribution of \"NA\" owing to the availability of negative training data on every task. \n\nFor the base model and experience reply, we use an event classifier and memory space like previous works for a fair comparison. \n\n**Q2**: I think the authors might also overly rely on mathematical formulas (which are welcome) but without always summarizing them in simple, commonly used language. For instance, in more than one place, the authors show a loss function that appears to be cross-entropy loss, but they never use that term. It would be helpful to understand if they are using standard cross-entropy loss, or if there is anything different about their implementation than would be the most common usage, to more easily interpret what they are conveying.\n\n**A2**: Thank you for pointing this out. We will add easily-understandable summaries of mathematical formulae in the revision. Specifically, Eqs. (2) and (11) in our paper are standard cross-entropy losses.\n\n**Q3**: Similarly, the attention feature distillation methodology is hard to follow. Some of the symbols in equation (3) are defined somewhat vaguely. I’m not sure what exactly the output (described as an “attentive feature”) represents. It’s a sum over all tokens of some average of self-attention weights multiplied by the token’s feature representation, but I’m not sure if the attention weights are averaged over the other tokens when the given token is the query or the key. It’s hard to connect the formula to the motivation. Is the goal to place greater constraint on features that came from other tokens that were assigned more weight by a token that was classified as one of the previous event type triggers? If so, why can’t the model simply constrain the learned features of the actual tokens that were predicted to be the event triggers?\n\n**A3**: Thank you for your question. The attention weights are not averaged over the other tokens. Please allow us to introduce how to generate an attention feature in detail. To obtain an attention feature for a token $x_j$ , we leverage the self-attention score function from BERT to get the attention weight between $x_j$ and each token x in the sentence. We multiply the attention weight and the feature representation for each token x and sum them up. At last, the result divided by the number of tokens in the sentence is the attention feature for the token $x_j$ . Note that the self-attention score function is calculated as the average of the self-attention weights from the last $L$ layers of BERT and we set $L$ as 3 to get the best performance. \n\nWe clarify the connection between Eq. (3) and the motivation below. Our motivation is that the tokens associated closely with the event trigger should be assigned higher weights than others. For this purpose, we leverage the self-attention score function to sharpen the feature of associated tokens in this equation. Therefore, in the feature distillation, the current model would pay more attention to the associated tokens and obtain the critical and precise knowledge of these tokens from the previous model to remember the seen event types. Moreover, with the lower attention on the irrelevant tokens, the current model avoids being confused by their irrelevant semantics.\n\nIf the model simply constrains the learned features of the actual tokens that are predicted to be the event triggers, the model would be disturbed by the irrelevant tokens and learn inadequate knowledge from the previous model, leading to catastrophic forgetting.\n\n**Q4**: At times, I would read through multiple paragraphs unsure what a term meant, until the authors did eventually provide a sufficiently clear definition. For instance, while continual learning might be a growing area of research, the structure of the problem wasn’t even briefly outlined in the Intro or Related Works sections (despite discussion in the latter section about how others have approached it) until the Task Definition in section 3. I felt similarly with some of the terms of for the authors’ main contributions, like pivotal knowledge distillation and prototype knowledge transfer. Simpler, more explicit, high level definitions of concepts earlier on would help to reduce the burden on the reader, rather than relying on niche terms for much of setup, that imply the reader needs to dig through cited works or scour the full paper once just to get the big picture of what’s going on.\n\n**A4**: We appreciate your suggestions on how to improve our paper. We will add some brief and simple descriptions of continual learning and each module of our methods in the introduction. We will also revise the definitions of concepts in the revision to make them clearer.\n\nAgain, we appreciate the reviewers for the positive feedback and constructive suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper contains multiple innovations in improving continual event extraction. Some of the authors’ contributions are perhaps more incremental tweaks on tops of previous methods, and some seem to be more wholesale additions to their model. Both types of innovations seem to improve their model’s performance, and could be valuable to other researchers working on similar problems.\n\nThe authors draw inspiration both from past research on the same task, and on continual learning applied to other tasks. Their contributions might be valuable to others working on continual learning not only in event extraction, but other areas as well. This type of work does seem particularly valuable in event extraction, because EE systems are so reliant on painstakingly labeled data for very specific types of events, and new applications might not share the same event types.\n\nOverall, the methodology appears to be sound and the results are impressive. The ablation study is thorough and helps to address clear questions one might have about what specific revisions or additions to past models contributed most to the improved performance. This break-down should help other researchers considering each of the authors’ contributions as they might be relevant in other settings.", "reasons_to_reject": "There is a lot going on in this paper, and it is at times hard to follow the exact technical details. I think that arises partly because the authors rely heavily on citations to define many of the methods they’re building on, without always summarizing what exactly they use from other papers. They include some comments about what is different, as in when a key aspect of their approach was not done by past researchers, but I wasn’t always able to follow how much of the authors’ specific method was borrowed from past work.\n\nI think the authors might also overly rely on mathematical formulas (which are welcome) but without always summarizing them in simple, commonly used language. For instance, in more than one place, the authors show a loss function that appears to be cross-entropy loss, but they never use that term. It would be helpful to understand if they are using standard cross-entropy loss, or if there is anything different about their implementation than would be the most common usage, to more easily interpret what they are conveying.\n\nSimilarly, the attention feature distillation methodology is hard to follow. Some of the symbols in equation (3) are defined somewhat vaguely. I’m not sure what exactly the output (described as an “attentive feature”) represents. It’s a sum over all tokens of some average of self-attention weights multiplied by the token’s feature representation, but I’m not sure if the attention weights are averaged over the other tokens when the given token is the query or the key. It’s hard to connect the formula to the motivation. Is the goal to place greater constraint on features that came from other tokens that were assigned more weight by a token that was classified as one of the previous event type triggers? If so, why can’t the model simply constrain the learned features of the actual tokens that were predicted to be the event triggers?\n\nAt times, I would read through multiple paragraphs unsure what a term meant, until the authors did eventually provide a sufficiently clear definition. For instance, while continual learning might be a growing area of research, the structure of the problem wasn’t even briefly outlined in the Intro or Related Works sections (despite discussion in the latter section about how others have approached it) until the Task Definition in section 3. I felt similarly with some of the terms of for the authors’ main contributions, like pivotal knowledge distillation and prototype knowledge transfer. Simpler, more explicit, high level definitions of concepts earlier on would help to reduce the burden on the reader, rather than relying on niche terms for much of setup, that imply the reader needs to dig through cited works or scour the full paper once just to get the big picture of what’s going on.\n\nOverall, I think the paper’s contributions are quite interesting and intuitive (as far as I understand them), and the results appear to be impressive. But this sometimes difficulty of understanding the exact details makes me uncertain about my interpretation, which makes it harder to enthusiastically recommend the work for publication. I might be missing clear methodological weaknesses, or the paper might confuse other researchers as well, rather than helping to advance their own work.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on continual event extraction, in which event extraction models are consecutively trained on a sequence of different training datasets, each labeled with a different set of event types. When evaluated on each dataset’s corresponding test set, a key concern is that the models not forget the event types they learned earlier, even if those older event types are not labeled in the latest train and test data. The specific tasks the authors focus on are classifying event trigger tokens into event types, and argument tokens into event argument roles.\n\nThe authors build on past research in continual learning and specifically continual event extraction. They add several innovations on top of recent state of the art methods. One contribution is to add “pseudo labels” to each new training dataset by applying a model trained on previous datasets to automatically label the previously learned event types in the new data.\n\nAnother is to constrain learned features and predictions to not vary far from previous models, but only for tokens assigned to previously learned event types. And another contribution is to identify prototypical feature vectors for long-tail event types, based on the average and standard deviation of feature vectors for observed training examples with that event type, then use those prototypical feature vectors for additional learning.\n\nThey authors provide thorough experimental results, comparing their model to relevant baselines and replications of previous state of the art models for this task. They also provide an ablation study to show the added value of each of their specific innovations. Their model performs better than previous approaches overall, on three different event extraction datasets, and each of their main contributions appears to be beneficial to the overall result.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on continual event extraction which sequentially learn different event types to extract. This paper incorporates "pseudo labels" into each fresh training dataset. These labels are generated by employing a model previously trained on earlier datasets to automatically annotate the previously acquired event categories in the new data. This approach helps prevent the model from forgetting previously learned information. Another technique involves constraining the learned features and predictions to remain consistent with past models. Additionally, they make a valuable contribution by identifying characteristic feature vectors for infrequently occurring event types. This is accomplished by computing the average and standard deviation of feature vectors from observed training examples with those specific event types and subsequently employing these prototypical feature vectors for further learning. From my perspective, this is a solid work on continual event extraction. It builds upon some proven techniques from the field of continual learning and validate them within the context of event extraction. As the reviewer mentioned, there is room for improvement in the writing, particularly in the elaboration of technical details.
Decision: Accept-Main