Learning with Partial Annotations for Event Detection
Year: 2023

Abstract:
Event detection (ED) seeks to discover and classify event instances in plain texts. Previous methods for ED typically adopt supervised learning, requiring fully labeled and high-quality training data. However, in a realworld application, we may not obtain clean training data but only partially labeled one, which could substantially impede the learning process. In this work, we conduct a seminal study for learning with partial annotations for ED. We propose a new trigger localization formulation using contrastive learning to distinguish ground-truth triggers from contexts, showing a decent robustness for addressing partial annotation noise. Impressively, in an extreme scenario where more than 90% of events are unlabeled, our approach achieves an F1 score of over 60%. In addition, we reannotate and make available two fully annotated subsets of ACE 2005 to serve as an unbiased benchmark for event detection. We hope our approach and data will inspire future studies on this vital yet understudied problem.

heading: 1 Introduction
text: Deep learning models have shown impressive performance in event detection (ED) since large amounts of event data have become available (Chen et al., 2015; Nguyen and Grishman, 2015). However, such models require fully labelled and highquality data — in practice, we cannot ensure that every event is identified, and as a result, we often face the partial annotation issue, as depicted in Figure 1. We show a high rate of partial annotation in real-world event datasets. For example, in ACE 2005, which is widely used as a benchmark for ED evaluation (Christopher Walker and Maeda, 2006), nearly 20% of events are not labelled (see Table 2). Using a partially labelled dataset as a fully labelled one for training runs the risk of mis-training on false negatives, and using a partially labelled dataset for evaluation biases comparison. How-
ever, this issue is still understudied in the existing literature (Liu, 2018; Liu et al., 2020b).
In this work, we present a seminal study of learning with partial annotations for ED, with contributions in methodology, data, and practical applications. In our method, to reduce the risk of mis-training on false negatives, we propose a contrastive learning framework (Chopra et al., 2005; Chen et al., 2020) to distinguish ground-truth triggers from contexts, which is shown to be more tolerant of partial annotation noise than the traditional hard classification paradigm (Ji and Grishman, 2008; Chen et al., 2015). In addition, to succeed in the partial annotation scenario, we augment the model with a self-correction regime to recognize false negatives during the training stage.
Figure 2 visualizes the core of our method, which is a de facto trigger localization formulation that uses sentence-wise normalization (prompted by event types) to find event triggers. Compared to hard classification methods that add up individual losses (as shown at the top of Figure 2), our approach instead forms a contrastive learning paradigm by raising the scores of ground-truth triggers while lowering the scores of context words. As a result, even with a significant number of false negatives in training, it can still maintain a good separation between triggers and contexts (§ 6.1). In addition, we suggest that adding a margin softmax (Wang et al., 2018) with a Gaussian-based distance regularization can further improve learning.
508
In addition to the noise-tolerance mechanism described above, we propose a self-correction regime with the motive that when a model recognizes a false negative with high confidence, it should correct its labels for the subsequent training stage. Nevertheless, modeling the confidence of deep learning models is challenging since their predictions are poorly calibrated (i.e., a model often outputs a high prediction probability even if the prediction is incorrect (Guo et al., 2017)). To address this issue, we propose an uncertainty-guided retraining mechanism based on MC-Dropout (Gal and Ghahramani, 2016), which can output prediction confidence to guide the self-correction process. We explain the relationship between this paradigm and an expectation-maximization (EM) framework.
In addition to the methodology contribution, we re-annotate and release the ACE 2005 development and test sets as a data contribution. On the revised benchmark (and an extra MAVEN (Wang et al., 2020) benchmark), we demonstrate the impressive performance of our models — in particular, even in an extreme case with 90% of events unlabeled, our approach achieves more than 60% in F1, yielding a 40% definite improvement over previous methods. In addition to simulation tests, we also conduct a real-world annotation test on WikiEvents (Li et al., 2021a), where the results suggest the practical applicability of our approach.
Contributions. Our contributions are three-fold: (i) To the best of our knowledge, this is the first work addressing the potential partial annotation issue in ED, which may spark further research interest. (ii) We highlight a new learning paradigm for ED based on a trigger localization formulation and show that it works effectively with a wide range of partial annotation settings. (iii) We re-annotated the ACE 2005 development and test datasets and released them to the community to serve as an unbiased benchmark. (IV) In addition to simulation experiments, we conduct real-world annotation experiments to validate the effectiveness of our approach for practical use.

heading: 2 Related Work
text: ED and the Partial Annotation Issue. Event detection (ED) is a crucial subtask of event extraction that aims to identify event instances in texts (Grishman, 1997; Ahn, 2006). The existing ED methods can be divided as feature-based (Ahn, 2006; Li et al., 2013; Liao and Grishman, 2010;
Hong et al., 2011) and deep learning-based (Chen et al., 2015; Nguyen and Grishman, 2015; Nguyen et al., 2016; Liu et al., 2018a; Feng et al., 2016; Chen et al., 2018; Yang et al., 2019; Liu et al., 2020a; Du and Cardie, 2020; Lu et al., 2021; Liu et al., 2019a), and there has been a growing interest in applying these methods to specific scenarios (Liu et al., 2019b, 2022b,a). Nevertheless, most of such methods adopt supervised learning and assume the availability of clean datasets. To date, only a few studies have considered the partial annotation issue in ED: Liu et al. (2020b) identify several unlabeled cases in the ACE test set for error analysis; Liu (2018), in the PhD proposal, suggest that the Chinese portion of ACE 2005 is partially labeled. Unfortunately, neither work stands in a methodology perspective for addressing the issue. Our research, on the other hand, introduces a solution for learning with partial annotations. Our trigger localization formulation also relates to using prompts for event information extraction (Wang et al., 2022a; Hsu et al., 2022; Liu et al., 2022c; Wang et al., 2022b), but different from them focusing on improving the overall performance, our work stands in a point addressing the partial annotation issue.
Learning with Partial Annotations. Learning with partial annotations, also known as positive and unlabeled learning (Li et al., 2009), is an important problem in machine learning community (Elkan and Noto, 2008; Liu et al., 2002, 2003, 2005). In the domain of natural language processing (NLP), researchers have examined a number of tasks including named entity recognition (NER) (Jie et al., 2019; Mayhew et al., 2019; Peng et al., 2019), Chinese word segmentation (Yang and Vozila, 2014),
ED ModelSentence
(Sentence, Revised Label)
A man died when a heavy tank devastated the hotel.
BERT
[CLS] DIE [SEP] A man died when a heavy tank …
(Sentence, Annotation)
[CLS] ATTACK [SEP] A man … tank devastated …
λ+
O O DIE O O O O O O O
BERT
Sentence :
Annotation:
ED Model
ED ModelSentence
A Normal Training Step
An Uncertainty-Guided Interfering Step
(Sentence, Revised Label)
λ-(Sentence, Annotation)
and others (Tsuboi et al., 2008). The efforts for NER relate the most to our work, where a seminal work (Jie et al., 2019) treats the labels of negative instances as latent variables and infers them using partial Conditional Random Fields (Bellare and McCallum, 2007). Later works have devised downweighing mechanisms (Mayhew et al., 2019), confidence estimation methods (Liu et al., 2021), and negative sampling (Li et al., 2021b) for learning. In this study, we offer a new trigger localization formulation for the task of ED and demonstrate promising results in a wide range of partial annotation settings.

heading: 3 Proposed Method
text: LetX = [w1, · · · , wN ] be a sentence withN words and Y = [y1, · · · , yN ] be the ground-truth event label sequence, where yi ∈ T ∪ {O} is the event label of wi (Here T is a set of all event types and O is a special type for non-trigger words). Then the partial annotation issue can be formulated as: due to the neglect of human annotators, some yi 6= O are not identified, and this results in a partial annotation sequence Ỹ = [ỹ1, · · · , ỹN ]. Clearly, directly training a model on (X , Ỹ ) risks outputting a noisy detector. Here we propose a new learning framework to address this issue (as shown in Figure 3), which consists of a noise-tolerant learning mechanism with margin softmax (§ 3.2) and uncertainty-guided retraining mechanism (§ 3.3).

heading: 3.1 Input Representation
text: Given a sentence X , for each event type t ∈ T , we learn their joint representations for further processing. Particularly, we concatenate t and X as the
input1 of a BERT encoder (Devlin et al., 2019):
[CLS] t [SEP] The sentence X︷ ︸︸ ︷ w1 w2 · · · wN
and consider the output of BERT to be the joint representations, denoted as H(t,X) ∈ RM×d, with M being the length of the input sequence2 and d being the dimension of BERT.

heading: 3.2 Noise-Tolerant Learning via Margin Softmax
text: Based on H(t,X), we next locate event triggers of type t in the sentence. This can be achieved using sentence-level softmax, and here we introduce a margin softmax (Wang et al., 2018) to better address partial annotations. Specifically, we first map H(t,X) to a score vector st ∈ RM using st = H(t,X) w, where w ∈ Rd is a shared vector parameter, and then we distinguish between the following two cases for learning:
Case 1. A positive instance (i.e., a labeled trigger) of type t is found in the label sequence Ỹ (If many triggers are found, we address each one individually). Assume j is the labeled trigger’s index. Here we employ a positive margin λ+ and maximize the following objective:
p+(X, Ỹ , j) = exp(st(j) − λ+)
exp(st(j) − λ+) + ∑M m 6=j exp(s t (m)) (1)
where st(j) denotes the j th word’s score in the score vector st. This objective encourages a margin of at least λ+ (Wang et al., 2018) in scores of triggers
1[CLS] and [SEP] are special tokens used in BERT. 2A word in BERT may broke down into many subwords (Sennrich et al., 2016), and here we only consider the first subword to make H(t,X) have the same length as the input.
and context words, which therefore makes groundtruth triggers more separable. Note in this case, a sentence may contain “hidden” false negatives. Motivated by the fact that triggers are generally sparsely distributed (Lin et al., 2018), we employ a Gaussian regularization to reduce the penalty. Particularly, we obtain a new score vector ŝt with:
ŝt(m) = s t (m) ×N (|m− j|) (2)
where N (·) indicates the standard univariate Gaussian density function, and |m − j| is the distance between the mth word wm and the labeled trigger. This new score vector ŝt put small weights on words far away from the labeled trigger and is shown to marginally improve learning (§ 6.2).
Case 2. There is no trigger of type t found in Ỹ . In this case, we use the [SEP] token as a “no-event-existing” indicator and optimize to give it the highest score. It should be noted, however, that such a case may contain false negatives. To address them, we introduce a negative margin λ−3 to reduce the penalty:
p−(X, Ỹ ) = exp(st∆ + λ−)
exp(st∆ + λ−) + ∑M m 6=∆ exp(s t (m)) (3)
where ∆ indicates the index of [SEP]. In this way, the negative margin λ− instead loosens the gap between the indicator [SEP] and other words. As a result, the model is more forgiving of situations when certain words score higher than the “no event exists” indicator [SEP].
Training and Testing Protocols. The overall loss function for learning is:
L = − ∑
(X,Ỹ )∈D
∑ t∈T [ δt × ∑
j:yj=t
log p+(X, Ỹ , j) +
(1− δt)× log p−(X, Ỹ ) ] (4)
where (X, Ỹ ) ∈ D ranges over each instance in the training set D; t enumerates each event type; δt is a Dirac delta function:
δt = { 1 If a trigger of type t is found (Case 1) 0 Otherwise (Case 2)
(5)
In the inference stage, given a test sentence X , we compute a normalized probability vector pt:
pt = softmax(st) (6)
3Note that λ− has a positive value. We name it “negative margin” just to distinguish from the positive margin λ+.
Algorithm 1: Uncertainty-guided retraining regime
Input :The training dataset D = {Xi, Ỹi}ni=1; Output :The optimal model parameter Θ;
1 while not convergence do 2 Sample a training example (X , Ỹ ) from D; 3 if It is a burn-in or a normal training stage then 4 Update Θ on (X , Ỹ ) using Equation (4) 5 else 6 Build an uncertainty-regularized label sequence Ỹ ′ with MC-Dropout (§ 3.3); 7 Update Θ on (X , Ỹ ′) using Equation (4) 8 end if 9 end while
and then compose a set for event triggers of type t as: {wi | pt(i) > τ ; i 6=∆}, where τ is a threshold defined as 1 / N , with N being the length of the sentence (namely, when the predictive probability of a token is above a uniform distribution, we consider it as a trigger).

heading: 3.3 Uncertainty-Guided Retraining Regime
text: In addition to the noise-tolerant learning paradigm, we also design an uncertainty-guided retraining mechanism, in which we correct the potential labels for optimization (Algorithm 1).
Assume (X , Ỹ ) is a training example. In the uncertainty interfering stage, we assume that X is an unlabeled sentence and re-predict the event label sequence using the current model. We use Monte Carlo Dropout (MC-Dropout) (Gal and Ghahramani, 2016) to assess the model’s uncertainty on the prediction. Particularly, for each event type t, we predict the event triggers K times with dropout layers activated. Assume the resulting prediction set is {qi}Ki=1, where qi is the i
th prediction and N(qi) is its frequency4. We then create a categorical distribution using N(qi)/K as parameters and sample out a prediction from the categorical distribution as the predict result (This benefits predictions that the model is more confident in). We finally convert the prediction as a label sequence Y ′ and train a model on (X , Ỹ ′). We alternate between this uncertainty interfering stage and a standard training stage after several burn-in steps.
Connection to EM Algorithm. Intuitively, our approach can be viewed as an expectation maximization (EM) algorithm (Dempster et al., 1977) using MC-Dropout to approximate the posterior.
4For instance, for the sentence shown in Figure 3, if we consider the Attack type and set K = 5, we may get a prediction set: {[SEP], devastated, [SEP], devastated, devastated}. In this case, N([SEP]) = 2 and N (devastated) = 3.
Denote the original log-likelihood function as logL(Θ; D), where Θ and D indicate the model parameter and partially labeled data respectively. Let Θ(t) be the parameter at the tth iteration. We can view our method as introducing a hidden variable Z to represent the labels of false negatives and then alternating between two steps: (i) An expectation (E) step, which uses the category distribution generated by MC-Dropout as an approximate of the intractable posterior p(Z|D,Θ(t)). (ii) A maximization (M) step, which maximizes the expectation Ep(Z|D,Θ(t))[logL(Θ; D,Z)] for optimization.

heading: 4 Experimental Setups
text: Datasets. We conduct our experiments on ACE 2005 and MAVEN5 (Wang et al., 2020), with data statistics shown in Table 1. In light of the partial annotation issue in ACE, we re-annotate its development and test sets, using a method combining automatic potential case identification and human validation (The details are shown in Appendix A). To facilitate a fine-grained analysis, we also split up all potential cases into two categories: (i) a challenge set, which consists of unlabeled words where more than half of the ED models predict that they act as triggers, and (ii) a control set, which consists of unlabeled words where fewer than half of the ED models predict that they act as triggers. Table 2 gives the final results, indicating the partial annotation issue is crucial — for instance, on the test set the unlabeled ratio is 19.3%.
Implementations. In our approach, we use BERT-large architecture for ACE 2005 (Lin et al., 2020; Nguyen et al., 2021), and BERT-base for MAVEN (Wang et al., 2020). As for hyper-parameters, the batch size is set to 10 for ACE 2005 and 20 for MAVEN respectively, chosen from [2, 5, 10, 20, 30]. The learning rate is set at 1e-5 for both datasets, chosen from [5e-5, 1e5, 5e-6, 1e-6]. In the margin softmax regime, the positive margin λ+ is set to 10, and the negative margin λ− is set to 1; these values are chosen from [0.1, 0.5, 1, 5, 10, 50, 100]. In the uncertaintyguided retraining mechanism, the number of prediction times K is empirically set to 20 for a tradeoff between speed and efficiency. We release the data and the code at https://github.com/ jianliu-ml/partialED.
5It should be noted that MAVEN provides a candidate trigger set for prediction, so the evaluation problem caused by partial annotation on this dataset is not a concern.
Evaluation Settings. We investigate three evaluation settings: (i) A full training setting, in which we use the original training set for learning. Yet, because the original training set inevitably contains unlabeled events, this setting is still a partial learning setting. (ii) A data removal setting, in which we exclude a portion of events from the training setting to study whether the performance drop is caused by a degraded number of positive examples. (iii) A data masking setting, in which we remove the labeling information of some events (by replacing their labels to O) to simulate a more serious partial annotation scene. We use precision (P), recall (R), and F1 as evaluation metrics following previous studies (Ji and Grishman, 2008; Li et al., 2013), and to against randomness, we report experimental results based on a 5-run average.
Baselines. We compare our approach to supervised and partial learning methods. For ACE 2005, we consider the following supervised learning methods: Hybrid (Feng et al., 2016), which combines Recurrent Neural Networks and Convolutional Neural Networks; SeqBERT (Yang et al., 2019), which introduces BERT representations; BERTQA (Du and Cardie, 2020; Liu et al., 2020a), which frames ED as a question answering problem; OneIE (Lin et al., 2020), which uses Graph Neural Networks to learn document-level clues; FourIE (Nguyen et al., 2021), which uses an interaction network to combine four information extraction tasks jointly. For MAVEN, we consider DMBERT (Wang et al., 2019) and BERT-CRF (Wang et al.,
2020) as baselines. We consider the following partial learning methods: (1) HiddenCRF (Jie et al., 2019), which treats missing labels as latent variables and infers them using a CRF model (We follow the original paper and use SeqBERT for parameter initialization); (2) NegSPL (Li et al., 2021b), which applies negative sampling for learning and shows good results on NER (we use the same strategy to tune the sampling hyper-parameter). (3) Self-Pu (Chen et al., 2020), which is a self-training boosted method for general positive and unlabeled learning. Our approach is denoted by PromptLoc.

heading: 5 Experimental Results
text: Results in the Full Training Setting. Tables 3 and 4 show results in the full training setting. Accordingly, our method achieves the best F1 on the clean ACE test set and the MAVEN development/test set, suggesting its efficacy. Comparing the results on the original ACE test set is interesting: our method has lower precision than other methods, but when applied to the revised set, the precision is greatly boosted — this implies that our
model does predict triggers not annotated in the original test set. Lastly, partial learning approaches generally outperform supervised methods, showing that the partial annotation issue is a practical concern to be addressed in the ED task.
Results in the Data Removal Setting. Finally, we consider the data removal setting to study the impact of a lack of positive examples, and we show results in Figure 4. According to the results, while our model consistently outperforms others, the gap is small, implying that a reduced number of positive instances is not a major factor impeding learning, especially when there are relatively abundant training examples (e.g., p > 60%) or the pre-trained language models are applied (It does have a significant impact on non-BERT models e.g., Hybrid).
Results in the Data Masking Setting. We then consider the data masking setting and we show results6 in Figure 5. Here p denotes the ratio of remaining examples (i.e., we mask the labels of 1 - p events). According to the results, our approach outperforms prior methods by significant margins. For example, on the ACE 2005, when 70% of triggers are masked (p = 30%), our approach obtains 70% in F1, outperforming previous methods by 30% in F1;
6We use the development set for MAVEN because the official site has a submission limit of only 5 per day.
when 90% are masked (p = 10%), our approach still achieves 60% in F1, yet previous methods achieves only 20% in F1. Another interesting finding is that our approach yields better results than in the data removal setting (+2.4% and +1.7% in F1 on ACE 2005 and MAVEN). This directly demonstrates our approach’s ability to learn from unlabeled events.

heading: 6 Qualitative Analysis

heading: 6.1 Insights of the Formulation
text: We conduct a sanity check experiment to understand why our trigger localization paradigm works. First, we randomly select an event type and collect N sentences (200 in our experiments) with events of this type. Then, we create two training examples from each sentence: one with original labels and one with all labels masked — this results in a highly mislabeled dataset. Finally, we train two models — one for trigger localization and the other for hard classification — and evaluate them on a leave-out test set. Table 5 gives the results, where we note that even in this extreme partial annotation scene, our trigger localization paradigm performs well, yielding 70.5% in F1 compared to 73.0% in F1 using clean dataset for training. The hard classification based approach, on the other hand, behaves poorly, yielding a drop of 30% in F1.
In Figure 6, we visualize the learned probabilities of two models on ground-truth triggers and contexts. According to the results, our method can maintain a separation between ground-truth triggers and context words in this extreme partial annotation scene. However, the hard classificationbased model is very sensitive to partial annotation noise and can not obtain a clear boundary between the ground-truth triggers and context words. For the above reason, incorporating an adaptive τ has little effect on the performance (Table 5).

heading: 6.2 Ablations on Margin Softmax and Uncertainty Retraining
text: Table 6 (Top) shows an ablation study on the margin softmax regime, based on the data masking settings, where we study the impact of positive margin λ+, negative margin λ−, and Gaussian regularization respectively. According to the results, we find that the negative margin λ− is the most effective, yet the effects of different components are complimentary. An ablation on the multiple triggers are shown in Appendix C.
In Table 6 (Bottom), we conduct an ablation study on our uncertainty-guided retraining mechanism and compare it to: (i) w/o uncertainty, which excludes the uncertainty interfering stage for learning, (ii) DirectPred, which retains the stage but uses predicted labels directly for model retraining, and (iii) BoostLearn, which considers half of the dataset to be clean and the other half to be unlabeled and conducts a bootstrapping process (Grézl and Karafiát, 2013). The results have ver-
ified the effectiveness of our method — without the uncertainty mechanism (w/o uncertainty), the performance drops 2.4% in F1 on average for ACE and 2.1% for MAVEN. The major advantage of our method lies in that it can select reliable predictions for training — as evidence, we have checked the predictions with high probability (> 0.9) in the categorical distribution and found that 91.4% of them are correct. Finally, the results suggest that our uncertainty-guided mechanism can also promote OneIE and NegSPL, particularly in scenes with large unlabeled rates (e.g., p = 10% and p = 50%).

heading: 6.3 Analysis of Unlabeled Cases
text: We explore common patterns of unlabeled events in Table 7. Indeed, 51.1% of them lack a discernible pattern, which could just be due to the annotator’s negligence. For example, in case 2, the genocide event is labeled only in the first sentence but not in the subsequent one. The other patterns we find include light verbs (20.7%), such as got in case 4, rare words (25.2%), such as pummel in case 8, and co-reference based triggers (3%), such as it in case 11. These examples are hard for human annotators. We have also investigated the suspicious cases encountered in our re-annotation procedure. Aside from 11% merely mis-predicted by a model, we find two prevalent patterns: (i) compound nouns (54%), such as “election” in “create an election code”, which does not refer to events, and (ii) definition violation (35%), such as “lobby” in “Bush plan to lobby allies ...” — though many event detectors predict “lobby” as a Meet event, but in the ACE event ontology, a Meet event is defined as “a meeting event is physically located somewhere”. The comparison of cases of the control and challenge set is shown in Appendix A.2.

heading: 6.4 A Real-World Annotation Test
text: Finally, we conduct a real-world annotation test to investigate the practical applicability of our approach. Particularly, we use WikiEvents (Li et al., 2021a) as the test bed and employ two annotators to annotate events in 100 randomly selected training documents. For tractability, we only consider 10 most frequent event types and limit the annotation time to 4 hours. After deleting incorrect labels, we obtain A1 and A2, two sets with annotation rates of 67% and 52%, respectively. We then train models on A1, A2, and the original 100 labeled documents respectively and test them on the test set. The performances of different models are shown in Figure 8. According to the results, when trained on A1 and A2, previous models exhibit a significant drop in F1 (more than 25%). By comparison, our method achieves a good performance and performs comparably to methods that use the original training set for learning. This indicates its efficacy in dealing with the partial annotation issue.

heading: 7 Conclusion
text: In this study, we investigate the partial annotation problem in ED, a critical yet less-explored problem. We motivate a new learning model for ED and investigate its effectiveness in a variety of par-
tial annotation settings. We also provide two reannotated subsets of ACE 2005 to the community as a data contribution in order to establish a fair evaluation. In the future, we plan to investigate the theoretical aspects of our approach and increase its scope by applying it to other information extraction tasks suffering the partial annotation issue, such as named entity recognition and relation extraction.

heading: 8 Limitations
text: There are two limitations of this study that could be addressed in future research. First, this study focuses solely on the ED task. In the future, we seek to extend it to the overall event extraction (EE) task, which also includes the event argument extraction task, where a complete annotation is more challenging than in ED. Second, our study models the partially labeled training data instead of annotators. Indeed, the annotators produce the data, so building a model for annotators may be an essential way to address the partial learning problem. For example, an annotator may be more careless than others and generate more noisy data. Consequently, a robust model for the task should give a lower belief in the data of this annotator to improve learning. Lastly, our research raises no ethical issues because it focuses solely on the technical aspects of a normal information extraction problem.

heading: Acknowledgments
text: This work is supported by the National Natural Science Foundation of China (No.62106016), the Open Projects Program of the State Key Laboratory of Multimodal Artificial Intelligence Systems, and the Tencent Open Fund.

heading: A ACE 2005 Dataset Revision
text: Given that the ACE 2005 dataset is partially annotated and that comparing models on a partially annotated test set results in biased results, we revised the ACE 2005 development and test sets to create a fair benchmark. Specifically, we create an automatic method that incorporates (i) a potential false negative identification stage to identify all possible unlabeled cases and (ii) a human validation stage to manually validate each case.
A.1 Potential False Negative Identification
To identify potential unlabeled cases, we first train a set of 20 different ED models with diverse archi-
tectures7 ranging from Feed-Forward Network Networks (Liu et al., 2017), Convolutional Network Networks (Chen et al., 2015), Recurrent Neural Networks (Nguyen et al., 2016), Graph Convolutional neural networks (Liu et al., 2018b) to pretrained language models (Yang et al., 2019), and then check their predictions on the development and test sets. The model details are shown in Table 8. Our intuition is that a wide range of ED models with various architectures can integrate a variety of inductive biases, and we regard any predicted trigger whose original label is O to be a potentially unlabeled example. Consequently, we uncover 112 and 136 potentially unlabeled cases on the ACE 2005 development and test sets respectively. To undertake a finer-grained analysis, we divide all the potential cases further into two groups: (i) a challenge set, in which more than half of the ED models predict an event label for a word whose initial label is O, and (ii) a control set in which fewer than half of the models do.
A.2 Human Validation
In the human validation stage, we manually check each potential case following the ACE Event Annotation Guidelines8. Specifically, we employ two annotators9 to analyze each case independently, resulting in an inter-annotator agreement κ=0.81, and a third annotator to resolve the conflict. Table 9 summarizes the final human validation results — on the ACE 2005 development set, 53 unlabeled cases are finally confirmed (with a verification rate
7For each architecture, we create one model using the hyper-parameters specified in the original paper, as well as three variations with additional hidden layers.
8https://www.ldc.upenn.edu/ sites/www.ldc.upenn.edu/files/ english-events-guidelines-v5.4.3.pdf
9The annotators were recruited from a pool of students who had completed a high-level NLP class at the author’s institution. Each annotator is compensated with around $70 for validating the total of 248 cases, resulting in an approximate payment of $0.28 per case. In § 6.3, a similar recruitment process and composition are adopted for annotations.
of 47.3%), producing a 10.5% percent ratio of unlabeled examples to labeled ones; on the ACE 2005 test set, 86 unlabeled cases are identified (with a verification rate of 60.2%), producing a 19.3 percent ratio of unlabeled examples to labeled ones. The high unlabeled ratio shows that the partial annotation problem is critical for the ACE 2005 corpus. Interestingly, we also note the challenge set has a lower validation rate than the control set. One reason for this is that the challenge set contains many spurious cases, such as compound nouns that are not event triggers, lowering the validation rate, whereas the control set contains many difficult cases, such as light verbs and unusual words that are ground-truth triggers missed by annotators, boosting the validation rate. We discuss the specific examples in Section 6.3.

heading: B Ablation on Prompting Strategy
text: We compare different prompting strategies, including “No Prompting”, which does not uses prompting strategy, but build separate model for each event type. “Prompting w Type”, which is our approach using event type as prompt. “Prompting w Description”, which uses the event type description as the prompt. According to the results in Table 10, the prompting mechanism is not an important factor for improvement — the method without prompting (No Prompting) also yields good results. However, unlike the prompting method, which allows for natural parameter sharing, it necessitates the building of individual models for each event type, which may be costly in a real-world setting. Furthermore, we note that there is no noticeable difference when type or description are used as prompts.

heading: C Performance with Multiple Triggers
text: We next investigate how well our approach performs in cases where the sentence contain multiple triggers. In the original ACE dataset, 25.1% (790
out of 3136) of all sentences containing events have more than one event trigger. However, because our method treats different event types separately, it may only be impacted by sentences that contain two triggers of the same event type — such cases account for only 7% (245 out of 3136). Figure 8 shows the results of our approach for sentences with a single trigger and sentences with multiple triggers. The gap between the two is very small, indicating that our approach is effective for sentences with multiple triggers.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 8

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 8

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: Appendix A

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Appendix A, ACE 2005.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix A

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix A
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A
C 3 Did you run computational experiments? Section 5 and 6

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants?

heading: Appendix A

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix A

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix A
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: There are two limitations of this study that could be addressed in future research. First, this study focuses solely on the ED task. In the future, we seek to extend it to the overall event extraction (EE) task, which also includes the event argument extraction task, where a complete annotation is more challenging than in ED. Second, our study models the partially labeled training data instead of annotators. Indeed, the annotators produce the data, so building a model for annotators may be an essential way to address the partial learning problem. For example, an annotator may be more careless than others and generate more noisy data. Consequently, a robust model for the task should give a lower belief in the data of this annotator to improve learning. Lastly, our research raises no ethical issues because it focuses solely on the technical aspects of a normal information extraction problem