Venue: NAACL 2021

A Context-Dependent Gated Module for Incorporating Symbolic Semantics into Event Coreference Resolution

Tuan Lai, Heng Ji

University of Illinois at Urbana-Champaign

{tuanml2, hegnji}@illinois.edu

Trung Bui, Quan Hung Tran, Franck Dernoncourt, Walter Chang

Adobe Research

{bui, qtran, franck.dernoncourt, wachang}@adobe.com

###### Abstract

Event coreference resolution is an important research problem with many applications. Despite the recent remarkable success of pre-trained language models, we argue that it is still highly beneficial to utilize symbolic features for the task. However, as the input for coreference resolution typically comes from upstream components in the information extraction pipeline, the automatically extracted symbolic features can be noisy and contain errors. Also, depending on the specific context, some features can be more informative than others. Motivated by these observations, we propose a novel context-dependent gated module to adaptively control the information flows from the input symbolic features. Combined with a simple noisy training method, our best models achieve state-of-the-art results on two datasets: ACE 2005 and KBP 2016.1

Footnote 1: The code is publicly available at [https://github.com/laiutan245/eventcoref](https://github.com/laiutan245/eventcoref).

+
Footnote †: _Corresponding of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3491–3499 June 6–11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Within-document event coreference resolution is the task of clustering event mentions in a text that refer to the same real-world events Lu and Ng (2018). It is an important research problem, with many applications Vanderwende et al. (2004); Ji and Grishman (2011); Choubey et al. (2018). Since the trigger of an event mention is typically the word or phrase that most clearly describes the event, virtually all previous approaches employ features related to event triggers in one form or another. To achieve better performance, many methods also need to use a variety of additional symbolic features such as event types, attributes, and arguments Chen et al. (2009); Chen and Ji (2009); Zhang et al. (2015); Sammons et al. (2015); Lu and Ng (2016); Chen and Ng (2016); Duncan et al. (2017). Previous neural methods Nguyen et al. (2016); Choubey and Huang (2017); Huang et al. (2019) also use noncontextual word embeddings such as word2vec Mikolov et al. (2013) or GloVe Pennington et al. (2014).

With the recent remarkable success of language models such as BERT Devlin et al. (2019) and SpanBERT Joshi et al. (2020), one natural question is whether we can simply use these models for coreference resolution without relying on any additional features. We argue that it is still highly beneficial to utilize symbolic features, especially when they are clean and have complementary information. Table 1 shows an example in the ACE 2005 dataset, where our baseline SpanBERT model incorrectly predicts the highlighted event mentions to be coreferential. The event triggers are semantically similar, making it challenging for our model to distinguish. However, notice that the event \(\{\text{head out}\}_{\text{evl}}\) is mentioned as if it was a real occurrence, and so its modality attribute is ASSERTED LDC (2005). In contrast, because of the phrase "were set to", we can infer that the event \(\{\text{leave}\}_{\text{ev2}}\) did not actually happen (i.e., its modality attribute is OTHER). Therefore, our model should be able to avoid the mistake if it utilizes additional symbolic features such as the modality attribute in this case.

There are several previous methods that use contextual embeddings together with type-based or argument-based information Lu et al. (2020); Yu et al. (2020). For example, Lu et al. (2020) proposes a new mechanism to better exploit event type information for coreference resolution. Despite their impressive performance, these methods are specific to one particular type of additional information.

In this paper, we propose general and effective methods for incorporating a wide range of sym

\begin{table}
\begin{tabular}{p{227.6pt}} ... we are seeing these soldiers \(\{\text{head out}\}_{\text{eff}}\) ... \\ ... these soldiers were set to \(\{\text{leave}\}_{\text{red}}\) in January ... \\ ev1 (Movement:Transport): Modality = ASSERTED \\ ev2 (Movement:Transport): Modality = OTHER \\ \end{tabular}
\end{table}
Table 1: An example of using the modality attribute to improve event coreference resolution.

 

bolic features into event coreference resolution. Simply concatenating symbolic features with contextual embeddings is not optimal, since the features can be noisy and contain errors. Also, depending on the context, some features can be more informative than others. Therefore, we design a novel context-dependent gated module to extract information from the symbolic features selectively. Combined with a simple regularization method that randomly adds noise into the features during training, our best models achieve state-of-the-art results on ACE 2005 Walker et al. (2006) and KBP 2016 Mitamura et al. (2016) datasets. To the best of our knowledge, our work is the first to explicitly focus on dealing with various noisy symbolic features for event coreference resolution.

## 2 Methods

### Preliminaries

We focus on within-document event coreference resolution. The input to our model is a document \(D\) consisting of \(n\) tokens and \(k\) (predicted) event mentions \(\{m_{1},m_{2},\dots,m_{k}\}\). For each \(m_{i}\), we denote the start and end indices of its trigger by \(s_{i}\) and \(e_{i}\) respectively. We assume the mentions are ordered based on \(s_{i}\) (i.e., If \(i\leq j\) then \(s_{i}\leq s_{j}\)).

We also assume each \(m_{i}\) has \(K\) (predicted) categorical features \(\{c^{(1)}_{i},c^{(2)}_{i},\ldots,c^{(K)}_{i}\}\), with each \(c^{(u)}_{i}\in\{1,2,\dots,N_{u}\}\) taking one of \(N_{u}\) different discrete values. Table 2 lists the symbolic features we consider in this work. The definitions of the features and their possible values are in ACE and Rich ERE guidelines LDC (2005; Mitamura et al., 2016). The accuracy scores of the symbolic feature predictors are also shown in Table 2. We use OneIE Lin et al. (2020) to identify event mentions along with their subtypes. For other symbolic features, we train a joint classification model based on SpanBERT. The appendix contains more details.

### Single-Mention Encoder

Given a document \(D\), our model first forms a contextualized representation for each input token using a Transformer encoder Joshi et al. (2020). Let \(\mathbf{X}=(\mathbf{x}_{1},...,\mathbf{x}_{n})\) be the output of the encoder, where \(\textbf{x}_{i}\in\mathbb{R}^{d}\). Then, for each mention \(m_{i}\), its trigger's representation \(\textbf{t}_{i}\) is defined as the average of its token embeddings:

\[\textbf{t}_{i}=\sum_{j=s_{i}}^{e_{i}}\frac{\textbf{x}_{j}}{e_{i}-s_{i}+1} \tag{1}\]

Next, by using \(K\) trainable embedding matrices, we convert the symbolic features of \(m_{i}\) into \(K\) vectors \(\{\textbf{h}^{(1)}_{i},\textbf{h}^{(2)}_{i},\dots,\textbf{h}^{(K)}_{i}\}\), where \(\textbf{h}^{(u)}_{i}\in\mathbb{R}^{l}\).

### Mention-Pair Encoder and Scorer

Given two event mentions \(m_{i}\) and \(m_{j}\), we define their trigger-based pair representation as:

\[\textbf{t}_{ij}=\text{FFNN}_{\text{t}}\big{(}\big{[}\textbf{t}_{i},\textbf{t} _{j},\textbf{t}_{i}\circ\textbf{t}_{j}\big{]}\big{)} \tag{2}\]

where \(\text{FFNN}_{\text{t}}\) is a feedforward network mapping from \(\mathbb{R}^{3\times d}\rightarrow\mathbb{R}^{p}\), and \(\circ\) is element-wise multiplication. Similarly, we can compute their feature-based pair representations \(\{\textbf{h}^{(1)}_{ij},\textbf{h}^{(2)}_{ij},\dots,\textbf{h}^{(K)}_{ij}\}\) as follows:

\[\textbf{h}^{(u)}_{ij}=\text{FFNN}_{\text{u}}\big{(}\big{[}\textbf{h}^{(u)}_{ i},\textbf{h}^{(u)}_{j},\textbf{h}^{(u)}_{i}\circ\textbf{h}^{(u)}_{j}\big{]} \big{)} \tag{3}\]

where \(u\in\{1,2,\dots,K\}\), and \(\text{FFNN}_{\text{u}}\) is a feedforward network mapping from \(\mathbb{R}^{3\times l}\rightarrow\mathbb{R}^{p}\).

Now, the most straightforward way to build the final pair representation \(\textbf{f}_{ij}\) of \(m_{i}\) and \(m_{j}\) is to simply concatenate the trigger-based representation and all the feature-based representations together:

\[\textbf{f}_{ij}=[\textbf{t}_{ij},\textbf{h}^{(1)}_{ij},\textbf{h}^{(2)}_{ij}, \dots,\textbf{h}^{(K)}_{ij}] \tag{4}\]

However, this approach is not always optimal. First, as the symbolic features are predicted, they can be noisy and contain errors. The performance of most symbolic feature predictors is far from perfect (Table 2). Also, depending on the specific context, some features can be more useful than others.

Inspired by studies on gated modules Lin et al. (2019); Lai et al. (2019), we propose **Context-Dependent Gated Module** (CDGM), which uses a gating mechanism to extract information from the input symbolic features selectively (Figure 1). Given two mentions \(m_{i}\) and \(m_{j}\), we use their trigger feature vector \(\textbf{t}_{ij}\) as the main controlling context to compute the filtered representation \(\overline{\textbf{h}}^{(u)}_{ij}\):

\[\overline{\textbf{h}}^{(u)}_{ij}=\text{CDGM}^{(u)}\big{(}\textbf{t}_{ij}, \textbf{h}^{(u)}_{ij}\big{)} \tag{5}\]

\begin{table}
\begin{tabular}{|c|l|c|c|c|} \hline Dataset & Features & Acc. & Acc. & Acc. \\  & & (Train) & (Dev) & (Test) \\ \hline \multirow{4}{*}{ACE} & Type & 0.999 & 0.945 & 0.953 \\ \cline{2-5}  & Polarity & 0.999 & 0.994 & 0.988 \\ \cline{2-5}  & Modality & 0.999 & 0.856 & 0.884 \\ \cline{2-5}  & Genericity & 0.999 & 0.865 & 0.872 \\ \cline{2-5}  & Tense & 0.984 & 0.802 & 0.763 \\ \hline \multirow{2}{*}{KBP} & Type & 0.960 & 0.874 & 0.818 \\ \cline{2-5}  & Realis & 0.979 & 0.845 & 0.840 \\ \hline \end{tabular}
\end{table}
Table 2: Symbolic features we consider in this work.

 

where \(u\in\{1,2,\ldots,K\}\). More specifically:

\[\begin{split}\textbf{g}_{ij}^{(u)}&=\sigma\big{(} \text{FFNN}_{\textbf{g}}^{(u)}\big{(}\big{[}\textbf{t}_{ij},\textbf{h}_{ij}^{( u)}\big{]}\big{)}\big{)}\\ \textbf{o}_{ij}^{(u)},\textbf{p}_{ij}^{(u)}&=\text{ DECOMPOSE}\big{(}\textbf{t}_{ij},\textbf{h}_{ij}^{(u)}\big{)}\\ \overline{\textbf{h}}_{ij}^{(u)}&=\textbf{g}_{ij}^{( u)}\circ\textbf{o}_{ij}^{(u)}+\left(1-\textbf{g}_{ij}^{(u)}\right)\circ \textbf{p}_{ij}^{(u)}\end{split} \tag{6}\]

where \(\sigma\) denotes sigmoid function. FFNN\({}_{\textbf{g}}^{(u)}\) is a mapping from \(\mathbb{R}^{2\times p}\rightarrow\mathbb{R}^{p}\). At a high level, \(\textbf{h}_{ij}^{(u)}\) is decomposed into an orthogonal component and a parallel component, and \(\overline{\textbf{h}}_{ij}^{(u)}\) is simply the fusion of these two components. In order to find the optimal mixture, **g\({}_{ij}\)** is used to control the composition. The decomposition unit is defined as:

\[\begin{split}\textit{Parallel}&\quad\textbf{p}_{ ij}^{(u)}=\frac{\textbf{h}_{ij}^{(u)}\cdot\textbf{t}_{ij}}{\textbf{t}_{ij} \cdot\textbf{t}_{ij}}\;\textbf{t}_{ij}\\ \textit{Orthogonal}&\quad\textbf{o}_{ij}^{(u)}= \textbf{h}_{ij}^{(u)}-\textbf{p}_{ij}^{(u)}\end{split} \tag{7}\]

where \(\cdot\) denotes dot product. The parallel component \(\textbf{p}_{ij}^{(u)}\) is the projection of \(\textbf{h}_{ij}^{(u)}\) on \(\textbf{t}_{ij}\). It can be viewed as containing information that is already part of \(\textbf{t}_{ij}\). In contrast, \(\textbf{o}_{ij}^{(u)}\) is orthogonal to \(\textbf{t}_{ij}\), and so it can be viewed as containing _new_ information. Intuitively, when the original symbolic feature vector \(\textbf{h}_{ij}^{(u)}\) is very clean and has complementary information, we want to utilize the new information in \(\textbf{o}_{ij}^{(u)}\) (i.e., we want \(\textbf{g}_{ij}^{(u)}\approx\textbf{1}\)), and vice versa.

Finally, after using CDGMs to distill symbolic features, the final pair representation \(\textbf{f}_{ij}\) of \(m_{i}\) and \(m_{j}\) can be computed as follows:

\[\textbf{f}_{ij}=[\textbf{t}_{ij},\overline{\textbf{h}}_{ij}^{(1)},\overline{ \textbf{h}}_{ij}^{(2)},\dots,\overline{\textbf{h}}_{ij}^{(K)}] \tag{8}\]

And the coreference score \(s(i,j)\) of \(m_{i}\) and \(m_{j}\) is:

\[s(i,j)=\text{FFNN}_{\textrm{a}}(\textbf{f}_{ij}) \tag{9}\]

where FFNN\({}_{\textrm{a}}\) is a mapping from \(\mathbb{R}^{(K+1)\times p}\rightarrow\mathbb{R}\).

### Training and Inference

```
Input: Document \(D\) Hyperparameters: \(\{\epsilon_{1},\epsilon_{2},\cdots,\epsilon_{K}\}\) for\(i=1~{}\dots~{}k\)do for\(u=1~{}\dots~{}K\)do  With prob. \(\epsilon_{u}\), replace \(c_{i}^{(u)}\) by \(\hat{c}_{i}^{(u)}\sim\textit{Uniform}(N_{u})\) end for  end for
```

**Algorithm 1**Noise Addition for Symbolic Features

TrainingWe use the same loss function as in Lee et al. (2017). Also, notice that the training accuracy of a feature predictor is typically much higher than its accuracy on the dev/test set (Table 2). If we simply train our model without any regularization, our CDGMs will rarely come across noisy symbolic features during training. Therefore, to encourage our CDGMs to actually learn to distill reliable signals, we also propose a simple but effective **noisy training** method. Before passing a training data batch to the model, we randomly add noise to the predicted features. More specifically, for each document \(D\) in the batch, we go through every symbolic feature of every event mention in \(D\) and consider sampling a new value for the feature. The operation is described in Algorithm 1 (we use the same notations mentioned in Section 2.1). \(\{\epsilon_{1},\epsilon_{2},\cdots,\epsilon_{K}\}\) are hyperparameters determined by validation. In general, the larger the discrepancy between the train and test accuracies, the larger \(\epsilon\).

InferenceFor each (predicted) mention \(m_{i}\), our model will assign an antecedent \(a_{i}\) from all pre

Figure 1: Overall architecture of our mention-pair encoder, which uses CDGMs to incorporate symbolic features.

 

ceding mentions or a dummy antecedent \(\epsilon\): \(a_{i}\in Y(i)=\{\epsilon,m_{1},m_{2}\ldots,m_{i-1}\}\). Basically, \(a_{i}=\operatorname*{arg\,max}_{j<i}s(i,j)\). The dummy antecedent \(\epsilon\) represents two possible cases: (1) \(m_{i}\) is not actually an event mention (2) \(m_{i}\) is indeed an event mention but it is not coreferent with any previous extracted mentions. In addition, we fix \(s(i,\epsilon)\) to be 0.

## 3 Experiments and Results

Data and Experiments SetupWe evaluate our methods on two English datasets: ACE2005 Walker et al. (2006) and KBP2016 Ji et al. (2016); Mitamura et al. (2016). We report results in terms of F1 scores obtained using the CoNLL and AVG metrics. By definition, these metrics are the summary of other standard coreference metrics, including B\({}^{3}\), MUC, CEAF\({}_{e}\), and BLANC Lu and Ng (2018). We use SpanBERT (spanbert-base-cased) as the Transformer encoder Wolf et al. (2020); Joshi et al. (2020). More details about the datasets and hyperparameters are in the appendix. We refer to models that use only trigger features as [Baseline]. In a baseline model, **t\({}_{ij}\)** is simply **t\({}_{ij}\)** (Eq. 2). We refer to models that use only the simple concatenation strategy as [Simple] (Eq. 4), and models that use the simple concatenation strategy and the noisy training method as [Noise].

Overall Results (on Predicted Mentions)Table 3 and Table 4 show the overall end-to-end results on ACE2005 and KBP2016, respectively. We use OneIE Lin et al. (2020) to extract event mentions and their types. Other features are predicted by a simple Transformer model. Overall, our full model outperforms the baseline model by a large margin and significantly outperforms state-of-the-art on KBP 2016. Our ACE 2005 scores are not directly comparable with previous work, as Peng et al. (2016) conducted 10-fold cross-validation and essentially used more training data. Nevertheless, the magnitude of the differences in scores between our best model and the state-of-the-art methods indicates the effectiveness of our methods.

Overall Results (on Ground-truth Triggers)The overall results on ACE 2005 using ground-truth triggers and predicted symbolic features are shown in Table 5. The performance of our full model is comparable with previous state-of-the-art result in Yu et al. (2020). To better analyze the usefulness of symbolic features as well as the effectiveness of our methods, we also conduct experiments using _ground-truth_ triggers and _ground-truth_ symbolic features (Table 6). First, when the symbolic features are clean, incorporating them using the simple concatenation strategy can already boost the performance significantly. The symbolic features contain information complementary to that in the SpanBERT contextual embeddings. Second, we also see that the noisy training method is not helpful when the symbolic features are clean. Unlike other regularization methods such as dropout Srivastava et al. (2014) and weight decay Krogh and Hertz (1992), the main role of our noisy training method is not to reduce overfitting in the traditional sense. Its main function is to help CDGMs learn to distill reliable signals from noisy features.

\begin{table}
\begin{tabular}{l c c} \hline ACE (Test Data) & CoNLL & AVG \\ \hline Baseline & 81.62 & 81.49 \\ Simple (All Features) & 85.75 & 85.40 \\ CDGM (All Features) & **87.90** & **88.30** \\ CDGM + Noise (All Features) & 85.40 & 85.38 \\ \hline \end{tabular}
\end{table}
Table 6: Results on ACE 2005 using gold triggers and ground-truth symbolic features.

\begin{table}
\begin{tabular}{l c c} \hline System & CoNLL & AVG \\ \hline UTD’s system Liu et al. (2015) & 32.69 & 30.08 \\ Joint Learning Liu et al. (2017b) & 35.77 & 33.08 \\ E\({}^{3}\)C Liu et al. (2020) & 41.97 & 38.66 \\ \hline Baseline & 40.57 & 37.59 \\ \hline Simple (All Features) & 41.40 & 38.58 \\ CDGM + Noise (All Features) & **43.55** & **40.61** \\ \hline \end{tabular}
\end{table}
Table 4: End-to-end results on KBP 2016 (using predicted triggers and predicted symbolic features).

\begin{table}
\begin{tabular}{l c c} \hline ACE (Cross-Validation) & CoNLL & AVG \\ \hline SSED + Supervised\({}_{\text{Extended}}\) (2016) & 55.23 & 52.53 \\ SSED + MSEP Liu et al. (2016) & 53.80 & 51.38 \\ \hline ACE (Test Data) & CoNLL & AVG \\ \hline Baseline & 58.93 & 55.78 \\ \hline Simple (All Features) & 57.55 & 54.79 \\ CDGM (All Features) & 58.99 & 56.32 \\ Noise (All Features) & 60.43 & 57.85 \\ CDGM + Noise (All Features) & **62.07** & **59.76** \\ \hline \end{tabular}
\end{table}
Table 3: End-to-end results on ACE 2005 (using predicted triggers and predicted symbolic features).

\begin{table}
\begin{tabular}{l c c} \hline ACE (Test Data) & CoNLL & AVG \\ \hline PAIREDRL Lerman et al. (2020) & 84.65 & - \\ \hline Baseline & 81.62 & 81.49 \\ Simple (All Features) & 75.32 & 74.94 \\ CDGM + Noise (All Features) & **84.76** & **83.95** \\ \hline \end{tabular}
\end{table}
Table 5: Results on ACE 2005 using gold triggers and predicted symbolic features.

 

Impact of Different Symbolic FeaturesTable 7 shows the results of incorporating different types of symbolic features on the ACE 2005 dataset. Overall, our methods consistently perform better than the simple concatenation strategy across all feature types. The gains are also larger for more noisy features than clean features (feature prediction accuracies were shown in Table 2). This suggests that our methods are particularly useful in situations where the symbolic features are noisy.

Comparison with Multi-Task LearningWe also investigate whether we can incorporate symbolic semantics into coreference resolution by simply doing multi-task training. We train our baseline model to jointly perform coreference resolution and symbolic feature prediction. The test AVG score on ACE 2005 is only 56.5. In contrast, our best model achieves an AVG score of 59.76 (Table 3).

Qualitative ExamplesTable 8 shows few examples from the ACE 2005 dataset that illustrate how incorporating symbolic features using our proposed methods can improve the performance of event conference resolution. In each example, our baseline model incorrectly predicts the highlighted event mentions to be coreferential.

Remaining ChallengesPrevious studies suggest that there exist different types and degrees of event coreference Recasens et al. (2011); Hovy et al. (2013). Many methods (including ours) focus on the full strict coreference task, but other types of coreference such as partial coreference have remained underexplored. Hovy et al. (2013) defines two core types of partial event coreference relations: subevent relations and membership relations. Subevent relations form a stereotypical sequence of events, whereas membership relations represent instances of an event collection. We leave tackling the partial coreference task to future work.

## 4 Related Work

Several previous approaches to within-document event coreference resolution operate by first applying a mention-pair model to compute pairwise distances between event mentions, and then they apply a clustering algorithm such as agglomerative clustering or spectral graph clustering Chen et al. (2009); Chen and Ji (2009); Chen and Ng (2014); Nguyen et al. (2016); Huang et al. (2019). In addition to trigger features, these methods use a variety of additional symbolic features such as event types, attributes, arguments, and distance. These approaches do not use contextual embeddings such as BERT and SpanBERT Devlin et al. (2019); Joshi et al. (2020). Recently, there are several studies that use contextual embeddings together with type-based or argument-based information Lu et al. (2020); Yu et al. (2020). These methods design networks or mechanisms that are specific to only one type of symbolic features. In contrast, our work is more general and can be effectively applied to a wide range of symbolic features.

## 5 Conclusions and Future Work

In this work, we propose a novel gated module to incorporate symbolic semantics into event coreference resolution. Combined with a simple noisy training technique, our best models achieve competitive results on ACE 2005 and KBP 2016. In the future, we aim to extend our work to address more general problems such as cross-lingual cross-document coreference resolution.

## Acknowledgement

This research is based upon work supported in part by U.S. DARPA KAIROS Program No. FA8750-19-2-1004, U.S. DARPA AIDA Program No. FA8750-18-2-0014, and Air Force No. FA8650-17-C-7715. The views and conclusions contained herein are those of the authors and should not be

\begin{table}
\begin{tabular}{l c c c} \hline \multirow{2}{*}{Features} & \multicolumn{2}{c}{AVG} & \multicolumn{2}{c}{AVG} \\  & (Simple) & (CDGM + Noise) & \(\Delta_{\text{avg}}\) \\ \hline Subtype & 56.41 & 57.02 & **+0.61** \\ Polarity & 56.06 & 57.03 & **+0.97** \\ Modality & 54.81 & 58.54 & **+3.73** \\ Genericity & 54.70 & 57.82 & **+3.12** \\ Tense & 54.28 & 56.62 & **+2.34** \\ \hline \end{tabular}
\end{table}
Table 7: Impact of Symbolic Features (ACE 2005)

\begin{table}
\begin{tabular}{l l} ... \{Negotiations\}_ori between Washington and ... ... ... think that this will affect the \{elections\}_v\% unless ... \\ ev1 (Contact:Meet) \\ ev2 (Personnel:Elect) \\ ... since you are **not** directly \{elected\}_ev1, it would be ... ... Az-Zaman daily that \{elections\}_v\%2 should be held ... \\ ev1 (Personnel:Elect): Polarity = NEGATIVE \\ ev2 (Personnel:Elect): Polarity = POSITIVE \\ ... told reporters after his \{appeal\}_ev1 was rejected ... ... most junior of the court of \{appeal\}_ev2, and its ... \\ ev1 (Justice:Appeal): Genericity = SPECIFIC \\ ev2 (Justice:Appeal): Genericity = GENERIC \\ \hline \end{tabular}
\end{table}
Table 8: Examples of using symbolic features to improve event coreference resolution.

 

interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

## References

* Chen and Ng (2014) C. Chen and Vincent Ng. 2014. Sinocoreferencer: An end-to-end chinese event coreference resolver. In _LREC_.
* Chen and Ng (2016) Chen Chen and Vincent Ng. 2016. Joint inference over a lightly supervised information extraction pipeline: Towards event coreference resolution for resource-scar languages. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_, AAAI'16, page 2913-2920. AAAI Press.
* Chen et al. (2015) Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multi-pooling convolutional neural networks. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 167-176, Beijing, China. Association for Computational Linguistics.
* Chen and Ji (2009) Zheng Chen and Heng Ji. 2009. Graph-based event coreference resolution. In _Proceedings of the 2009 Workshop on Graph-based Methods for Natural Language Processing (TextGraphs-4)_, pages 54-57, Suntec, Singapore. Association for Computational Linguistics.
* Chen et al. (2009) Zheng Chen, Heng Ji, and Robert Haralick. 2009. A pairwise event coreference model, feature impact and evaluation for event coreference resolution. In _Proceedings of the Workshop on Events in Emerging Text Types_, pages 17-22, Borovets, Bulgaria. Association for Computational Linguistics.
* Choubey and Huang (2017) Prafulla Kumar Choubey and Ruihong Huang. 2017. Event coreference resolution by iteratively unfolding inter-dependencies among events. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2124-2133, Copenhagen, Denmark. Association for Computational Linguistics.
* Choubey et al. (2018) Prafulla Kumar Choubey, Kaushik Raju, and Ruihong Huang. 2018. Identifying the most dominant event in a news article by mining event coreference relations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 340-345, New Orleans, Louisiana. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Duncan et al. (2017) Chase Duncan, Liang-Wei Chan, Haoruo Peng, Hao Wu, Shyam Upadhyay, Nitish Gupta, Chen-Tse Tsai, Mark Sammons, and Dan Roth. 2017. UICCG TACKBP2017 submissions: Entity discovery and linking, and event nugget detection and co-reference. In _Proceedings of the 2017 Text Analysis Conference, TAC 2017, Gaitherspurg, Maryland, USA, November 13-14, 2017_. NIST.
* Harris et al. (2020) C. Harris, K. J. Millman, S. Walt, Ralf Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, Nathaniel J. Smith, R. Kern, Matti Piccus, S. Hoyer, M. Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fern'andez del Rio, Mark Wiebe, P. Peterson, Pierre G'erard-Marchant, K. Sheppard, Teddy, W. Veckesser, H. Abbasi, Christoph Goldhke, and T. E. Oliphant. 2020. Array programming with numpy. _Nature_, 585 7825:357-362.
* Hovy et al. (2013) Eduard Hovy, Teruko Mitamura, Felisa Verdejo, Jun Araki, and Andrew Philpot. 2013. Events are not simple: Identity, non-identity, and quasi-identity. In _Workshop on Events: Definition, Detection, Coreference, and Representation_, pages 21-28, Atlanta, Georgia. Association for Computational Linguistics.
* Huang et al. (2019) Yin Jou Huang, Jing Lu, Sadao Kurohashi, and Vincent Ng. 2019. Improving event coreference resolution by learning argument compatibility from unlabeled data. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 785-795, Minneapolis, Minnesota. Association for Computational Linguistics.
* Ji and Grishman (2011) Heng Ji and Ralph Grishman. 2011. Knowledge base population: Successful approaches and challenges. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_, pages 1148-1158, Portland, Oregon, USA. Association for Computational Linguistics.
* Ji et al. (2016) Heng Ji, Joel Nothman, H Trang Dang, and Sydney Informatics Hub. 2016. Overview of tac-kbp2016 trilingual edl and its impact on end-to-end cold-start kbp. _Proceedings of TAC_.
* Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. _Transactions of the Association for Computational Linguistics_, 8(0):64-77 

Anders Krogh and John Hertz. 1992. A simple weight decay can improve generalization. In _Advances in Neural Information Processing Systems_, volume 4. Morgan-Kaufmann.
* Lai et al. (2019) Tuan Lai, Quan Hung Tran, Trung Bui, and Daisuke Kahara. 2019. A gated self-attention memory network for answer selection. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 5953-5959, Hong Kong, China. Association for Computational Linguistics.
* LDC (2005) LDC. 2005. English annotation guidelines for events version 5.4. 3. _Linguistic Data Consortium_, 1.
* Lee et al. (2017) Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 188-197, Copenhagen, Denmark. Association for Computational Linguistics.
* Lin et al. (2020) Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020. A joint neural model for information extraction with global features. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7999-8009, Online. Association for Computational Linguistics.
* Lin et al. (2019) Ying Lin, Liyuan Liu, Heng Ji, Dong Yu, and Jiawei Han. 2019. Reliability-aware dynamic feature composition for name tagging. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 165-174, Florence, Italy. Association for Computational Linguistics.
* Lu and Ng (2017a) J. Lu and Vincent Ng. 2017a. Learning antecedent structures for event coreference resolution. _2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA)_, pages 113-118.
* Lu and Ng (2015) Jing Lu and Vincent Ng. 2015. Utd's event nugget detection and coreference system at KBP 2015. In _Proceedings of the 2015 Text Analysis Conference, TAC 2015, Gaithersburg, Maryland, USA, November 16-17, 2015, 2015_. NIST.
* Lu and Ng (2016) Jing Lu and Vincent Ng. 2016. Event coreference resolution with multi-pass sieves. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)_, pages 3996-4003, Portoroz, Slovenia. European Language Resources Association (ELRA).
* Lu and Ng (2017b) Jing Lu and Vincent Ng. 2017b. Joint learning for event coreference resolution. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 90-101, Vancouver, Canada. Association for Computational Linguistics.
* Lu and Ng (2018) Jing Lu and Vincent Ng. 2018. Event coreference resolution: A survey of two decades of research. In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18_, pages 5479-5486. International Joint Conferences on Artificial Intelligence Organization.
* Lu et al. (2020) Yaojie Lu, Hongyu Lin, Jialong Tang, Xianpei Han, and Le Sun. 2020. End-to-end neural event coreference resolution. _arXiv preprint arXiv:2009.08153_.
* Volume 2_, NIPS'13, page 3111-3119, Red Hook, NY, USA. Curran Associates Inc.
* Mitamura et al. (2016) Teruko Mitamura, Zhengzhong Liu, and Eduard H. Hovy. 2016. Overview of TAC-KBP 2016 event nugget track. In _Proceedings of the 2016 Text Analysis Conference, TAC 2016, Gaithersburg, Maryland, USA, November 14-15, 2016_. NIST.
* Nguyen et al. (2016) Thien Huu Nguyen, Adam Meyers, and Ralph Grishman. 2016. New york university 2016 system for KBP event nugget: A deep learning approach. In _Proceedings of the 2016 Text Analysis Conference, TAC 2016, Gaithersburg, Maryland, USA, November 14-15, 2016_. NIST.
* Paszke et al. (2019) Adam Paszke, S. Gross, Francisco Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, Alban Desmaison, Andreas Kopf, E. Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, B. Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. _ArXiv_, abs/1912.01703.
* Peng et al. (2016) Haoruo Peng, Yangqiu Song, and Dan Roth. 2016. Event detection and co-reference with minimal supervision. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 392-402, Austin, Texas. Association for Computational Linguistics.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
* Recasens et al. (2011) M. Recasens, E. Hovy, and M. Marti. 2011. Identity, non-identity, and near-identity: Addressing the complexity of coreference. _Lingua_, 121:1138-1152.
* Sammons et al. (2015) Mark sammons, Haoruo Peng, Yangqiu Song, Shyam Upadhyay, Chen-Tse Tsai, Pavankumar Reddy, Subhro Roy, and Dan Roth. 2015. Illinois CCG TAC 2015 event nugget, entity discovery and linking, and slot filler validation systems. In _Proceedings of the 

2015 Text Analysis Conference, TAC 2015, Gaithersburg, Maryland, USA, November 16-17, 2015, 2015_. NIST.
* Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. _Journal of Machine Learning Research_, 15(56):1929-1958.
* Vanderwende et al. (2004) Lucy Vanderwende, Michele Banko, and Arul Menezes. 2004. Event-centric summary generation. In _Working notes of the Document Understanding Conference 2004_. ACL.
* Walker et al. (2006) Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. Ace 2005 multilingual training corpus. _Linguistic Data Consortium, Philadelphia_, 57:45.
* Wolf et al. (2020a) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jurien, Julian Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariana Drame, Quentin Hoest, and Alexander Rush. 2020a. Transformers: State-of-the-art natural language processing: The 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Wolf et al. (2020b) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierre Cristac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jerneit, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariana Drame, Quentin Loost, and Alexander Rush. 2020b. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Yu et al. (2020) Xiaodong Yu, Wenpeng Yin, and Dan Roth. 2020. Paired representation learning for event and entity coreference. _arXiv preprint arXiv:2010.12808_.
* Zhang et al. (2015) Tongtao Zhang, Hongzhi Li, Heng Ji, and Shih-Fu Chang. 2015. Cross-document event coreference resolution based on cross-media features. In _Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP2015)_.

## Appendix A Appendix

Section A.1 describes our symbolic feature predictors. Section A.2 provides the details of the datasets we used. Section A.3 describes the hyperparameters and their value ranges that were explored. Section A.4 presents our reproducibility checklist.

### Symbolic Feature Predictors

In an end-to-end setting, we train and use OneIE Lin et al. (2020) to identify event mentions along with their subtypes. For other symbolic features, we train a simple joint model. More specifically, given a document, our joint model first forms contextualized representations for the input tokens using SpanBERT Joshi et al. (2020). Each event mention's representation is then defined as the average of the embeddings of the tokens in its trigger. After that, we feed the mentions' representations into classification heads for feature value prediction. Each classification head is a standard multi-layer feedforward network with softmax output units.

The event detection performance of OneIE on the test set of ACE 2005 is 74.7 (Type-F1 score). OneIE's performance on KBP 2016 is 55.20 (Type-F1 score). For reference, the performance of the event detection component of E\({}^{3}\)C Lu et al. (2020) on KBP 2016 is 55.38 (Type-F1 score).

### Datasets Description

In this work, we use two English within-document coreference datasets: ACE 2005 and KBP 2016. The ACE 2005 English corpus contains fine-grained event annotations for 599 articles from a variety of sources. We use the same split as that stated in Chen et al. (2015), where there are 529/30/40 documents in the train/dev/test split. In ACE, a strict notion of event coreference is adopted, which requires two event mentions to be coreferential if and only if they had the same agent(s), patient(s), time, and location. For KBP 2016, we follow the setup of Lu and Ng (2017), where there are 648 documents that can be used for training and 169 documents for testing. We train our model on 509 documents randomly chosen from the training documents and tune parameters on the remaining 139 training documents. Different from ACE, KBP adopts a more relaxed definition of event coreference, where two event mentions can be coreferent as long as they intuitively refer to the same real-world event. Table 9 summarizes the basic statistics of the datasets.

\begin{table}
\begin{tabular}{l|c|c|c} Dataset & Train (\# Docs) & Dev (\# Docs) & Test (\# Docs) \\ \hline ACE-2005 & 529 & 30 & 40 \\ KBP 2016 & 509 & 139 & 169 \\ \hline \end{tabular}
\end{table}
Table 9: Basic statistics of the datasets.

 



### Hyperparameters

We use SpanBERT (spanbert-base-cased) as the Transformer encoder Wolf et al. (2020a, Joshi et al., 2020). We did hyperparameter tuning using the datasets' dev sets. For all the experiments, we pick the model which achieves the best AVG score on the dev set, and then evaluate it on the test set. For each of our models, two different learning rates are used, one for the lower pretrained Transformer encoder and one for the upper layer. The optimal hyperparameter values are variant-specific, and we experimented with the following range of possible values: \(\{8,16\}\) for batch size, {3e-5, 4e-5, 5e-5} for lower learning rate, {1e-4, 2.5e-4, 5e-4} for upper learning rate, and \(\{50,100\}\) for number of training epochs. Table 10 shows the value of \(\epsilon\) we used for each symbolic feature type. In general, the larger the discrepancy between the train and test accuracies, the larger the value of \(\epsilon\).

### Reproducibility Checklist

We present the reproducibility information of the paper. Due to license reason, we cannot provide downloadable links for ACE 2005 and KBP 2016.

Implementation Dependencies LibrariesPytorch 1.6.0 Paszke et al. (2019), Transformers 3.0.2 Wolf et al. (2020b), Numpy 1.19.1 Harris et al. (2020), CUDA 10.2.

Computing InfrastructureThe experiments were conducted on a server with Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz and NVIDIA Tesla V100 GPUs. The allocated RAM is 187G. GPU memory is 16G.

Average RuntimeTable 11 shows the estimated average run time of our full model.

Number of Model ParametersThe number of parameters in a baseline model is about 109.7M parameters. The number of parameters in a full model trained on the KBP 2016 dataset is about 111.4M parameters. The number of parameters in a full model trained on the ACE 2005 dataset is about 113.8M parameters.

Hyperparameters of Best-Performing ModelsTable 12 summarizes the hyperparameter configurations of best-performing models. Note that Table 10 already showed the hyperparameters used for the noisy training method.

Expected Validation PerformanceWe repeat training five times for each best-performing model. We show the average validation performance in Table 13. Our validation scores on KBP 2016 are not comparable to that of Lu et al. (2020), because we split the original 648 training documents into the final train set and dev set randomly. We still use the same test set. For each best-performing model, we report the test performance of the checkpoint with the best AVG score in the main paper.

\begin{table}
\begin{tabular}{l|c|c|c} Dataset & Avg. CoNLL score & Avg. AVG score \\ \hline ACE 2005 & 60.20 & 58.80 \\ (end-to-end) & \multicolumn{1}{r}{54.86} \\ \cline{2-4}   KBP 2016 & \multirow{2}{*}{54.86} & \multirow{2}{*}{48.83} \\ (end-to-end) & \multicolumn{1}{r}{58.02} \\ \cline{1-1} \cline{3-4}   ACE 2005 & & & \\ (gold mentions) & \multicolumn{1}{r}{81.9} & \multicolumn{1}{r}{83.02} \\ \hline \end{tabular}
\end{table}
Table 13: Average validation performance.

\begin{table}
\begin{tabular}{|p{56.9pt}|p{56.9pt}|p{56.9pt}|p{56.9pt}|} \hline Dataset & Features & \(\epsilon\) (predicted mentions) & \(\epsilon\) (gold mentions) \\ \hline \multirow{4}{*}{ACE 2005} & Type & 0.00 & 0.10 \\ \cline{2-4}  & Polarity & 0.00 & 0.02 \\ \cline{2-4}  & Modality & 0.15 & 0.20 \\ \cline{2-4}  & Genericity & 0.15 & 0.20 \\ \cline{2-4}  & Tense & 0.25 & 0.30 \\ \hline \multirow{2}{*}{KBP 2016} & Type & 0.05 & - \\ \cline{2-4}  & Realis & 0.10 & - \\ \hline \end{tabular}
\end{table}
Table 10: The value of \(\epsilon\) for each feature type.