SlowGuess/NAACL_2021_OCR huggingface dataset:
Tuan Lai, Heng Ji  
University of Illinois at Urbana-Champaign  
{tuanml2, hengji} @illinois.edu  
Trung Bui, Quan Hung Tran, Franck Dernoncourt, Walter Chang  
Adobe Research  
{bui, qtran, franck.dernoncourt, wachang} @adobe.com
Event coreference resolution is an important research problem with many applications. Despite the recent remarkable success of pretrained language models, we argue that it is still highly beneficial to utilize symbolic features for the task. However, as the input for coreference resolution typically comes from upstream components in the information extraction pipeline, the automatically extracted symbolic features can be noisy and contain errors. Also, depending on the specific context, some features can be more informative than others. Motivated by these observations, we propose a novel context-dependent gated module to adaptively control the information flows from the input symbolic features. Combined with a simple noisy training method, our best models achieve state-of-the-art results on two datasets: ACE 2005 and KBP 2016.
Within-document event coreference resolution is the task of clustering event mentions in a text that refer to the same real-world events (Lu and Ng, 2018). It is an important research problem, with many applications (Vanderwende et al., 2004; Ji and Grishman, 2011; Choubey et al., 2018). Since the trigger of an event mention is typically the word or phrase that most clearly describes the event, virtually all previous approaches employ features related to event triggers in one form or another. To achieve better performance, many methods also need to use a variety of additional symbolic features such as event types, attributes, and arguments (Chen et al., 2009; Chen and Ji, 2009; Zhang et al., 2015; Sammons et al., 2015; Lu and Ng, 2016; Chen and Ng, 2016; Duncan et al., 2017). Previous neural methods (Nguyen et al., 2016; Choubey and Huang, 2017; Huang et al., 2019) also use noncontextual word embeddings such as word2vec
... we are seeing these soldiers  $\{\mathrm{head~out}\}_{\mathrm{ev1}}$  ...   
... these soldiers were set to  $\{\mathrm{leave}\}_{\mathrm{ev2}}$  in January ...   
ev1 (Movement:Transport): Modality  $=$  ASSERTED   
ev2 (Movement:Transport): Modality  $=$  OTHER
Table 1: An example of using the modality attribute to improve event coreference resolution.
(Mikolov et al., 2013) or GloVe (Pennington et al., 2014).
With the recent remarkable success of language models such as BERT (Devlin et al., 2019) and SpanBERT (Joshi et al., 2020), one natural question is whether we can simply use these models for coreference resolution without relying on any additional features. We argue that it is still highly beneficial to utilize symbolic features, especially when they are clean and have complementary information. Table 1 shows an example in the ACE 2005 dataset, where our baseline SpanBERT model incorrectly predicts the highlighted event mentions to be coreferential. The event triggers are semantically similar, making it challenging for our model to distinguish. However, notice that the event  $\{\text{head out}\}_{\text{ev1}}$  is mentioned as if it was a real occurrence, and so its modality attribute is ASSERTED (LDC, 2005). In contrast, because of the phrase "were set to", we can infer that the event  $\{\text{leave}\}_{\text{ev2}}$  did not actually happen (i.e., its modality attribute is OTHER). Therefore, our model should be able to avoid the mistake if it utilizes additional symbolic features such as the modality attribute in this case.
There are several previous methods that use contextual embeddings together with type-based or argument-based information (Lu et al., 2020; Yu et al., 2020). For example, Lu et al. (2020) proposes a new mechanism to better exploit event type information for coreference resolution. Despite their impressive performance, these methods are specific to one particular type of additional information.
In this paper, we propose general and effective methods for incorporating a wide range of sym
The code is publicly available at https://github.com/laituan245/eventcoref.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3491-3499 June 6-11, 2021. ©2021 Association for Computational Linguistics
bolic features into event coreference resolution. Simply concatenating symbolic features with contextual embeddings is not optimal, since the features can be noisy and contain errors. Also, depending on the context, some features can be more informative than others. Therefore, we design a novel context-dependent gated module to extract information from the symbolic features selectively. Combined with a simple regularization method that randomly adds noise into the features during training, our best models achieve state-of-the-art results on ACE 2005 (Walker et al., 2006) and KBP 2016 (Mitamura et al., 2016) datasets. To the best of our knowledge, our work is the first to explicitly focus on dealing with various noisy symbolic features for event coreference resolution.
We focus on within-document event coreference resolution. The input to our model is a document  $D$  consisting of  $n$  tokens and  $k$  (predicted) event mentions  $\{m_1, m_2, \ldots, m_k\}$ . For each  $m_i$ , we denote the start and end indices of its trigger by  $s_i$  and  $e_i$  respectively. We assume the mentions are ordered based on  $s_i$  (i.e., if  $i \leq j$  then  $s_i \leq s_j$ ).
We also assume each  $m_{i}$  has  $K$  (predicted) categorical features  $\{c_i^{(1)},c_i^{(2)},\ldots ,c_i^{(K)}\}$ , with each  $c_{i}^{(u)}\in \{1,2,\dots,N_u\}$  taking one of  $N_{u}$  different discrete values. Table 2 lists the symbolic features we consider in this work. The definitions of the features and their possible values are in ACE and Rich ERE guidelines (LDC, 2005; Mitamura et al., 2016). The accuracy scores of the symbolic feature predictors are also shown in Table 2. We use OneIE (Lin et al., 2020) to identify event mentions along with their subtypes. For other symbolic features, we train a joint classification model based on SpanBERT. The appendix contains more details.
2.2 Single-Mention Encoder
Given a document  $D$ , our model first forms a contextualized representation for each input token using a Transformer encoder (Joshi et al., 2020). Let  $\mathbf{X} = (\mathbf{x}_1, \dots, \mathbf{x}_n)$  be the output of the encoder, where  $\mathbf{x}_i \in \mathbb{R}^d$ . Then, for each mention  $m_i$ , its trigger's representation  $\mathbf{t}_i$  is defined as the average of its token embeddings:
$$
\mathbf {t} _ {i} = \sum_ {j = s _ {i}} ^ {e _ {i}} \frac {\mathbf {x} _ {j}}{e _ {i} - s _ {i} + 1} \tag {1}
$$
Dataset	Features	Acc. (Train)	Acc. (Dev)	Acc. (Test)
ACE 2005	Type	0.999	0.945	0.953
Polarity	0.999	0.994	0.988
Modality	0.999	0.856	0.884
Genericity	0.999	0.865	0.872
Tense	0.984	0.802	0.763
KBP 2016	Type	0.960	0.874	0.818
Realis	0.979	0.845	0.840
Table 2: Symbolic features we consider in this work.
Next, by using  $K$  trainable embedding matrices, we convert the symbolic features of  $m_{i}$  into  $K$  vectors  $\{\mathbf{h}_i^{(1)},\mathbf{h}_i^{(2)},\ldots ,\mathbf{h}_i^{(K)}\}$ , where  $\mathbf{h}_i^{(u)}\in \mathbb{R}^l$ .
2.3 Mention-Pair Encoder and Scorer
Given two event mentions  $m_{i}$  and  $m_{j}$ , we define their trigger-based pair representation as:
$$
\mathbf {t} _ {i j} = \operatorname {F F N N} _ {\mathrm {t}} \left(\left[ \mathbf {t} _ {i}, \mathbf {t} _ {j}, \mathbf {t} _ {i} \circ \mathbf {t} _ {j} \right]\right) \tag {2}
$$
where  $\mathrm{FFNN_t}$  is a feedforward network mapping from  $\mathbb{R}^{3\times d}\to \mathbb{R}^p$  , and  $\circ$  is element-wise multiplication. Similarly, we can compute their featurebased pair representations  $\{\mathbf{h}_{ij}^{(1)},\mathbf{h}_{ij}^{(2)},\dots ,\mathbf{h}_{ij}^{(K)}\}$  as follows:
$$
\mathbf {h} _ {i j} ^ {(u)} = \operatorname {F F N N} _ {\mathrm {u}} \left(\left[ \mathbf {h} _ {i} ^ {(u)}, \mathbf {h} _ {j} ^ {(u)}, \mathbf {h} _ {i} ^ {(u)} \circ \mathbf {h} _ {j} ^ {(u)} \right]\right) \tag {3}
$$
where  $u\in \{1,2,\ldots ,K\}$ , and  $\mathrm{FFNN_u}$  is a feedforward network mapping from  $\mathbb{R}^{3\times l}\to \mathbb{R}^p$
Now, the most straightforward way to build the final pair representation  $\mathbf{f}_{ij}$  of  $m_{i}$  and  $m_{j}$  is to simply concatenate the trigger-based representation and all the feature-based representations together:
$$
\mathbf {f} _ {i j} = \left[ \mathbf {t} _ {i j}, \mathbf {h} _ {i j} ^ {(1)}, \mathbf {h} _ {i j} ^ {(2)}, \dots , \mathbf {h} _ {i j} ^ {(K)} \right] \tag {4}
$$
However, this approach is not always optimal. First, as the symbolic features are predicted, they can be noisy and contain errors. The performance of most symbolic feature predictors is far from perfect (Table 2). Also, depending on the specific context, some features can be more useful than others.
Inspired by studies on gated modules (Lin et al., 2019; Lai et al., 2019), we propose Context-Dependent Gated Module (CDGM), which uses a gating mechanism to extract information from the input symbolic features selectively (Figure 1). Given two mentions  $m_{i}$  and  $m_{j}$ , we use their trigger feature vector  $\mathbf{t}_{ij}$  as the main controlling context to compute the filtered representation  $\overline{\mathbf{h}}_{ij}^{(u)}$ :
$$
\overline {{\mathbf {h}}} _ {i j} ^ {(u)} = \operatorname {C D G M} ^ {(u)} \left(\mathbf {t} _ {i j}, \mathbf {h} _ {i j} ^ {(u)}\right) \tag {5}
$$
Figure 1: Overall architecture of our mention-pair encoder, which uses CDGMs to incorporate symbolic features.
where  $u\in \{1,2,\ldots ,K\}$  . More specifically:
$$
\mathbf {g} _ {i j} ^ {(u)} = \sigma \left(\operatorname {F F N N} _ {\mathrm {g}} ^ {(u)} \left(\left[ \mathbf {t} _ {i j}, \mathbf {h} _ {i j} ^ {(u)} \right]\right)\right)
$$
$$
\mathbf {o} _ {i j} ^ {(u)}, \mathbf {p} _ {i j} ^ {(u)} = \operatorname {D E C O M P O S E} \left(\mathbf {t} _ {i j}, \mathbf {h} _ {i j} ^ {(u)}\right) \tag {6}
$$
$$
\overline {{\mathbf {h}}} _ {i j} ^ {(u)} = \mathbf {g} _ {i j} ^ {(u)} \circ \mathbf {o} _ {i j} ^ {(u)} + (1 - \mathbf {g} _ {i j} ^ {(u)}) \circ \mathbf {p} _ {i j} ^ {(u)}
$$
where  $\sigma$  denotes sigmoid function.  $\mathrm{FFNN}_{\mathbf{g}}^{(u)}$  is a mapping from  $\mathbb{R}^{2\times p}\rightarrow \mathbb{R}^p$ . At a high level,  $\mathbf{h}_{ij}^{(u)}$  is decomposed into an orthogonal component and a parallel component, and  $\overline{\mathbf{h}}_{ij}^{(u)}$  is simply the fusion of these two components. In order to find the optimal mixture,  $\mathbf{g}_{ij}$  is used to control the composition. The decomposition unit is defined as:
$$
\text {P a r a l l e l} \quad \mathbf {p} _ {i j} ^ {(u)} = \frac {\mathbf {h} _ {i j} ^ {(u)} \cdot \mathbf {t} _ {i j}}{\mathbf {t} _ {i j} \cdot \mathbf {t} _ {i j}} \mathbf {t} _ {i j} \tag {7}
$$
$$
\text {O r t h o g o n a l} \quad \mathbf {o} _ {i j} ^ {(u)} = \mathbf {h} _ {i j} ^ {(u)} - \mathbf {p} _ {i j} ^ {(u)}
$$
where  $\cdot$  denotes dot product. The parallel component  $\mathbf{p}_{ij}^{(u)}$  is the projection of  $\mathbf{h}_{ij}^{(u)}$  on  $\mathbf{t}_{ij}$ . It can be viewed as containing information that is already part of  $\mathbf{t}_{ij}$ . In contrast,  $\mathbf{o}_{ij}^{(u)}$  is orthogonal to  $\mathbf{t}_{ij}$  and so it can be viewed as containing new information. Intuitively, when the original symbolic feature vector  $\mathbf{h}_{ij}^{(u)}$  is very clean and has complementary information, we want to utilize the new information in  $\mathbf{o}_{ij}^{(u)}$  (i.e., we want  $\mathbf{g}_{ij}^{(u)} \approx \mathbf{1}$ ), and vice versa.
Finally, after using CDGMs to distill symbolic features, the final pair representation  $\mathbf{f}_{ij}$  of  $m_{i}$  and  $m_j$  can be computed as follows:
$$
\mathbf {f} _ {i j} = \left[ \mathbf {t} _ {i j}, \overline {{\mathbf {h}}} _ {i j} ^ {(1)}, \overline {{\mathbf {h}}} _ {i j} ^ {(2)}, \dots , \overline {{\mathbf {h}}} _ {i j} ^ {(K)} \right] \tag {8}
$$
And the coreference score  $s(i,j)$  of  $m_{i}$  and  $m_{j}$  is:
$$
s (i, j) = \operatorname {F F N N} _ {\mathrm {a}} \left(\mathbf {f} _ {i j}\right) \tag {9}
$$
where  $\mathrm{FFNN_a}$  is a mapping from  $\mathbb{R}^{(K + 1)\times p}\to \mathbb{R}$ .
2.4 Training and Inference
Algorithm 1: Noise Addition for Symbolic Features
Input: Document  $D$
Hyperparameters:  $\{\epsilon_1, \epsilon_2, \dots, \epsilon_K\}$
for  $i = 1\ldots k$  do
for  $u = 1\ldots K$  do With prob.  $\epsilon_{u}$  , replace  $c_{i}^{(u)}$  by  $\hat{c}_i^{(u)}\sim \mathrm{Uniform}(N_u)$  end
Training We use the same loss function as in (Lee et al., 2017). Also, notice that the training accuracy of a feature predictor is typically much higher than its accuracy on the dev/test set (Table 2). If we simply train our model without any regularization, our CDGMs will rarely come across noisy symbolic features during training. Therefore, to encourage our CDGMs to actually learn to distill reliable signals, we also propose a simple but effective noisy training method. Before passing a training data batch to the model, we randomly add noise to the predicted features. More specifically, for each document  $D$  in the batch, we go through every symbolic feature of every event mention in  $D$  and consider sampling a new value for the feature. The operation is described in Algorithm 1 (we use the same notations mentioned in Section 2.1).  $\{\epsilon_1,\epsilon_2,\dots ,\epsilon_K\}$  are hyperparameters determined by validation. In general, the larger the discrepancy between the train and test accuracies, the larger  $\epsilon$
Inference For each (predicted) mention  $m_{i}$ , our model will assign an antecedent  $a_{i}$  from all pre
ACE (Cross-Validation)	CoNLL	AVG
SSED + SupervisedExtended (2016)	55.23	52.53
SSED + MSEP (2016)	53.80	51.38
ACE (Test Data)	CoNLL	AVG
Baseline	58.93	55.78
Simple (All Features)	57.55	54.79
CDGM (All Features)	58.99	56.32
Noise (All Features)	60.43	57.85
CDGM + Noise (All Features)	62.07	59.76
Table 3: End-to-end results on ACE 2005 (using predicted triggers and predicted symbolic features).
System	CoNLL	AVG
UTD's system (2015)	32.69	30.08
Joint Learning (2017b)	35.77	33.08
E³C (2020)	41.97	38.66
Baseline	40.57	37.59
Simple (All Features)	41.40	38.58
CDGM + Noise (All Features)	43.55	40.61
ceding mentions or a dummy antecedent  $\epsilon$ :  $a_{i} \in Y(i) = \{\epsilon, m_{1}, m_{2}, \ldots, m_{i-1}\}$ . Basically,  $a_{i} = \arg \max_{j < i} s(i, j)$ . The dummy antecedent  $\epsilon$  represents two possible cases: (1)  $m_{i}$  is not actually an event mention (2)  $m_{i}$  is indeed an event mention but it is not coreferent with any previous extracted mentions. In addition, we fix  $s(i, \epsilon)$  to be 0.
Data and Experiments Setup We evaluate our methods on two English datasets: ACE2005 (Walker et al., 2006) and KBP2016 (Ji et al., 2016; Mitamura et al., 2016). We report results in terms of F1 scores obtained using the CoNLL and AVG metrics. By definition, these metrics are the summary of other standard coreference metrics, including  $\mathbf{B}^3$ , MUC,  $\mathrm{CEAF}_e$ , and BLANC (Lu and Ng, 2018). We use SpanBERT (spanbert-base-cased) as the Transformer encoder (Wolf et al., 2020a; Joshi et al., 2020). More details about the datasets and hyperparameters are in the appendix. We refer to models that use only trigger features as [Baseline]. In a baseline model,  $\mathbf{f}_{ij}$  is simply  $\mathbf{t}_{ij}$  (Eq. 2). We refer to models that use only the simple concatenation strategy as [Simple] (Eq. 4), and models that use the simple concatenation strategy and the noisy training method as [Noise].
Overall Results (on Predicted Mentions) Table 3 and Table 4 show the overall end-to-end results on ACE2005 and KBP2016, respectively. We
Table 4: End-to-end results on KBP 2016 (using predicted triggers and predicted symbolic features).
ACE (Test Data)	CoNLL	AVG
PAIREDRL (2020)	84.65	-
Baseline	81.62	81.49
Simple (All Features)	75.32	74.94
CDGM + Noise (All Features)	84.76	83.95
Table 5: Results on ACE 2005 using gold triggers and predicted symbolic features.
ACE (Test Data)	CoNLL	AVG
Baseline	81.62	81.49
Simple (All Features)	85.75	85.40
CDGM (All Features)	87.90	88.30
CDGM + Noise (All Features)	85.40	85.38
Table 6: Results on ACE 2005 using gold triggers and ground-truth symbolic features.
use OneIE (Lin et al., 2020) to extract event mentions and their types. Other features are predicted by a simple Transformer model. Overall, our full model outperforms the baseline model by a large margin and significantly outperforms state-of-the-art on KBP 2016. Our ACE 2005 scores are not directly comparable with previous work, as Peng et al. (2016) conducted 10-fold cross-validation and essentially used more training data. Nevertheless, the magnitude of the differences in scores between our best model and the state-of-the-art methods indicates the effectiveness of our methods.
Overall Results (on Ground-truth Triggers) The overall results on ACE 2005 using ground-truth triggers and predicted symbolic features are shown in Table 5. The performance of our full model is comparable with previous state-of-the-art result in (Yu et al., 2020). To better analyze the usefulness of symbolic features as well as the effectiveness of our methods, we also conduct experiments using ground-truth triggers and ground-truth symbolic features (Table 6). First, when the symbolic features are clean, incorporating them using the simple concatenation strategy can already boost the performance significantly. The symbolic features contain information complementary to that in the SpanBERT contextual embeddings. Second, we also see that the noisy training method is not helpful when the symbolic features are clean. Unlike other regularization methods such as dropout (Srivastava et al., 2014) and weight decay (Krogh and Hertz, 1992), the main role of our noisy training method is not to reduce overfitting in the traditional sense. Its main function is to help CDGMs learn to distill reliable signals from noisy features.
Features	AVG (Simple)	AVG (CDGM + Noise)	ΔAVG
Subtype	56.41	57.02	+0.61
Polarity	56.06	57.03	+0.97
Modality	54.81	58.54	+3.73
Genericity	54.70	57.82	+3.12
Tense	54.28	56.62	+2.34
Impact of Different Symbolic Features Table 7 shows the results of incorporating different types of symbolic features on the ACE 2005 dataset. Overall, our methods consistently perform better than the simple concatenation strategy across all feature types. The gains are also larger for more noisy features than clean features (feature prediction accuracies were shown in Table 2). This suggests that our methods are particularly useful in situations where the symbolic features are noisy.
Comparison with Multi-Task Learning We also investigate whether we can incorporate symbolic semantics into coreference resolution by simply doing multi-task training. We train our baseline model to jointly perform coreference resolution and symbolic feature prediction. The test AVG score on ACE 2005 is only 56.5. In contrast, our best model achieves an AVG score of 59.76 (Table 3).
Qualitative Examples Table 8 shows few examples from the ACE 2005 dataset that illustrate how incorporating symbolic features using our proposed methods can improve the performance of event conference resolution. In each example, our baseline model incorrectly predicts the highlighted event mentions to be coreferential.
Remaining Challenges Previous studies suggest that there exist different types and degrees of event coreference (Recasens et al., 2011; Hovy et al., 2013). Many methods (including ours) focus on the full strict coreference task, but other types of coreference such as partial coreference have remained underexplored. Hovy et al. (2013) defines two core types of partial event coreference relations: subevent relations and membership relations. Subevent relations form a stereotypical sequence of events, whereas membership relations represent instances of an event collection. We leave tackling the partial coreference task to future work.
Several previous approaches to within-document event coreference resolution operate by first ap
Table 7: Impact of Symbolic Features (ACE 2005)
...{Negotiations}ev1 between Washington and ...
...think that this will affect the{elections}ev2 unless ...
ev1(Contact:Meet)
ev2(Personnel:Elect)
...since you are not directly{elected}ev1, it would be ...
...Az-Zaman daily that{elections}ev2 should be held ...
ev1(Personnel:Elect): Polarity = NEGATIVE
ev2(Personnel:Elect): Polarity = POSITIVE
...told reporters after his{appeal}ev1 was rejected ...
...most junior of the court of{appeal}ev2, and its ...
ev1(Justice:Apeal): Genericity = SPECIFIC
ev2(Justice:Apeal): Genericity = GENERIC
Table 8: Examples of using symbolic features to improve event coreference resolution.
plying a mention-pair model to compute pairwise distances between event mentions, and then they apply a clustering algorithm such as agglomerative clustering or spectral graph clustering (Chen et al., 2009; Chen and Ji, 2009; Chen and Ng, 2014; Nguyen et al., 2016; Huang et al., 2019). In addition to trigger features, these methods use a variety of additional symbolic features such as event types, attributes, arguments, and distance. These approaches do not use contextual embeddings such as BERT and SpanBERT (Devlin et al., 2019; Joshi et al., 2020). Recently, there are several studies that use contextual embeddings together with type-based or argument-based information (Lu et al., 2020; Yu et al., 2020). These methods design networks or mechanisms that are specific to only one type of symbolic features. In contrast, our work is more general and can be effectively applied to a wide range of symbolic features.
In this work, we propose a novel gated module to incorporate symbolic semantics into event coreference resolution. Combined with a simple noisy training technique, our best models achieve competitive results on ACE 2005 and KBP 2016. In the future, we aim to extend our work to address more general problems such as cross-lingual cross-document coreference resolution.
This research is based upon work supported in part by U.S. DARPA KAIROS Program No. FA8750-19-2-1004, U.S. DARPA AIDA Program No. FA8750-18-2-0014, and Air Force No. FA8650-17-C-7715. The views and conclusions contained herein are those of the authors and should not be
interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.