SlowGuess/NAACL_2021_OCR huggingface dataset:
Jing Lu and Vincent Ng
{ljwinnie,vince}@hlt.utdallas.edu
We propose a neural event coreference model in which event coreference is jointly trained with five tasks: trigger detection, entity coreference, anaphoricity determination, realis detection, and argument extraction. To guide the learning of this complex model, we incorporate cross-task consistency constraints into the learning process as soft constraints via designing penalty functions. In addition, we propose the novel idea of viewing entity coreference and event coreference as a single coreference task, which we believe is a step towards a unified model of coreference resolution. The resulting model achieves state-of-the-art results on the KBP 2017 event coreference dataset.
Event coreference resolution is the task of determining whether two event mentions in a document refer to the same real-world event. For two event mentions to be coreferent, their triggers (i.e., the words realizing the occurrence of the events) should have the same subtype and their corresponding arguments (e.g., the times, places, and people involved) have to be entity-coreferent. However, identifying potential arguments (which is performed by an entity extraction system), linking arguments to their event mentions (which is also performed by an event extraction system), and determining whether two event arguments are coreferent (which is the job of an entity coreference resolver), are all nontrivial tasks. Hence, a key challenge in designing an event coreference resolver involves determining how to integrate these noisy components.
One of the most common approaches to event coreference resolution is pipelined approaches, where a trigger detection component, which identifies triggers and assigns event subtypes to them, is followed by an event coreference component, which clusters coreferent event mentions. It should therefore not be surprising that errors propagate
from the trigger detection component to the event coreference component. To avoid aggravating this error propagation problem, knowledge provided by other information extraction (IE) components (e.g., entity coreference, event arguments) is typically employed as features for training event coreference models (Chen et al., 2009; McConky et al., 2012; Cybulska and Vossen, 2013; Araki et al., 2014; Liu et al., 2014; Peng et al., 2016; Krause et al., 2016; Choubey and Huang, 2017). Oftentimes, these features provide limited improvements to event coreference models as they are too noisy to be useful.
Though less popular than pipelined approaches, bootstrapping approaches have been used for event coreference resolution, where an event coreference model is bootstrapped with models trained for one or more related IE tasks. For instance, Lee et al. (2012) incrementally build clusters of coreferent event and entity mentions by iteratively bootstrapping event coreference output using entity coreference output and vice versa. While in pipelined approaches only upstream tasks can influence downstream tasks, in bootstrapping approaches different tasks can influence each other. Nevertheless, errors made in earlier iterations of the bootstrapping process cannot be undone in later iterations.
Joint learning approaches have recently emerged as promising approaches to event coreference owing to their ability to address error propagation. In these approaches, two or more tasks are jointly trained. For instance, Araki and Mitamura (2015) learn a joint model for trigger detection and event coreference using a structured perceptron, and Lu and Ng (2017) learn a joint model for trigger detection, event coreference, and anaphoricity determination using a structured conditional random field. The key advantage of these models is that the tasks involved can benefit from each other during training. However, since a jointly learned model involves multiple tasks, it is typically complex. In fact, it is by no means easy to scale such a model
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4504-4514 June 6-11, 2021. ©2021 Association for Computational Linguistics
to a large number of tasks because of the high computational complexity involved in learning.
Joint inference approaches have also been applied to event coreference resolution. For instance, Chen and Ng (2016) and Lu et al. (2016) first train separate models for entity coreference, trigger detection, argument extraction, and event coreference, then use Integer Linear Programming or Markov Logic Networks to jointly infer the outputs of these tasks subject to (mostly) hard cross-task consistency constraints. For instance, one such hard constraint says that two coreferent event mentions should have the same event subtype. Since the models are trained independently, they cannot benefit from each other and could be noisy. Worse still, performing joint inference using hard constraints over (very) noisy outputs could do more harm than good. For instance, if two event mentions are correctly classified as coreferent but one of their subtypes is misclassified, then enforcing the aforementioned constraint might cause the joint inference procedure to incorrectly infer that the two are not coreferent. This explains why joint inference approaches have become less popular than joint learning approaches in recent years.
In light of the above discussion, we seek to advance the state of the art in event coreference resolution by proposing a model that jointly learns six tasks: trigger detection, event coreference, entity coreference, anaphoricity determination, argument extraction, and realis detection. As noted above, joint learning typically presents a serious computational challenge, and training a complex joint model involving six tasks would not have been possible without the advent of the neural NLP era.
While multi-task learning in a neural network typically allows the different tasks involved to benefit from each other via learning shared representations, we hypothesize that the model would benefit additional guidance given that the learning task, which involves six tasks, is so complex. Consequently, we propose to guide the learning process by exploiting cross-task consistency constraints. As mentioned above, such consistency constraints are typically employed in joint inference and rarely in joint learning. Moreover, unlike in joint inference where such constraints are typically implemented as hard constraints, we provide flexibility by implementing them as soft constraints. Specifically, we design penalty functions for penalizing outputs that violate a constraint, where the degree
of penalty depends on the extent of the violation.
Another contribution of our work involves proposing the idea of a unified coreference model. So far, entity and event coreference have always been viewed as two separate tasks, where links between entity mentions are distinguished from links between event mentions. However, their similarity has led us to hypothesize that they could be viewed as a single task, where coreference links are established between a set of mentions without distinguishing between entity and event mentions.
Traditional resolvers. Many existing event coreference resolvers, including those that employ the four approaches described in the introduction, are developed in the pre-neural era. resolvers. For a detailed overview of these non-neural resolvers and the wide variety of hand-engineered features they employ, we refer the reader to Lu and Ng (2018).
Neural resolvers. Of particular relevance to our work are neural event coreference models (e.g., Nguyen et al. (2016), Choubey and Huang (2017, 2018), Huang et al. (2019)). Unlike their traditional counterparts, neural coreference models can leverage the knowledge learned from large unlabeled corpora through pretrained word embeddings or transfer learning. Existing neural event coreference models are pipeline-based and seek to learn word representations so that coreferent event mentions have similar word embeddings, effectively making the rather unrealistic assumption that an event trigger is composed of a single token (Nguyen et al., 2016). In contrast, our neural resolver is a joint model and seeks to learn the representations of text spans, each of which corresponds to a candidate event trigger and may be composed of more than one token, so that coreferent event mentions have similar span representations.
Constrained learning in neural models. Another line of related work concerns the use of constraints in neural models (Li et al., 2019; Wang et al., 2020), where constraints are represented as first order logic formulas and compiled into the loss functions. These models are typically trained to minimize the weighted sum of task losses and constraint losses. Rather than introduce additional terms in the loss function, we employ constraints as penalty terms when learning to score how likely two event mentions are coreferent, effectively making the two mentions less likely to be coreferent if
Table 1: Event coreference example.
a constraint is violated.
Before formally defining the six tasks in the next section, we introduce several related definitions.
- An event mention is an explicit occurrence of an event consisting of a textual trigger, arguments or participants (if any), and the event subtype, and can optionally be characterized by a set of attributes and their values.
- An entity mention is an explicit mention of an entity in a text that has an entity type.
- An event trigger is a string of text that most clearly expresses the occurrence of an event, usually a word or a multi-word phrase.
- An event argument is an argument filler that plays a certain role in an event.
- Realis denotes whether an event actually happened or will happen in the future, or whether it is a generic event. Its value can be ACTUAL, GENERIC or OTHER.
- An event/entity coreference chain is a group of event/entity mentions that refer to the same real-world event/entity.
To better understand these definitions, consider the example in Table 1. The four event mentions (ev1, ev2, ev3, ev4) are triggered by "hacking", "death", "murder", and "murder" respectively. The first three have ACTUAL as their reals and the last one belongs to OTHER. While ev2 has LIFE_DIE as its subtype, the remaining ones all have subtype CONFLICT_attack. Among the eight entity mentions (en1, ..., en8), en3 has FACILITY as its type and the remaining ones are all PERSONs. en1 and en2 are the arguments of ev1 filling the roles of ATTACKER and TARGET respectively, whereas en4 is the argument of ev3 having the role ATTACKER. There are two entity coreference chains (one composed of en1 and en4 and the other en5 and en7) and one event coreference chain (ev1 and ev3).
We design a span-based neural model for event coreference resolution owing to its ability to effectively learn representations of text spans. While
span-based models have been successfully applied to a variety of entity-based IE tasks such as entity coreference (Lee et al., 2017; Joshi et al., 2020) and relation extraction (Luan et al., 2019), they have not been applied to event coreference.
More formally, our model takes as input a document  $D$  represented as a sequence of word tokens, from which we extract all possible intra-sentence spans of up to length  $L$ . It simultaneously learns six tasks, which we define below.
The trigger detection task aims to assign each span  $i$  a subtype label  $y_{i}$ . Each  $y_{i}$  takes a value in a subtype inventory or NONE, which indicates that  $i$  is not a trigger. The model predicts  $i$ 's subtype to be  $y_{i}^{*} = \arg \max_{y_{t}} s_{t}(i, y_{t})$ , where  $s_{t}$  is a scoring function suggesting  $i$ 's likelihood of having  $y_{i}$  as its subtype.
The event coreference resolution task aims to assign span  $i$  an antecedent  $y_{c}$ , where  $y_{c} \in \{1, \ldots, i - 1, \epsilon\}$ . In other words, the value of  $y_{c}$  is the id of  $i$ 's antecedent, which can be one of the preceding spans or a dummy antecedent  $\epsilon$  (if the event mention underlying  $i$  starts a new cluster). We define the following scoring function:
$$
s _ {c} (i, j) = \left\{ \begin{array}{l l} 0 & j = \epsilon \\ s _ {m} (i) + s _ {m} (j) + s _ {p} (i, j) & j \neq \epsilon \end{array} \right. \tag {1}
$$
where  $s_m(i)$  is the score suggesting span  $i$ 's likelihood of being a trigger and  $s_p(i,j)$  is a pairwise coreference score computed over span  $i$  and a preceding span  $j$ . The model predicts the antecedent of  $i$  to be  $y_c^* = \arg \max_{j\in \mathcal{Y}(i)}s_c(i,j)$ , where  $\mathcal{Y}(i)$  is the set of  $i$ 's candidate antecedents.
The entity coreference resolution task involves identifying entity mentions that refer to the same real-world entity. Intuitively, entity coreference is useful for event coreference: two event mentions are not likely to be coreferent if there exists an argument role (e.g., ATTACKER) for which the corresponding arguments in the two event mentions are not entity-coreferent. In our model, it is defined in the same way as the event coreference resolution task except that it operates on the spans identified by the entity mention detection component rather than the trigger detection component. The entity mention detection task is defined in the same way
{Two men}en1 accused of {hacking}ev1 {a British soldier}en2 to {death}ev2 last month appeared in {separate courts}en3 for hearings. {The men}en4, {Michael Adebolajo}en5, 28, and {Michael Adebowale}en6, 22, face {murder}ev3 charges. {Adebolajo}en7 was also charged with other offenses, including the attempted {murder}ev4 of {two police officers}en8.
as the trigger detection task except that it aims to assign each span  $i$  an entity type label.
The anaphoricity determination task aims to assign each span  $i$  an anaphoricity label  $y_{a}$ , where  $y_{a}$  can be ANAPHORIC, which indicates that the mention having span  $i$  is coreferent with a preceding mention, or NON-ANAPHORIC. The model  $s_{a}(i)$  predicts the mention having span  $i$  as anaphoric if and only if  $s_{a}(i) \geq 0$ . To train this model, we set the target value to 1 for anaphoric mentions and -1 for non-anaphoric mentions. Anaphoricity is useful for coreference: it prevents non-anaphoric mentions from being resolved.
The realis detection task aims to assign each span  $i$  a realis label  $y_{r}$ , where  $y_{r} \in \{\mathrm{ACTUAL}, \mathrm{GENERIC}, \mathrm{OTHER}, \mathrm{ENTITY}, \mathrm{AND} \mathrm{NONE}\}$ . As mentioned in Section 3, ACTUAL, GENERIC, and OTHER are labels used for event mention spans. To enable every span  $i$  to be assigned a realis label, we augment the realis label set to include ENTITY and NONE. Specifically, ENTITY is a label that is exclusively reserved for spans that correspond to entity mentions, and NONE indicates that  $i$  does not correspond to a mention. The model predicts the realis type of  $i$  to be  $y_{r}^{*} = \arg \max_{y_{r}} s_{r}(i, y_{r})$ , where  $s_{r}$  is a scoring function suggesting  $i$ 's likelihood of having realis type  $y_{r}$ . Realis detection is useful for event coreference: two event mentions cannot be coreferent if their realis labels are different.
The argument extraction task aims to assign an argument role label  $y_{o}$  to a candidate argument  $k$  of a candidate event mention span  $i$ , where (1)  $k$  is a candidate entity mention span, and (2)  $y_{o}$  is a role taken from an argument role inventory or NONE, which indicates that the token is not an argument of  $i$ . We consider (1)  $k$  to be a candidate argument of  $i$  if and only if it appears within the same sentence as  $i$ ; and (2) a span to be a candidate event/entity mention span if it is assigned a non-NONE event/entity type by the Mention Prediction Layer, which we will describe shortly. For each candidate argument  $k$  of  $i$ , the model predicts its role in  $i$  to be  $y_{o}^{*} = \arg \max_{y_{o}} s_{o}(i, k, y_{o})$ , where  $s_{o}$  is a scoring function suggesting token  $k$ 's likelihood of being an argument of  $i$  having role  $y_{o}$ . Arguments, when combined with entity coreference chains, would be useful for event coreference.
The model structure, which is shown in Figure 1, is described in detail below.
Span Representation Layer We adapt the independent version of Joshi et al.'s (2019) state-of-the-art entity coreference resolver to event coreference resolution. Specifically, we divide an input document into non-overlapping regions, each of which has size  $L_{d}$ . The word sequence in each region serves as an input training sequence. We then pass the sequence into a pretrained transformer encoder used in SpanBERT-large (Joshi et al., 2020) to encode tokens and their contexts. Finally, we set  $\mathbf{g}_i$ , the representation of span  $i$ , to  $[\mathbf{h}_{start(i)}; \mathbf{h}_{end(i)}; \mathbf{h}_{head(i)}; \mathbf{f}_i]$ , where  $\mathbf{h}_{start(i)}$  and  $\mathbf{h}_{end(i)}$  are the hidden vectors of the start and end tokens of the span,  $\mathbf{h}_{head(i)}$  is an attention-based head vector and  $\mathbf{f}_i$  is a span width feature embedding. To maintain computational tractability, we first compute a score  $s_m$  for each span  $i$ :
$$
s _ {m} (i) = \operatorname {F F N N} _ {m} (\mathbf {g} _ {i}) \tag {2}
$$
where FFNN is a standard feedforward neural network. Then we retain only the top  $N\%$  of the spans for further processing.
Trigger Prediction Layer For each span  $i$  that survives the filtering, we pass its representation  $\mathbf{g}_i$  to a FFNN, which outputs a vector  $\mathbf{ot}_i$  of dimension  $T$ , where  $T$  is the number of possible event subtypes (including NONE).  $\mathbf{ot}_i(y)$ , the  $y$ th element of  $\mathbf{ot}_i$ , is a score indicating  $i$ 's likelihood of belonging to event subtype  $y$ . Specifically:
$$
\mathbf {o t} _ {i} = \operatorname {F F N N} _ {t} (\mathbf {g} _ {i}) \tag {3}
$$
$$
s _ {t} (i, y) = \mathbf {o t} _ {i} (y) \tag {4}
$$
Anaphoricity Prediction Layer We predict the anaphoricity value of each top span  $i$  as follows. Since the anaphoricity of a mention is dependent on its preceding context, we first concatenate the average of the representations of the 25 tokens immediately preceding  $i$  (to approximate  $i$ 's preceding context) with the span representation  $\mathbf{g}_i$ . We then pass the resulting vector,  $\mathbf{c}\mathbf{x}_i$ , to a FFNN, which outputs an anaphoricity value. Specifically:
$$
s _ {a} (i) = \operatorname {F F N N} _ {a} (\mathbf {c x} _ {i}) \tag {5}
$$
Realis Prediction Layer To predict the realis value of each top span  $i$ , we pass its representation  $\mathbf{g}_i$  to a FFNN, which outputs a vector  $\mathbf{or}_i$  of length 5.  $\mathbf{or}_i(y)$ , the  $y$ th element of  $\mathbf{or}_i$ , is a score indicating  $i$ 's likelihood of having realis type  $y$ :
$$
\mathbf {o r} _ {i} = \mathrm {F F N N} _ {r} (\mathbf {g} _ {i}) \tag {6}
$$
Figure 1: Model structure.
Two men accused of hacking a British soldier .... face murder charges ...
$$
s _ {r} (i, y) = \mathbf {o r} _ {i} (y) \tag {7}
$$
Coreference Prediction Layer To predict event coreference links, we define the pairwise score between span  $i$  and span  $j$  as follows:
$$
s _ {p} (i, j) = \operatorname {F F N N} _ {c} ([ \mathbf {g} _ {i}; \mathbf {g} _ {j}; \mathbf {g} _ {i} \circ \mathbf {g} _ {j}, \mathbf {u} _ {i j} ]) \tag {8}
$$
where  $\circ$  denotes element-wise multiplication,  $\mathbf{g}_i\circ \mathbf{g}_j$  encodes the similarity between  $i$  and  $j$ , and  $\mathbf{u}_{ij}$  is a feature embedding encoding the distance between them. We can then compute the full coreference score defined in Equation 1 using Equations 2 and 8. To improve running time, we follow Lee et al. (2018) and use their antecedent pruning method, coarse-to-fine pruning, to reduce the number of candidate antecedents for each anaphor.
Incorporating Entity Coreference The most straightforward way to incorporate entity coreference information into our model would be to have (1) an entity mention detection model that is architecturally identical to the trigger detection model except that it assigns entity type (rather than event subtype) labels to each span, and (2) an entity coreference model that is architecturally identical to the event coreference model described above except that it identifies antecedents for spans provided by the entity mention detection (rather than trigger detection) component. While this would allow entity coreference to interact with event coreference and other tasks via the shared Span Representation Layer, the two coreference tasks would otherwise be learned independently of each other.
Towards the goal of building a unified model of coreference, we propose a novel idea: we seek
to learn entity and event coreference simultaneously by viewing them as a single coreference task. From a learning perspective, there is only one task to be learned, which is coreference resolution over a set of mentions. To do so, we extend the Span Representation Layer, the Trigger Prediction Layer, and the Coreference Prediction Layer as follows. First, the Span Representation Layer will identify spans corresponding to mentions that are composed of both entity mentions and event mentions even though the model doesn't know (and doesn't need to know) which ones are entity mentions and which ones are event mentions. Second, the Trigger Prediction Layer will assign each mention span a semantic type, which is taken from a type inventory consisting of both entity types and event subtypes (and NONE, if the span is not a mention). In other words, the Trigger Prediction Layer, which is essentially extended to a Mention Prediction Layer, now extracts both entity and event mention spans. Third, the Coreference Prediction Layer computes coreference chains based on the predicted mention spans and their semantic types. Since all the learner sees are mentions, it doesn't know (and doesn't need to know) which coreference chains it computes are entity-based and which ones are event-based. Similarly, it doesn't know (and doesn't need to know) which types in the type inventory are entity types and which ones are event subtypes. A key advantage of this unified model of coreference is that it allows entity and event coreference to be tightly coupled via parameter sharing.
When we apply this model to a test document, we need to distinguish which coreference relations
it identifies are entity-based and which ones are event-based. This can be done easily based on the semantic type associated with the mentions underlying the extracted coreference relation under consideration. If the semantic type is an entity type, the corresponding coreference relation is regarded as an entity coreference relation; otherwise, it is regarded as an event coreference relation.
Argument Prediction Layer To predict arguments and their roles, we pair each top span  $i$  and each candidate argument  $k$  to form an input vector  $\mathbf{v}\mathbf{a}_{ik} = [\mathbf{g}_i;\mathbf{t}_i;\mathbf{g}_k;\mathbf{t}_k]$ , where  $\mathbf{g}_i$  is the span representation of  $i$ ,  $\mathbf{t}_i$  is the one-hot subtype vector of  $i$ ,  $\mathbf{g}_k$  is the span representation of argument candidate  $k$ , and  $\mathbf{t}_k$  is the one-hot subtype vector of  $k$ . During training, we use the gold subtype label to derive the subtype vector. During inference, we derive the subtype vector from the output of the Mention Detection Layer. We feed the resulting vector into a FFNN, which outputs a vector  $\mathbf{o}\mathbf{a}_{ik}$  of dimension 21.  $\mathbf{o}\mathbf{a}_{ik}(y)$ , the  $y$ th element of  $\mathbf{o}\mathbf{a}_{ik}$ , is a score indicating  $k$ 's likelihood of being an argument of  $i$  with role  $y$ :
$$
\mathbf {o a} _ {i k} = \mathrm {F F N N} _ {o a} (\mathbf {v a} _ {i k}) \tag {9}
$$
$$
s _ {o} (i, k, y) = \mathbf {o a} _ {i k} (y) \tag {10}
$$