SlowGuess/NAACL_2021_OCR huggingface dataset:
Incorporating Consistency Constraints As noted before, we propose to guide the learning process by incorporating commonsense knowledge that encodes cross-task consistency constraints on coreference and the auxiliary tasks. We begin by incorporating two consistency constraints on the outputs of coreference and mention detection:
C1: If two spans are coreferent, they should have the same semantic type. C2: If a span has an antecedent that is not the dummy antecedent, its semantic type shouldn't be NONE.
We incorporate each constraint into the model via a scoring function that computes how much two spans  $i$  (an anaphor) and  $j$  (a candidate antecedent of  $i$ ) should be penalized if a constraint is violated. For constraint C1, we define a cost function,  $c_{1}$ , which is computed as follows:
$$
c _ {1} (i, j) = \min  \left(\left| s _ {t} (i, y _ {i}) - s _ {t} (i, y _ {j}) \right|, \left| s _ {t} (j, y _ {j}) - s _ {t} (j, y _ {i}) \right| \right. \tag {11}
$$
where  $y_{i} = \arg \max_{y_{t}} s_{t}(i, y_{t})$  and  $y_{j} = \arg \max_{y_{t}} s_{t}(j, y_{t})$ . Intuitively,  $c_{1}$  provides an estimate of the least amount of adjustment needed to make  $i$ 's semantic type the same as  $j$ 's or the
other way round. In particular,  $c_{1}$  returns 0 (i.e., no penalty) if the two spans have the same type.
Similarly, for constraint C2, we define a cost function  $c_2$ , which is computed as follows:
$$
c _ {2} (i, j) = \left\{ \begin{array}{l l} 0 & \underset {y \in \mathcal {Y}} {\arg \max  s _ {t} (i, y)} \neq \text {N o n e} \\ s _ {t} (i, \text {N o n e}) - \underset {y \in \mathcal {Y} \backslash \{\text {N o n e} \}} {\max } s _ {t} (i, y) & \text {o t h e r w i s e} \end{array} \right. \tag {12}
$$
where  $\mathcal{V}$  is the set of possible types. Intuitively,  $c_{2}$  estimates the minimum amount that needs to be adjusted so that anaphor  $j$ 's type is not NONE.
Finally, we incorporate  $c_{1}$  and  $c_{2}$  into the model as penalty terms in  $s_{c}$  (Equation 1). Specifically, we redefine  $s_{c}$  as follows:
$$
s _ {c} (i, j) = \left\{ \begin{array}{l l} 0 & j = \epsilon \\ s _ {m} (i) + s _ {m} (j) + s _ {p} (i, j) - \left[ \beta_ {1} c _ {1} (i, j) + \beta_ {2} c _ {2} (i, j) \right] & j \neq \epsilon \end{array} \right. \tag {13}
$$
where  $\beta_{1}$  and  $\beta_{2}$  are positive constants that control the hardness of the constraints. The smaller a  $\beta_{i}$  is, the softer the corresponding constraint is. Intuitively, if a constraint is violated,  $s_c(i,j)$  will be lowered by one or more of the penalty terms, and  $j$  will less likely be selected as the antecedent of  $i$ .
In addition, we enforce the following consistency constraints. Like C1 and C2, each of them will be accompanied by a cost function that will eventually be incorporated into  $s_c$  as a penalty term. Coreference and anaphoricity. C3: If a span's antecedent is not the dummy antecedent, its anaphoricity value should be ANAPHORIC. C4: If a span has a dummy antecedent, its anaphoricity value should be NON-ANAPHORIC.
Coreference and realis detection. C5: If two spans are coreferent, they should have the same realis value. C6: If a span's antecedent is not the dummy antecedent, its realis value should not be NONE.
Coreference and argument extraction. C7: If two event mention spans are coreferent, their same-role arguments, if any, should be entity-coreferent.
The loss function we use,  $L(\Theta)$ , is composed of the losses of the six tasks, and is defined as follows:
$$
L (\Theta) = \sum_ {i = 1} ^ {d} \left(\lambda_ {c} L _ {c} + \lambda_ {t} L _ {t} + \lambda_ {a} L _ {a} + \lambda_ {r} L _ {r} + \lambda_ {o} L _ {o}\right) \tag {14}
$$
where the hyperparameters (i.e., the  $\lambda$ 's) determine the trade-off between the task losses. The model is trained to minimize  $L(\Theta)$ , whereas the hyperparameters are tuned using grid search to maximize AVG-F (the standard event coreference evaluation metric; see the next section) on development data.
Task Losses We employ a max-margin loss for each of the six tasks.
Defining the coreference loss is slightly tricky since the coreference annotations for each document are provided in the form of clusters. We adopt the coreference loss function previously defined by Wiseman et al. (2015) for entity coreference resolution. Specifically, let  $\mathrm{GOLD}_c(i)$  denote the set of spans preceding span  $i$  that are coreferent with  $i$ , and  $y_c^l$  be  $\arg \max_{y \in \mathrm{GOLD}_c(i)} s_c(i, y)$ . In other words,  $y_c^l$  is the highest scoring (latent) antecedent of  $i$  according to  $s_c$  among all the antecedents of  $i$ . The loss function for coreference is defined as:
$$
L _ {c} (\Theta) = \sum_ {i = 1} ^ {n} \max  _ {j \in \mathcal {Y} (i)} \left(\Delta_ {c} (i, j) \left(1 + s _ {c} (i, j) - s _ {c} (i, y _ {c} ^ {l})\right) \right. \tag {15}
$$
where  $\Delta_c(i,j)$  is a mistake-specific cost function that returns the cost associated with a particular type of error (Durrett and Klein, 2013). Intuitively, the loss function penalizes a span  $i$  if the predicted antecedent  $j$  has a higher score than the correct latent antecedent  $y_c^l$ .
We similarly define the loss for trigger detection:
$$
L _ {t} (\Theta) = \sum_ {i = 1} ^ {n} \sum_ {\hat {l} \neq y _ {t}} \max  (0, \Delta_ {t} (i, \hat {l}) (1 + s _ {t} (i, \hat {l}) - s _ {t} (i, y _ {t}))) \tag {16}
$$
where  $\Delta_t(i,\hat{l})$  is a mistake-specific cost function that returns the cost associated with a particular type of error. Intuitively, the loss function penalizes each span for which each of the wrong subtypes  $\hat{l}$  has a higher score than the correct subtype  $y_{t}$  according to  $s_t$ .
The task losses for anaphoricity determination, realis detection, and argument extraction are all max-margin losses that are defined similarly as the one used for trigger detection.
5.1.1 Corpora
We perform training and evaluation on the English corpora used in the TAC KBP 2017 Event Nugget Detection and Coreference task. There are no official training sets: the task organizers simply made available a number of event coreference-annotated corpora for training. We use LDC2015E29, E68, E73, E94, and LDC2016E64 as our training set, which contain 817 documents
with 22894 event mentions distributed over 13146 coreference chains<sup>2</sup>. Among these 817 documents, we reserve 82 documents for parameter tuning and use the remaining documents for model training. We report results on the official test set, which consists of 167 documents with 4375 event mentions distributed over 2963 coreference chains.
5.1.2 Evaluation Metrics
Results of event coreference, trigger detection and realis detection are obtained using version 1.8 of the official scorer provided by the KBP 2017 organizers. For event coreference, the scorer employs four scoring metrics, MUC (Vilain et al., 1995),  $B^3$  (Bagga and Baldwin, 1998),  $\mathrm{CEAF}_e$  (Luo, 2005) and BLANC (Recasens and Hovy, 2011), as well as the unweighted average of their F-scores (AVG-F). Results of trigger detection and realis detection are both expressed in terms of Precision (P), Recall (R) and F-score. The scorer considers (1) a trigger correctly detected if it has an exact match with a gold trigger in terms of boundary and event subtype, and (2) a realis label correctly classified if it has an exact match with a gold trigger in terms of boundary and realis value.
Additionally, we express results of both argument extraction and anaphoricity determination in terms of Precision, Recall and F-score. We consider an event argument correctly extracted if it has an exact match with a gold trigger-argument pair in terms of trigger boundary, event subtype, argument head and argument role. We consider an anaphoric mention correct if it has an exact match with the boundary of a gold anaphoric mention.
Finally, we report entity coreference results in terms of CoNLL score, which is the unweighted average of MUC,  $B^3$ , and  $\mathrm{CEAF}_e$ .
5.1.3 Implementation Details
We use the SpanBERT-large model in the Span Representation Layer. For each document, we split it into segments of length 512 and generate all spans of length up to 10. Each FFNN has one hidden layer of size 2000. The size of the width feature embedding is 20. For span pruning, we keep the top  $50\%$  of the spans. For candidate antecedent pruning, we keep the top 15 antecedents.
<sup>1</sup>Space limitations preclude a description of these error types. See Durrett and Klein (2013) for details.
$^{2}$ LDC2015E73 and E94 don't have annotations for entity detection, entity coreference resolution and argument extraction. We set the losses of these three tasks to 0 during training.  $^{3}$ https://github.com/facebookresearch/SpanBERT
Event Coreference	Trigger	Anaphoricity	Realis	Argument	Entity Coref.
MUC	B3	CEA	BLA	AVG	P	R	F	P	R	F	P	R	F	P	R	F	CoNLL
Jiang et al. (2017)	30.6	43.8	39.9	27.0	35.3	56.8	55.6	56.2	-	-	-	48.0	46.9	47.4	-	-	-	-
Huang et al. (2019)	35.7	43.2	40.0	32.4	36.8	56.8	46.4	51.1	-	-	-	-	-	-	-	-	-	-
Lu and Ng (2020)	37.1	44.5	40.0	29.9	37.9	64.5	46.9	54.3	-	-	-	-	-	-	-	-	-	-
Knowledge-lean	37.6	52.3	51.7	33.6	43.8	71.5	55.3	62.4	-	-	-	-	-	-	-	-	-	-
Pipeline	38.6	53.0	53.0	35.0	44.9	73.9	56.1	63.8	43.0	44.5	43.8	70.0	53.1	60.3	36.9	29.9	33.0	72.6
Full Joint	45.2	54.7	53.8	38.2	48.0	71.6	58.7	64.5	50.4	45.3	47.7	63.7	52.0	57.3	32.4	24.5	27.9	68.7
Table 2: Results of different resolvers on event coreference and related tasks. Results in rows 1-3 are copied verbatim from the original papers; - indicates the corresponding result is not available.
For training, we use document sized minibatches and apply a dropout rate of 0.3. Following Joshi et al. (2019), we use different learning rates for training the task parameters and the SpanBERT parameters. Specifically, the task learning rate is  $1 \times 10^{-5}$  and is decayed linearly, whereas the learning rate for SpanBERT is  $2 \times 10^{-4}$  and is decayed linearly. The hyperparameters in the loss function,  $\lambda_{c}, \lambda_{t}, \lambda_{a}, \lambda_{r}$ , and  $\lambda_{o}$ , are 1, 1, 0.05, 0.5, and 0.05.
Results are shown in Table 2. To gauge the performance of our model, we employ five baselines. Row 1 shows the results of our first baseline, Jiang et al.'s (2017) resolver, which is the highest-scoring system participating in KBP 2017. Rows 2 and 3 show the performance of our next two baselines, a neural resolver (Huang et al., 2019) and a non-neural resolver (Lu and Ng, 2020) that have achieved the best results to date on the KBP 2017 test set. Hence, these three baselines can be viewed as the prior state of the art. As we can see, while Jiang et al. have the best trigger detector (56.2 F-score), the best event coreference performance is achieved by Lu and Ng's resolver (37.9 AVG-F).
Row 4 shows our fourth baseline, which is our model except that (1) three prediction layers (argument, realis, and anaphoricity) are removed, and (2) the remaining layers are trained to identify event mentions only (i.e., without entity mentions). This baseline mimics typical knowledge-lean approaches to event coreference resolution, which perform only trigger detection and event coreference, but is the first knowledge-lean event coreference approach implemented in a span-based framework. As we can see, this baseline outperforms Lu and Ng's resolver by  $5.9\%$  points in AVG-F for event coreference. A closer inspection of the coreference evaluation metrics reveals that in comparison to Lu and Ng, this baseline's  $B^{3}$ ,  $\mathrm{CEAF}_{e}$  and BLANC scores increase substantially while its MUC score
barely changes. Since MUC only rewards successful identification of coreference links, the fact that the MUC score is more or less unchanged implies that the improvement does not arise from link identification; rather, the fact that the  $B^3$ ,  $\mathrm{CEAF}_e$  and BLANC scores improve suggests that the improvement arises from successful identification of singleton clusters. This is further supported by the improvement in trigger detection: the baseline's trigger detection module achieves an F-score of 62.4, outperforming Lu and Ng's trigger detection module by  $8.1\%$  points in F-score. This huge improvement should not be surprising, as SpanBERT is specifically designed to extract text spans. Overall, despite the encouraging  $6\%$ -point improvement in event coreference AVG-F score, we cannot say that the successes of span-based models on entity coreference can be extended to event coreference as it largely fails to establish event coreference links.
Row 5 shows the result of our fifth baseline, which is a pipelined version of our model designed to gauge the benefits of our joint model. Here, we first train a trigger detector, which is the same as the Mention Prediction Layer of our model trained to assign event subtypes to top spans. The resulting triggers are used to train an anaphoricity model (same as our model's Anaphoricity Prediction Layer) and a realis detection model (same as our model's Realis Prediction Layer). Next, we train an entity coreference model, which is the same as our third baseline except that it is trained to operate on entity rather than event mention spans. Then, we train an argument extraction model (same as our model's Argument Prediction Layer) using the extracted entity mentions as candidate arguments for the triggers identified by the trigger detection model. Finally, the outputs of these models are used to enforce the seven constraints in our model as hard constraints: any candidate antecedent of an anaphor that violates any of the constraints is filtered prior to event coreference resolution. Over
all, this baseline outperforms the fourth baseline by  $0.6\%$  points in AVG-F for event coreference and  $1.4\%$  points in F-score for trigger detection.
Row 6 shows the result of our full model, which outperforms the Pipeline model by  $3.1\%$  points in AVG-F for event coreference and establishes new state-of-the-art results. Encouragingly, the gains in AVG-F are accompanied by improvements w.r.t. all four coreference scoring metrics. In particular, the MUC score improves considerably by  $6.6\%$  points, which means that the full model has successfully identified event coreference links. In addition, we see a  $0.7\%$  point improvement in trigger detection over Pipeline, and a  $12.9\%$  point improvement in realis detection in comparison to Jiang et al. For bookkeeping purposes, we also report the scores for each component of our model. Overall, the fact that our joint model outperforms Pipeline suggests the benefits of joint modeling.
To evaluate the contribution of the different components in our model, we show in Table 3 ablation results, which we obtain by removing one component at a time from the model and retraining it.
Consistency constraints. Ablating the consistency constraints means removing all the penalty terms from  $s_c$ . The ablated system resembles what one would usually see in a multi-task learning setup, where the different tasks involved has a shared representation. As we can see from row 2, event coreference performance drops by  $1\%$  point, suggesting the usefulness of using consistency constraints in a multi-task setup. While it is perhaps not surprising that the consistency constraints have the largest impact on event coreference performance, it is somewhat interesting to see that there is one task whose performance improves when consistency constraints are ablated, realis detection.
Entity coreference. Next, we ablate the entity coreference component. The ablation of entity coreference necessitates the removal of the argument extraction component and the associated constraints since the latter relies on the outputs of entity coreference. We see from row 3 that event coreference performance drops precipitously by  $2.7\%$  points. This suggests that entity coreference has a considerable positive impact on event coreference.
The next question is: will coreference performance go up or down if we treat entity and event coreference as two separate tasks that are learned
Event Coref.	Tri.	Ana.	Rea.	Arg.	Entity Coref.
AVG	F	F	F	F	CoNLL
1	Full Model	48.0	64.5	47.7	57.3	27.9	68.7
2	- constraints	47.0	64.5	47.6	57.9	27.9	68.5
3	- entity coref.	45.3	63.5	45.0	58.2	-	-
4	sep. entity coref.	47.2	65.1	47.8	56.3	26.0	65.7
5	- anaphoricity	47.5	64.9	46.9	58.1	28.4	69.3
6	- realis	46.6	64.8	46.7	-	29.6	69.3
7	- argument	47.4	64.3	48.6	58.5	-	66.7
Table 3: Ablation results of the full model.
in a typical multi-task setup? As we can see from row 4, the performances of event coreference and entity coreference drop by  $0.8\%$  points and  $3\%$  points respectively. These results suggest that our viewing the two tasks as a single task is beneficial.
Anaphoricity determination. Next, we ablate the anaphoricity component, which involves removing both its task loss and the associated constraints. From row 5, we see that event coreference performance drops by  $0.5\%$  points, and anaphoricity determination performance drops  $0.8\%$  points.
Realis detection. When we ablate realis detection, both the task loss and the associated consistency are removed. The performances of event coreference and anaphoricity drop precipitously, by  $1.4\%$  points and  $1.0\%$  point respectively, suggesting the usefulness of realis detection for both event coreference and anaphoricity detection.
Argument extraction. Finally, when the argument extraction component is ablated, event coreference performance drops by  $0.6\%$  points. These results illustrate the importance of argument extraction for event coreference.
Overall, these results suggest that each component contributes positively to event coreference.
We proposed the first neural model for event coreference resolution that (1) jointly learned six tasks, (2) used consistency constraints to guide learning, and (3) viewed entity and event coreference as a single task. Our model outperformed several strong baselines and achieved state-of-the-art results on the KBP 2017 event coreference dataset.
Jun Araki, Zhengzhong Liu, Eduard Hovy, and Teruko Mitamura. 2014. Detecting subevent structure for event coreference resolution. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, pages 4553-4558.
Jun Araki and Teruko Mitamura. 2015. Joint event trigger identification and event coreference resolution with structured perceptron. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2074-2080.
Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of the LREC Workshop on Linguistic Coreference, pages 563-566.
Chen Chen and Vincent Ng. 2016. Joint inference over a lightly supervised information extraction pipeline: Towards event coreference resolution for resource-scarce languages. In Proceedings of the 30th AAAI Conference on Artificial Intelligence, pages 2913-2920.
Zheng Chen, Heng Ji, and Robert Haralick. 2009. A pairwise event coreference model, feature impact and evaluation for event coreference resolution. In Proceedings of the Workshop on Events in Emerging Text Types, pages 17-22.
Prafulla Kumar Choubey and Ruihong Huang. 2017. Event coreference resolution by iteratively unfolding inter-dependencies among events. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2124-2133.
Prafulla Kumar Choubey and Ruihong Huang. 2018. Improving event coreference resolution by modeling correlations between event coreference chains and document topic structures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 485-495.
Agata Cybulska and Piek Vossen. 2013. Semantic relations between events and their time, locations and participants for event coreference resolution. In Proceedings of the International Conference Recent Advances in Natural Language Processing, pages 156-163.
Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1971-1982.
Yin Jou Huang, Jing Lu, Sadao Kurohashi, and Vincent Ng. 2019. Improving event coreference resolution by learning argument compatibility from unlabeled data. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 785-795.
Shanshan Jiang, Yihan Li, Tianyi Qin, Qian Meng, and Bin Dong. 2017. SRCB entity discovery and linking (EDL) and event nugget systems for TAC 2017. In Proceedings of the 2017 Text Analysis Conference.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, (8):64-77.
Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5803-5808.
Sebastian Krause, Feiyu Xu, Hans Uszkoreit, and Dirk Weissenborn. 2016. Event linking with sentential features from convolutional neural networks. In Proceedings of the Conference of Computataional Natural Language Learning, pages 239-249.
Heeyoung Lee, Marta Recasens, Angel Chang, Mihai Surdeanu, and Dan Jurafsky. 2012. Joint entity and event coreference resolution across documents. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 489-500.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188-197.
Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 687-692.
Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Srikumar. 2019. A logic-driven framework for consistency of neural models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3924-3935.
Zhengzhong Liu, Jun Araki, Eduard Hovy, and Teruko Mitamura. 2014. Supervised within-document event coreference using information propagation. In Proceedings of the 9th International Conference on Language Resources and Evaluation, pages 4539-4544.
Jing Lu and Vincent Ng. 2017. Joint learning for event coreference resolution. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 90-101.