SlowGuess/NAACL_2021_OCR huggingface dataset:
Juntao Yu $^{1}$ , Nafise Sadat Moosavi $^{2}$ , Silviu Paun $^{1}$ , Massimo Poesio $^{1}$
$^{2}$ UKP Lab, Technische Universität Darmstadt
$^{1}\{juntao.yu, s.paun, m.poesio\} @qmul.ac.uk$
$^{2}$ moosavi@ukp.informatik.tu-darmstadt.de
The state-of-the-art on basic, single-antecedent anaphora has greatly improved in recent years. Researchers have therefore started to pay more attention to more complex cases of anaphora such as split-antecedent anaphora, as in Time-Warner is considering a legal challenge to Telecommunications Inc's plan to buy half of Showtime Networks Inc-a move that could lead to all-out war between the two powerful companies. Split-antecedent anaphora is rarer and more complex to resolve than single-antecedent anaphora; as a result, it is not annotated in many datasets designed to test coreference, and previous work on resolving this type of anaphora was carried out in unrealistic conditions that assume gold mentions and/or gold split-antecedent anaphors are available. These systems also focus on split-antecedent anaphors only. In this work, we introduce a system that resolves both single and split-antecedent anaphors, and evaluate it in a more realistic setting that uses predicted mentions. We also start addressing the question of how to evaluate single and split-antecedent anaphors together using standard coreference evaluation metrics. $^{1}$
Thanks in part to the latest developments in deep neural network architectures and contextual word embeddings (e.g., ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019)), the performance of models for single-antecedent anaphora resolution has greatly improved (Wiseman et al., 2016; Clark and Manning, 2016b; Lee et al., 2017, 2018; Kantor and Globerson, 2019; Joshi et al., 2020). So recently, the attention has turned to more complex cases of anaphora, such as anaphora requiring some sort of commonsense knowledge as in the Winograd Schema Challenge (Rahman and Ng,
2012; Peng et al., 2015; Liu et al., 2017; Sakaguchi et al., 2020); pronominal anaphors that cannot be resolved purely using gender (Webster et al., 2018), bridging reference (Hou, 2020; Yu and Poesio, 2020), discourse deixis (Kolhatkar and Hirst, 2014; Marasović et al., 2017; Kolhatkar et al., 2018) and, finally, split-antecedent anaphora (Zhou and Choi, 2018; Yu et al., 2020a) - plural anaphoric reference in which the two antecedents are not part of a single noun phrase.
However, a number of hurdles have to be tackled when trying to study these cases of anaphora, ranging from the lack of annotated resources to the rarity of some of these phenomena in the existing ones. Thus, most previous work on resolving these anaphoric relations focused on developing dedicated systems for the specific task. The systems are usually enhanced by transfer-learning to utilise extra resources, as those anaphoric relations are sparsely annotated. The most frequently used extra resource is single-antecedent anaphors. Due to the complexity of these tasks, previous work is usually based on assuming that either gold anaphors (Hou, 2020; Yu et al., 2020a) or gold mentions (Zhou and Choi, 2018; Yu and Poesio, 2020) are provided. By contrast, in this work we introduce a system that resolves both single and split-antecedent anaphors, and is evaluated in a more realistic setting that does not rely on gold anaphors/mentions. We evaluate our system on the ARRAU corpus (Poesio and Artstein, 2008; Uryupina et al., 2020), in which both single and split-antecedent anaphors are annotated, although the latter are much rarer than the former. We use the state-of-the-art coreference resolution system on ARRAU (Yu et al., 2020b) as our base system for single-antecedent anaphors. This cluster-ranking system interprets single-antecedent anaphors, singletons and non-referring expressions jointly. In this work, we extend the system to resolve split-antecedent anaphors. The extended part of the system shares mention representations
The code is available at https://github.com/juntaoy/dali-full-anaphora
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4174-4184. June 6-11, 2021. ©2021 Association for Computational Linguistics
and candidate clusters with the base system, and outputs binary decisions between a mention and individual candidate clusters. We configure our system to learn the split-antecedent part and the base system in both JOINT and PRE-TRAINED fashion. The results show both versions work much better than naive baselines based on heuristics and random selection. The PRE-TRAINED version works equally well as the JOINT version on split-antecedent anaphors, but it is better for the other aspects of anaphoric interpretation.
In the paper we also begin to address the question of how a system carrying out both single and split-antecedent anaphora resolution should be evaluated. Specifically, we introduce an extended version of LEA (Moosavi and Strube, 2016), a standard coreference metric which can be used to give partial credit for resolution, to evaluate single and split-antecedent anaphors together. Using this metric, we find that our best model achieves a better LEA score than the baselines.
We further evaluate our best system in the gold setting to compare with the Yu et al. (2020a) system. The model achieved better performance when compared to their system that is designed solely for split-antecedent task.
2.1 Neural Approaches to Single-antecedent Anaphora Resolution
Single-antecedent anaphora resolution is an active research topic. The first neural model was introduced by Wiseman et al. (2015) and later extended in (Wiseman et al., 2016). Clark and Manning (2016b) introduced a hybrid cluster/mention-ranking approach, whereas Clark and Manning (2016a) adapted reinforcement learning to a mention-ranking model. Lee et al. (2017) introduced the first end-to-end system, performing mention detection and coreference resolution jointly. The Lee et al. (2017) system was also simpler than previous systems, using only a small number of hand-coded features. As a result, the Lee et al. (2017) system has become the blueprint for most subsequent systems. Lee et al. (2018) and Kantor and Globerson (2019) showed that employing contextual ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019) embeddings in the system by Lee et al. (2017) can significantly improve performance. (Joshi et al., 2019, 2020) fine-tuned BERT and SpanBERT to further improve performance.
Recently, Wu et al. (2020) framed coreference resolution task as question answering and showed that the additional pre-training on a large question answering dataset can further improve performance. However, those systems are only focused on single-antecedent anaphors and do not consider the other anaphoric relations.
2.2 Other Aspects of Anaphoric Interpretation
Interpreting nominal expressions with respect to a discourse model is not simply a matter of identifying identity links; it also involves recognizing that certain potential anaphors are in fact non-referring, or singletons; other expressions refer to entities which have to be introduced in the discourse model via accomodation processes involving for instance the construction of a plural object out of other entities, as in the case of split-antecedent anaphors; other expressions again are related to existing entities by associative relations, as in one-anaphora or bridging reference. These other anaphoric interpretation processes are much less studied, primarily because the relevant information is not annotated in the dominant corpus for coreference, OntoNotes (Pradhan et al., 2012). Systems such as the Stanford Deterministic Coreference Resolver (Lee et al., 2013) do use linguistically-based heuristic rules to recognize and filter singletons and non-referring expressions, but these aspects of the system are not evaluated. Carrying out such an evaluation requires a corpus with richer anaphoric annotations, such as ARRAU (Uryupina et al., 2020).
Yu et al. (2020b) is the only neural system that targets singletons and non-referring expressions. The system uses the mention representation from Lee et al. (2018); Kantor and Globerson (2019) and applies a cluster-ranking algorithm to incrementally attach mentions directly to their clusters. Yu et al. (2020b) showed that performance on single-antecedent anaphors improves by up to 1.4 p.p. when jointly training the model with non-referring expressions and singletons. We use Yu et al. (2020b) as our base system, and extend it to resolve split-antecedent anaphors.
A few systems resolving split-antecedent anaphors have been proposed in recent years. Vala et al. (2016) introduced a system to resolve plural pronouns they and them in a fiction corpus they themselves annotated. Zhou and Choi (2018) introduced an entity-linking corpus based on the transcripts of the Friends sitcom. The mentions (in
cluding plural mentions) are annotated if they are linked to the main characters. Coreference clusters are then created for mentions linked to the same entities. One issue with this corpus is that it is mainly created for entity-linking, so it is problematic as a coreference dataset, as many mentions are linked to general entities that are not annotated in the text. Zhou and Choi (2018) trained a CNN classifier to determine the relation between mention pairs, jointly performing single and split-antecedent resolution.
Another issue with this work is evaluation. Zhou and Choi (2018) evaluate their system using the standard CONLL scorer; in order to do this, they encode split-antecedent anaphora by adding the plural mention to each cluster. So, for instance, in John met Mary. They went to the movies, they would have two gold clusters: {John, They} and {Mary, They}. This is clearly problematic, as They is not a mention of the individual entity John, but of the set consisting of John and Mary. In this work, we propose an alternative, an extended version of LEA (Moosavi and Strube, 2016) that does joint evaluation of single/split-antecedent anaphors by explicitly representing plural entities.
Yu et al. (2020a) introduced the first system to resolve all split-antecedent anaphors annotated in the ARRAU corpus. Their work focuses on the data sparsity problem; split-antecedent anaphora resolution is helped using four auxiliary corpora created from a crowdsourced corpus and other anaphoric annotations in the ARRAU corpus. However, their approach focuses on split-antecedent anaphora only, and assumes gold split-antecedent anaphors and gold mentions are provided during the evaluation, which is not realistic. In this work, we resolve both single and split-antecedent anaphora and evaluate our system on predicted mentions.
3 The Resolution Method
3.1 The Base System
In this work, we use the system of Yu et al. (2020b) as starting point, and extend it to handle split-antecedent anaphora. Yu et al. (2020b) is a cluster-ranking system that jointly processes single-antecedent anaphors, singletons and non-referring expressions. The system uses the same mention representations as in Lee et al. (2018); Kantor and Globerson (2019). The input to the system is a concatenation of contextual BERT (Devlin et al., 2019) embeddings, context-independent GLOVE embeddings (Pennington et al., 2014) and learned
character-level embeddings based on convolutional neural network (CNNs). The system then uses a multi-layer BILSTM to encode the document at the sentence level to create the word representations  $(T_{i})$ . The candidate mention representations  $(M_{i})$  are created by the concatenation of the word representations at the start/end positions of the mention as well as a weighted sum of all the tokens within the mention boundary. After that, the candidate mentions are pruned according to their mention scores  $(s_m(i))$  computed by applying a feedforward neural network (FFNN) to the  $M_{i}$ . The top-ranked candidate mentions are then used by the cluster-ranking model to form the entity clusters and to identify the non-referring expressions.
The cluster-ranking model incrementally links the candidate mentions to the clusters according to the scoring function  $(s(i,j))$  between candidate mention  $M_{i}$  and partial clusters created so far  $(C_{i - 1}^{j})$ . More precisely,  $s(i,j)$  is defined as:
$$
s (i, j) = \left\{ \begin{array}{l l} s _ {n o} (i) & j = \mathrm {N O} \\ s _ {n r} (i) + s _ {m} (i) & j = \mathrm {N R} \\ s _ {d n} (i) + s _ {m} (i) & j = \mathrm {D N} \\ s _ {m} (i) + s _ {c} (j) + s _ {m c} (i, j) & j \in C _ {i - 1} \end{array} \right.
$$
where  $s_{no}(i), s_{nr}(i)$  and  $s_{dn}(i)$  are the likelihood for a candidate mention to be a non-mention (NO), a non-referring expression (NR) or a discourse new mention (DN) respectively.  $s_m(i), s_c(j)$  and  $s_{mc}(i,j)$  are the mention scores (computed for mention pruning), cluster scores (a weighted sum of  $s_m$  for the mentions in the cluster) and cluster-mentation pairwise scores. The system employs additional methods to enhance performance-e.g., keeping cluster histories and training the system on the oracle clusters. We refer the reader to (Yu et al., 2020b) for more details. We use the default settings of the system in our experiments.
3.2 Resolving Split-antecedent Anaphors
To resolve split-antecedent anaphors, we follow Yu et al. (2020a) who framed the task as a binary classification task. The system uses a scoring function to assign each cluster-mention pair a score  $s_p(i,j)$  specifying the likelihood that that cluster is one of the split-antecedents of the mention. During training, we add a dummy score  $(s_{\epsilon}(i) = 0)$  for the cases in which a mention is not a split-antecedent anaphor. Formally,  $s_p(i,j)$  is calculated as follows:
$$
s _ {p} (i, j) = \left\{ \begin{array}{l l} 0 & j = \epsilon \\ s _ {m} (i) + s _ {c} (j) + s _ {p m c} (i, j) & j \in C _ {i - 1} \end{array} \right.
$$
The extension for split-antecedents uses the same mention/cluster representations as well as the can
didate mentions/clusters of the single-antecedent component. This benefits the split-antecedent anaphors part of the system, that can share the representations learned from more numerous single-antecedent anaphors. As a result, the extension shares the same  $s_m(i)$  and  $s_c(j)$  scores as the base system.  $s_{pmc}$  is calculated by applying a FFNN to the cluster-mention pairwise representations. At test time, we convert  $s_p(i,j)$  into probabilities  $(p_p(i,j))$ , and assign split-antecedents to plural mentions when the  $p_p(i,j)$  between the plural mentions and the candidate clusters are above the threshold (e.g., 0.5).  $p_p(i,j)$  is calculated by applying a sigmoid function to  $s_p(i,j)$ :
$$
p _ {p} (i, j) = \frac {1}{1 + e ^ {- s _ {p} (i , j)}}
$$
To make sure the final system outputs (single-antecedent anaphors, singletons, non-referring expressions and split-antecedent anaphors) do not contradict each other, we only allow discourse-new mentions to become split-antecedent anaphors. We also constrain split-antecedent anaphors to have at least two and at most five antecedents.
Since we are working with predicted clusters, to evaluate using lenient and strict scores as in Yu et al. (2020a), we need to find a way to align the predicted clusters with the gold clusters. Here we use the standard coreference alignment function  $\mathrm{CEAF}_{\phi 4}$  to align predicted and gold clusters. The alignment between predicted and gold clusters is at the centre of the  $\mathrm{CEAF}_{\phi 4}$  scores, which gives exactly what we need for our evaluation.
To train, we add to the original loss  $(loss_{s})$  a second dedicated loss  $(loss_{p})$  for split-antecedent anaphors. We use marginal log-likelihood loss, and optimize on all oracle clusters that belong to the gold split-antecedent cluster list  $\mathrm{GOLD}_p(i)$  of split-antecedent anaphors  $M_{i}$ . Formally,
$$
l o s s _ {p} = l o g \prod_ {j = 1} ^ {N} \sum_ {\hat {c} \in \mathrm {G O L D} _ {p} (j)} s _ {p} (\hat {c}, j)
$$
Since the vast majority of mentions (99%) are negative examples (non-split-antecedent anaphors), training is highly imbalanced. So during training we also use the mentions from the same cluster as the split-antecedent anaphors as additional positive examples. In this way we managed to nearly double the number of positive training examples. We multiply the losses of the negative examples an adjustment parameter  $\alpha$  to balance the training.
We train our system both in JOINT and PRE-TRAINED mode. For JOINT learning, we train our system on the sum of two losses and weigh them by a  $\beta$  factor that determines the relative importance of the losses. Formally, we compute the joint loss as follows:
$$
l o s s _ {j} = (1 - \beta) l o s s _ {s} + \beta l o s s _ {p}
$$
To use a joint loss the split-antecedent part of the system can have an impact on the mention representations hence might lead to better performance.
Our PRE-TRAINED approach is based on the hypothesis that mention/cluster representations trained on the single-antecedent anaphors are sufficient as pre-trained embeddings for downstream tasks like split-antecedent anaphors. The PRE-TRAINED approach minimises the changes to the base system, and one can even reuse the models trained solely for the base system. The training for the split-antecedent part is inexpensive. We use the pre-trained models for our base system to supply mention/cluster representations and other necessary information and optimise the split-antecedent part of the system solely on  $loss_{p}$ .
4 Evaluating Coreference Chains with Split Antecedents
If the interpretation of a split-antecedent anaphor were only given credit when all antecedents are correctly detected and grouped together, without giving any reward to systems that find at least some of the antecedents, systems that get closer to the gold would be unfairly penalized, particularly for the cases with 3 or more split antecedents (25% in our data). Consider example 4.1, in which "their $_{i,j}$ " refers to the set {"Mary", "John"}, and "they $_{i,j,p}$ " to the set {"Mary", "John", "Jane)}. And take two systems A and B that resolve "their $_{i,j}$ " to {"Alex", "Jane"} and {"Mary", "Jane"}, respectively and "they $_{i,j,p}$ " to {"Alex"} and {"Mary", "John"}}, respectively. Neither system is perfect, but intuitively, system B is more accurate in resolving split-antecedent anaphors (it correctly identifies 1 antecedent of "their $_{i,j}$ " and 2 of "they $_{i,j,p}$ ", versus 0 for A)-yet both systems will receive the same 0 score if only a perfect match is credited.
Example 4.1. Mary  $i$  and John  $j$  were on their way to visit Alex  $k$  when Mary  $i$  saw  $\mathrm{Jane}_p$  on their  $i,j$  way and realized they  $i,j,p$  all wore the same shirt.
This example indicates that in order to score a system carrying out both single and split-
antecedent resolution three issues have to be addressed. First of all, it is necessary to have some way to represent plural entities. Second, we need some way of ensuring that systems that propose different but equivalent resolutions for split-antecedent plurals score the same. Third, we need a metric allowing some form of partial credit.<sup>2</sup> We discuss how we addressed each issue in turn.
Plural mentions First of all, we propose to have two types of mentions in our coreference chains: in addition to the standard individual mentions ("Mary"), we also allow plural mentions (\{"Mary", "Jane"\}).
Normalizing coreference chains As Example 4.1 shows, a text may contain multiple individual mentions of the same entity that participate in a plural mention (e.g. 'Mary'). Plural mentions whose antecedents are mentions of the same entity should be equivalent. To do this, we use the first mention of each gold coreference chains as the representative of the entity. We normalize every plural mention in a system- produced coreference chain by (i) aligning the system-produced coreference chains for the individual mentions in the plural mention to the gold coreference chains using CEAF, and (ii) replacing each individual mention in the plural mention with the first mention in the aligned gold coreference chains.
Partial credit A natural way to obtain a scorer for coreference resolution giving partial credit is to extend the LEA evaluation metric (Moosavi and Strube, 2016) to handle split-antecedents. For each entity  $e$ , LEA evaluates (a) how important is  $e$ , and (b) how well it is resolved. Thus, for computing recall, LEA evaluates a set of system-detected entities  $E$  as follows:3
$$
\frac {\sum_ {e \in E} \operatorname {i m p o r t a n c e} (e) * \operatorname {r e s o l u t i o n - s c o r e} (e)}{\sum_ {e \in E} \operatorname {i m p o r t a n c e} (e)} \tag {1}
$$
where resolution-score is the ratio of correctly resolved coreference links in the entity, and the importance measures how important is entity  $e$  in the given text. In the default implementation, importance is set to the size of the entity. However, it can be adjusted based on the use case.
Let  $e$  be an entity in the system output  $E$  consisting of  $n$  mentions, and  $K$  be the set of gold entities. The resolution-score  $(RS)$  of  $e$  is computed as:
$$
R S (e) = \frac {1}{| \mathbb {L} (e) |} \sum_ {l \in \mathbb {L} (e)} \mathbb {B} (l, K) \tag {2}
$$
where  $\mathbb{L}(e)$  is the set of all coreference links in  $e^4$  and  $\mathbb{B}(l, K)$  is defined as
$$
\mathbb {B} (l, K) = \left\{ \begin{array}{l l} 1 & \{\exists_ {k \in K} | l \in \mathbb {L} (k) \} \\ 0 & \text {o t h e r w i s e} \end{array} \right. \tag {3}
$$
(3) states that for each coreference link  $l$  in system entities, the system receives a reward of one if  $l$  also exists in gold entities, and zero otherwise. If any of the mentions that are connected by  $l$  is a partially resolved plural mention, the system receives a zero score.
To extend LEA to handle split-antecedents, we change  $\mathbb{B}$  to also reward a system if any of the corresponding mentions of  $l$ , i.e., mentions that are connected by  $l$ , is a plural mention and is partially resolved. Let  $\hat{\mathbb{P}}(m)$  be an ordered list of all subsets of  $m$ , including  $m$ , by descending order of their size. If  $m$  is a singular mention,  $\hat{\mathbb{P}}$  will only contain  $\{m\}$ . If  $m$  is a plural mention,  $\hat{\mathbb{P}}$  will contain  $m$  as well as all the subsets of  $m$ 's containing mentions. For instance,  $\hat{\mathbb{P}}(\{"Mary", "John"\}) = [\{"Mary", "John"\}, {"John"\}, {"Mary"\}]$ . Assuming the corresponding mentions of  $l$  are  $m_i$  and  $m_j$ , we update  $\mathbb{B}(l, K)$  as follows:
$$
\left\{ \begin{array}{l l} \frac {| s _ {i} | * | s _ {j} |}{| m _ {i} | * | m _ {j} |} & \{\exists_ {k \in K, s _ {i} \in \hat {\mathbb {P}} (m _ {i}), s _ {j} \in \hat {\mathbb {P}} (m _ {j})} | l _ {s _ {i}, s _ {j}} \in \mathbb {L} (k) \} \\ \frac {| m _ {i} | * | m _ {j} |}{| m _ {k} | * | m _ {p} |} & \{\exists_ {k \in K, m _ {i} \in \hat {\mathbb {P}} (m _ {k}), m _ {j} \in \hat {\mathbb {P}} (m _ {p})} | l _ {m _ {k}, m _ {p}} \in \mathbb {L} (k) \} \\ 0 & \text {o t h e r w i s e} \end{array} \right.
$$
where  $l_{s_i,s_j}$  is the link connecting  $s_i$  and  $s_j$  that are the largest subset of  $\hat{\mathbb{P}}(m_i)$  and  $\hat{\mathbb{P}}(m_j)$ , respectively, that exist in gold entities and are coreferent.  $m_k$  and  $m_p$  are gold coreferring mentions that  $m_i$  and  $m_j$  are a subset of, respectively.
For instance, consider the system chain  $\{m_1 = \{\text{"Mary","Jane"}\} ,m_2 = \text{"their}_{i,j}\}$  for Example 4.1. The coreference link between  $m_{1}$  and  $m_{2}$  does not exist in the gold entities. However,  $m_{1}$  is a subset of a gold mention, i.e.,  $m_{k} = \{\text{"Mary", "John", "Jane"}\}$ , and  $m_{1}\subset \hat{\mathbb{P}} (m_{k})$ . Therefore, system B receives a reward of  $\frac{2*1}{3*1}$  for resolving the coreference link between  $m_{1}$  and  $m_{2}$  based on  $RS$ .
4 There are  $\frac{n(n - 1)}{2}$  coreference links in  $e$ .