SlowGuess/ACL_2020_OCR huggingface dataset:
Wei Wu\*, Fei Wang\*, Arianna Yuan\*, Fei Wu\* and Jiwei Li\*\*
$\diamond$  Computer Science Department, Stanford University
ShannonAI
xfyuan@stanford.edu, wufei@zju.edu.cn
{wei_wu, fei_wang,jiwei_li} @shannonai.com
In this paper, we present CorefQA, an accurate and extensible approach for the coreference resolution task. We formulate the problem as a span prediction task, like in question answering: A query is generated for each candidate mention using its surrounding context, and a span prediction module is employed to extract the text spans of the coreferences within the document using the generated query. This formulation comes with the following key advantages: (1) The span prediction strategy provides the flexibility of retrieving mentions left out at the mention proposal stage; (2) In the question answering framework, encoding the mention and its context explicitly in a query makes it possible to have a deep and thorough examination of cues embedded in the context of coreferent mentions; and (3) A plethora of existing question answering datasets can be used for data augmentation to improve the model's generalization capability. Experiments demonstrate significant performance boost over previous models, with 83.1 (+3.5) F1 score on the CoNLL-2012 benchmark and 87.5 (+2.5) F1 score on the GAP benchmark.
Recent coreference resolution systems (Lee et al., 2017, 2018; Zhang et al., 2018a; Kantor and Globerson, 2019) consider all text spans in a document as potential mentions and learn to find an antecedent for each possible mention. There are two key issues with this paradigm, in terms of task formalization and the algorithm.
At the task formalization level, mentions left out at the mention proposal stage can never be recovered since the downstream module only operates on the proposed mentions. Existing models often suffer from mention proposal (Zhang et al.,
Original Passage
In addition, many people were poisoned when toxic gas was released. They were poisoned and did not know how to protect themselves against the poison.
Our formulation
Q1: Who were poisoned when toxic gas was released?
A1: [They, themselves]
Q2: What was released when many people were poisoned?
A2: [the poison]
Q3: Who were poisoned and did not know how to protect themselves against the poison?
A3: [many people, themselves]
Q4: Whom did they not know how to protect against the poison?
A4: [many people, They]
Q5: They were poisoned and did not know how to protect themselves against what?
A5: [toxic gas]
Figure 1: An illustration of the paradigm shift from coreference resolution to query-based span prediction. Spans with the same format represent coreferent mentions.
2018a). The coreference datasets can only provide a weak signal for spans that correspond to entity mentions because singleton mentions are not explicitly labeled. Due to the inferiority of the mention proposal model, it would be favorable if a coreference framework had a mechanism to retrieve left-out mentions.
At the algorithm level, existing end-to-end methods (Lee et al., 2017, 2018; Zhang et al., 2018a) score each pair of mentions only based on mention representations from the output layer of a contextualization model. This means that the
https://github.com/ShannonAI/CorefQA
6953
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6953-6963 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
model lacks the connection between mentions and their contexts. Semantic matching operations between two mentions (and their contexts) are performed only at the output layer and are relatively superficial. Therefore it is hard for their models to capture all the lexical, semantic and syntactic cues in the context.
To alleviate these issues, we propose CorefQA, a new approach that formulates the coreference resolution problem as a span prediction task, akin to the question answering setting. A query is generated for each candidate mention using its surrounding context, and a span prediction module is further employed to extract the text spans of the coreferences within the document using the generated query. Some concrete examples are shown in Figure 1.
This formulation provides benefits at both the task formulation level and the algorithm level. At the task formulation level, since left-out mentions can still be retrieved at the span prediction stage, the negative effect of undetected mentions is significantly alleviated. At the algorithm level, by generating a query for each candidate mention using its surrounding context, the CorefQA model explicitly considers the surrounding context of the target mentions, the influence of which will later be propagated to each input word using the self-attention mechanism. Additionally, unlike existing end-to-end methods (Lee et al., 2017, 2018; Zhang et al., 2018a), where the interactions between two mentions are only superficially modeled at the output layer of contextualization, span prediction requires a more thorough and deeper examination of the lexical, semantic and syntactic cues within the context, which will potentially lead to better performance.
Moreover, the proposed question answering formulation allows us to take advantage of existing question answering datasets. Coreference annotation is expensive, cumbersome and often requires linguistic expertise from annotators. Under the proposed formulation, the coreference resolution has the same format as the existing question answering datasets (Rajpurkar et al., 2016a, 2018; Dasigi et al., 2019a). Those datasets can thus readily be used for data augmentation. We show that pre-training on existing question answering datasets improves the model's generalization and
transferability, leading to additional performance boost.
Experiments show that the proposed framework significantly outperforms previous models on two widely-used datasets. Specifically, we achieve new state-of-the-art scores of 83.1 (+3.5) on the CoNLL-2012 benchmark and 87.5 (+2.5) on the GAP benchmark.
2.1 Coreference Resolution
Coreference resolution is a fundamental problem in natural language processing and is considered as a good test of machine intelligence (Morgenstern et al., 2016). Neural network models have shown promising results over the years. Earlier neural-based models (Wiseman et al., 2016; Clark and Manning, 2015, 2016) rely on parsers and hand-engineered mention proposal algorithms. Recent work (Lee et al., 2017, 2018; Kantor and Globerson, 2019) tackled the problem in an end-to-end fashion by jointly detecting mentions and predicting coreferences. Based on how entity-level information is incorporated, they can be further categorized as (1) entity-level models (Björkelund and Kuhn, 2014; Clark and Manning, 2015, 2016; Wiseman et al., 2016) that directly model the representation of real-world entities and (2) mention-ranking models (Durrett and Klein, 2013; Wiseman et al., 2015; Lee et al., 2017) that learn to select the antecedent of each anaphoric mention. Our CorefQA model is essentially a mention-ranking model, but we identify coreference using question answering.
2.2 Formalizing NLP Tasks as question answering
Machine reading comprehension is a general and extensible task form. Many tasks in natural language processing can be framed as reading comprehension while abstracting away the task-specific modeling constraints.
McCann et al. (2018) introduced the decaNLP challenge, which converts a set of 10 core tasks in NLP to reading comprehension. He et al. (2015) showed that semantic role labeling annotations could be solicited by using question-answer pairs to represent the predicate-argument structure. Levy et al. (2017) reduced relation extraction to answering simple reading comprehension questions, yielding models that generalize better in the
2This is an illustration of the question formulation. The actual operation is described in Section 3.4.
Figure 2: The overall architecture of our CorefQA model. The input passage is first fed into the Mention Proposal Module 3.3 to obtain candidate mentions. Then the Mention Linking Module 3.4 is used to extract coreferent mentions from the passage for each proposed mention. The coreference clusters are obtained using the scores produced in the above two stages.
zero-shot setting. Li et al. (2019a,b) cast the tasks of named entity extraction and relation extraction as a reading comprehension problem. In parallel to our work, Aralikatte et al. (2019) converted coreference and ellipsis resolution in a question answering format, and showed the benefits of training joint models for these tasks. Their models are built under the assumption that gold mentions are provided at inference time, whereas our model does not need that assumption – it jointly trains the mention proposal model and the coreference resolution model in an end-to-end manner.
2.3 Data Augmentation
Data augmentation is a strategy that enables practitioners to significantly increase the diversity of data available for training models. Data augmentation techniques have been explored in various fields such as question answering (Talmor and Berant, 2019), text classification (Kobayashi, 2018) and dialogue language understanding (Hou et al., 2018). In coreference resolution, Zhao et al. (2018); Emami et al. (2019); Zhao et al. (2019) focused on debiasing the gender bias problem; Aralikatte et al. (2019) explored the effectiveness of joint modeling of ellipsis and coreference resolution. To the best of our knowledge, we are the first to use existing question answering datasets as data augmentation for coreference resolution.
In this section, we describe our CorefQA model in detail. The overall architecture is illustrated in Figure 2.
Given a sequence of input tokens  $X = \{x_{1}, x_{2}, \ldots, x_{n}\}$  in a document, where  $n$  denotes the length of the document.  $N = n * (n + 1) / 2$  denotes the number of all possible text spans in  $X$ . Let  $e_{i}$  denotes the  $i$ -th span representation  $1 \leq i \leq N$ , with the start index FIRST(i) and the end index LAST(i).  $e_{i} = \{x_{\text{FIRST}(i)}, x_{\text{FIRST}(i) + 1}, \ldots, x_{\text{LAST}(i) - 1}, x_{\text{LAST}(i)}\}$ .
The task of coreference resolution is to determine the antecedents for all possible spans. If a candidate span  $e_i$  does not represent an entity mention or is not coreferent with any other mentions, a dummy token  $\epsilon$  is assigned as its antecedent. The linking between all possible spans  $e$  defines the final clustering.
3.2 Input Representations
We use the SpanBERT model  ${}^{3}$  to obtain input representations following Joshi et al. (2019a). Each token  ${x}_{i}$  is associated with a SpanBERT representation  ${\mathbf{x}}_{i}$  . Since the speaker information is indispensable for coreference resolution, previous methods (Wiseman et al., 2016; Lee et al., 2017; Joshi et al., 2019a) usually convert the speaker information into binary features indicating whether two mentions are from the same speaker. However, we use a straightforward strategy that directly concatenates the speaker's name with the corresponding utterance. This strategy is inspired by recent research in personalized dialogue modeling that use persona information to represent speakers (Li et al., 2016; Zhang et al., 2018b;
$^{3}$ https://github.com/facebookresearch/ SpanBERT
Mazaré et al., 2018). In subsection 5.2, we will empirically demonstrate its superiority over the feature-based method in Lee et al. (2017).
To fit long documents into SpanBERT, we use a sliding-window approach that creates a  $T$ -sized segment after every  $T/2$  tokens. Segments are then passed to the SpanBERT encoder independently. The final token representations are derived by taking the token representations with maximum context.
3.3 Mention Proposal
Similar to Lee et al. (2017), our model considers all spans up to a maximum length  $L$  as potential mentions. To improve computational efficiency, we further prune the candidate spans greedily during both training and evaluation. To do so, the mention score of each candidate span is computed by feeding the first and the last of its constituent token representations into a feed-forward layer:
$$
s _ {\mathrm {m}} (i) = \operatorname {F F N N} _ {\mathrm {m}} \left(\left[ \boldsymbol {x} _ {\text {F I R S T} (i)}, \boldsymbol {x} _ {\text {L A S T} (i)} \right]\right) \tag {1}
$$
where  $\mathbf{x}_{\mathrm{FIRST}(i)}$  and  $\mathbf{x}_{\mathrm{LAST}(i)}$  represent the first and the last token representation of the  $i$ -th candidate span.  $\mathrm{FFNN_m}()$  denotes the feed-forward neural network that computes a nonlinear mapping from the input vector to the mention score. We only keep up to  $\lambda n$  (where  $n$  is the document length) spans with the highest mention scores.
3.4 Mention Linking as Span Prediction
Given a mention  $e_i$  proposed by the mention proposal network, the role of the mention linking network is to give a score  $s_a(i,j)$  for any text span  $e_j$ , indicating whether  $e_i$  and  $e_j$  are coreferent. We propose to use the question answering framework as the backbone to compute  $s_a(i,j)$ . It operates on the triplet  $\{ \text{context}(\mathbf{X}), \text{query}(\mathbf{q}), \text{answers}(\mathbf{a}) \}$ . The context  $X$  is the input document. The query  $q(e_i)$  is constructed as follows: given  $e_i$ , we use the sentence that  $e_i$  resides in as the query, with the minor modification that we encapsulates  $e_i$  with special tokens  $< \text{mention} > < / \text{mention} >$ . The answers  $a$  are the coreferent mentions of  $e_i$ .
Following Devlin et al. (2019), we represent the input query and the context as a single packed sequence. Since a mention can have multiple coreferent mentions, we follow Li et al. (2019a,b) and generate a  $BIO$  tag for each token.  $BIO$  tags respectively mark the beginning(B), inside(I) and outside(O) of a coreferent mention. It is worth
noting that there exist unanswerable queries where labels for tokens in  $X$  are all  $O$ .<sup>4</sup> A query is considered unanswerable in the following scenarios: (1) the candidate span  $e_i$  does not represent an entity mention or (2) the candidate span  $e_i$  represents an entity mention but is not coreferent with any other mentions in  $X$ .
The probability of assigning a tag  $\in B, I, O$  is computed as follows:
$$
p _ {i} ^ {\text {t a g}} = \operatorname {s o f t m a x} \left(\operatorname {F F N N} _ {\text {t a g}} \left(\boldsymbol {x} _ {i}\right)\right) \tag {2}
$$
$\mathrm{FFNN}_{\mathrm{tag}}()$  represents the feed-forward neural network that computes a nonlinear mapping from the input vector to the tag logit.
We further extend the token-level score in Eq. 2 to the span level. The anaphora score  $s_a(j|i)$ , the compatibility score of span  $j$  being a answer for span  $i$ , is calculated by the log probability of its beginning word taking the  $B$  tag and the rest taking the  $I$  tag:
$$
s _ {a} (j \mid i) = \frac {1}{\left| e _ {j} \right|} \left[ \log p _ {\text {F I R S T} (j)} ^ {B} + \sum_ {k = \text {F I R S T} (j) + 1} ^ {k = \text {L A S T} (j)} \log p _ {\mathrm {k}} ^ {I} \right] \tag {3}
$$
A closer look at Eq.3 reveals that it only models the uni-directional coreference relation from  $e_i$  to  $e_j$ , i.e.,  $e_j$  is the answer for query  $q(e_i)$ . This is suboptimal since if  $e_i$  is a coreference mention of  $e_j$ , then  $e_j$  should also be the coreference mention  $e_i$ . We thus need to optimize the bi-directional relation between  $e_i$  and  $e_j$ . The final score  $s_a(i, j)$  is thus given as follows:
$$
s _ {a} (i, j) = \frac {1}{2} \left(s _ {a} (j \mid i) + s _ {a} (i \mid j)\right) \tag {4}
$$
$s_a(i|j)$  can be computed in the same way as  $s_a(j|i)$ , in which  $q(e_j)$  is used as the query. For a pair of text span  $e_i$  and  $e_j$ , the premises for them being coreferent mentions are (1) they are mentions and (2) they are coreferent. This makes the overall score  $s(i,j)$  for  $e_i$  and  $e_j$  the combination of Eq.1 and Eq.4:
$$
s (i, j) = s _ {\mathrm {m}} (i) + s _ {\mathrm {m}} (j) + s _ {a} (i, j) \tag {5}
$$
4 In the rare cases where coreferent answers are nested, we simply treat all tokens of the inner mentions as  $I$ .
This bidirectional relationship is actually referred to as mutual dependency and has shown to benefit a wide range of NLP tasks such as machine translation (Hassan et al., 2018) or dialogue generation (Li et al., 2015).
6956
3.5 Antecedent Pruning
Given a document  $X$  with length  $n$  and the number of spans  $O(n^{2})$ , the computation of Eq.5 for all mention pairs is intractable with the complexity of  $O(n^{4})$ . Given an extracted mention  $e_{i}$ , the computation of Eq.5 for  $(e_{i}, e_{j})$  regarding all  $e_{j}$  is still extremely intensive since the computation of the backward span prediction score  $s_{a}(i|j)$  requires running question answering models on all query  $q(e_{j})$ . A further pruning procedure is thus needed: For each query  $q(e_{i})$ , we collect  $C$  span candidates only based on the  $s_{a}(j|i)$  scores.
3.6 Training
For each mention  $e_i$  proposed by the mention proposal network, it is associated with  $C$  potential spans proposed by the mention linking network based on  $s(j|i)$ , we aim to optimize the marginal log-likelihood of all correct antecedents implied by the gold clustering. Following Lee et al. (2017), we append a dummy token  $\epsilon$  to the  $C$  candidates. The model will output it if none of the  $C$  span candidates is coreferent with  $e_i$ . For each mention  $e_i$ , the model learns a distribution  $P(\cdot)$  over all possible antecedent spans  $e_j$  based on the global score  $s(i,j)$  from Eq. 5:
$$
P \left(e _ {j}\right) = \frac {e ^ {s (i , j)}}{\sum_ {j ^ {\prime} \in C} e ^ {s (i , j ^ {\prime})}} \tag {6}
$$
The mention proposal module and the mention linking module are jointly trained in an end-to-end fashion using training signals from Eq.6, with the SpanBERT parameters shared.
3.7 Inference
Given an input document, we can obtain an undirected graph using the overall score, each node of which represents a candidate mention from either the mention proposal module or the mention linking module. We prune the graph by keeping the edge whose weight is the largest for each node based on Eq.6. Nodes whose closest neighbor is the dummy token  $\epsilon$  are abandoned. Therefore, the mention clusters can be decoded from the graph.
3.8 Data Augmentation using Question Answering Datasets
We hypothesize that the reasoning (such as synonymy, world knowledge, syntactic variation, and multiple sentence reasoning) required to answer
the questions are also indispensable for coreference resolution. Annotated question answering datasets are usually significantly larger than the coreference datasets due to the high linguistic expertise required for the latter. Under the proposed QA formulation, coreference resolution has the same format as the existing question answering datasets (Rajpurkar et al., 2016a, 2018; Dasigi et al., 2019a). In this way, they can readily be used for data augmentation. We thus propose to pretrain the mention linking network on the Quoref dataset (Dasigi et al., 2019b), and the SQuAD dataset (Rajpurkar et al., 2016b).
3.9 Summary and Discussion
Comparing with existing models (Lee et al., 2017, 2018; Joshi et al., 2019b), the proposed question answering formalization has the flexibility of retrieving mentions left out at the mention proposal stage. However, since we still have the mention proposal model, we need to know in which situation missed mentions could be retrieved and in which situation they cannot. We use the example in Figure 1 as an illustration, in which {many people, They, themselves} are coreferent mentions: If partial mentions are missed by the mention proposal model, e.g., many people and They, they can still be retrieved in the mention linking stage when the not-missed mention (i.e., themselves) is used as query. But, if all the mentions within the cluster are missed, none of them can be used for query construction, which means they all will be irreversibly left out. Given the fact that the proposal mention network proposes a significant number of mentions, the chance that mentions within a mention cluster are all missed is relatively low (which exponentially decreases as the number of entities increases). This explains the superiority (though far from perfect) of the proposed model. However, how to completely remove the mention proposal network remains a problem in the field of coreference resolution.
The special tokens used to denote the speaker's name (< speaker >/speaker>) and the special tokens used to denote the queried mentions (< mention >/mention>) are initialized by randomly taking the unused tokens from the SpanBERT vocabulary. The sliding window size
6957
$T = 512$ , and the mention keep ratio  $\lambda = 0.2$ . The maximum length  $L$  for mention proposal  $= 10$  and the maximum number of antecedents kept for each mention  $C = 50$ . The SpanBERT parameters are updated by the Adam optimizer (Kingma and Ba, 2015) with initial learning rate  $1 \times 10^{-5}$  and the task parameters are updated by the Range optimizer with initial learning rate  $2 \times 10^{-4}$ .
We compare the CorefQA model with previous neural models that are trained end-to-end:
- e2e-coref (Lee et al., 2017) is the first end-to-end coreference system that learns which spans are entity mentions and how to best cluster them jointly. Their token representations are built upon the GLoVe (Pennington et al., 2014) and Turian (Turian et al., 2010) embeddings.
- c2f-coref + ELMo (Lee et al., 2018) extends Lee et al. (2017) by combining a coarse-to-fine pruning with a higher-order inference mechanism. Their representations are built upon ELMo embeddings (Peters et al., 2018).
- c2f-coref + BERT-large(Joshi et al., 2019b) builds the c2f-coref system on top of BERT (Devlin et al., 2019) token representations.
- EE + BERT-large (Kantor and Globerson, 2019) represents each mention in a cluster via an approximation of the sum of all mentions in the cluster.
- c2f-coref + SpanBERT-large (Joshi et al., 2019a) focuses on pre-training span representations to better represent and predict spans of text.
4.3 Results on CoNLL-2012 Shared Task
The English data of CoNLL-2012 shared task (Pradhan et al., 2012) contains 2,802/343/348 train/development/test documents in 7 different genres. The main evaluation is the average of three metrics - MUC (Vilain et al., 1995),  $\mathbf{B}^3$  (Bagga and Baldwin, 1998), and  $\mathrm{CEAF}_{\phi_4}$  (Luo, 2005) on the test set according to the official CoNLL-2012 evaluation scripts<sup>7</sup>.
We compare the CorefQA model with several baseline models in Table 1. Our CorefQA system achieves a huge performance boost over existing systems: With SpanBERT-base, it achieves an F1 score of 79.9, which already outperforms the previous SOTA model using SpanBERT-large by 0.3. With SpanBERT-large, it achieves an F1 score of 83.1, with a 3.5 performance boost over the previous SOTA system.
4.4 Results on GAP
The GAP dataset (Webster et al., 2018) is a gender-balanced dataset that targets the challenges of resolving naturally occurring ambiguous pronouns. It comprises 8,908 coreference-labeled pairs of (ambiguous pronoun, antecedent name) sampled from Wikipedia.
We follow the protocols in Webster et al. (2018); Joshi et al. (2019b) and use the off-the-shelf resolver trained on the CoNLL-2012 dataset to get the performance of the GAP dataset. Table 2 presents the results. We can see that the proposed CorefQA model achieves state-of-the-art performance on all metrics on the GAP dataset.
5 Ablation Study and Analysis
We perform comprehensive ablation studies and analyses on the CoNLL-2012 development dataset. Results are shown in Table 3.
5.1 Effects of Different Modules in the Proposed Framework
Effect of SpanBERT Replacing SpanBERT with vanilla BERT leads to a 3.8 F1 degradation. This verifies the importance of span-level pre-training for coreference resolution and is consistent with previous findings (Joshi et al., 2019a).