SlowGuess/ACL_2020_OCR huggingface dataset:
Yufang Hou
IBM Research Europe
yhou@ie.ibm.com
Most previous studies on bridging anaphora resolution (Poesio et al., 2004; Hou et al., 2013b; Hou, 2018a) use the pairwise model to tackle the problem and assume that the gold mention information is given. In this paper, we cast bridging anaphora resolution as question answering based on context. This allows us to find the antecedent for a given anaphor without knowing any gold mention information (except the anaphor itself). We present a question answering framework (BARQA) for this task, which leverages the power of transfer learning. Furthermore, we propose a novel method to generate a large amount of "quasi-bridging" training data. We show that our model pre-trained on this dataset and fine-tuned on a small amount of in-domain dataset achieves new state-of-the-art results for bridging anaphora resolution on two bridging corpora (ISNotes [Markert et al., 2012] and BASHI [Rösiger, 2018]).
Anaphora accounts for text cohesion and is crucial for text understanding. An anaphor is a noun phrase (NP) that usually refers back to the same or a different entity (the antecedent) in text. Anaphora resolution is the task to determine the antecedent for a given anaphor. While direct anaphora resolution attracts a lot of attention in the NLP community recently, such as Winograd Schema Challenge (Rahman and Ng, 2012; Opitz and Frank, 2018; Kocijan et al., 2019), indirect anaphora resolution or bridging anaphora resolution is less well studied.
In this paper, we focus on bridging anaphora resolution where bridging anaphors and their antecedents are linked via various lexico-semantic, frame or encyclopedic relations. Following Hou et al. (2013b) and Rösiger et al. (2018), we mainly consider "referential bridging" in which bridging
anaphors are truly anaphoric and bridging relations are context-dependent. In Example  $1^{1}$ , both "her building" and "buildings with substantial damage" are plausible antecedent candidates for the bridging anaphor "residents" based on lexical semantics. In order to find the antecedent (buildings with substantial damage), we have to take the meaning of the broader discourse context into account.
(1) In post-earthquake parlance, her building is a "red". After being inspected, buildings with substantial damage were color-coded. Green allowed residents to re-enter; yellow allowed limited access; red allowed residents one last entry to gather everything they could within 15 minutes.
Most previous studies on bridging anaphora resolution (Poesio et al., 2004; Lassalle and Denis, 2011; Hou et al., 2013b; Hou, 2018a) tackle the problem using the pairwise model and assume that the gold mention information is given. Most work (Poesio et al., 2004; Lassalle and Denis, 2011; Hou et al., 2013b) uses syntactic patterns to measure semantic relatedness between the head nouns of an anaphor and its antecedent. Hou (2018a) proposes a simple deterministic algorithm that also considers the semantics of modifications for head nouns. These approaches, however, do not take the broader context outside of noun phrases (i.e., anaphors and antecedent candidates) into account and often fail to resolve context-dependent bridging anaphors as demonstrated in Example 1.
Resolving bridging anaphors requires context-dependent text understanding. Recently, Gardner et al. (2019) argue that question answering (QA) is a natural format to model tasks that require question understanding. In this paper, we cast bridging anaphora resolution as question answering based
<sup>1</sup>All examples, if not specified otherwise, are from ISNotes (Markert et al., 2012). Bridging anaphors are typed in boldface, antecedents in italics throughout this paper.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1428-1438 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
on context. We develop a QA system (BARQA) for the task based on BERT (Devlin et al., 2019). Given a context as shown in Example 1, we first rephrase every anaphor as a question, such as "res-idents of what?". By answering the question, the system then identifies the span of the antecedent from the context. Compared to the pairwise model, our QA system does not require the gold or system mention information as the antecedent candidates. In addition, this framework allows us to integrate context outside of NPs when choosing antecedents for bridging anaphors. For instance, "Green" and "damage were color-coded" are among the top predicted answers for the above question.
Different from coreference resolution, there are no large-scale corpora available for referential bridging resolution due to its complexity. In this paper we propose a new method to generate a large amount of "quasi-bridging" training data from the automatically parsed Gigaword corpus (Parker et al., 2011; Naples et al., 2012). We demonstrate that our "quasi-bridging" training data is a better pre-training choice for bridging anaphora resolution compared to the SQuAD corpus (Rajpurkar et al., 2016). Moreover, we show that our model pre-trained on this dataset and fine-tuned on a small amount of in-domain dataset achieves new state-of-the-art results for bridging anaphora resolution on two bridging corpora (i.e., ISNotes (Markert et al., 2012) and BASHI (Rösiger, 2018)).
To summarize, the main contributions of our work are: (1) we formalize bridging anaphora resolution as a question answering problem and propose a QA model to solve the task; (2) we explore a new method to generate a large amount of "quasi-bridging" training dataset and demonstrate its value for bridging anaphora resolution; and (3) we carefully carry out a series of experiments on two referential bridging corpora and provide some error analysis to verify the effectiveness of our QA model to resolve the context-dependent bridging anaphors in ISNotes. We release the code and all experimental datasets at https://github.com/IBM/bridging-resolution.
Bridging Anaphora Resolution. Since the '90s, the empirical corpus studies related to bridging have been carried out on various genres and different languages (Fraurud, 1990; Poesio and Vieira, 1998; Poesio, 2004; Nissim et al., 2004; Gardent
and Manuelian, 2005; Nedoluzhko et al., 2009; Eckart et al., 2012; Markert et al., 2012; Rösiger, 2018; Poesio et al., 2018). Among those datasets, ISNotes (Markert et al., 2012), BASHI (Rösiger, 2018) and ARRAU (Poesio et al., 2018) are recent three public English corpora which contain medium- to large-sized bridging annotations and have been used to evaluate systems' performance on bridging anaphora recognition (Hou et al., 2013a; Hou, 2016; Rösiger et al., 2018), bridging anaphora resolution (Poesio et al., 2004; Lassalle and Denis, 2011; Hou et al., 2013b; Hou, 2018a), as well as full bridging resolution (Hou et al., 2014, 2018; Rösiger et al., 2018). In this paper, we focus exclusively on the task of antecedent selection.
It is worth noting that the bridging definition in the ARRAU corpus is different from the one used in the other two datasets. Rösiger et al. (2018) pointed out that ISNotes and BASHI contain "referential bridging" where bridging anaphors are truly anaphoric and bridging relations are context-dependent, while in ARRAU, most bridging links are purely lexical bridging pairs which are not context-dependent (e.g., Europe - Spain or Tokyo - Japan). In this paper, we focus on resolving referential bridging anaphors.
Regarding the algorithm for bridging anaphora resolution, most previous work uses the pairwise model for the task. The model assumes gold or system mention information (NPs) is given beforehand. It creates (positive/negative) training instances by pairing every anaphor  $a$  with its preceding mention  $m$ . Usually,  $m$  is from a set of antecedent candidates which is formed using a fixed window size. Poesio et al. (2004) and Lassalle and Denis (2011) trained such pairwise models to resolve mereological bridging anaphors in the English GNOME corpus<sup>2</sup> and the French DEDE corpus (Gardent and Manuelian, 2005), respectively. One exception is Hou et al. (2013b), which proposed a joint inference framework to resolve bridging anaphors in ISNotes. The framework is built upon the pairwise model and predicts all semantically related bridging anaphors in one document together.
Recently, Hou (2018a) generated a word representation resource for bridging (i.e., embeddings_bridging) and proposed a simple deterministic algorithm to find antecedents for bridging anaphors in ISNotes and BASHI. The word representation resource is learned from a large corpus
2The GNOME corpus is not publicly available.
Figure 1: Resolving bridging anaphors in Example 1 using BARQA.
and it captures the common-sense knowledge (i.e., semantic relatedness) between NPs.
Different from the algorithms mentioned above, our QA model does not require the extracted or gold mentions (NPs) as the input, and it predicts the span of the antecedent for a bridging anaphor directly.
Question Answering. Reading comprehension or question answering based on context has attracted much attention within the NLP community, in particular since Rajpurkar et al. (2016) released a large-scale dataset (SQuAD) consisting of  $100,000+$  questions on a set of paragraphs extracted from Wikipedia articles. Previous work has cast a few traditional NLP tasks as question answering, such as textual entailment (McCann et al., 2018), entity-relation extraction (Li et al., 2019), and coreference resolution (Wu et al., 2020). However, unlike these tasks, we do not have large scale training datasets for bridging. As a result, we form the questions for our task in a more natural way in order to leverage the existing QA datasets (e.g., SQuAD) that require common-sense reasoning. In addition, we generate a large-scale training dataset of "quasi-bridging" and demonstrate that it is a good pre-training corpus for bridging anaphora resolution.
Recently, Gardner et al. (2019) argue that we should consider question answering as a format instead of a task in itself. From this perspective, our work can be seen as a specific probing task to test a QA model's ability to understand bridging anaphora based on context.
Winograd Schema Challenge. Bridging anaphora resolution shares some similarities with Winograd Schema Challenge (WSC). Specifically, in both tasks, one has to understand the context to find the antecedents for anaphors. However, the antecedent search space in bridging anaphora resolution is much bigger than the one in WSC. This is because an anaphor (pronoun) and its antecedent in WSC are usually from the same sentence, while bridging pairs usually require cross-sentence inference. For instance, in ISNotes, only around  $26\%$  of anaphors have antecedents occurring in the same sentence, and  $23\%$  of anaphors have antecedents that are more than two sentences away.
Recently, Kocijan et al. (2019) use some heuristics to generate a large-scale WSC-like dataset and report that the model pre-trained on this dataset achieves the best results on several WSC datasets after being fine-tuned on a small in-domain dataset. We find similar patterns of results for bridging anaphora resolution (see Section 5.3).
3 BARQA: A QA System for Bridging Anaphora Resolution
In this section, we describe our QA system (called BARQA) for bridging anaphora resolution in detail. Figure 1 illustrates how BARQA predicts antecedents for bridging anaphors in Example 1.
We formulate bridging anaphora resolution as a context-based QA problem. More specifically, given a bridging anaphor  $a$  and its surrounding
context  $c_{a}$ , we rephrase  $a$  as a question  $q_{a}$ . The goal is to predict a text span  $s_{a}$  from  $c_{a}$  that is the antecedent of  $a$ . We propose to use the span-based QA framework to extract  $s_{a}$ . In general, our BARQA system is built on top of the vanilla BERT QA framework (Devlin et al., 2019). We further modify the inference algorithm to guarantee that the answer span  $s_{a}$  should always appear before the bridging anaphor  $a$  (see Section 3.4 for more details).
Following Devlin et al. (2019), we present the input question  $q_{a}$  and the context  $c_{a}$  as a single packed sequence "[cls]  $q_{a}$  [sep]  $c_{a}$ " and calculate the probabilities of every word in  $c_{a}$  being the start and end of the answer span. The training objective is the log-likelihood of the correct start and end positions.
3.2 Question Generation
In English, the preposition "of" in the syntactic structure "np1 of np2" encodes different associative relations between noun phrases that cover a variety of bridging relations. For instance, "the chairman of IBM" indicates a professional function in an organization, and "the price of the stock" indicates an attribute of an object. Poesio et al. (2004) also used such patterns to estimate the part-of bridging relations. These patterns reflect how we explain bridging anaphora as human beings. It seems that the most natural way to understand the meaning of a bridging anaphor  $a$  is to find the answer for the question "a of what?" from the surrounding context of  $a$ .
As a result, in order to generate the corresponding question  $q_{a}$  for a bridging anaphor  $a$ , we first create  $a'$  by removing all words appearing after the head of  $a$ , we then concatenate  $a'$  with "of what?" to form the question. This is because, as pointed by Hou (2018a), premodifiers of bridging anaphors are essential elements to understand bridging relations. For instance, for the bridging anaphor "a painstakingly documented report, based on hundreds of interviews with randomly selected refugees", the corresponding question is "a painstakingly documented report of what?".
3.3 Answer Generation
For each bridging anaphor  $a$  together with its corresponding question  $q_{a}$  and context  $c_{a}$  described above, we construct a list of answers  $A$  that contains all antecedents of  $a$  occurring in the context
$c_{a}$ . In addition, for every NP antecedent  $n$  from  $A$ , we add the following variations which represent the main semantics of  $n$  into the answer list:
- the head of  $n$  (e.g., last week's earthquake)
-  $n'$  which is created by removing all postmodifiers from  $n$  (e.g., the preliminary conclusion from a survey of 200 downtown high-rises)
-  $n''$  which is created by removing all postmodifiers and the determiner from  $n$  (e.g., the total potential claims from the disaster)
It is worth noting that if the context  $c_{a}$  does not contain any antecedent for the bridging anaphora  $a$  (e.g., some anaphors do not have antecedents occurring in  $c_{a}$  if we use a small window size to construct it), we put "no answer" into the answer list  $A$ .
Different from the SQuAD-style question answering where there is no specific requirement for the position of the predicted span, in bridging anaphora resolution, an anaphor must appear after its antecedent. Therefore in the inference stage, for each bridging anaphor  $a$ , we first identify the position of  $a$  in its context  $c_{a}$ , then we only predict text spans which appear before  $a$ . We further prune the list of predicted text spans by only keeping the top  $k$  span candidates that contain at most  $l$  words ( $k$  and  $l$  are empirically set to 20 and 5, respectively). We also prune span predictions that are function words (e.g.,  $a$ ,  $an$ , the, this, that).
During the training process, we first use Span-BERT (Joshi et al., 2019) to initialize our BARQA model because it shows promising improvements on SQuAD 1.1 compared to the vanilla BERT embeddings. We then continue to train our model using different pre-training and fine-tuning strategies. Section 5.3 describes different training strategies in detail.
For every training strategy, we train BARQA for five epochs with a learning rate of 3e-5 and a batch size of 24. During training and testing, the maximum text length is set to 128 tokens.
<sup>3</sup>In ISNotes and BASHI, we use gold coreference annotations from OntoNotes (Weischedel et al., 2011) to identify all possible antecedents for every bridging anaphor.
In general, the small learning rate (i.e., 3e-5, 4e-5, and 5e-5) and small fine-tuning epochs are common practices for fine-tuning BERT models. We test the combination of these
In a search for new evidence of obstruction of justice by the president, Republicans seek documents concerning several figures from the campaign fund-raising scandal.
Today 's hearing into crimes of perjury is an attempt to focus the nation's attention on whether to remove Clinton from office for allegedly lying under oath about his relationship with the former White House intern and then obstructing justice and tampering with witnesses to conceal it.
Generated Quasi-bridging example
Sentence s.y: Today 's hearing into crimes of perjury is an attempt to focus the nation's attention on whether to remove Clinton from office for allegedly lying under oath about his relationship with the former White House intern and then obstructing justice and tampering with witnesses to conceal it.
Sentence  $s_x$ : In a search for new evidence of the obstruction by the president, Republicans seek documents concerning several figures from the campaign fund-raising scandal.
Figure 2: Examples of generating "quasi-bridging" training data.
4 Generate "Quasi-bridging" Training Data
Bridging anaphora is a complex phenomenon, and there are no large-scale corpora available for referential bridging. In this section, we describe how we generate a large scale "quasi-bridging" dataset.
Hou (2018b) explores the syntactic prepositional and possessive structures of NPs to train word embeddings for bridging. Inspired by this work, we first use these structures to identify "bridging anaphors" and the corresponding "antecedents". Next, we map them back to the discourse to create bridging-like examples.
More specifically, given a text, we first extract NPs containing the prepositional structure (e.g.,  $\mathbf{X}$  preposition  $\mathbf{Y}$ ) or the possessive structure (e.g.,  $\mathbf{Y}$ 's  $\mathbf{X}$ ). In order to have a high-quality set of automatically generated bridging annotations, we apply an additional constraint to the above NPs, i.e.,  $\mathbf{X}$  and  $\mathbf{Y}$  should not contain any other NP nodes in the constituent tree. For instance, we do not consider NPs such as "the political value of imposing sanctions against South Africa" or "the cost of repairing the region's transportation system".
Figure 2 illustrates how we generate a bridging annotation with a sentence pair  $\{s_y, s_x\}$  from a raw text: we first extract the NP "obstruction of justice" from the sentence  $s_i$  and identify  $\mathbf{X} / \mathbf{Y}$  in this extracted NP (i.e.,  $\mathbf{X} =$  obstruction,  $\mathbf{Y} =$  justice). Next, we collect a list of sentences  $S$  from the
whole text. Every sentence in  $S$  contains  $\mathbf{Y}$  but does not contain  $\mathbf{X}$ . If  $S$  contains more than one sentence, we choose the one which is the closest to  $s_i$  as  $s_y$ . This is because close sentences are more likely semantically related. Finally, we generate the sentence  $s_x$  by replacing "obstruction of justice" in the original sentence  $s_i$  with "the obstruction". This gives us a quasi-bridging example with two adjacent sentences (i.e.,  $s_y$  and  $s_x$ ) and a bridging link (i.e., justice - the obstruction).
As a result, we obtain a large amount of "quasi-bridging" training data (i.e., around 2.8 million bridging pairs) by applying the method described above to the NYT19 section of the automatically parsed Gigaword corpus.
In order to understand the quality of our "quasi-bridging" training dataset, we randomly sample 100 quasi-bridging sentence pairs and manually check bridging annotations in these instances. We score each bridging annotation using a scale of 0-2: "2" means that the bridging annotation is correct and the sentence pair sounds natural; "1" indicates that the example makes sense, but it does not sound natural in English; and "0" denotes that the annotation is unacceptable. Overall, we find that  $25\%$  of instances and  $37\%$  of instances have a score of 2 and 1, respectively. And the remaining  $38\%$  of instances are scored as zero. In general, our noisy "quasi-bridging" training dataset does contain a large number of diverse bridging pairs.
We use four datasets for experiments. The first dataset is  $\mathrm{ISNotes}^6$  released by Markert et al.
parameters for various training configurations on a small set (10 documents) of the ISNotes corpus and the BASHI corpus, respectively. On both corpora, we observed that a learning rate of 3e-5, 4e-5, or 5e-5 has minimal impact on results; and for each learning rate, the result continues improving at the beginning (epochs = 1,2,3,4,5), but the performances stays more or less the same after epochs > 5.
5The raw text is from the Gigaword corpus (Parker et al., 2011; Naples et al., 2012).
http://www.h-its.org/en/research/nlp/isnotes-corpus
(2012). This dataset contains 50 texts with 663 referential bridging NPs from the World Street Journal (WSJ) portion of the OntoNotes corpus (Weischedel et al., 2011). The second dataset is called BASHI from Rösiger (2018). It contains 459 bridging NPs<sup>7</sup> with 344 referential anaphors from 50 WSJ texts<sup>8</sup>. Note that bridging anaphors in these two corpora are not limited to definite NPs as in previous work (Poesio et al., 1997, 2004; Lassalle and Denis, 2011) and bridging relations are not limited to the prototypical whole-part relation or set-element relation. We consider these two corpora as expert-annotated in-domain datasets.
We assume that some reasoning skills (e.g., world knowledge, word relatedness) required to answer questions in SQuAD can also be applied for bridging anaphora resolution. Therefore we include the SQuAD 1.1 training data (Rajpurkar et al., 2016) as one training dataset. Another training dataset is the large scale quasi-bridging corpus (QuasiBridging) described in Section 4.
Table 1 summarizes the four datasets mentioned above. Note that in ISNotes and BASHI, the number of QA pairs is more than the number of bridging anaphors. This is because an anaphor can have multiple antecedents (e.g., coreferent mentions of the same antecedent entity).
Following Hou (2018a), we use accuracy on the number of bridging anaphors to measure systems' performance for resolving bridging anaphors on ISNotes and BASHI. It is calculated as the number of the correctly resolved bridging anaphors divided by the total number of bridging anaphors.
We measure two types of accuracy: lenient accuracy and strict accuracy. In strict accuracy, only the original gold antecedent annotations are counted as the correct answers. For lenient accuracy, we add the additional variations of the original antecedent annotations (described in Section 3.3) into the correct answer list. For instance, suppose that the gold antecedent annotation is "the Four Seasons restaurant", and the predicted span is "Four Seasons restaurant", we count this prediction as an incorrect prediction in strict accuracy evaluation. However, it is a correct prediction in lenient accuracy evaluation.
It is worth noting that our lenient accuracy corresponds to the "exact match" metric in SQuAD (Rajpurkar et al., 2016). The correct answer lists that are generated as described in Section 3.3 can partially address the evaluation problem of imperfect system mention predictions. We do not report F1 score because it will give partial credit for a prediction that does not capture the main semantics of the original gold annotation, such as "the Four Seasons".
During evaluation, for every bridging anaphor  $a$ , let  $s_a$  be the sentence containing  $a$ , we use the first sentence of the text, the previous two sentences of  $s_a$ , as well as  $s_a$  to form  $a$ 's surrounding context  $c_a$ . This is in line with Hou (2018a)'s antecedent candidate selection strategy.
5.3 Results on ISNotes and BASHI Using Different Training Strategies