SlowGuess/NAACL_2021_OCR huggingface dataset:
Inspired by this observation, we formulate bridging anaphora resolution as a cloze task. Specifically, given a bridging anaphor and its context, we insert "of [MASK]" after the head word of the anaphor (see Example 1). We then calculate the probability of each candidate to be filled as the mask token. The highest scoring candidate is selected as the predicted antecedent for the anaphor. One of the advantages of our formulation is that we can easily control the scope of the context for each bridging anaphor (e.g., no-context, local context or global context). This allows us to test the effect of different types of context for bridging inference.
(1) Original context: The survey found that over a three-year period  $22\%$  of the firms said employees or owners had been robbed on their way to or from work or while on the job. Seventeen percent reported their customers being robbed.
Cloze test context: The survey found that over a three-year period  $22\%$  of the firms said employees
or owners had been robbed on their way to or from work or while on the job. Seventeen percent of [MASK] reported their customers being robbed.
Recall that in our Of-Cloze test, antecedent candidates are provided and the highest scoring candidate is selected as the predicted antecedent. These candidates are formed by considering mentions which are occurring prior to the anaphor. We design two different experiment sets based on the scope of antecedent candidates and the surrounding context.
Candidates Scope In the first set of experiments, we consider two different sets of antecedent candidates for an anaphor  $a$ . The first set contains salient and nearby mentions as antecedent candidates. Here, mentions only from the first sentence of the document, previous two sentences preceding  $a$  and the sentence containing  $a$  are considered as candidates. This setup follows previous work on selecting antecedent candidates (Hou, 2020a). The second set contains all mentions occurring before the anaphor  $a$  from the whole document. The second setup of forming antecedent candidates is more challenging than the first one because the number of candidates increases which makes selecting the correct antecedent difficult.
Next, we provide the same context for anaphors in both of the experiments described above. We construct the context  $c$  for the bridging anaphor  $a$ . Precisely,  $c$  contains the first sentence of the document, the previous two sentences occurring before  $a$ , as well as the sentence containing  $a$ . We replace the head of  $a$  as "of [MASK].
We also compare this fill-in-the-gap probing approach with the attention heads-based approach for resolving bridging anaphors. Specifically, we use the prominent heads in BERT for identifying bridging relations from Section 4. Here, we obtained attention weights from an anaphor head to all antecedent candidate heads by adding attentions from prominent heads 5:1, 9:12, 11:3, and 12:2-4. Then the highest scoring candidate is predicted as the antecedent for the anaphor.
Context Scope In the second set of experiments, we concentrate on probing the behavior of language models at capturing bridging relations with different contexts. We experiment with the following four settings:
- a. Only anaphor: in this setup, only the
anaphor phrase (with "of [MASK]" being inserted after the anaphor's head word) is given as the input to the model.
- b. Anaphor sentence: the sentence containing the anaphor is provided. The phrase "of [MASK]" is inserted after the head word of the anaphor.
c. Ante+Ana sentence: on top of b, the sentence containing the antecedent is also included in the context.
- d. More context: on top of b, the first sentence from the document as well as the previous two sentences preceding the anaphor are included.
Without "of" Context To test the effect of the strong bridging indicating signal "of", we further execute another set of experiments. Specifically, We remove "of" from "anaphorhead [MASK]" and instead, provide "anaphorhead [MASK]" for each type of the context described above.
Perturbed Context In this setting, we perturb the context by randomly shuffling the words in the context except for the anaphor and antecedent phrases for each type of the context mentioned above. Note that we still have the “of” indicator in this setup.
5.3.1 Results on Candidates Scope
Table 2 shows the accuracy of using only the prominent heads and our Of-Cloze test approach for bridging anaphora resolution. All experiments are based on the same context (i.e., the sentence containing an anaphor, the previous two sentences preceding the anaphor as well as the first sentence from the document).
We find that the Of-Cloze probing approach achieves higher result in comparison to the prominent attention head approach (31.64% vs. 20.15%) under the same conditions. One reason might be that although other attention heads do not significantly attend to bridging relations but cumulatively they are effective.
We also observe that in the Of-Cloze test, the results of using salient/nearby mentions as antecedent candidates are better than choosing antecedents from all previous mentions (Row (2) vs. Row (3), and Row (2) vs. Row (4)). This is because the model has to choose from a smaller number of candidates in the first case as the average number of
Antecedent Candidate Scope	No. Anaphors	BERT-Base	BERT-Large	RoBERTa-Base	RoBERTa-Large
Prominent attention heads
(1) Salient/nearby mentions	531	20.15	-	-	-
Of-Cloze Test
(2) Salient/nearby mentions	531	31.64	33.71	34.08	34.65
(3) All previous mentions	622	26.36	28.78	27.49	29.90
Of-Cloze Test: Anaphors with antecedents in the provided contexts
(4) All previous mentions	531	29.00	30.88	30.32	32.39
Of-Cloze Test: Anaphors with antecedents outside of the provided contexts
(5) All previous mentions	91	10.98	16.48	10.98	15.38
Table 2: Result of selecting antecedents for anaphors with two different probing approaches (Prominent attention heads and Of-Cloze Test) based on the same context. Accuracy is calculated over a different number of anaphors.
Distance	Accuracy
salient*	38.65
0	26.92
1	20.58
2	17.30
>2	10.98
Table 3: Anaphor-antecedent distance-wise accuracy with the BERT-base-cased model. * indicates that the antecedent is in the first sentence of the document.
Context Scope	with “of”	without “of”	perturb
only anaphor	17.20	5.62	-
ana sent.	22.82	7.71	10.28
ana+ante sent.	27.81	9.61	10.93
more context	26.36	12.21	11.41
Table 4: Accuracy of selecting antecedents with different types of context using BERT-of-Cloze Test.
antecedent candidates are only 22 per anaphor as opposed to 148 in the later case.
We further divide 622 anaphors in Row (3) into two groups (Row (4) and Row (5) in Table 2) depending on whether the corresponding antecedents occur in the provided contexts. It can be seen that the performance is significantly better when antecedents occur in the contexts.
Finally, when comparing the results of each language model in each row separately, it seems that the bigger models are always better at capturing bridging information. In general, the RoBERTa-large model performs better than other models except when antecedents do not occur in the provided contexts (Row (5)).
Note that the results in Table 2 are not calculated over all 663 anaphors in ISNotes. Therefore, if the results are normalized over all anaphors then we get the best result with the RoBERTa-large model (28.05%), which is reasonably fine in comparison with the state-of-the-art result of 50.08% (Hou, 2020a) given that the model is not fine-tuned for the bridging task.
5.3.2 Results on Ana-Ante Distance
We further analyze the results of choosing antecedents obtained using the BERT-base-cased model with all previous mentions as the antecedent candidate scope in our Of-Cloze test probing experiment (Row (3) in Table 2) to understand the effect of distance between anaphors and antecedents. The results are shown in Table 3.
In general, it seems that the accuracy decreases as the distance between anaphors and antecedents increases except when antecedents are from the first sentences of the documents. This is related to the position bias in news articles from ISNotes. Normally globally salient entities are often introduced in the beginning of a new article and these entities are preferred as antecedents.
The other reason for the lower results in case of antecedents being away for more than two sentences might be that these antecedents are absent from the provided context.
5.3.3 Results on Different Contexts
The results of experiments with different types of context are shown in Table 4. All experiments are based on the BERT-base-cased model with all pre
vious mentions as the antecedent candidate scope. We refer to this model as BERT-Of-Cloze in the following discussion.
In the first column of the table, BERT-Of-Cloze achieves an accuracy score of  $17.20\%$  with only the anaphor information plus "of [mask]". We can see that the results improve incrementally with the addition of context. More specifically, the accuracy score improves from  $17.20\%$  to  $22.82\%$  by adding sentences containing anaphors. Adding sentences which contain antecedents (ana + ante sent.) further improves the accuracy score to  $27.81\%$ . Finally, adding more local context and the first sentence leads to an accuracy score of  $26.36\%$ . Note that compared to "ana + ante sent.", "more context" represents a more realistic scenario in which we do not assume that the antecedent position information is known beforehand. In general, the results in the first column of Table 4 indicate that the model can leverage context information when predicting antecedents for bridging anaphors.
Results reduce drastically when "of" is removed from the "anaphor of [MASK]" phrase (Table 4, column:2) from all context scopes. Without this indicator, the language model cannot make sense of two adjacent tokens such as "consultant company".
It is interesting to see that the results reduced drastically as well when we perturb the context between the anaphor and antecedent (Table 4, last column). This establishes the importance of meaningful context for performing bridging inference effectively in transformer language models.
5.4 Error Analysis: Of-Cloze test
We analyzed anaphor-antecedent pairs that are linked wrongly by the Of-Cloze formulation and observed some common errors.
Failure at capturing sophisticated commonsense knowledge: We found that the pre-trained transformer language model such as BERT acquires simple common-sense knowledge, therefore it can link anaphor-antecedent pairs such as "sand-dunes" and "principal-school". But it fails at capturing sophisticated knowledge, such as "consultant-Delmed (a company)" and "pool-OPEC (Organization of petroleum countries)". This might be happening because of the rare co-occurrences of these pairs in the original text on which BERT is pre-trained. Also, BERT has inherent limitations at acquiring such structured knowledge (Park et al., 2020).
Language modelling bias: In our Of-Cloze test probing, we use pre-trained transformer language models without fine-tuning. As a result, the model fills masked tokens that are fit according to the language modeling objective, not for bridging resolution. Thus, sometimes, the selected token perfectly makes sense in the single sentence but the choice is incorrect in the broader context. Consider the example, "Only  $22\%$  of [MASK] supported private security patrols [...]". BERT predicts "police" as a suitable antecedent that produces a meaningful local sentence. However, the correct antecedent is "correspondents" according to the surrounding context of this sentence.
Unsuitable formulation for set-relations: Our Of-Cloze formulation produces awkward phrases for some bridging pairs that possess set-relations. Considering a bridging pair – “One man - employees”, in this case the model should assign high score for the phrase – “One man of employees”. But, as this phrase is quite clumsy, BERT naturally being a language model assigns low scores for these pairs.
We investigated the effectiveness of pre-trained transformer language models in capturing bridging relation inference by employing two distinct but complementary approaches.
In the first approach, we probed individual attention heads in BERT and observed that attention heads from higher layers prominently captured bridging compared to the middle and lower layers and some specific attention heads consistently looked for bridging relation. In our second approach, we considered using language models for bridging anaphora resolution by formulating the task as a Of-Cloze test. We carefully designed experiments to test the influence of different types of context for language models to resolve bridging anaphors. Our results indicate that pre-trained transformer language models encode substantial information about bridging.
Finally, in this work, we only focus on understanding the capacity of the pre-trained language models for bridging inference. Based on the insights we gained from the current probing study, in the future, we plan to explore how to better use pretrained transformer language models for bridging resolution.
We thank the three anonymous reviewers for their comments and feedback. This work was partially supported by the French National Research Agency via grant no ANR-16-CE33-0011-01 as well as by CPER Nord-Pas de Calais/FEDER DATA Advanced data science and technologies 2015-2020.
Nicholas Asher and Alex Lascarides. 1998. Bridging. Journal of Semantics, 15:83-113.
Johan Bos, Paul Buitelaar, and Anne Marie Mineur. 1995. Bridging as coercive accomodation. In Working Notes of the Edinburgh Conference on Computational Logic and Natural Language Processing (CLNLP-95), Human Communications Research Centre, University of Edinburgh, Edinburgh, U.K.
Herbert H. Clark. 1975. Bridging. In Proceedings of the Conference on Theoretical Issues in Natural Language Processing, Cambridge, Mass., June 1975, pages 169-174.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy. Association for Computational Linguistics.
Jeff Da and Jungo Kasai. 2019. Cracking the contextual commonsense code: Understanding commonsense reasoning aptitude of deep contextual representations. In Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, pages 1-12, Hong Kong, China. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yoav Goldberg. 2019. Assessing BERT's syntactic abilities. ArXiv, abs/1901.05287.
Yufang Hou. 2018a. A deterministic algorithm for bridging anaphora resolution. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1938-1948, Brussels, Belgium. Association for Computational Linguistics.
Yufang Hou. 2018b. Enhanced word representations for bridging anaphora resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 1-7, New Orleans, Louisiana. Association for Computational Linguistics.
Yufang Hou. 2020a. Bridging anaphora resolution as question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1428-1438, Online. Association for Computational Linguistics.
Yufang Hou. 2020b. Fine-grained information status classification using discourse context-aware BERT. In Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain. Association for Computational Linguistics.
Yufang Hou, Katja Markert, and Michael Strube. 2013. Global inference for bridging anaphora resolution. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 907-917, Atlanta, Georgia. Association for Computational Linguistics.
Phu Mon Htut, Jason Phang, Shikha Bordia, and Samuel R. Bowman. 2019. Do attention heads in bert track syntactic dependencies? ArXiv, abs/1911.12246.
Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543-3556, Minneapolis, Minnesota. Association for Computational Linguistics.
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651-3657, Florence, Italy. Association for Computational Linguistics.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4365-4374, Hong Kong, China. Association for Computational Linguistics.
Katja Markert, Yufang Hou, and Michael Strube. 2012. Collective classification for fine-grained information status. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 795-804, Jeju Island, Korea. Association for Computational Linguistics.
S. Park, J. Son, S. w. Hwang, and K. Park. 2020. Bert is not all you need for commonsense inference. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8204-8208.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
Ina Roesiger, Arndt Riester, and Jonas Kuhn. 2018. Bridging resolution: Task definition, corpus resources and rule-based experiments. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3516-3528, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Ionut-Teodor Sorodoc, Kristina Gulordava, and Gemma Boleda. 2020. Probing for referential information in language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4177-4189, Online. Association for Computational Linguistics.
Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020. oLMPics - on what language model pre-training captures. Transactions of the Association for Computational Linguistics, 8.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.