Venue: NAACL 2021

# Evaluating the Impact of a Hierarchical Discourse Representation on

Entity Coreference Resolution Performance

Sopan Khosla  James Fiacco  Carolyn Rose

Language Technologies Institute

Carnegie Mellon University, USA

{sopank,jfiacco,cprose}@cs.cmu.edu

###### Abstract

Recent work on entity coreference resolution (CR) follows current trends in Deep Learning applied to embeddings and relatively simple task-related features. SOTA models do not make use of hierarchical representations of discourse structure. In this work, we leverage automatically constructed discourse parse trees within a neural approach and demonstrate a significant improvement on two benchmark entity coreference-resolution datasets. We explore how the impact varies depending upon the type of mention.

+
Footnote †: _Corresponding of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1645–1651

June 6-11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Historically, theories of discourse coherence [9, 38, 7] have offered elaborate expositions on how the patterns of anaphoric references in discourse are constrained by limitations in human capacity to manage attention and resolve ambiguity. hobbs2019multitask has been taken that these human limitations have meant that coreference resolution in natural text can be achieved with relatively high accuracy using a combination of recency and simple semantic constraints. State-of-the-art neural approaches for coreference resolution [10, 20, 11] have therefore not surprisingly shown strong performance relying on surface-level features and local-context (i.e., extracted from a small text window around the mention). Traditional approaches, on the other hand, make an attempt to formally model the process of managing attention, for example, the stack in grosz2018learning's model. Their stack-based model suggests specific places where recency might fail while a more explicit model of discourse structure might make a correct prediction, for example, where an anaphor and a nearby potential (but incorrect) antecedent are in adjacent but separate discourse segments. Because of the potential existence of such cases, we hypothesize that formally incorporating a representation of discourse structure would have a small but non-random positive impact on the ability to correctly resolve anaphoric references. This effect might vary depending upon the semantic informativeness of alternative types of anaphoric expressions, since they impose different constraints on where their antecedent can be located within a hierarchical discourse structure. There is also a danger that the level of accuracy with which the hierarchical structure of discourse can be obtained in practice might reduce the positive impact still further.

The contribution of this paper is an empirical investigation of the impact of including a representation of the hierarchical structure of discourse within a neural entity coreference approach. To this end, we leverage a state-of-the-art RST discourse-parser to convert a flat document into a tree-like structure from which we can derive features that model the structural constraints. We embed this representation within an architecture that is enabled to learn to use this information deferentially depending upon the type of mention. The results demonstrate that this level of nuance enables a small but significant improvement in coreference accuracy, even with automatically constructed RST trees.

## 2 Related Work

Though recency is the strongest predictor for coreference resolution (CR), prior work in CR has benefited from the inclusion of semantic features such as type-information on top of the surface and syntax-level features. soon2001natural; bengtsson2008natural used dictionaries like WordNet to extract the semantic class for a noun. More recently, khosla2020multitask showed that adding NER style type-information to lee2017neural substantially improves performance across multiple datasets.

Discourse-level features have been successfully

employed in multiple downstream NLP tasks like summarization Louis et al. (2010), sentiment analysis Somasundaran et al. (2009), and student writing evaluation Burstein et al. (2013). For coreference resolution, Cristea et al. (1999) showed that the potential of natural language systems to correctly determine co-referential links, can be increased by exploiting the hierarchical structure of texts. Their discourse model was informed by Vein Theory Fox (1987), which identifies chains of elementary discourse units, over discourse structure trees that are built according to the RST Mann and Thompson (1987) requirements. Haghighi and Klein (2010) proposed an entity-centered model that leveraged discourse features like dependency-parse tree distance, sentence distance, and the syntactic positions (subject, object, and oblique) of the mention and antecedent to perform coreference.

In this work, we use Yu et al. (2018)'s RST parser to convert documents into RST discourse-structure trees Mann and Thompson (1987); Taboada and Mann (2006). From these trees, we derive distance and coverage-based features to model the discourse-level structural constraints, which are passed as input to a neural-network based coreference resolver. To our knowledge, ours is the first work that tries to explicitly incorporate discourse-level constraints for coreference resolution in a neural setting.

## 3 Model

In this section, we explain how we introduce discourse-level features into a neural CR system.

### Baseline

We leverage Lee et al. (2017) as our baseline. We replace the word-embeddings with a BERT encoder. A preprocessing step for CR is to identify the mentions within the text that need to be resolved. Following Bamman et al. (2020) and Khosla and Rose (2020), we remove this possible source of error from our evaluation of entity coreference accuracy by using gold-standard mentions.

The baseline model's prediction of coreference for a pair of mentions, \({\bf S}\left(m_{i},m_{j}\right)\), is computed as follows. The representations of the two mentions \(m_{i}\) and \(m_{j}\) along with their element-wise product (\(m_{i}\odot m_{j}\)) and other features like distance between the mentions (\(d_{m}\)), and distance between the sentences that contain the mentions (\(d_{s}\)), are joined together and passed through a fully-connected layer **F** (blue boxes in Figure 1).

\[mm_{ij}=[m_{i};m_{j};m_{i}\odot m_{j};d_{m};d_{s};...]\] \[{\bf S}\left(m_{i},m_{j}\right)={\bf F}\left(mm_{ij}\right)\]

### Incorporating Discourse-level Features

By incorporating a representation of the hierarchical discourse structure into the representation that is input to the neural model, we seek to add the capability for reasoning that is not possible in the baseline for each mention-pair (\(mm_{ij}\)). None of the features included in the baseline distinguish between pairs that occur within the same or different discourse segments, for example. The closest feature in the baseline that approximates document-level relationships is \(d_{s}\), since it can be assumed that mentions are less likely to occur within the

Figure 1: Schematic diagram of our discourse-informed neural architecture. Discourse (yellow) and mention-type features (green) are concatenated with baseline features (blue) to obtain the mention-pair representation for scoring.

 

same segment the further apart they are in the discourse. But this is not universally true.

RST (Mann and Thompson, 1987) offers a theoretical framework in which documents can be parsed into trees that capture the hierarchical discourse structure of the text. In this work, we incorporate structural features from such discourse trees, obtained automatically from Yu et al. (2018). We concatenate three structural features, extracted from the discourse-tree of the document, with \(mm_{ij}\) to model these constraints (as shown in Figure 1). We use binarized RST-trees to represent the discourse hierarchy and relationships within each document. Discourse-units identified by the parser occur at the leaves (\(l\)) of the output tree.

Consider the document under consideration \(doc\) and its RST-tree \(t_{doc}\). For the current mention \(m_{j}\) and candidate mention \(m_{i}\), and the position of the smallest discourse-unit they belong to in the tree (\(l_{u_{m_{j}}}\) and \(l_{u_{m_{i}}}\) respectively):

**DistLCA (d\({}^{1}_{\textbf{ICA}}\))** encodes the distance between \(l_{u_{m_{j}}}\) and \(LCA(l_{u_{m_{i}}},l_{u_{m_{j}}})\). This feature provides information about the amount of generality required to have the two mentions in the same discourse subtree. The smaller the _DistLCA_, the closer the two mentions are assumed to be in the discourse.

**LeafCoverageLCA (lc\({}_{\textbf{ica}}\))** encodes the number of sentences that are covered by the discourse sub-tree with \(LCA(l_{u_{m_{i}}},l_{u_{m_{j}}})\) as its root. This feature captures the coverage of the level of discourse that encloses both mentions. The larger the _LeafCoverageLCA_, the more the document area that needs to be covered to include both mentions.

**WordCoverageLCA (wc\({}_{\textbf{ica}}\))** encodes the number of words that are covered by the discourse subtree with \(LCA(l_{u_{m_{i}}},l_{u_{m_{j}}})\) as its root. This feature is analogous to _LeafCoverageLCA_ but operates on word-level rather than the discourse-unit-level.

### Mention Types and Cognitive Load

Across different types of anaphoric mentions, depending upon how much information about the antecedent is made apparent, there are differences with respect to the cognitive load imposed on the reader. Because this places differential constraints on the interpretation process, we hypothesize that enabling the model to learn different strategies depending upon the mention-type will be advantageous. We divide mentions into three types (\(type\)) motivated by the above-mentioned intuition: (i) pronouns (low lexical information, high cognitive load on the reader), (ii) named-entities (already grounded mentions), and (iii) all other noun phrases. A mention is put in the second category if it contains at least one named-entity as predicted by an off-the-shelf NER system.1 To identify pronouns, we compare the mention against a manually curated list of English pronouns. Ultimately, the discourse and mention-type features are concatenated with \(mm_{ij}\) and passed through a fully-connected layer for scoring (Figure 1).

Footnote 1: [https://demo.allennlp.org/named-entity-recognition](https://demo.allennlp.org/named-entity-recognition)

## 4 Experimental Setup

In this section, we describe the datasets and evaluation metrics we use in our experiments.

### Datasets

We gauge the benefits of using RST-tree features on two state-of-the-art entity CR datasets discussed below. Since, our off-the-shelf RST parser (Yu et al., 2018) is trained on news articles, the choice of datasets is motivated by the attempt at reducing the distribution shift between training and inference while ensuring that the parser was trained on different data than we are using for testing. We use the English subset of **OntoNotes**(Pradhan et al., 2012). The corpus contains multiple sub-genres ranging from news articles to telephone conversations. We also evaluate our approach on a subset of the RST sub-genre of the ARRAU corpus (Poesio et al., 2018) (**A-RST(gt)**), which contains RST ground-truth parse-tree annotations in the RST Discourse-Treebank (Carlson et al., 2003). Following Yu et al. (2018), we keep 347 A-RST(gt) articles for training (out of which we set aside 22 articles for development), and 38 articles for testing. Although ARRAU also annotates bridging (Clark, 1975) and abstract anaphora (Webber, 1991), in this work, we only focus on entity anaphora.

### Evaluation Metrics

Both OntoNotes and A-RST(gt) are input to the system in the CoNLL 2012 format. We evaluate the systems on the F1-score for MUC, B3, and CEAF metrics using the CoNLL-2012 official scripts. However, we only show the average F1-score of the above-mentioned metrics in this 

paper for brevity. We report the mean score of 5 independent runs with different seeds.3

Footnote 2: We leave the evaluation of the impact of including RST structural features in the end-to-end CR setting as future work.

Footnote 3: Refer to Appendix A for hyperparameter values and Appendix B for detailed results.

## 5 Results

**Ground-truth RST-Trees:** To establish an upperbound for the improvement through introduction of the discourse-tree features, we use features extracted from ground-truth trees. We evaluate the upper-bound performance on A-RST(gt) as it contains documents with annotations for coreference as well as RST-structures. Our results show that incorporating ground-truth tree features along with the mention's type (**+ disc(gt) + type**) gives a boost of \(0.90\) Avg. F1 (\(p<0.01\)) over the baseline (Table 1), suggesting that discourse-level features are beneficial on A-RST(gt). Furthermore, we also find that removing \(d_{s}\) from this discourse-informed model does not cause a statistically-significant drop in performance. We believe that this happens because when discourse-structure features are included in the model, the signal from \(d_{s}\) becomes redundant and sub-optimal.

**Predicted RST-Trees:** In our second set of experiments we use discourse-trees extracted using Yu et al. (2018)'s RST-parser. As shown in Table 1, adding predicted discourse-tree features improves over the baseline on both datasets, with A-RST(gt) corpus witnessing the highest absolute gain of \(0.71\) Avg. F1 points. Please note that the results are statistically significant with \(p<0.01\).4 The relative improvement on OntoNotes is smaller than A-RST(gt) (\(0.53\) absolute Avg. F1 points). This could partially be explained by the fact that the RST-parser is trained on news articles, and therefore, might not generalize well on conversational sub-genres of OntoNotes like tc or bc.

Footnote 4: We performed a one-tailed t-test to evaluate significance.

**Ablation Study:** To evaluate the contribution of each feature separately, we also perform an ablation study (Table 1). On A-RST(gt), we find that the \(type\) feature by itself does not provide a considerable boost over the baseline. Use of RST-tree based structural features, on the other hand, shows statistically significant improvements (\(p<0.01\)), however, the jump is small (from \(85.80\) to \(86.19\)). Our final model which includes both RT-tree features and \(type\) gives the best results. **+ disc + type** performs much better than **+ disc** on both datasets (improvement of \(0.32\) Avg. F1 points on A-RST(gt) and \(0.26\) points on Onto) suggesting that the use of \(type\) as a feature enhances the discriminative power of discourse-tree features.

**Mention Type Analysis:** To study the influence of different mention-types on the discriminative power of discourse features, we analyze the distribution of \(d_{lca}\) across different mention-pair categories in the A-RST(gt) training set.

_Setup._ To this end, we firstly extract relevant coreferent mention-pairs from the ground-truth clusters. To create a pair for each mention \(m_{j}\), we choose the mention \(m_{i}\) that belongs to the same cluster (\(C\)) as \(m_{j}\), occurs before it in the document (\(i<j\)), and is the closest instance of \(C\) to \(m_{j}\). Pairs created using this algorithm do not have other supporting mentions from the same cluster in between them. We then extract three types of mention-pairs from these relevant pairs for our analysis: (i) \(m_{j}\) is a pronoun and \(m_{i}\) is not a pronoun (**PRP-N**); (ii) \(m_{j}\) contains a named entity and \(m_{i}\) is not a pronoun (**NE-N**); and (iii) \(m_{j}\) is neither a pronoun

Figure 2: Distribution of \(d_{lca}\) for the three different categories of anaphoric mention-pairs.

 

nor contains a named entity, \(m_{i}\) is not a pronoun, and \(m_{i},m_{j}\) have no lexical overlap (**NP-N**).

ResultsFigure 2 shows that there is indeed a dependence between \(d_{lca}\) and mention-pair type. Most of the PRP-N pairs have a \(d_{lca}<5\) even though the the full RST-tree of a document can be as deep as \(24\) levels. This corroborates our intuition that anaphors with higher ambiguity occur closer to their antecedents in the discourse. For NP-N, we find that \(90\%\) of the pairs have \(d_{lca}<8\), whereas, \(d_{lca}\) can go as large as \(10\) for NE-N. This trend explains, at least partially, the difference between the performance of discourse-informed models with and without the mention-type feature.

## 6 Conclusion

In this paper, we show that a representation of hierarchical discourse structure is beneficial for entity coreference resolution. Our proposed discourse-informed model observes small but statistically significant improvements over a state-of-the-art neural baseline on two coreference resolution datasets. Our analysis shows that the impact of the representation on performance is related to the cognitive load imposed by the type of anaphoric mention.

While the model proposed in this work could serve as a useful baseline for the benefits of including discourse structure-based features in neural coreference resolution models, we realize that there is potential for achieving additional improvements by including more complex constraints (e.g. Right Frontier Constraint Asher et al. (2003)). We plan to study the affect of such features in future work.

## Acknowledgements

We thank the anonymous NAACL reviewers for their insightful comments. We are also grateful to the members of the TELEDIA group at LTI, CMU for the invaluable feedback. This work was funded in part by NSF grants 1949110, 1822831, 1546393, and funding from Dow Chemical.

## References

* A. Asher, N. M. Asher, and A. Lacarides (2003)Logics of conversation. Cambridge University Press. Cited by: SS1.
* D. Bamman, O. Lewke, and A. Mansoor (2020)An annotated dataset of coreference in english literature. In Proceedings of The 12th Language Resources and Evaluation Conference, pp. 44-54. Cited by: SS1.
* E. Bengtson and D. Roth (2008)Understanding the value of features for coreference resolution. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pp. 294-303. Cited by: SS1.
* J. Burstein, J. Tetreault, and M. Chodorow (2013)Holistic discourse coherence annotation for noisy essay writing. Dialogue & Discourse4 (2), pp. 34-52. External Links: Document, Link Cited by: SS1.
* L. Carlson, D. Marcu, and M. Ellen Okurowski (2003)Building a discourse-tagged corpus in the framework of rhetorical structure theory. In Current and new directions in discourse and dialogue, pp. 85-112. Cited by: SS1.
* W. Chafe (1976)Givenness, contrastiveness, definiteness, subjects, topics, and point of view. Subject and topic. Cited by: SS1.
* H. H. Clark (1975)Bridging. In Proceedings of TINLAP, Cited by: SS1.
* H. H. Clark and S. E. Brennan (1991)Grounding in communication. Perspectives on socially shared cognition. Cited by: SS1.
* D. Cristea, N. Ide, D. Marcu, and V. Talblan (1999)Discourse structure and co-reference: an empirical study. In The relation of discourse/dialogue structure and reference, Cited by: SS1.
* B. A. Fox (1987)Discourse Structure and Anaphora: Written and Conversational English. Cambridge Studies in Linguistics, Cambridge University Press. Cited by: SS1.
* Barbara Grosz and C. L Sidner (1986)Attention, intentions, and the structure of discourse. Computational linguistics. Cited by: SS1.
* A. Haghighi and D. Klein (2010)Coreference resolution in a modular, entity-centered model. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 385-393. Cited by: SS1.
* J. R. Hobbs (1979)Coherence and coreference. Cognitive science3 (1), pp. 67-90. Cited by: SS1.
* M. Joshi, D. Chen, Y. Liu, D. S. Weld, L. Zettlemoyer, and O. Levy (2020)Spanbert: improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics8, pp. 64-77. Cited by: SS1.
* M. Joshi, O. Levy, L. Zettlemoyer, and D. S. Weld (2019)BERT for coreference resolution: baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5807-5812.

 

* Khosla and Rose (2020) Sopan Khosla and Carolyn Rose. 2020. Using type information to improve entity coreference resolution. In _Proceedings of the First Workshop on Computational Approaches to Discourse_, pages 20-31.
* Lee et al. (2017) Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 188-197.
* Louis et al. (2010) Annie Louis, Aravind Joshi, and Ani Nenkova. 2010. Discourse indicators for content selection in summarization. In _Proceedings of the SIGDIAL 2010 Conference_, pages 147-156.
* Mann and Thompson (1987) William C Mann and Sandra A Thompson. 1987. _Rhetorical structure theory: A theory of text organization_. University of Southern California, Information Sciences Institute Los Angeles.
* Poesio et al. (2018) Massimo Poesio, Yulia Grishina, Varada Kolhatkar, Nafise Sadat Moosavi, Ina Roesiger, Adam Roussel, Fabian Simonjetz, Alexandra Uma, Olga Urupina, Juntao Yu, et al. 2018. Anaphora resolution with the arrau corpus. In _Proceedings of the First Workshop on Computational Models of Reference, Anaphora and Coreference_, pages 11-22.
* Shared Task_, CoNLL '12, page 1-40, USA. Association for Computational Linguistics.
* Somasundaran et al. (2009) Swapna Somasundaran, Galileo Namata, Janyce Wiebe, and Lise Getoor. 2009. Supervised and unsupervised methods in employing discourse relations for improving opinion polarity classification. In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing_, pages 170-179.
* Soon et al. (2001) Wee Meng Soon, Hwee Tou Ng, and Daniel Chung Yong Lim. 2001. A machine learning approach to coreference resolution of noun phrases. _Computational linguistics_, 27(4):521-544.
* Taboada and Mann (2006) Maite Taboada and William C Mann. 2006. Applications of rhetorical structure theory. _Discourse studies_, 8(4):567-588.
* Webber (1991) Bonnie Lynn Webber. 1991. Structure and ostension in the interpretation of discourse deixis. _Language and Cognitive processes_, 6(2):107-135.
* Yu et al. (2018) Nan Yu, Meishan Zhang, and Guohong Fu. 2018. Transition-based neural rst parsing with implicit syntax features. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 559-570.

## Appendix A Hyperparameters

## Appendix B Detailed Results

## Appendix C Results on Validation Set

## Appendix D Computational Infrastructure

All our experiments are performed on a single Nvidia GeForce GTX 1080 Ti GPU. Training took

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Model** & **OntoNotes** & **A-RST(gt)** \\ \hline
**Lee et al. (2017)** & 83.42 & 85.98 \\
**+ type** & 84.01 & 86.15 \\
**+ disc** & 83.91 & 86.40 \\
**+ disc** + type** & **84.38** & **86.74** \\ \hline
* & 86.68 \\
* & **87.02** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance (Avg. F1) of discourse-informed model variants (gold-mentions) on OntoNotes and A-RST(gt) validation set.

\begin{table}
\begin{tabular}{l r} \hline \hline
**Hyperparameter** & **Value** \\ \hline BERT & base-cased \\ BERT weights & freeze \\ BiLSTM hidden dim & 200 \\ \(d_{ca}\) embedding-size & 20 \\ \(loc_{ca}\) embedding-size & 20 \\ \(wcl_{ca}\) embedding-size & 20 \\ \(type\) embedding-size & 20 \\ FC-layer 1 size & 150 \\ FC-layer 2 size & 150 \\ Dropout & 0.2 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Hyperparameter values for our model. Reader is referred to [https://github.com/dbamman/lrec2020](https://github.com/dbamman/lrec2020)–coref for the implementation of the baseline model.

\begin{table}
\begin{tabular}{l|r r|r r r} \hline \hline
**Model** & \multicolumn{3}{c|}{**OntoNotes**} & \multicolumn{3}{c}{**A-RST(gt)**} \\ \hline  & **MUC** & **B\({}^{3}\)****CEAF** & **MUC** & **B\({}^{3}\)****CEAF** \\ \hline
**Lee et al. (2017)** & 90.8 & 82.3 & 77.1 & 79.0 & 89.0 & 89.3 \\
**+ type** & 91.1 & 82.7 & 77.3 & 79.3 & 89.0 & 89.4 \\
**+ disc** & 91.0 & 82.5 & 77.4 & 79.5 & 89.2 & 89.7 \\

- & 79.4 & 88.5 & 89.3 \\ \(\cdot d_{s}\) **+ disc(gt) + type** &
- & 80.2 & 89.6 & 90.2 \\ \hline

- & 79.8 & 89.5 & 89.9 \\

- & 80.2 & 89.8 & 90.1 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Detailed Performance (F1 score) of discourse-informed model variants (gold-mentions) on OntoNotes and A-RST(gt).
 

20-22 hours on OntoNotes, and 5-7 hours on A-RST(gt). We trained the models for 100 epochs with an early-stopping criteria on the Avg. F1 performance on the validation set.