Venue: NAACL 2022

# Towards Understanding Large-Scale Discourse Structures in Pre-Trained

and Fine-Tuned Language Models

Patrick Huber and Giuseppe Carenini

Department of Computer Science

University of British Columbia

Vancouver, BC, Canada, V6T 1Z4

{huberpat, carenini}@cs.ubc.ca

###### Abstract

With a growing number of BERTology works analyzing different components of pre-trained language models, we extend this line of research through an in-depth analysis of discourse information in pre-trained and fine-tuned language models. We move beyond prior work along three dimensions: First, we describe a novel approach to infer discourse structures from arbitrarily long documents. Second, we propose a new type of analysis to explore where and how accurately intrinsic discourse is captured in the BERT and BART models. Finally, we assess how similar the generated structures are to a variety of baselines as well as their distributions within and between models.

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2376 - 2394

## 1 Introduction

Transformer-based machine learning models are an integral part of many recent improvements in Natural Language Processing (NLP). With their rise spearheaded by Vaswani et al. (2017), the pre-training/fine-tuning paradigm has gradually replaced previous approaches based on architecture engineering, with transformer models such as BERT Devlin et al. (2019), BART Lewis et al. (2020), RoBERTa Liu et al. (2019) and others delivering state-of-the-art performance on a wide variety of tasks. Besides their strong empirical results on most real-world problems, such as summarization Zhang et al. (2020); Xiao et al. (2021), question-answering Joshi et al. (2020); Oguz et al. (2021) and sentiment analysis Adhikari et al. (2019); Yang et al. (2019), uncovering what kind of linguistic knowledge is captured by this new type of pretrained language models (PLMs) has become a prominent question by itself. As part of this line of research, called _BERTology_Rogers et al. (2020), researchers explore the amount of linguistic understanding encapsulated in PLMs, exposed through either external probing tasks Raganato and Tiedemann (2018); Zhu et al. (2020); Koto et al. (2021) or unsupervised methods Wu et al. (2020); Pandia et al. (2021). Previous work thereby either focuses on analyzing the syntactic structures (e.g., Hewitt and Manning (2019); Wu et al. (2020)), relations Papanikolaou et al. (2019), ontologies Michael et al. (2020) or, to a more limited extend, discourse related behaviour Zhu et al. (2020); Koto et al. (2021); Pandia et al. (2021).

Generally speaking, while most previous _BERTology_ works has focused on either sentence level phenomena or connections between adjacent sentences, large-scale semantic and pragmatic structures (oftentimes represented as discourse trees or graphs) have been less explored. These structures (e.g., discourse trees) play a fundamental role in expressing the intent of multi-sentential documents and, not surprisingly, have been shown to benefit many NLP tasks such as summarization Gerani et al. (2019), sentiment analysis Bhatia et al. (2015); Nejat et al. (2017); Hogenboom et al. (2015) and text classification Ji and Smith (2017).

With multiple different theories for discourse proposed in the past, the RST discourse theory Mann and Thompson (1988) and the lexicalized discourse grammar Webber et al. (2003) (underlying PDTB Prasad et al. (2008)) have received most attention. While both theories propose tree-like structures, the PDTB framework postulates partial trees up to the between-sentence level, while RST-style discourse structures consist of a single rooted tree covering whole documents, comprising of: (1) The tree structure, combining clause-like sentence fragments (Elementary Discourse Units, short: EDUs) into a discourse constituency tree, (2) Nuclearity, assigning every tree-branch primary (_Nucleus_) or peripheral (_Satellite_) importance in a local context and (3) Relations, defining the type of connection holding between siblings in the tree.

Given the importance of large-scale discourse structures, we extend the area of _BERtology_ research with novel insights regarding the amount of

intrinsic discourse information captured in established PLMs. More specifically, we aim to better understand to what extend RST-style discourse information is stored as latent trees in encoder self-attention matrices1. While we focus on the RST formalism in this work, our presented methods are theory-agnostic and, hence, applicable to discourse structures in a broader sense, including other tree-based theories, such as the lexicalized discourse grammar. Our contributions in this paper are:

Footnote 1: Please note that we focus on discourse structure and nuclearity here, leaving relation classification for future work.

**(1)** A novel approach to extract discourse information from arbitrarily long documents with standard transformer models, inherently limited by their input size. This is a non-trivial issue, which has been mostly by-passed in previous work through the use of proxy tasks like connective prediction, relation classification, sentence ordering, EDU segmentation, cloze story tests and others.

**(2)** An exploration of discourse information locality across pre-trained and fine-tuned language models, finding that discourse structures are consistently captured in a fixed subset of self-attention heads.

**(3)** An in-depth analysis of the discourse quality in pre-trained language models and their fine-tuned extensions. We compare constituency and dependency structures of 2 PLMs fine-tuned on 4 tasks and 7 fine-tuning datasets to gold-standard discourse trees, finding that the captured discourse structures outperform simple baselines by a large margin, even showing superior performance compared to distantly supervised models.

**(4)** A similarity analysis between PLM inferred discourse trees and supervised, distantly supervised and simple baselines. We reveal that PLM constituency discourse trees do align relatively well with previously proposed supervised models, but also capture complementary information.

**(5)** A detailed look at information redundancy in self-attention heads to better understand the structural overlap between self-attention matrices and models. Our results indicate that similar discourse information is consistently captured in the same heads, even across fine-tuning tasks.

## 2 Related Work

At the base of our work are two of the most popular and frequently used PLMs: BERT Devlin et al. (2019) and BART Lewis et al. (2020). We choose these two popular approaches in our study due to their complementary nature (encoder-only vs. encoder-decoder) and based on previous work by Zhu et al. (2020) and Koto et al. (2021), showing the effectiveness of BERT and BART models for discourse related tasks.

Our work is further related to the field of discourse parsing. With a rich history of traditional machine learning models (e.g., Hernault et al. (2010); Ji and Eisenstein (2014); Joty et al. (2015); Wang et al. (2017)_inter alia_), recent approaches slowly shifted to successfully incorporate a variety of PLMs into the process of discourse prediction, such as ELMo embeddings Kobayashi et al. (2019), XLNet Nguyen et al. (2021), BERT Koto et al. (2021), RoBERTa Guz et al. (2020) and SpanBERT Guz and Carenini (2020). Despite these works showing the usefulness of PLMs for discourse parsing, all of them cast the task into a "local" problem, using only partial information through the shift-reduce framework Guz et al. (2020); Guz and Carenini (2020), natural document breaks (e.g. paragraphs Kobayashi et al. (2020)) or by framing the task as an inter-EDU sequence labelling problem on partial documents Koto et al. (2021). However, we believe that the true benefit of discourse information emerges when complete documents are considered, leading us to propose a new approach to connect PLMs and discourse structures in a "global" manner, superseding the local proxy-tasks with a new methodology to explore arbitrarily long documents.

Aiming to better understand what information is captured in PLMs, the line of _BERTology_ research has recently emerged Rogers et al. (2020), with early work mostly focusing on the syntactic capacity of PLMs Hewitt and Manning (2019); Jawahar et al. (2019); Kim et al. (2020), in parts also exploring the internal workings of transformer-based models (e.g., self-attention matrices Raganato and Tiedemann (2018); Marecek and Rosa (2019)). More recent work started to explore the alignment of PLMs with discourse information, encoding semantic and pragmatic knowledge. Along those lines, Wu et al. (2020) present a parameter-free probing task for both, syntax and discourse. With their tree inference approach being computationally expensive and limited to the exploration of the outputs of the BERT model, we significantly extend this line of research by exploring the internal self-attention matrices of PLMs with a more computationally feasible approach. More tradi 

tionally, Zhu et al. (2020) use \(24\) hand-crafted rhetorical features to execute three different supervised probing tasks, showing promising performance of the BERT model. Similarly, Pandia et al. (2021) aim to infer pragmatics through the prediction of discourse connectives by analyzing the model inputs and outputs and Koto et al. (2021) analyze discourse in seven PLMs through seven supervised probing tasks, finding that BART and BERT contain most information related to discourse. In contrast to the approach taken by both Zhu et al. (2020) and Koto et al. (2021), we use an unsupervised methodology to test the amount of discourse information stored in PLMs (which can also conveniently be used to infer discourse structures for new and unseen documents) and extend the work by pandia2021learning by taking a closer look at the internal workings of the self-attention component. Looking at prior work analyzing the amount of discourse information in PLMs, structures are solely explored through the use of proxy tasks, such as connective prediction [35], relation classification [67], and others [98]. However, despite the difficulties of encoding arbitrarily long documents, we believe that to systematically explore the relationship between PLMs and discourse, considering complete documents is imperative. Along these lines, recent work started to tackle the inherent input-length limitation of general transformer models through additional recurrence in the Transformer-XL model [66], compression modules [14] or sparse patterns (e.g., as in the Reformer [13], BigBird [15], and Longformer [12] models). While all these approaches to extend the maximum document length of transformer-based models are important to create more globally inspired models, the document-length limitation is still practically and theoretically in place, with models being limited to a fixed number of pre-defined tokens the model can process. Furthermore, with many proposed systems still based on more established PLMs (e.g., BERT) and with no single dominant solution for the general problem of the input length-limitation yet, we believe that even with the restriction being actively tackled, an in-depth analysis of traditional PLMs with discourse is highly valuable to establish a solid understanding of the amount of semantic and pragmatic information captured.

Besides the described _BERTology_ work, we got encouraged to explore fine-tuned extensions of standard PLMs through previous work showing the benefit of discourse parsing for many downstream tasks, such as summarization [10, 12, 13] and text classification [14]. Conversely, we recently showed promising results when inferring discourse structures from related downstream tasks, such as sentiment analysis [15] and summarization [16]. Given this bidirectional synergy between discourse and the mentioned downstream tasks, we move beyond traditional experiments focusing on standard PLMs and additionally explore discourse structures of PLMs fine-tuned on a variety of auxiliary tasks.

## 3 Discourse Extraction Method

With PLMs rather well analyzed according to their syntactic capabilities, large-scale discourse structures have been less explored. One reason for this is the input length constraint of transformer models. While this is generally not prohibitive for intra

Figure 1: Small-scale example of the discourse extraction approach. Purple=EDUs, green=sub-word embeddings, red=input slices of size \(t_{max}\), orange=PLM, blue=self-attention values, grey-scale=frequency count.

 

sentence syntactic structures (e.g., presented in Wu et al. (2020)), it does heavily influence large-scale discourse structures, operating on complete (potentially long) documents. Overcoming this limitation is non-trivial, since traditional transformer-based models only allow for fixed, short inputs.

Aiming to systematically explore the ability of PLMs to capture discourse, we investigate a novel way to effectively extract discourse structures from the self-attention component of the BERT and BART models. We thereby extend our previously proposed tree-generation methodology Xiao et al. (2021) to support the input length constraints of standard PLMs using a sliding-window approach in combination with matrix frequency normalization and an EDU aggregation method. Figure 1 visualizes the complete process on a small scale example with 3 EDUs and 7 sub-word embeddings.

The Tree Generation Procedurewe previously proposed in Xiao et al. (2021) explores a two-stage approach to obtain discourse structures from a transformer model, by-passing the input-length constraint. Using the intuition that the self-attention score between any two EDUs is an indicator of their semantic/pragmatic relatedness, influencing their distance in a projective discourse tree, they use the CKY dynamic programming approach Jurafsky and Martin (2014) to generate constituency trees based on the internal self-attention of the transformer model. To generate dependency trees, we apply the same intuition used to infer discourse trees with the Eisner algorithm Eisner (1996). Since we explore the discourse information captured in standard PLMs, we can't directly transfer our two-stage approach in Xiao et al. (2021), first encoding individual EDUs using BERT and subsequently feeding the dense representations into a fixed-size transformer model. Instead, we propose a new method to overcome the length-limitation of the transformer model2.

Footnote 2: For more information on the general tree-generation approach using the Eisner algorithm we refer interested readers to Xiao et al. (2021).

The Sliding-Window Approachist at the core of our new methodology to overcome the input-length constraint. We first tokenize arbitrarily long documents with \(n\) EDUs \(E=\{e_{1},...,e_{n}\}\) into the respective sequence of \(m\) sub-word tokens \(T=\{t_{1},...t_{m}\}\) with \(n\ll m\), according to the PLM tokenization method WordPiece for BERT, Byte-Pair-Encoding for BART), as show at the top of Figure 1. Using the sliding window approach, we subdivide the \(m\) sub-word tokens into sequences of maximum input length \(t_{max}\), defined by the PLM (\(t_{max}=512\) for BERT, \(t_{max}=1024\) for BART). Using a stride of \(1\), we generate \((m-t_{max})+1\) sliding windows \(W\), feed them into the PLM, and extract the resulting \(t_{max}\times t_{max}\) partial square self-attention matrices (\(M_{P}\) in Figure 1) for a specific self-attention head3.

Footnote 3: We omit the self-attention indexes for better readability.

The Frequency Normalization Methodallows us to combine the partially overlapping self-attention matrices \(M_{P}\) into a single document-level matrix \(M_{D}\) of size \(m\times m\). To this end, we combine multiple overlapping windows, generated due to the stride size of \(1\), by adding up the self-attention cells, while keeping track of the number of overlaps in a separate \(m\times m\) frequency matrix \(M_{F}\). We then divide \(M_{D}\) by the frequency matrix \(M_{F}\), to generate a frequency normalized self-attention matrix \(M_{A}\) (see bottom of Figure 1).

The EDU Aggregationis the final processing step to obtain the document-level self-attention matrix. In this step, the \(m\) sub-word tokens \(T=\{t_{1},...t_{m}\}\) are aggregated back into \(n\) EDUs \(E=\{e_{1},...,e_{n}\}\) by computing the average bidirectional self-attention score between any two EDUs in \(M_{A}\). For example, in Figure 1, we aggregate the scores in cells \(M_{A}[0{:}1,5{:}6]\) to compute the final output of cell \([0,2]\) (purple matrix in Figure 1) and \(M_{A}[5{:}6,0{:}1]\) to generate the value of cell \([0,2]\). This way, we obtain the average bidirectional self-attention scores between \(EDU_{1}\) and \(EDU_{3}\). We use the resulting \(n\times n\) matrix as the input to the CKY/Eisner discourse tree generation methods.

## 4 Experimental Setup

### Pre-Trained Models

We select the _BERT-base_ (\(110\) million parameters) and _BART-large_ (\(406\) million parameters) models for our experiments. We choose these models for their diverse objectives (encoder-only vs. encoder-decoder), popularity for diverse fine-tuning tasks, and their prior successful exploration in regards to discourse information Zhu et al. (2020); Koto et al. (2021). For the _BART-large_ model, we limit our analysis to the encoder, as motivated in Koto et al. (2021), leaving experiments with the decoder and cross-attention for future work.

 



### Fine-Tuning Tasks and Datasets

We explore the BERT model fine-tuned on two classification tasks, namely sentiment analysis and natural language inference (NLI). For our analysis on BART, we select the abstractive summarization and question answering tasks. Table 1 summarizes the 7 datasets used to fine-tune PLMs in this work, along with their underlying tasks and domains4.

Footnote 4: We exclusively analyze published models provided on the huggingface platform, further specified in Appendix A.

### Evaluation Treebanks

**RST-DT**Carlson et al. (2002) is the largest English RST-style discourse treebank, containing \(385\) Wall-Street-Journal articles, annotated with full constituency discourse trees. To generate additional dependency trees, we apply the conversion algorithm proposed in Li et al. (2014).

**GUM**Zeldes (2017) is a steadily growing treebank of richly annotated texts. In the current version \(7.3\), the dataset contains \(168\) documents from \(12\) genres, annotated with full RST-style constituency and dependency discourse trees.

All evaluations shown in this paper are executed on the \(38\) and \(20\) documents in the RST-DT and GUM test-sets, to be comparable with previous baselines and supervised models. A similarly-sized validation-set is used where mentioned to determine the best performing self-attention head.

### Baselines and Evaluation Metrics

**Simple Baselines:** We compare the inferred constituency trees against right- and left-branching structures. For dependency trees, we evaluate against simple chain and inverse chain structures.

**Distantly Supervised Baselines:** We compare our results obtained in this paper against our previous approach presented in Xiao et al. (2021), using similar CKY and Eisner tree-generation methods to infer constituency and dependency tree structures from a summarization model trained on the CNN-DM and New York Times (NYT) corpora (referred to as _SumCNN-DM_ and _SumNT_)5.

Footnote 5: www.github.com/Wendy-Xiao/summ_guided_disco_parser

**Supervised Baseline:** We select the popular Two-Stage discourse parser Wang et al. (2017) as our supervised baseline, due to its strong performance, available model checkpoints and code6, as well as the traditional architecture. We use the published Two-Stage parser checkpoint on RST-DT (from here on called _Two-StageBT-DT_) and re-train the discourse parser on GUM (_Two-StageGUM_). We convert the generated constituency structures into dependency trees following Li et al. (2014).

Footnote 6: www.github.com/yizhongw/StageDP

**Evaluation Metrics:** We apply the original parseval score to compare discourse constituency structures with gold-standard treebanks, as argued in Morey et al. (2017). To evaluate the generated dependency structures, we use the Unlabeled Attachment Score (UAS).

## 5 Experimental Results

### Discourse Locality

Our discourse tree generation approach described in section 3 directly uses self-attention matrices to generate discourse trees. The standard BERT

Figure 2: Constituency (top) and dependency (bottom) discourse tree evaluation of BERT (a) and BART (b) models on GUM. Purple=high score, Blue=low score. Left-to-right: self-attention heads, top-to-bottom: high layers to low layers. + indicates fine-tuning dataset.



model contains \(144\) of those self-attention matrices (\(12\) layers, \(12\) self-attention heads each), all of which potentially encode discourse structures. For the BART model, this number is even higher, consisting of \(12\) layers with \(16\) self-attention heads each. With prior work suggesting the locality of discourse information in PLMs (e.g., Raganato and Tiedemann (2018); Marecek and Rosa (2019); Xiao et al. (2021b)), we analyze every self-attention matrix individually to gain a better understanding of their alignment with discourse information.

Besides investigating standard PLMs, we also explore the robustness of discourse information across fine-tuning tasks. We believe that this is an important step to better understand if the captured discourse information is general and robust, or if it is "re-learned" from scratch for downstream tasks. To the best of our knowledge, no previous analysis of this kind has been performed in the literature.

To this end, Figure 2 shows the constituency and dependency structure overlap of the generated discourse trees from individual self-attention heads with the gold-standard tree structures of the GUM dataset7. The heatmaps clearly show that constituency discourse structures are mostly captured in higher layers, while dependency structures are more evenly distributed across layers. Comparing the patterns between models, we find that, despite being fine-tuned on different downstream tasks, the discourse information is consistently encoded in the same self-attention heads. Even though the best performing self-attention matrix is not consistent, discourse information is clearly captured in a "local" subset of self-attention heads across all presented fine-tuning tasks. This plausibly suggests that the discourse information in pre-trained BERT and BART models is robust and general, requiring only minor adjustments depending on the fine-tuning task.

Footnote 7: The analysis on RST-DT shows similar trends and can be found in Appendix B.

### Discourse Quality

We now focus on assessing the discourse information captured in the single best-performing self-attention head. In Table 2, we compare the discourse structure quality of pre-trained and fine-tuned PLMs in the context of supervised models, distantly supervised approaches and simple baselines. We show the oracle-picked best head on the test-set, analyzing the upper-bound for the potential performance of PLMs on RST-style discourse structures. This is not a realistic scenario, as the best performing head is generally not known apriori. Hence, we also explore the performance using a small-scale validation set to pick the best-performing self-attention matrix. In this more realistic scenario for discourse parsing, we find that scores on average drop by \(1.55\) points for BERT and \(1.33\%\) for BART compared to the oracle-picked performance of a single self-attention matrix. We show detailed results of this degradation in Appendix C8. Our results in Table 2 are separated into three sub-tables, showing the results for BERT, BART and baseline models on the RST-DT and GUM treebanks, respectively. In the BERT and BART sub-table, we further annotate each performance with \(\uparrow\), \(\bullet\), \(\downarrow\), indicating the relative performance to the standard pre-trained model as sup

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Model & \multicolumn{2}{c}{RST-DT} & \multicolumn{2}{c}{GUM} \\  & Span & UAS & Span & UAS \\ \hline \multicolumn{5}{c}{BERT} \\ \hline rand. init & \(\downarrow\) 25.5 & \(\downarrow\) 13.3 & \(\downarrow\) 23.2 & \(\downarrow\) 12.4 \\ PLM & \(\bullet\) 35.7 & \(\star\) 45.3 & \(\star\) 33.0 & \(\star\) 45.2 \\ + IMDB & \(\downarrow\) 35.4 & \(\downarrow\) 42.8 & \(\star\) 33.0 & \(\downarrow\) 43.3 \\ + Yelp & \(\downarrow\) 34.7 & \(\downarrow\) 42.3 & \(\downarrow\) 32.6 & \(\downarrow\) 43.7 \\ + SST-2 & \(\downarrow\) 35.5 & \(\downarrow\) 42.9 & \(\downarrow\) 32.6 & \(\downarrow\) 43.5 \\ + MNLI & \(\downarrow\) 34.8 & \(\downarrow\) 41.8 & \(\downarrow\) 32.4 & \(\downarrow\) 43.3 \\ \hline \multicolumn{5}{c}{BART} \\ \hline rand. init & \(\downarrow\) 25.3 & \(\downarrow\) 12.5 & \(\downarrow\) 23.2 & \(\downarrow\) 12.2 \\ PLM & \(\bullet\) 39.1 & \(\star\) 41.7 & \(\star\) 31.8 & \(\star\) 41.8 \\ + CNN-DM & \(\dagger\) 40.9 & \(\dagger\) 44.3 & \(\dagger\) 32.7 & \(\dagger\) 42.8 \\ + XSUM & \(\dagger\) 40.1 & \(\dagger\) 41.9 & \(\dagger\) 32.1 & \(\downarrow\) 39.9 \\ + SQuAD & \(\dagger\) 40.1 & \(\dagger\) 43.2 & \(\downarrow\) 31.3 & \(\downarrow\) 40.7 \\ \hline \multicolumn{5}{c}{Baselines} \\ \hline RB / Chain & 9.3 & 40.4 & 9.4 & 41.7 \\ LB / Chain\({}^{1}\) & 7.5 & 12.7 & 1.5 & 12.2 \\ Sum\({}_{\text{CNN-DM}}\) & 21.4 & 20.5 & 17.6 & 15.8 \\ Sum\({}_{\text{NYT}}\) & 24.0 & 15.7 & 18.2 & 12.6 \\ Two-Stage\({}_{\text{RST-DT}}\) & 72.0 & 71.2 & 54.0 & 54.5 \\ Two-Stage\({}_{\text{GUM}}\) & 65.4 & 61.7 & 58.6 & 56.7 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Original parseval (Span) and Unlabelled Attachment Score (UAS) of the single best performing self-attention matrix of the BERT and BART models compared with baselines and previous work. \(\uparrow\), \(\bullet\), \(\downarrow\) indicate better, same, worse performance compared to the PLM. “rand. init”=Randomly initialized transformer model of similar architecture as the PLM, RB=Right-Branching, LB=Left-Branching, Chain\({}^{1}\)=Inverse chain.

 

rior, equal, or inferior.

Taking a look at the top sub-table (BERT) we find that, as expected, the randomly initialized transformer model achieves the worst performance. Fine-tuned models perform equal or worse than the standard PLM. Despite the inferior results of the fine-tuned models, the drop is rather small, with the sentiment analysis models consistently outperforming NLI. This seems reasonable, given that the sentiment analysis objective is intuitively more aligned with discourse structures (e.g., long-form reviews with potentially complex rhetorical structures) than the between-sentence NLI task, not involving multi-sentential text.

In the center sub-table (BART), a different trend emerges. While the worst performing model is still (as expected) the randomly initialized system, fine-tuned models mostly outperform the standard PLM. Interestingly, the model fine-tuned on the CNN-DM corpus consistently outperforms the BART baseline, while the XSUM model performs better on all but the GUM dependency structure evaluation. On one hand, the superior performance of both summarization models on the RST-DT dataset seems reasonable, given that the fine-tuning datasets and the evaluation treebank are both in the news domain. The strong results of the CNN-DM model on the GUM treebank, yet inferior performance of XSUM, potentially hints towards dependency discourse structures being less prominent when fine-tuning on the extreme summarization task, compared to the longer summaries in the CNN-DM corpus. The question-answering task evaluated through the SQuAD fine-tuned model underperforms the standard PLM on GUM, however reaches superior performance on RST-DT. Since the SQuAD corpus is a subset of Wikipedia articles, more aligned with news articles than the 12 genres in GUM, we believe the stronger performance on RST-DT (i.e., news articles) is again reasonable, yet shows weaker generalization capabilities across domains (i.e., on the GUM corpus). Interestingly, the question-answering task seems more aligned with dependency than constituency trees, in line with what would be expected from a factoid-style question-answering model, focusing on important entities, rather than global constituency structures.

Directly comparing the BERT and BART models, the former performs better on three out of four metrics. At the same time, fine-tuning hurts the performance for BERT, however, improves BART models. Plausibly, these seemingly unintuitive results may be caused by the following co-occurring circumstances: (1) The inferior performance of BART can potentially be attributed to the decoder component capturing parts of the discourse structures, as well as the larger number of self-attention heads "diluting" the discourse information. (2) The different trends regarding fine-tuned models might be directly influenced by the input-length limitation to \(512\) (BERT) and \(1024\) (BART) subword tokens during the fine-tuning stage, hampering the ability to capture long-distance semantic and pragmatic relationships. This, in turn, limits the amount of discourse information captured, even for document-level datasets (e.g., Yelp, CNN-DM, SQuAD). With this restriction being more prominent in BERT, it potentially explains the comparably low performance of the fine-tuned models.

Finally, the bottom sub-table puts our results in the context of previously proposed supervised and distantly-supervised models, as well as simple baselines. Compared to simple right- and left-branching trees (Span), the PLM-based models reach clearly superior performance. Looking at the chain/inverse chain structures (UAS), the improvements are generally lower, however, the vast majority still outperforms the baseline. Comparing the first two sub-tables against completely supervised methods (Two-Stagerst-dt, Two-StageGUM), the BERT- and BART-based models are, unsurprisingly, inferior. Lastly, compared to the distantly supervised SumCNN-DM and SummNYT models, the PLM-based discourse performance shows clear improvements over the 6-layer, 8-head standard transformer.

Figure 3: PLM discourse constituency (left) and dependency (right) structure overlap with baselines and gold trees (e.g., BERT \(\leftrightarrow\) Two-Stage (RST-DT)) according to the original parseval and UAS metrics.

 



### Discourse Similarity

Further exploring what kind of discourse information is captured in the PLM self-attention matrices, we directly compare the emergent discourse structures with trees inferred from existing discourse parsers and simple baselines. This way, we aim to better understand if the information encapsulated in PLMs is complementary to existing methods, or if the PLMs solely capture trivial discourse phenomena and simple biases (e.g., resemble right-branching constituency trees). Since the GUM dataset contains a more diverse set of test documents (12 genres) than the RST-DT corpus (exclusively news articles), we perform our experiments from here on only on the GUM treebank.

Figure 3 shows the micro-average structural overlap of discourse constituency (left) and dependency (right) trees between the PLM-generated discourse structures and existing methods, baselines, as well as gold-standard trees. Noticeably, the generated constituency trees (on the left) are most aligned with the structures predicted by supervised discourse parsers, showing only minimal overlap to simple structures (i.e., right- and left-branching trees). Taking a closer look at the generated dependency structures presented on the right side in Figure 3, the alignment between PLM inferred discourse trees and the simple chain structure is predominant, suggesting a potential weakness in regards to the discourse exposed by the Eisner algorithm in the BERT and BART model. Not surprisingly, the highest overlap between PLM-generated trees and the chain structure occurs when fine-tuning on the CNN-DM dataset, well-known to contain a strong lead-bias Xing et al. (2021).

To better understand if the PLM-based constituency structures are complementary to existing, supervised discourse parsers, we further analyze the correctly predicted overlap. More specifically, we compute the intersection between PLM generated structures and gold-standard trees as well as previously proposed models and the gold-standard. Subsequently, we intersect the two resulting sets (e.g., BERT \(\cap\) Gold Trees \(\leftrightarrow\) Two-Stage (RST-DT) \(\cap\) Gold Trees). This way, we explore if the correctly predicted PLM discourse structures are a subset of the correctly predicted trees by supervised approaches, or if complementary discourse information is captured. We find that \(>20\%\) and \(>16\%\) of the correctly predicted constituency and dependency structures of our PLM discourse inference approach are not captured by supervised models, making the exploration of ensemble methods a promising future avenue. A detailed version of Fig. 3 as well as more specific results regarding the correctly predicted overlap of discourse structures are shown in Appendix E.

### Discourse Redundancy

Up to this point, our quantitative analysis of the ability of PLMs to capture discourse information has been limited to the single best-performing head. However, looking at individual models, the discourse performance distribution in Figure 2 suggests that a larger subset of self-attention heads performs similarly well (i.e., there are several dark purple cells in each heatmap). This leads to the interesting questions if the information captured

Figure 4: Nested aggregation approach for discourse similarity. (a) Grey cells contain same-head, white cells indicate different heads. (b) Grey cells contain same-model, white cells indicate different models. Column indices equal row indices.

Figure 5: BERT self-attention similarities on GUM.



in different, top-performing self-attention heads is redundant or complementary. Similarly, Figure 2 indicates that the same heads perform well across different fine-tuning tasks, leading to the question if the discourse structures captured in a single self-attention matrix of different fine-tuned models is consistent, or varies depending on the underlying task. Hence, we take a detailed look at the similarity of model self-attention heads in regards to their alignment with discourse information and explore if (1) the top performing heads \(h_{i},...,h_{k}\) of a specific model \(m_{m}\) capture redundant discourse structures, and if (2) the discourse information captured by a specific head \(h_{i}\) across different models \(m_{m},...,m_{o}\) contain similar discourse information.

Specifically, we pick the top \(10\) best performing self-attention matrices of each model, remove self-attention heads that don't appear in at least two models (since no comparisons can be made), and compare the generated discourse structures in a nested aggregation approach.

Figure 4 shows a small-scale example of our nested visualization methodology. For the self-attention head-aligned approach (Figure 4 (a)), high similarity values (calculated as the micro-average structural overlap) along the diagonal (grey cells) would be expected if the same head \(h_{i}\) encodes consistent discourse information across different fine-tuning tasks and datasets. Inversely, the model-aligned matrix (Figure 4 (b)) should show high values along the diagonal if different heads \(h_{i},...,h_{k}\) in the same model \(m_{k}\) capture redundant discourse information. Besides the visual inspection methodology presented in Figure 4, we also compare aggregated similarities between the same head (=Head) against different heads (\(\neq\)Head) and between the same model (=Model) against different models (\(\neq\)Model) (i.e., grey cells (=) and white cells (\(\neq\)) in Figure 4 (a) and (b)). In order to assess the statistical significance of the resulting differences in the underlying distributions, we compute a two-sided, independent t-test between same/different models and same/different heads9.

Footnote 9: Prior to running the t-test we confirm similar variance and the assumption of normal distribution (Shapiro-Wilk test).

The resulting redundancy evaluations for BERT are presented in Figure 510. It appears that the same self-attention heads \(h_{i}\) consistently encode similar discourse information across models indicated by: (1) High similarities (yellow) along the diagonal in heatmaps \(I& III\) and (2) through the statistically significant difference in distributions at the bottom of Figure 5 (a) and (b). However, different self-attention heads \(h_{i},...,h_{k}\) of the same model \(m_{m}\) encode different discourse information (heatmaps \(II\& IV\)). While the trend is stronger for constituency tree structures, there is a single dependency self-attention head which does generally not align well between models and heads (purple line in heatmap \(III\)). Plausibly, this specific self-attention head encodes fine-tuning task specific discourse information, making it a prime candidate for further investigations in future work. Furthermore, the similarity patterns observed in Figure 5 (a) and (b) point towards an opportunity to combine model self-attention heads to improve the discourse inference performance compared to the scores shown in Table 2, where each self-attention head was assessed individually, in future work.

Footnote 10: Evaluations for BART can be found in Appendix F.

## 6 Conclusions

In this paper, we extend the line of _BERTology_ work by focusing on the important, yet less explored, alignment of pre-trained and fine-tuned PLMs with large-scale discourse structures. We propose a novel approach to infer discourse information for arbitrarily long documents. In our experiments, we find that the captured discourse information is consitently local and general, even across a collection of fine-tuning tasks. We compare the inferred discourse trees with supervised, distantly supervised and simple baselines to explore the structural overlap, finding that constituency discourse trees align well with supervised models, however, contain complementary discourse information. Lastly, we individually explore self-attention matrices to analyze the information redundancy. We find that similar discourse information is consistently captured in the same heads.

In the future, we intend to explore additional discourse inference strategies based on the insights we gained in this analysis. Specifically, we want to explore more sophisticated methods to extract a single discourse tree from multiple self-attention matrices, rather than only the single best-performing head. Further, we want to investigate the relationship between supervised discourse parsers and PLM generated discourse trees and more long term, we plan to analyze PLMs with enhanced input-length limitations.

 



## Acknowledgements

We thank the anonymous reviewers and the UBC NLP group for their insightful comments and suggestions. This research was supported by the Language & Speech Innovation Lab of Cloud BU, Huawei Technologies Co., Ltd and the Natural Sciences and Engineering Research Council of Canada (NSERC). Nous remercions le Conseil de recherches en sciences naturelles et en genie du Canada (CRSNG) de son soutien.

## References

* A. Adhikari, A. Ram, R. Tang, and J. Lin (2019)DocBERT: Bert for document classification. arXiv preprint arXiv:1904.08398. Cited by: SS5.
* I. Beltagy, M. E. Peters, and A. Cohan (2020)Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150. Cited by: SS6.
* P. Bhatia, Y. Ji, and J. Eisenstein (2015)Better document-level sentiment analysis from rst discourse parsing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, pp. 2212-2218. External Links: Link Cited by: SS1, SS2.
* L. Carlson, M. Ellen Okurowski, and D. Marcu (2002)RST discourse treebank. Linguistic Data Consortium, University of Pennsylvania. Cited by: SS1, SS2.
* Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. Le, and R. Salakhutdinov (2019)Transformer-XL: attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 2978-2988. External Links: Link Cited by: SS1, SS2.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 4171-4186. External Links: Link Cited by: SS1, SS2.
* Q. Diao, M. Qiu, C. Wu, A. J. Smola, J. Jiang, and C. Wang (2014)Jointly modeling aspects, ratings and sentiments for movie recommendation (jmars). In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '14, New York, NY, USA, pp. 193-202. External Links: ISBN 978-1-4503-3811-1, Document, Link Cited by: SS1, SS2.
* J. M. Eisner (1996)Three new probabilistic models for dependency parsing: an exploration. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics, Cited by: SS1, SS2.
* S. Gerani, G. Carenini, and R. T. Ng (2019)Modeling content and structure for abstractive review summarization. Computer Speech & Language53, pp. 302-331. Cited by: SS1, SS2.
* G. Guz and G. Carennii (2020)Coreference for discourse parsing: a neural approach. In Proceedings of the First Workshop on Computational Approaches to Discourse, Online, pp. 160-167. External Links: Link Cited by: SS1, SS2.
* H. Hernault, H. Prendinger, M. Ishizuka, et al. (2010)Hilda: a discourse parser using support vector machine classification. Dialogue & Discourse1(3). Cited by: SS1, SS2.
* J. Hewitt and C. D. Manning (2019)A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 4129-4138. External Links: Link Cited by: SS1, SS2.
* A. Hogenboom, F. Frasincar, F. dong, and U. Kaymak (2015)Using rhetorical structure in sentiment analysis. Commun. ACM58 (7), pp. 69-77. External Links: ISSN 0018-907X, Link, Document Cited by: SS1, SS2.
* P. Huber and G. Carenini (2020)MEGA rest discourse treebanks with structure and nuclearity from scalable distant sentiment supervision. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 7442-7457. External Links: Link Cited by: SS1, SS2.
* G. Jawahar, B. Sagot, and D. Seddah (2019)What does BERT learn about the structure of language?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 3651-3657. External Links: Link Cited by: SS1, SS2.
* Y. Ji and J. Eisenstein (2014)Representation learning for text-level discourse parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Baltimore, Maryland, pp. 13-24. External Links: Link Cited by: SS1, SS2.



* Ji and Smith (2017) Yangfeng Ji and Noah A. Smith. 2017. Neural discourse structure for text categorization. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 996-1005, Vancouver, Canada. Association for Computational Linguistics.
* Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. _Transactions of the Association for Computational Linguistics_, 8:64-77.
* Joty et al. (2015) Shafiq Joty, Giuseppe Carenini, and Raymond T. Ng. 2015. CODRA: A novel discriminative framework for rhetorical analysis. _Computational Linguistics_, 41(3):385-435.
* Jurafsky and Martin (2014) Dan Jurafsky and James H Martin. 2014. _Speech and language processing_, volume 3. Pearson London.
* Kim et al. (2020) Taeuk Kim, Jihun Choi, Daniel Edmiston, and Sang goo Lee. 2020. Are pre-trained language models aware of phrases? simple but strong baselines for grammar induction. In _International Conference on Learning Representations_.
* Kitaev et al. (2020) Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In _International Conference on Learning Representations_.
* Kobayashi et al. (2020) Naoki Kobayashi, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2020. Topdown rst parsing utilizing granularity levels in documents. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 34, pages 8099-8106.
* Kobayashi et al. (2019) Naoki Kobayashi, Tsutomu Hirao, Keng Nakamura, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2019. Split or merge: Which is better for unsupervised RST parsing? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 5797-5802, Hong Kong, China. Association for Computational Linguistics.
* Koto et al. (2021a) Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2021a. Discourse probing of pretrained language models. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3849-3864, Online. Association for Computational Linguistics.
* Koto et al. (2021b) Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2021b. Top-down discourse parsing via sequence labelling. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 715-726, Online. Association for Computational Linguistics.
* Kurfali and Ostling (2021) Murathan Kurfali and Robert Ostling. 2021. Probing multilingual language models for discourse. In _Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021)_, pages 8-19, Online. Association for Computational Linguistics.
* Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7871-7880, Online. Association for Computational Linguistics.
* Li et al. (2014) Sujian Li, Liang Wang, Ziqiang Cao, and Wenjie Li. 2014. Text-level discourse dependency parsing. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 25-35, Baltimore, Maryland. Association for Computational Linguistics.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Mann and Thompson (1988) William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. _Text-Interdisciplinary Journal for the Study of Discourse_, 8(3):243-281.
* Marecek and Rosa (2019) David Marecek and Rudolf Rosa. 2019. From balustrades to pierre vinken: Looking for syntax in transformer self-attentions. In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 263-275, Florence, Italy. Association for Computational Linguistics.
* Michael et al. (2020) Julian Michael, Jan A. Botha, and Ian Tenney. 2020. Asking without telling: Exploring latent ontologies in contextual representations. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6792-6812, Online. Association for Computational Linguistics.
* Morey et al. (2017) Mathieu Morey, Philippe Muller, and Nicholas Asher. 2017. How much progress have we made on RST discourse parsing? a replication study of recent results on the RST-DT. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 1319-1324, Copenhagen, Denmark. Association for Computational Linguistics.
* Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caplar Guliceher, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
* Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In _Proceedings of the 2018 

Conference on Empirical Methods in Natural Language Processing_, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
* Nejat et al. (2017) Bita Nejat, Giuseppe Carenini, and Raymond Ng. 2017. Exploring joint neural model for sentence level discourse parsing and sentiment analysis. In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_, pages 289-298, Saarbrucken, Germany. Association for Computational Linguistics.
* Nguyen et al. (2021) Thanh-Tung Nguyen, Xuan-Phi Nguyen, Shafiq Joty, and Xiaoli Li. 2021. RST parsing from scratch. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1613-1625, Online. Association for Computational Linguistics.
* Oguz et al. (2021) Barlas Oguz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xiulun Chen, Sebastian Riedel, Wen-tau Yih, Sonal Gupta, et al. 2021. Domain-matched pretraining tasks for dense retrieval. _arXiv preprint arXiv:2107.13602_.
* Pandia et al. (2021) Lalchand Pandia, Yan Cong, and Allyson Etinger. 2021. Pragmatic competence of pre-trained language models through the lens of discourse connectives. In _Proceedings of the 25th Conference on Computational Natural Language Learning_, pages 367-379, Online. Association for Computational Linguistics.
* Papanikolaou et al. (2019) Yannis Papanikolaou, Ian Roberts, and Andrea Pierleoni. 2019. Deep bidirectional transformers for relation extraction without supervision. In _Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019)_, pages 67-75, Hong Kong, China. Association for Computational Linguistics.
* Prasad et al. (2008) Rashmi Prasad, Nikhit Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In _Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)_, Marrakech, Morocco. European Language Resources Association (ELRA).
* Rae et al. (2020) Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In _International Conference on Learning Representations_.
* Raganato and Tiedemann (2018) Alessandro Raganato and Jorg Tiedemann. 2018. An analysis of encoder representations in transformer-based machine translation. In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 287-297, Brussels, Belgium. Association for Computational Linguistics.
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
* Rogers et al. (2020) Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. _Transactions of the Association for Computational Linguistics_, 8:842-866.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, volume 30. Curran Associates, Inc.
* Wang et al. (2017) Yizhong Wang, Sujian Li, and Houfeng Wang. 2017. A two-stage parsing method for text-level discourse analysis. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 184-188, Vancouver, Canada. Association for Computational Linguistics.
* Webber et al. (2003) Bonnie Webber, Matthew Stone, Aravind Joshi, and Alistair Knott. 2003. Anaphora and discourse structure. _Computational Linguistics_, 29(4):545-587.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
* Wu et al. (2020) Zhiyang Wu, Yun Chen, Ben Kao, and Qun Liu. 2020. Perturbed masking: Parameter-free probing for analyzing and interpreting BERT. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4166-4176, Online. Association for Computational Linguistics.
* Xiao et al. (2021a) Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. 2021a. Primer: Pyramid-based masked sentence pre-training for multi-document summarization. _arXiv preprint arXiv:2110.08499_.
* Xiao et al. (2021b) Wen Xiao, Patrick Huber, and Giuseppe Carenini. 2021b. Predicting discourse trees from transformer-based neural summarizers. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4139-4152, Online. Association for Computational Linguistics 

Linzi Xing, Wen Xiao, and Giuseppe Carenini. 2021. Demoting the lead bias in news summarization via alternating adversarial learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Online, pp. 948-954. External Links: Link Cited by: SS3.
* Z. Yang, X. Dai, Y. Yang, J. Carbonell, R. R Salakhutdinov, and Q. V. Le (2019)Xlnet: generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, Vol. 32, pp. . External Links: Document Cited by: SS1, SS2.
* M. Zaheer, G. Prashanth Guruganesh, A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, and A. Mahmoud El Houssieny Ahmed (2020)Big bird: transformers for longer sequences. In Advances in Neural Information Processing Systems, Cited by: SS1, SS2.
* A. Zeldes (2017)The gum corpus: creating multilayer resources in the classroom. Language Resources and Evaluation51 (3), pp. 581-612. External Links: ISSN 0378-0221, Link, Document Cited by: SS1, SS2.
* J. Zhang, Y. Zhao, M. Saleh, and P. Liu (2020)Pegasus: pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pp. 11328-11339. External Links: Link, Document Cited by: SS1, SS2.
* Volume 1, NIPS'15, Cambridge, MA, USA, pp. 649-657. External Links: ISBN 978-3-540-3871-1, Link, Document Cited by: SS1, SS2.
* Z. Zhu, C. Pan, M. Abdalla, and F. Rudzicz (2020)Examining the rhetorical capacities of neural language models. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, Online, pp. 16-32. External Links: Link, Document Cited by: SS1, SS2.

 

Huggingface Models

We investigate 7 fine-tuned BERT and BART models from the huggingface model library, as well as the two pre-trained models. The model names and links are provided in Table 3

## Appendix B Test-Set Results on RST-DT and GUM

Figure 6: Constituency (top) and dependency (bottom) discourse tree evaluation of BERT (a) and BART (b) models on RST-DT (test). Purple=high score, blue=low score. + indicates fine-tuning dataset.

 

* [26] M. C. Bader, J.-M. Chen, and S.-J. Lee, "A new approach to the detection of a large number of people in multi-person data using deep learning approaches," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1048-1053, 2017.
* [27] A. Dosovitskiy et al., "Deep residual networks for image recognition," _IEEE Transactions on Image Processing Systems_**9**, no. 1, 1023, 2016.
* [28] R. Guezel, L. Zhang, and Y. Bengio, "Learning from tiny images by exploiting sparse representations," _arXiv preprint arXiv:1512.00158_ (2015).
* [29] T. H. Nguyen, P. K. LeCun, and X. Bengio, "Image denoising with convolutional neural networks," _CoRR_**abs/1412.02142, 2014.
* [30] W. Li, Z. Wang, and Q. Sun, "Recurrent network for object detection via deep belief nets," _arXiv preprint arXiv:1512.00158_ (2015).
* [31] E. Maaten, N. Schlicht, and I. Sutskever, "Improving deep features from tiny images," _Neural Networks_**3**, no. 1, 1023, 2016.
* [32] F. Liu, Z. Yang, and Y. Bengio, "Deep residual networks for image classification," _arXiv preprint arXiv:1512.00158_ (2015).

[MISSING_PAGE_POST]

 



## Appendix C Oracle-picked self-attention head compared to validation-picked matrix

\begin{table}
\begin{tabular}{l r r r r} \hline \hline Model & \multicolumn{2}{c}{RST-DT} & \multicolumn{2}{c}{GUM} \\  & Span & UAS & Span & UAS \\ \hline \multicolumn{5}{c}{BERT} \\ \hline rand. init & 25.5 (-0.0) & 13.3 (-0.0) & 23.2 (-0.0) & 12.4 (-0.0) \\ PLM & 35.7 (-1.6) & 45.3 (-4.9) & 33.0 (-0.4) & 45.2 (-0.0) \\ + IMDB & 35.4 (-1.8) & 42.8 (-2.4) & 33.0 (-3.8) & 43.3 (-0.1) \\ + Yelp & 34.7 (-1.0) & 42.3 (-1.9) & 32.6 (-3.6) & 43.7 (-0.0) \\ + SST-2 & 35.5 (-1.9) & 42.9 (-2.5) & 32.6 (-0.3) & 43.5 (-0.9) \\ + MNLI & 34.8 (-1.7) & 41.8 (-1.4) & 32.4 (-0.3) & 43.3 (-0.5) \\ \hline \multicolumn{5}{c}{BART} \\ \hline rand. init & 25.3 (-0.0) & 12.5 (-0.0) & 23.2 (-0.0) & 12.2 (-0.0) \\ PLM & 39.1 (-0.4) & 41.7 (-2.7) & 31.8 (-0.3) & 41.8 (-0.0) \\ + CNN-DM & 40.9 (-0.0) & 44.3 (-4.0) & 32.7 (-0.3) & 42.8 (-0.7) \\ + XSUM & 40.1 (-0.9) & 41.9 (-3.4) & 32.1 (-1.7) & 39.9 (-0.0) \\ + SQuAD & 40.1 (-0.0) & 43.2 (-4.6) & 31.3 (-2.1) & 40.7 (-0.1) \\ \hline \multicolumn{5}{c}{Baselines} \\ \hline Right-Branch/Chain & 9.3 & 40.4 & 9.4 & 41.7 \\ Left-Branch/Chain-1 & 7.5 & 12.7 & 1.5 & 12.2 \\ Sum\({}_{\text{CNN-DM}}\)(2021b) & 21.4 & 20.5 & 17.6 & 15.8 \\ Sum\({}_{\text{NYT}}\)(2021b) & 24.0 & 15.7 & 18.2 & 12.6 \\ Two-Stage\({}_{\text{GST-DT}}\)(2017) & 72.0 & 71.2 & 54.0 & 54.5 \\ Two-Stage\({}_{\text{GUM}}\) & 65.4 & 61.7 & 58.6 & 56.7 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Original parseval (Span) and Unlabelled Attachment Score (UAS) of the single best performing oracle self-attention matrix and validation-set picked head (in brackets) of the BERT and BART models compared with baselines and previous work. “rand. init”=Randomly initialized transformer model of similar architecture as the PLM.





## Appendix D Detailed Self-Attention Statistics

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline Model & \multicolumn{4}{c}{Span} & \multicolumn{5}{c}{Eisner} \\  & Min & Med & Mean & Max & Min & Med & Mean & Max \\ \hline \multicolumn{10}{c}{RST-DT} \\ \hline rand. init & 20.3 & 23.3 & 23.3 & 25.3 & 8.5 & 10.6 & 10.6 & 12.5 \\ PLM & 20.3 & 28.3 & 28.5 & 39.1 & 4.1 & 15.8 & 19.2 & 41.7 \\ + CNN-DM & 20.5 & 28.6 & 28.7 & 40.9 & 3.6 & 15.2 & 19.2 & 44.3 \\ + XSUM & 20.2 & 27.6 & 28.3 & 40.1 & 4.8 & 14.8 & 18.7 & 41.9 \\ + SQuAD & 20.5 & 27.6 & 28.2 & 40.1 & 2.8 & 14.8 & 18.8 & 43.2 \\ \hline \multicolumn{10}{c}{GUM} \\ \hline rand. init & 18.6 & 21.0 & 21.0 & 23.2 & 8.0 & 10.2 & 10.2 & 12.2 \\ PLM & 16.7 & 23.4 & 23.8 & 31.5 & 2.6 & 15.2 & 18.7 & 41.8 \\ + CNN-DM & 15.9 & 23.7 & 24.1 & 32.4 & 3.7 & 14.7 & 18.9 & 42.8 \\ + XSUM & 16.4 & 23.2 & 23.9 & 31.8 & 3.0 & 14.1 & 18.1 & 39.9 \\ + SQuAD & 16.1 & 23.4 & 23.8 & 31.0 & 2.4 & 14.8 & 18.3 & 40.7 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Minimum, median, mean and maximum performance of the self-attention matrices on RST-DT and GUM for the BART model.

\begin{table}
\begin{tabular}{l l r r r r r r r} \hline \hline Model & \multicolumn{4}{c}{Span} & \multicolumn{4}{c}{Eisner} \\  & Min & Med & Mean & Max & Min & Med & Mean & Max \\ \hline \multicolumn{10}{c}{RST-DT} \\ \hline rand. init & 21.7 & 23.4 & 23.4 & 25.5 & 7.5 & 10.3 & 10.3 & 13.3 \\ PLM & 19.3 & 27.0 & 27.4 & 35.7 & 6.6 & 17.4 & 21.6 & 45.3 \\ + IMDB & 19.7 & 26.9 & 27.2 & 35.4 & 6.6 & 16.9 & 21.3 & 42.8 \\ + YELP & 20.2 & 26.6 & 26.9 & 34.7 & 7.0 & 16.5 & 21.0 & 42.3 \\ + SST-2 & 19.5 & 27.3 & 27.7 & 35.5 & 7.3 & 17.6 & 21.9 & 42.9 \\ + MNLI & 18.5 & 26.9 & 27.1 & 34.8 & 6.9 & 17.5 & 21.5 & 41.8 \\ \hline \multicolumn{10}{c}{GUM} \\ \hline rand. init & 18.6 & 21.0 & 21.0 & 23.2 & 7.9 & 10.1 & 10.1 & 12.4 \\ PLM & 17.8 & 24.2 & 24.3 & 32.6 & 6.7 & 16.0 & 21.2 & 45.2 \\ + IMDB & 18.1 & 23.8 & 24.1 & 32.7 & 6.1 & 15.9 & 21.0 & 43.3 \\ + YELP & 18.6 & 24.0 & 23.9 & 32.3 & 7.0 & 15.8 & 20.7 & 43.7 \\ + SST-2 & 18.2 & 24.6 & 24.7 & 32.3 & 6.5 & 16.5 & 21.6 & 43.5 \\ + MNLI & 17.4 & 23.9 & 24.2 & 32.1 & 6.8 & 16.6 & 21.3 & 43.3 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Minimum, median, mean and maximum performance of the self-attention matrices on RST-DT and GUM for the BERT model.



Details of Structural Discourse Similarity

Figure 8: Detailed PLM discourse constituency (left) and dependency (right) structure overlap with baselines and gold trees according to the original parseval and UAS metrics.

Figure 9: Detailed PLM discourse constituency (left) and dependency (right) structure performance of intersection with gold trees (e.g., BERT \(\cap\) Gold Trees \(\leftrightarrow\) Two-Stage (RST-DT) \(\cap\) Gold Trees) according to the original parseval and UAS metrics.

 

Intra- and Inter-Model Self-Attention Comparison

Figure 10: Top: Visual analysis of sorted heatmaps. Yellow=high score, purple=low score. Bottom: Aggregated similarity of same heads, same models, different heads and different models. *=Head/=Model significantly better than \(\neq\)Head/\(\neq\)Model performance with p-value \(<0.05\).