SlowGuess/NAACL_2022_OCR huggingface dataset:
Patrick Huber and Giuseppe Carenini
Vancouver, BC, Canada, V6T 1Z4
{huberpat, carenini}@cs.ubc.ca
With a growing number of BERTology works analyzing different components of pre-trained language models, we extend this line of research through an in-depth analysis of discourse information in pre-trained and finetuned language models. We move beyond prior work along three dimensions: First, we describe a novel approach to infer discourse structures from arbitrarily long documents. Second, we propose a new type of analysis to explore where and how accurately intrinsic discourse is captured in the BERT and BART models. Finally, we assess how similar the generated structures are to a variety of baselines as well as their distributions within and between models.
Transformer-based machine learning models are an integral part of many recent improvements in Natural Language Processing (NLP). With their rise spearheaded by Vaswani et al. (2017), the pre-training/fine-tuning paradigm has gradually replaced previous approaches based on architecture engineering, with transformer models such as BERT (Devlin et al., 2019), BART (Lewis et al., 2020), RoBERTa (Liu et al., 2019) and others delivering state-of-the-art performance on a wide variety of tasks. Besides their strong empirical results on most real-world problems, such as summarization (Zhang et al., 2020; Xiao et al., 2021a), question-answering (Joshi et al., 2020; Oğuz et al., 2021) and sentiment analysis (Adhikari et al., 2019; Yang et al., 2019), uncovering what kind of linguistic knowledge is captured by this new type of pretrained language models (PLMs) has become a prominent question by itself. As part of this line of research, called BERTology (Rogers et al., 2020), researchers explore the amount of linguistic understanding encapsulated in PLMs, exposed through either external probing tasks (Raganato and Tiedemann, 2018; Zhu et al., 2020; Koto et al., 2021a)
or unsupervised methods (Wu et al., 2020; Pandia et al., 2021). Previous work thereby either focuses on analyzing the syntactic structures (e.g., Hewitt and Manning (2019); Wu et al. (2020)), relations (Papanikolaou et al., 2019), ontologies (Michael et al., 2020) or, to a more limited extent, discourse related behaviour (Zhu et al., 2020; Koto et al., 2021a; Pandia et al., 2021).
Generally speaking, while most previous BERTology works has focused on either sentence level phenomena or connections between adjacent sentences, large-scale semantic and pragmatic structures (oftentimes represented as discourse trees or graphs) have been less explored. These structures (e.g., discourse trees) play a fundamental role in expressing the intent of multi-sentential documents and, not surprisingly, have been shown to benefit many NLP tasks such as summarization (Gerani et al., 2019), sentiment analysis (Bhatia et al., 2015; Nejat et al., 2017; Hogenboom et al., 2015) and text classification (Ji and Smith, 2017).
With multiple different theories for discourse proposed in the past, the RST discourse theory (Mann and Thompson, 1988) and the lexicalized discourse grammar (Webber et al., 2003) (underlying PDTB (Prasad et al., 2008)) have received most attention. While both theories propose tree-like structures, the PDTB framework postulates partial trees up to the between-sentence level, while RST-style discourse structures consist of a single rooted tree covering whole documents, comprising of: (1) The tree structure, combining clause-like sentence fragments (Elementary Discourse Units, short: EDUs) into a discourse constituency tree, (2) Nuclearity, assigning every tree-branch primary (Nucleus) or peripheral (Satellite) importance in a local context and (3) Relations, defining the type of connection holding between siblings in the tree.
Given the importance of large-scale discourse structures, we extend the area of BERTology research with novel insights regarding the amount of
Human Language Technologies, pages 2376 - 2394
intrinsic discourse information captured in established PLMs. More specifically, we aim to better understand to what extend RST-style discourse information is stored as latent trees in encoder self-attention matrices<sup>1</sup>. While we focus on the RST formalism in this work, our presented methods are theory-agnostic and, hence, applicable to discourse structures in a broader sense, including other tree-based theories, such as the lexicalized discourse grammar. Our contributions in this paper are:
(1) A novel approach to extract discourse information from arbitrarily long documents with standard transformer models, inherently limited by their input size. This is a non-trivial issue, which has been mostly by-passed in previous work through the use of proxy tasks like connective prediction, relation classification, sentence ordering, EDU segmentation, cloze story tests and others.
(2) An exploration of discourse information locality across pre-trained and fine-tuned language models, finding that discourse structures are consistently captured in a fixed subset of self-attention heads.
(3) An in-depth analysis of the discourse quality in pre-trained language models and their fine-tuned extensions. We compare constituency and dependency structures of 2 PLMs fine-tuned on 4 tasks and 7 fine-tuning datasets to gold-standard discourse trees, finding that the captured discourse structures outperform simple baselines by a large margin, even showing superior performance compared to distantly supervised models.
(4) A similarity analysis between PLM inferred discourse trees and supervised, distantly supervised and simple baselines. We reveal that PLM constituency discourse trees do align relatively well with previously proposed supervised models, but also capture complementary information.
(5) A detailed look at information redundancy in self-attention heads to better understand the structural overlap between self-attention matrices and models. Our results indicate that similar discourse information is consistently captured in the same heads, even across fine-tuning tasks.
At the base of our work are two of the most popular and frequently used PLMs: BERT (Devlin et al., 2019) and BART (Lewis et al., 2020). We choose these two popular approaches in our study
due to their complementary nature (encoder-only vs. encoder-decoder) and based on previous work by Zhu et al. (2020) and Koto et al. (2021a), showing the effectiveness of BERT and BART models for discourse related tasks.
Our work is further related to the field of discourse parsing. With a rich history of traditional machine learning models (e.g., Hernault et al. (2010); Ji and Eisenstein (2014); Joty et al. (2015); Wang et al. (2017), inter alia), recent approaches slowly shifted to successfully incorporate a variety of PLMs into the process of discourse prediction, such as ELMo embeddings (Kobayashi et al., 2019), XLNet (Nguyen et al., 2021), BERT (Koto et al., 2021b), RoBERTa (Guz et al., 2020) and SpanBERT (Guz and Carenini, 2020). Despite these works showing the usefulness of PLMs for discourse parsing, all of them cast the task into a "local" problem, using only partial information through the shift-reduce framework (Guz et al., 2020; Guz and Carenini, 2020), natural document breaks (e.g. paragraphs Kobayashi et al. (2020)) or by framing the task as an inter-EDU sequence labelling problem on partial documents (Koto et al., 2021b). However, we believe that the true benefit of discourse information emerges when complete documents are considered, leading us to propose a new approach to connect PLMs and discourse structures in a "global" manner, superseding the local proxy-tasks with a new methodology to explore arbitrarily long documents.
Aiming to better understand what information is captured in PLMs, the line of BERTology research has recently emerged (Rogers et al., 2020), with early work mostly focusing on the syntactic capacity of PLMs (Hewitt and Manning, 2019; Jawahar et al., 2019; Kim et al., 2020), in parts also exploring the internal workings of transformer-based models (e.g., self-attention matrices (Raganato and Tiedemann, 2018; Mareček and Rosa, 2019)). More recent work started to explore the alignment of PLMs with discourse information, encoding semantic and pragmatic knowledge. Along those lines, Wu et al. (2020) present a parameter-free probing task for both, syntax and discourse. With their tree inference approach being computationally expensive and limited to the exploration of the outputs of the BERT model, we significantly extend this line of research by exploring the internal self-attention matrices of PLMs with a more computationally feasible approach. More tradi
<sup>1</sup>Please note that we focus on discourse structure and nuclearity here, leaving relation classification for future work.
Figure 1: Small-scale example of the discourse extraction approach. Purple=EDUs, green=sub-word embeddings, red=input slices of size  $t_{max}$ , orange=PLM, blue=self-attention values, grey-scale=frequency count.
tionally, Zhu et al. (2020) use 24 hand-crafted rhetorical features to execute three different supervised probing tasks, showing promising performance of the BERT model. Similarly, Pandia et al. (2021) aim to infer pragmatics through the prediction of discourse connectives by analyzing the model inputs and outputs and Koto et al. (2021a) analyze discourse in seven PLMs through seven supervised probing tasks, finding that BART and BERT contain most information related to discourse. In contrast to the approach taken by both Zhu et al. (2020) and Koto et al. (2021a), we use an unsupervised methodology to test the amount of discourse information stored in PLMs (which can also conveniently be used to infer discourse structures for new and unseen documents) and extend the work by Pandia et al. (2021) by taking a closer look at the internal workings of the self-attention component. Looking at prior work analyzing the amount of discourse information in PLMs, structures are solely explored through the use of proxy tasks, such as connective prediction (Pandia et al., 2021), relation classification (Kurfali and Ostling, 2021), and others (Koto et al.,
2021a). However, despite the difficulties of encoding arbitrarily long documents, we believe that to systematically explore the relationship between PLMs and discourse, considering complete documents is imperative. Along these lines, recent work started to tackle the inherent input-length limitation of general transformer models through additional recurrence in the Transformer-XL model (Dai et al., 2019), compression modules (Rae et al., 2020) or sparse patterns (e.g., as in the Reformer (Kitaev et al., 2020), BigBird (Zaheer et al., 2020), and Longformer (Beltagy et al., 2020) models). While all these approaches to extend the maximum document length of transformer-based models are important to create more globally inspired models, the document-length limitation is still practically and theoretically in place, with models being limited to a fixed number of pre-defined tokens the model can process. Furthermore, with many proposed systems still based on more established PLMs (e.g., BERT) and with no single dominant solution for the general problem of the input length-limitation yet, we believe that even with the restriction being actively tackled, an in-depth analysis of traditional PLMs with discourse is highly valuable to establish a solid understanding of the amount of semantic and pragmatic information captured.
Besides the described BERTology work, we got encouraged to explore fine-tuned extensions of standard PLMs through previous work showing the benefit of discourse parsing for many downstream tasks, such as summarization (Gerani et al., 2019), sentiment analysis (Bhatia et al., 2015; Nejat et al., 2017; Hogenboom et al., 2015) and text classification (Ji and Smith, 2017). Conversely, we recently showed promising results when inferring discourse structures from related downstream tasks, such as sentiment analysis (Huber and Carenini, 2020) and summarization (Xiao et al., 2021b). Given this bidirectional synergy between discourse and the mentioned downstream tasks, we move beyond traditional experiments focusing on standard PLMs and additionally explore discourse structures of PLMs fine-tuned on a variety of auxiliary tasks.
3 Discourse Extraction Method
With PLMs rather well analyzed according to their syntactic capabilities, large-scale discourse structures have been less explored. One reason for this is the input length constraint of transformer models. While this is generally not prohibitive for intra
sentence syntactic structures (e.g., presented in Wu et al. (2020)), it does heavily influence large-scale discourse structures, operating on complete (potentially long) documents. Overcoming this limitation is non-trivial, since traditional transformer-based models only allow for fixed, short inputs.
Aiming to systematically explore the ability of PLMs to capture discourse, we investigate a novel way to effectively extract discourse structures from the self-attention component of the BERT and BART models. We thereby extend our previously proposed tree-generation methodology (Xiao et al., 2021b) to support the input length constraints of standard PLMs using a sliding-window approach in combination with matrix frequency normalization and an EDU aggregation method. Figure 1 visualizes the complete process on a small scale example with 3 EDUs and 7 sub-word embeddings.
The Tree Generation Procedure we previously proposed in Xiao et al. (2021b) explores a two-stage approach to obtain discourse structures from a transformer model, by-passing the input-length constraint. Using the intuition that the self-attention score between any two EDUs is an indicator of their semantic/pragmatic relatedness, influencing their distance in a projective discourse tree, they use the CKY dynamic programming approach (Jurafsky and Martin, 2014) to generate constituency trees based on the internal self-attention of the transformer model. To generate dependency trees, we apply the same intuition used to infer discourse trees with the Eisner algorithm (Eisner, 1996). Since we explore the discourse information captured in standard PLMs, we can't directly transfer our two-stage approach in Xiao et al. (2021b), first encoding individual EDUs using BERT and subsequently feeding the dense representations into a fixed-size transformer model. Instead, we propose a new method to overcome the length-limitation of the transformer model<sup>2</sup>.
The Sliding-Window Approach is at the core of our new methodology to overcome the input-length constraint. We first tokenize arbitrarily long documents with  $n$  EDUs  $E = \{e_1, \dots, e_n\}$  into the respective sequence of  $m$  sub-word tokens  $T = \{t_1, \dots, t_m\}$  with  $n \ll m$ , according to the PLM tokenization method (WordPiece for BERT, Byte-Pair-Encoding for BART), as show at the top of
Figure 1. Using the sliding window approach, we subdivide the  $m$  sub-word tokens into sequences of maximum input length  $t_{max}$ , defined by the PLM  $(t_{max} = 512$  for BERT,  $t_{max} = 1024$  for BART). Using a stride of 1, we generate  $(m - t_{max}) + 1$  sliding windows  $W$ , feed them into the PLM, and extract the resulting  $t_{max} \times t_{max}$  partial square self-attention matrices  $(M_P$  in Figure 1) for a specific self-attention head<sup>3</sup>.
The Frequency Normalization Method allows us to combine the partially overlapping self-attention matrices  $M_P$  into a single document-level matrix  $M_D$  of size  $m \times m$ . To this end, we combine multiple overlapping windows, generated due to the stride size of 1, by adding up the self-attention cells, while keeping track of the number of overlaps in a separate  $m \times m$  frequency matrix  $M_F$ . We then divide  $M_D$  by the frequency matrix  $M_F$ , to generate a frequency normalized self-attention matrix  $M_A$  (see bottom of Figure 1).
The EDU Aggregation is the final processing step to obtain the document-level self-attention matrix. In this step, the  $m$  sub-word tokens  $T = \{t_1, \dots, t_m\}$  are aggregated back into  $n$  EDUs  $E = \{e_1, \dots, e_n\}$  by computing the average bidirectional self-attention score between any two EDUs in  $M_A$ . For example, in Figure 1, we aggregate the scores in cells  $M_A[0:1, 5:6]$  to compute the final output of cell  $[0, 2]$  (purple matrix in Figure 1) and  $M_A[5:6, 0:1]$  to generate the value of cell  $[0, 2]$ . This way, we obtain the average bidirectional self-attention scores between  $EDU_1$  and  $EDU_3$ . We use the resulting  $n \times n$  matrix as the input to the CKY/Eisner discourse tree generation methods.
4.1 Pre-Trained Models
We select the BERT-base (110 million parameters) and BART-large (406 million parameters) models for our experiments. We choose these models for their diverse objectives (encoder-only vs. encoder-decoder), popularity for diverse fine-tuning tasks, and their prior successful exploration in regards to discourse information (Zhu et al., 2020; Koto et al., 2021a). For the BART-large model, we limit our analysis to the encoder, as motivated in Koto et al. (2021a), leaving experiments with the decoder and cross-attention for future work.
For more information on the general tree-generation approach using the Eisner algorithm we refer interested readers to Xiao et al. (2021b).
3We omit the self-attention indexes for better readability.
Dataset	Task	Domain
IMDB(2014)	Sentiment	Movie Reviews
Yelp(2015)	Sentiment	Reviews
SST-2(2013)	Sentiment	Movie Reviews
MNLI(2018)	NLI	Range of Genres
CNN-DM(2016)	Summarization	News
XSUM(2018)	Summarization	News
SQuAD(2016)	Question-Answering	Wikipedia
Table 1: The seven fine-tuning datasets used in this work along with the underlying tasks and domains.
4.2 Fine-Tuning Tasks and Datasets
We explore the BERT model fine-tuned on two classification tasks, namely sentiment analysis and natural language inference (NLI). For our analysis on BART, we select the abstractive summarization and question answering tasks. Table 1 summarizes the 7 datasets used to fine-tune PLMs in this work, along with their underlying tasks and domains<sup>4</sup>.
4.3 Evaluation Treebanks
RST-DT (Carlson et al., 2002) is the largest English RST-style discourse treebank, containing 385 Wall-Street-Journal articles, annotated with full constituency discourse trees. To generate additional dependency trees, we apply the conversion algorithm proposed in Li et al. (2014).
GUM (Zeldes, 2017) is a steadily growing treebank of richly annotated texts. In the current version 7.3, the dataset contains 168 documents from 12 genres, annotated with full RST-style constituency and dependency discourse trees.
All evaluations shown in this paper are executed on the 38 and 20 documents in the RST-DT and GUM test-sets, to be comparable with previous baselines and supervised models. A similarly-sized validation-set is used where mentioned to determine the best performing self-attention head.
4.4 Baselines and Evaluation Metrics
Simple Baselines: We compare the inferred constituency trees against right- and left-branching structures. For dependency trees, we evaluate against simple chain and inverse chain structures.
Distantly Supervised Baselines: We compare our results obtained in this paper against our previous approach presented in Xiao et al. (2021b), using similar CKY and Eisner tree-generation methods to infer constituency and dependency tree structures
(a) BERT: PLM, +IMDB, +Yelp, +SST-2, +MNLI
(b) BART: PLM, +CNN-DM, +XSUM, +SQuAD
Figure 2: Constituency (top) and dependency (bottom) discourse tree evaluation of BERT (a) and BART (b) models on GUM. Purple=high score, Blue=low score. Left-to-right: self-attention heads, top-to-bottom: high layers to low layers. + indicates fine-tuning dataset.
from a summarization model trained on the CNNDM and New York Times (NYT) corpora (referred to as  $\text{Sum}_{\text{CNN-DM}}$  and  $\text{Sum}_{\text{NYT}}$ )<sup>5</sup>.
Supervised Baseline: We select the popular Two-Stage discourse parser (Wang et al., 2017) as our supervised baseline, due to its strong performance, available model checkpoints and code<sup>6</sup>, as well as the traditional architecture. We use the published Two-Stage parser checkpoint on RST-DT (from here on called Two-StageRST-DT) and re-train the discourse parser on GUM (Two-StageGUM). We convert the generated constituency structures into dependency trees following Li et al. (2014).
Evaluation Metrics: We apply the original parseval score to compare discourse constituency structures with gold-standard treebanks, as argued in Morey et al. (2017). To evaluate the generated dependency structures, we use the Unlabeled Attachment Score (UAS).
5.1 Discourse Locality
Our discourse tree generation approach described in section 3 directly uses self-attention matrices to generate discourse trees. The standard BERT
4We exclusively analyze published models provided on the huggingface platform, further specified in Appendix A.
5www.github.com/Wendy-Xiao/summ Guided_disco_parser
$^{6}$ www.github.com/yizhongw/StageDP
model contains 144 of those self-attention matrices (12 layers, 12 self-attention heads each), all of which potentially encode discourse structures. For the BART model, this number is even higher, consisting of 12 layers with 16 self-attention heads each. With prior work suggesting the locality of discourse information in PLMs (e.g., Raganato and Tiedemann (2018); Mareček and Rosa (2019); Xiao et al. (2021b)), we analyze every self-attention matrix individually to gain a better understanding of their alignment with discourse information.
Besides investigating standard PLMs, we also explore the robustness of discourse information across fine-tuning tasks. We believe that this is an important step to better understand if the captured discourse information is general and robust, or if it is "re-learned" from scratch for downstream tasks. To the best of our knowledge, no previous analysis of this kind has been performed in the literature.
To this end, Figure 2 shows the constituency and dependency structure overlap of the generated discourse trees from individual self-attention heads with the gold-standard tree structures of the GUM dataset<sup>7</sup>. The heatmaps clearly show that constituency discourse structures are mostly captured in higher layers, while dependency structures are more evenly distributed across layers. Comparing the patterns between models, we find that, despite being fine-tuned on different downstream tasks, the discourse information is consistently encoded in the same self-attention heads. Even though the best performing self-attention matrix is not consistent, discourse information is clearly captured in a "local" subset of self-attention heads across all presented fine-tuning tasks. This plausibly suggests that the discourse information in pre-trained BERT and BART models is robust and general, requiring only minor adjustments depending on the fine-tuning task.
5.2 Discourse Quality
We now focus on assessing the discourse information captured in the single best-performing self-attention head. In Table 2, we compare the discourse structure quality of pre-trained and finetuned PLMs in the context of supervised models, distantly supervised approaches and simple baselines. We show the oracle-picked best head on the test-set, analyzing the upper-bound for the poten
Model	RST-DT	GUM
Span	UAS	Span	UAS
BERT
rand. init	↓25.5	↓13.3	↓23.2	↓12.4
PLM	●35.7	●45.3	●33.0	●45.2
+IMDB	↓35.4	↓42.8	●33.0	↓43.3
+Yelp	↓34.7	↓42.3	↓32.6	↓43.7
+SST-2	↓35.5	↓42.9	↓32.6	↓43.5
+MNLI	↓34.8	↓41.8	↓32.4	↓43.3
BART
rand. init	↓25.3	↓12.5	↓23.2	↓12.2
PLM	●39.1	●41.7	●31.8	●41.8
+CNN-DM	↑40.9	↑44.3	↑32.7	↑42.8
+XSUM	↑40.1	↑41.9	↑32.1	↓39.9
+SQuAD	↑40.1	↑43.2	↓31.3	↓40.7
Baselines
RB/Chain	9.3	40.4	9.4	41.7
LB/Chain-1	7.5	12.7	1.5	12.2
SumCNN-DM	21.4	20.5	17.6	15.8
SumNYT	24.0	15.7	18.2	12.6
Two-StageRST-DT	72.0	71.2	54.0	54.5
Two-StageGUM	65.4	61.7	58.6	56.7