<figure><img src="image_1.png" /></figure>

HAL
open science

# Discourse Structure Extraction from Pre-Trained and Fine-Tuned Language Models in Dialogues

Chuyuan Li, Patrick Huber, Wen Xiao, Maxime Amblard, Chloé Braud,
Giuseppe Carenini

► To cite this version:

Chuyuan Li, Patrick Huber, Wen Xiao, Maxime Amblard, Chloé Braud, et al. Discourse Structure Extraction from Pre-Trained and Fine-Tuned Language Models in Dialogues. Findings of the Association for Computational Linguistics: EACL 2023, May 2023, Dubrovnik, Croatia. pp.2562–2579, 10.18653/v1/2023.findings-eacl.194 . hal-04031267

HAL Id: hal-04031267

https://hal.univ-lorraine.fr/hal-04031267

Submitted on 15 Jun 2023

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Discourse Structure Extraction from Pre-Trained and Fine-Tuned Language Models in Dialogues

Chuyuan Li¹, Patrick Huber², Wen Xiao²

Maxime Amblard¹, Chloé Braud³, Giuseppe Carenini²

¹ Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France

² University of British Columbia, V6T 1Z4, Vancouver, BC, Canada

³ IRIT, Université de Toulouse, CNRS, ANITI, Toulouse, France

¹{firstname.name}@loria.fr, ³chloe.braud@irit.fr,

²{huberpat, xiaowen3, carenini}@cs.ubc.ca

## Abstract

Discourse processing suffers from data sparsity, especially for dialogues. As a result, we explore approaches to build discourse structures for dialogues, based on attention matrices from Pre-trained Language Models (PLMs). We investigate multiple tasks for fine-tuning and show that the dialogue-tailored Sentence Ordering task performs best. To locate and exploit discourse information in PLMs, we propose an unsupervised and a semi-supervised method. Our proposals thereby achieve encouraging results on the STAC corpus, with $F_1$ scores of 57.2 and 59.3 for the unsupervised and semi-supervised methods, respectively. When restricted to projective trees, our scores improved to 63.3 and 68.1.

## 1 Introduction

In recent years, the availability of accurate transcription methods and the increase in online communication have led to a vast rise in dialogue data, necessitating the development of automatic analysis systems. For example, summarization of meetings or exchanges with customer service agents could be used to enhance collaborations or analyze customers issues (Li et al., 2019; Feng et al., 2021); machine reading comprehension in the form of question-answering could improve dialogue agents' performance and help knowledge graph construction (He et al., 2021; Li et al., 2021). However, simple surface-level features are oftentimes not sufficient to extract valuable information from conversations (Qin et al., 2017). Rather, we need to understand the semantic and pragmatic relationships organizing the dialogue, for example through the use of discourse information.

Along this line, several discourse frameworks have been proposed, underlying a variety of annotation projects. For dialogues, data has been primarily annotated within the Segmented Discourse Representation Theory (SDRT) (Asher et al., 2003).

<figure><img src="image_3.png" /><figcaption>Figure 1: Excerpt of dependency structures in file s2-leagueM-game4, STAC. Red links are non-projective.</figcaption></figure>

Discourse structures are thereby represented as dependency graphs with arcs linking spans of text and labeled with semantico-pragmatic relations (e.g. Acknowledgment (Ack) or Question-Answer Pair (QAP)). Figure 1 shows an example from the Strategic Conversations corpus (STAC) (Asher et al., 2016). Discourse processing refers to the retrieval of the inherit structure of coherent text, and is often separated into three tasks: EDU segmentation, structure building (or attachment), and relation prediction. In this work, we focus on the automatic extraction of (naked) structures without discourse relations. This serves as a first critical step in creating a full discourse parser. It is important to note that naked structures have already been shown to be valuable features for specific tasks. Louis et al. (2010) mentioned that they are the most reliable indicator of importance in content selection. Xu et al. (2020); Xiao et al. (2020) on summarization, and Jia et al. (2020) on thread extraction, also demonstrated the advantages of naked structures.

Data sparsity has always been an issue for discourse parsing both in monologues and dialogues: the largest and most commonly used corpus annotated under the Rhetorical Structure Theory, the RST-DT (Carlson et al., 2001) contains 21,789 discourse units. In comparison, the largest dialogue discourse dataset (STAC) only contains 10,678
units. Restricted to domain and size, the perfor-
mance of supervised discourse parsers is still low,
especially for dialogues, with at best 73.8% F₁ for
the naked structure on STAC (Wang et al., 2021).
As a result, several transfer learning approaches
have been proposed, mainly focused on mono-
logues. Previous work demonstrate that discourse
information can be extracted from auxiliary tasks
like sentiment analysis (Huber and Carenini, 2020)
and summarization (Xiao et al., 2021), or repre-
sented in language models (Koto et al., 2021) and
further enhanced by fine-tuning tasks (Huber and
Carenini, 2022). Inspired by the latter approaches,
we are pioneering in addressing this issue for dia-
logues and introducing effective semi-supervised
and unsupervised strategies to uncover discourse
information in large pre-trained language models
(PLMs). We find, however, that the monologue-
inspired fine-tuning tasks are not performing well
when applied to dialogues. Dialogues are generally
less structured, interspersed with more informal
linguistic usage (Sacks et al., 1978), and have struc-
tural particularities (Asher et al., 2016). Thus, we
propose a new Sentence Ordering (SO) fine-tuning
task tailored to dialogues. Building on the pro-
posal in Barzilay and Lapata (2008), we propose
crucial, dialogue-specific extensions with several
novel shuffling strategies to enhance the pair-wise,
inter-speech block, and inter-speaker discourse in-
formation in PLMs, and demonstrate its effective-
ness over other fine-tuning tasks.

In addition, a key issue in using PLMs to ex-
tract document-level discourse information is how
to choose the best attention head. We hypothe-
size that the location of discourse information in
the network may vary, possibly influenced by the
length and complexity of the dialogues. Therefore,
we investigate methods that enables us to evaluate
each attention head individually, in both unsuper-
vised and semi-supervised settings. We introduce
a new metric called “Dependency Attention Sup-
port” (DAS), which measures the level of support
for the dependency trees generated by a specific
self-attention head, allowing us to select the opti-
mal head without any need for supervision. We
also propose a semi-supervised approach where a
small validation set is used to choose the best head.

Experimental results on the STAC dataset reveal
that our unsupervised and semi-supervised methods
outperform the strong LAST baseline (F₁ 56.8%,
Sec. 4), delivering substantial gains on the com-

plete STAC dataset (F₁ 59.3%, Sec. 5.2) and show
further improvements on the tree-structured subset
(F₁ 68.1%, Sec. 6.3).

To summarize, our contributions in this work are:
(1) Discourse information detection in pre-trained
and sentence ordering fine-tuned LMs; (2) Unsuper-
vised and semi-supervised methods for discourse
structure extraction from the attention matrices in
PLMs; (3) Detailed quantitative and qualitative
analysis of the extracted discourse structures.

## 2 Related Work

Discourse structures for complete documents have
been mainly annotated within the Segmented Dis-
course Representation Theory (SDRT) (Asher et al.,
2003) or the Rhetorical Structure Theory (RST)
(Mann and Thompson, 1988), with the latter lead-
ing to the largest corpora and many discourse
 parsers for monologues, while SDRT is the main
 theory for dialogue corpora, i.e., STAC (Asher
 et al., 2016) and Molweni (Li et al., 2020). In
 SDRT, discourse structures are dependency graphs
 with possibly non-projective links (see Figure 1)
 compared to constituent trees structures in RST.
 Early approaches to discourse parsing on STAC
 used varied decoding strategies, such as Maximum
 Spanning Tree algorithm (Muller et al., 2012; Li
 et al., 2014; Afantenos et al., 2012) or Integer Lin-
 ear Programming (Perret et al., 2016). Shi and
 Huang (2019) first proposed a neural architecture
 based on hierarchical Gated Recurrent Unit (GRU)
 and reported 73.2% F₁ on STAC for naked struc-
tures. Recently, Wang et al. (2021) adopted Graph
 Neural Networks (GNNs) and reported marginal
 improvements on the same test set (73.8% F₁).

Data sparsity being the issue, a new trend to-
wards semi-supervised and unsupervised discourse
parsing has emerged, almost exclusively for mono-
logues. Huber and Carenini (2019, 2020) leveraged
sentiment information and showed promising re-
sults in cross-domain setting with the annotation of
a silver-standard labeled corpus. Xiao et al. (2021)
extracted discourse trees from neural summarizers
and confirmed the existence of discourse informa-
tion in self-attention matrices. Another line of work
proposed to enlarge training data with a combina-
tion of several parsing models, as done in Jiang
et al. (2016); Kobayashi et al. (2021); Nishida and
Matsumoto (2022). In a fully unsupervised set-
ting, Kobayashi et al. (2019) used similarity and
dissimilarity scores for discourse tree creation, a
method that can not be directly used for discourse
graphs though. As for dialogues, transfer learn-
ing approaches are rare. Badene et al. (2019a,b)
investigated a weak supervision paradigm where
expert-composed heuristics, combined to a gen-
erative model, are applied to unseen data. Their
method, however, requires domain-dependent an-
notation and a relatively large validation set for
rule verification. Another study by Liu and Chen
(2021) focused on cross-domain transfer using
STAC (conversation during online game) and Mol-
weni (Ubuntu forum chat logs). They applied sim-
ple adaptation strategies (mainly lexical informa-
tion) on a SOTA discourse parser and showed im-
provement compared to bare transfer: trained on
Molweni and tested on STAC F₁ increased from
42.5% to 50.5%. Yet, their model failed to surpass
simple baselines. Very recently, Nishida and Mat-
sumoto (2022) investigated bootstrapping methods
to adapt BERT-based parsers to out-of-domain data
with some success. In comparison to all this previ-
ous work, to the best of our knowledge, we are the
first to propose a fully unsupervised method and its
extension to a semi-supervised setting.

As pre-trained language models such as BERT
(Devlin et al., 2019), BART (Lewis et al., 2020) or
GPT-2 (Radford et al., 2019) are becoming dom-
inant in the field, *BERTology* research has gained
much attention as an attempt to understand what
kind of information these models capture. Prob-
ing tasks, for instance, can provide fine-grained
analysis, but most of them only focus on sentence-
level syntactic tasks (Jawahar et al., 2019; Hewitt
and Manning, 2019; Mareček and Rosa, 2019; Kim
et al., 2019; Jiang et al., 2020). As for discourse,
Zhu et al. (2020) and Koto et al. (2021) applied
probing tasks and showed that BERT and BART
encoders capture more discourse information than
other models, like GPT-2. Very recently, Huber
and Carenini (2022) introduced a novel way to en-
code long documents and explored the effect of dif-
ferent fine-tuning tasks on PLMs, confirming that
pre-trained and fine-tuned PLMs both can capture
discourse information. Inspired by these studies on
monologues, we explore the use of PLMs to extract
discourse structures in dialogues.

3 Method: from Attention to Discourse

**3.1 Problem Formulation and Simplifications**

Given a dialogue *D* with *n* *Elementary Discourse Units* (EDUs) {*e*₁, *e*₂, *e*₃, ..., *e*ₙ}, which are the

<figure><img src="image_4.png" /><figcaption>Figure 2: Pipeline for discourse structure extraction.</figcaption></figure>

minimal spans of text (mostly clauses, at most
a sentence) to be linked by discourse relations,
the goal is to extract a Directed Acyclic Graph
(DAG) connecting the *n* EDUs that best represents
its SDRT discourse structure from attention matri-
ces in PLMs<fnref n="1" /> (see Figure 2 for an overview of the
process). In our proposal, we make a few simplifi-
cations, partially adopted from previous work. We
do not deal with SDRT *Complex Discourse Units*
(CDUs) following Muller et al. (2012) and Afan-
tenos et al. (2015), and do not tackle relation type
assignment. Furthermore, similar to Shi and Huang
(2019), our solution can only generate discourse
trees. Extending our algorithm to non-projective
trees (≈ 6% of edges are non-projectives in tree-
like examples) and graphs (≈ 5% of nodes with
multiple incoming arcs) is left as future work.

**3.2 Which kinds of PLMs to use?**

We explore both vanilla and fine-tuned PLMs, as
they were both shown to contain discourse informa-
tion for monologues (Huber and Carenini, 2022).

**Pre-Trained Models:** We select BART (Lewis et al., 2020), not only because its encoder has been shown to effectively capture discourse information, but also because it dominated other alternatives in preliminary experiments, including DialoGPT (Zhang et al., 2020) and DialogLM (Zhong et al., 2022) - language models pre-trained with conversational data².

<footnote n="1">¹For more details on extracting discourse information from attention mechanisms see Liu and Lapata (2018).
²See Appendix E for additional results with other PLMs.</footnote>
<figure><img src="image_5.png" /><figcaption>Figure 3: Shuffling strategies (left to right: partial, minimal-pair, block, speaker-turn) on a sequence of utterances 1 to 6, with A, B, C as the speakers.</figcaption></figure>

**Fine-Tuning Tasks:** We fine-tune BART on three discourse-related tasks:

(1) **Summarization:** we use BART fine-tuned on the popular CNN-DailyMail (CNN-DM) news corpus (Nallapati et al., 2016), as well as on the SAMSum dialogue corpus (Gliwa et al., 2019).

(2) **Question Answering:** we use BART fine-tuned on the latest version of the Stanford Question Answering Dataset (SQuAD 2.0) (Rajpurkar et al., 2018).

(3) **Sentence Ordering:** we fine-tune BART on the Sentence Ordering task – reordering a set of shuffled sentences to their original order. We use an in-domain and an out-of-domain dialogue datasets (Sec. 4) for this task. Since fully random shuffling showed very limited improvements, we considered additional strategies to support a more gradual training tailored to dialogues. Specifically, as shown in Figure 3, we explore: (a) *partial-shuf*: randomly picking 3 utterances in a dialogue (or 2 utterances if the dialogue is shorter than 4) and shuffling them while maintaining the surrounding context. (b) *minimal-pair-shuf*: shuffling minimal pairs, comprising of a pair of speech turns from 2 different speakers with at least 2 utterances. A speech turn marks the start of a new speaker's turn in the dialogue. (c) *block-shuf*: shuffling a block containing multiple speech turns. We divide one dialogue into [2, 5] blocks based on the number of utterances<fnref n="3" /> and shuffle between blocks. (d) *speaker-turn-shuf*: grouping all speech productions of one speaker together. The sorting task consists of ordering speech turns from different speakers' production. We evenly combine all permutations mentioned above to create our **mixed-shuf** data set and conduct the SO task.

**Choice of Attention Matrix:** The BART model contains three kinds of attention matrices: encoder, decoder and cross attention. We use the encoder attention in this work, since it has been shown to capture most discourse information (Koto et al., 2021) and outperformed the other alternatives in preliminary experiments on a validation set.

## 3.3 How to derive trees from attention heads?

Given an attention matrix $A^t \in \mathbb{R}^{k \times k}$ where $k$ is the number of tokens in the input dialogue, we derive the matrix $A^{edu} \in \mathbb{R}^{n \times n}$, with $n$ the number of EDUS, by computing $A^{edu}(i, j)$ as the average of the submatrix of $A^t$ corresponding to all the tokens of EDUS $e_i$ and $e_j$, respectively. As a result, $A^{edu}$ captures how much EDU $e_i$ depends on EDU $e_j$ and can be used to generate a tree connecting all EDUS by maximizing their dependency strength. Concretely, we find a Maximum Spanning Tree (MST) in the fully-connected dependency graph $A^{edu}$ using the Eisner algorithm (Eisner, 1996). Conveniently, since an utterance cannot be anaphorically and rhetorically dependent on following utterances in a dialogue, as they are previously unknown (Afantenos et al., 2012), we can further simplify the inference by applying the following hard constraint to remove all backward links from the attention matrix $A^{edu}$: $a_{ij} = 0$, if $i > j$.

## 3.4 How to find the best heads?

Xiao et al. (2021) and Huber and Carenini (2022) showed that discourse information is not evenly distributed between heads and layers. However, they do not provide a strategy to select the head(s) containing most discourse information. Here, we propose two effective selection methods: fully unsupervised or semi-supervised.

### 3.4.1 Unsupervised Best Head(s) Selection

**Dependency Attention Support Measure (DAS):** Loosely inspired by the confidence measure in Nishida and Matsumoto (2022), where the authors define the confidence of a teacher model based on predictive probabilities of the decisions made, we propose a DAS metric measuring the degree of support for the maximum spanning (dependency) tree (MST) from the attention matrix. Formally, given a dialogue *g* with *n* EDUS, we first derive the EDU matrix $A^{edu}$ from its attention matrix $A^g$ (see Sec. 3.3). We then build the MST $T^g$ by selecting $n-1$ attention links $l_{ij}$ from $A^{edu}$ based on the tree generation algorithm. DAS measures the strength

<footnote n="3">³Block size is designed to be as twice or 3 times bigger than “min-pair”, we thus set criteria aiming to have ≈ 6 EDUS per block: |utt.| < 12 : b = 2, |utt.| ∈ [12, 22] : b = 3, |utt.| ∈ [22, 33] : b = 4, |utt.| ≥ 33 : n = 5.</footnote>
of all those connections by computing the average score of all the selected links:

$$DAS(T^g) = \frac{1}{n-1} \sum_{i=1}^{n} \sum_{j=1}^{n} Sel(A^g, i, j) \quad (1)$$

with $Sel(A^g, i, j) = A_{ij}^g$, if $l_{ij} \in T^g$, 0 otherwise. Note that DAS can be easily adapted for a general graph by removing the restriction to $n-1$ arcs.

**Selection Strategy:** With DAS, we can now compute the degree of support from each attention head $h$ on each single example $g$ for the generated tree $DAS(T_h^g)$. We therefore propose two strategies to select attention heads based on the DAS measure, leveraging either global or local support. The **global** support strategy selects the head with highest averaged DAS score over all the data examples:

$$H_{global} = \arg \max_{h} \sum_{g=1}^{M} DAS(T_{h}^{g}) \quad (2)$$

where $M$ is the number of examples. In this way, we select the head that has a generally good performance on the target dataset.

The second strategy is more adaptive to each document, by only focusing on the **local** support. It does not select one specific head for the whole dataset, but instead selects the head/tree with the highest support for each single example $g$, i.e.,

$$H_{local}^{g} = \arg \max_{h} DAS(T_{h}^{g}) \quad (3)$$

### 3.4.2 Semi-Supervised Best Head(s) Selection

We also propose best heads selection using a few annotated examples. In conformity with real-world situations where labeled data is scarce, we sample three small subsets with {10, 30, 50} data points (i.e., dialogues) from the validation set. We examine every attention matrix individually, resulting in 12 layers × 16 heads candidate matrices for each dialogue. Then, the head with the highest micro-F₁ score on the validation set is selected to derive trees in the test set. We also consider layer-wise aggregation, with details in Appendix A.

## 4 Experimental Setup

**Datasets:** We evaluate our approach on predicting discourse dependency structures using the STAC corpus (Asher et al., 2016), a multi-party dialogue dataset annotated in the SDRT framework. For the summarization and question-answering

<table><caption>Table 1: Key statistics of datasets. Utt = sentences in DD or EDUS in STAC; Tok = tokens; Spk = speakers.</caption><thead><tr><th>Dataset</th><th>#Doc</th><th>#Utt/doc</th><th>#Tok/doc</th><th>#Spk/doc</th><th>Domain</th></tr></thead><tbody><tr><td>DailyDialog</td><td>13, 118</td><td>13</td><td>119</td><td>2</td><td>Daily</td></tr><tr><td>STAC</td><td>1, 161</td><td>11</td><td>50</td><td>3</td><td>Game</td></tr></tbody></table>

fine-tuning tasks, we use publicly available HuggingFace models (Wolf et al., 2020) (see Appendix F). For the novel sentence ordering task, we train BART model on the STAC corpus and the DailyDialog corpus (Li et al., 2017). The key statistics for STAC and DailyDialog can be found in Table 1. These datasets are split into train, validation, and test sets at 82%, 9%, 9% and 85%, 8%, 8% respectively. The Molweni corpus (Li et al., 2020) is not included in our experiments due to quality issues, as detailed in Appendix B.

**Baselines:** We compare against the simple yet strong unsupervised LAST baseline (Schegloff, 2007), attaching every EDU to the previous one. Furthermore, to assess the gap between our approach and supervised dialogue discourse parsers, we compare with the Deep Sequential model by Shi and Huang (2019) and the Structure Self-Aware (SSA) model by Wang et al. (2021).

**Metrics:** We report the micro-F₁ and the Unlabeled Attachment Score (UAS) for the generated naked dependency structures.

**Implementation Details:** We base our work on the transformer implementations from the Hugging-Face library (Wolf et al., 2020) and follow the text-to-marker framework proposed in Chowdhury et al. (2021) for the SO fine-tuning procedure. We use the original separation of train, validation, and test sets; set the learning rate to 5e-6; use a batch size of 2 for DailyDialog and 4 for STAC, and train for 7 epochs. All other hyper-parameters are set following Chowdhury et al. (2021). We do not do any hyper-parameter tuning. We omit 5 documents in DailyDialog during training since the documents lengths exceed the token limit. We replace speaker names with markers (e.g. Sam → “spk1”), following the preprocessing pipeline for dialogue utterances in PLMs.

## 5 Results

### 5.1 Results with Unsupervised Head Selection

Results using our novel unsupervised DAS method on STAC are shown in Table 2 for both the global
(H<sub>g</sub>) and local (H<sub>l</sub>) head selection strategies. These are compared to: (1) the unsupervised LAST baseline (at the top), which only predicts local attachments between adjacent EDUs. LAST is considered a strong baseline in discourse parsing (Muller et al., 2012), but has the obvious disadvantage of completely missing long-distance dependencies which may be critical in downstream tasks. (2) The supervised Deep Sequential parser by Shi and Huang (2019) and Structure Self-Aware model by Wang et al. (2021) (center of the table), both trained on STAC, reaching resp. 71.4%<sup>4</sup> and 73.8% in F<sub>1</sub>.

In the last sub-table we show unsupervised scores from pre-trained and fine-tuned LMs on three auxiliary tasks: summarization, question- answering and sentence ordering (SO) with the mixed shuffling strategy. We present the global head (H<sub>g</sub>) and local heads (H<sub>l</sub>) performances se- lected by the DAS score (see section 3.4.1). The best possible scores using an oracle head selector (H<sub>ora</sub>) are presented for reference.

Comparing the values in the bottom sub-table, we find that the pre-trained BART model under- performs LAST (56.8), with global head and lo- cal heads achieving similar performance (56.6 and 56.4 resp.). Noticeably, models fine-tuned on the summarization task (“+CNN”, "+SAMSum") and question-answering (“+SQuAD2”) only add marginal improvements compared to BART. In the last two lines of the sub-table, we explore our novel sentence ordering fine-tuned BART models. We find that the BART+SO approach surpasses LAST when using local heads (57.1 and 57.2 for Daily- Dialog and STAC resp.). As commonly the case, the intra-domain training performs best, which is further strengthened in this case due to the spe- cial vocabulary in STAC. Importantly, our PLM- based unsupervised parser can capture some long- distance dependencies compared to LAST (Sec- tion 6.2). Additional analysis regarding the chosen heads is in Section 6.1.

5.2 Results with Semi-Sup. Head Selection

While the unsupervised strategy only delivered min-
imal improvements over the strong LAST base-
line, Table 3 shows that if a few annotated exam-
ples are provided, it is possible to achieve substan-
tial gains. In particular, we report results on the
vanilla BART model, as well as BART model fine-

<table><caption>Table 2: Micro-F<sub>1</sub> on STAC for LAST, supervised SOTA models and unsupervised PLMs. H<sub>g</sub>/H<sub>l</sub>/H<sub>ora</sub>: global/local/oracle heads. Best (non-oracle) score in the 3<sup>rd</sup> block is in bold. DD: DailyDialog.</caption>
  <thead>
    <tr>
      <th colspan="2">Model</th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td colspan="2">Unsupervised Baseline</td>
      <td></td>
      <td></td>
    </tr>
    <tr>
      <td colspan="2">LAST</td>
      <td></td>
      <td>56.8</td>
    </tr>
    <tr>
      <td colspan="2">Supervised Models</td>
      <td></td>
      <td></td>
    </tr>
    <tr>
      <td colspan="2">Deep-Sequential (2019)</td>
      <td></td>
      <td>71.4</td>
    </tr>
    <tr>
      <td colspan="2">SSA-GNN (2021)</td>
      <td></td>
      <td>73.8</td>
    </tr>
    <tr>
      <td rowspan="2">Unsupervised PLMs</td>
      <td></td>
      <th>H<sub>g</sub></th>
      <th>H<sub>l</sub></th>
      <th>H<sub>ora</sub></th>
    </tr>
    <tr>
      <td>BART</td>
      <td>56.6</td>
      <td>56.4</td>
      <td>57.6</td>
    </tr>
    <tr>
      <td></td>
      <td>+ CNN</td>
      <td>56.8</td>
      <td>56.7</td>
      <td>57.1</td>
    </tr>
    <tr>
      <td></td>
      <td>+ SAMSum</td>
      <td>56.7</td>
      <td>56.6</td>
      <td>57.6</td>
    </tr>
    <tr>
      <td></td>
      <td>+ SQuAd2</td>
      <td>55.9</td>
      <td>56.4</td>
      <td>57.7</td>
    </tr>
    <tr>
      <td></td>
      <td>+ SO-DD</td>
      <td>56.8</td>
      <td>57.1</td>
      <td>58.2</td>
    </tr>
    <tr>
      <td></td>
      <td>+ SO-STAC</td>
      <td>56.7</td>
      <td><strong>57.2</strong></td>
      <td>59.5</td>
    </tr>
  </tbody>
</table>

<table><caption>Table 3: Micro-F<sub>1</sub> on STAC from BART and SO fine-
tuned BART with unsupervised and semi-supervised
approaches. Semi-supervised scores are averaged from
10 random runs. Subscription is standard deviation.</caption>
  <thead>
    <tr>
      <th>Train on →<br>Test with ↓</th>
      <th>BART<br>F<sub>1</sub></th>
      <th>+ SO-DD<br>F<sub>1</sub></th>
      <th>+ SO-STAC<br>F<sub>1</sub></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>LAST BSL</th>
      <td>56.8</td>
      <td>56.8</td>
      <td>56.8</td>
    </tr>
    <tr>
      <th>H<sub>ora</sub></th>
      <td>57.6</td>
      <td>58.2</td>
      <td>59.5</td>
    </tr>
    <tr>
      <th>Unsup H<sub>g</sub></th>
      <td><u>56.6</u></td>
      <td>56.8</td>
      <td>56.7</td>
    </tr>
    <tr>
      <th>Unsup H<sub>l</sub></th>
      <td>56.4</td>
      <td><u>57.1</u></td>
      <td><u>57.2</u></td>
    </tr>
    <tr>
      <th>Semi-sup 10</th>
      <td>57.0<sub>0.012</sub></td>
      <td>57.2<sub>0.012</sub></td>
      <td>57.1<sub>0.026</sub></td>
    </tr>
    <tr>
      <th>Semi-sup 30</th>
      <td>57.3<sub>0.005</sub></td>
      <td>57.3<sub>0.013</sub></td>
      <td>59.2<sub>0.009</sub></td>
    </tr>
    <tr>
      <th>Semi-sup 50</th>
      <td><strong>57.4</strong><sub>0.004</sub></td>
      <td><strong>57.7</strong><sub>0.005</sub></td>
      <td><strong>59.3</strong><sub>0.007</sub></td>
    </tr>
  </tbody>
</table>

tuned on DailyDialog (“+SO-DD”) and STAC itself (“+SO-STAC”). We execute 10 runs for each semi-supervised setting ([10, 30, 50]) and report average scores and the standard deviation.

The oracle heads (i.e., H<sub>ora</sub>) achieve superior per-
formance compared to LAST. Furthermore, using
a small scale validation set (50 examples) to se-
lect the best attention head remarkably improves
the F<sub>1</sub> score from 56.8% (LAST) to 59.3% (+SO-
STAC). F<sub>1</sub> improvements across increasingly large
validation-set sizes are consistent, accompanied by
smaller standard deviations, as would be expected.
The semi-supervised results are very encouraging:
with 30 annotated examples, we already reach a
performance close to the oracle result, and with
more examples we can further reduce the gap.

<footnote n="4">⁴We re-train the model, scores are slightly different due to different train-test splits, as in Wang et al. (2021).</footnote>
<figure><img src="image_6.png" /><figcaption>Figure 4: Heatmaps: DAS score matrices (layers: top to bottom=12 to 1, heads: left to right=1 to 16) for BART, BART+SO-DD, BART+SO-STAC. Darker purple=higher DAS score.</figcaption></figure>

Boxplot: Head-aggregated UAS scores for model BART (orange), BART+SO-DD (green) and BART+SO-STAC (red). Light green=head with highest UAS. Yellow=head with highest DAS score.

# 6 Analysis

## 6.1 Effectiveness of DAS

We now take a closer look at the performance degradation of our unsupervised approach based on DAS in comparison to the upper-bound defined by the performance of the oracle-picked head. To this end, Figure 4 shows the DAS score matrices (left) for three models with the oracle heads and DAS selected heads highlighted in green and yellow, respectively. These scores correspond to the global support strategy (i.e., $H_g$). It becomes clear that the oracle heads do not align with the DAS selected heads. Making a comparison between models, we find that discourse information is consistently located in deeper layers, with the oracle heads (light green) consistently situated in the same head for all three models. It is important to note that this information cannot be determined beforehand and can only be uncovered through a thorough examination of all attention heads.

While not aligning with the oracle, the top performing DAS heads (in yellow) are among the top 10% best heads in all three models, as shown in the box-plot on the right. Hence, we confirm that the DAS method is a reasonable approximation to find discourse intense self-attention heads among the 12 × 16 attention matrices.

## 6.2 Document and Arc Lengths

The inherent drawback of the simple, yet effective LAST baseline is its inability to predict indirect arcs. To test if our approach can reasonably predict distant arcs of different length in the dependency trees, we analyze our results in regards to the arc lengths. Additionally, since longer documents tend to contain more distant arcs, we also examine the performance across different document lengths.

**Arc Distance:** To examine the extracted discourse structures for data sub-sets with specific arc lengths, we present the UAS score plotted against different arc lengths on the left side in Figure 5. Our analysis thereby shows that direct arcs achieve high UAS score (>$ 80$%), independent of the model used. We further observe that the performance drops considerably for arcs of distance two and onwards, with almost all models failing to predict arcs longer than 6. BART+SO-STAC model correctly captures an arc of distance 13. Note that the presence for long-distance arcs (≥ 6) is limited, accounting for less than 5% of all arcs.

We further analyze the precision and recall scores when separating dependency links into *direct* (adjacent forward arcs) and *indirect* (all other non-adjacent arcs), following Xiao et al. (2021). For direct arcs, all models perform reasonably well (see Figure 6 at the bottom). The precision is higher (≈ +6% among all three BART models) and recall is lower than the baseline (100%), indicating that our models predict less direct arcs but more precisely. For indirect arcs (top in Figure 6), the best model is BART+SO-STAC (20% recall, 44% precision), closely followed by original BART (20% recall, 41% precision). In contrast, the LAST baseline model completely fails in this scenario (0 precision and recall).

**Document Length:** Longer documents tend to be more difficult to process because of the growing number of possible discourse parse trees. Hence, we analyze the UAS performance of documents in regards to their length, here defined as the number of EDUs. Results are presented on the right side in Figure 5, comparing the UAS scores for the three selected models and LAST for different document lengths. We split the document length range into 5 even buckets between the shortest (2 EDUs) and longest (37 EDUs) document, resulting in 60, 25, 16, 4 and 4 examples per bucket.

<figure><img src="image_7.png" /><figcaption>Figure 5: Left: UAS and arcs' distance. x axis: arc distance. Right: averaged UAS for different length of document. x axis: #EDUs in a document. y axis: UAS.</figcaption></figure>
<figure><img src="image_8.png" /><figcaption>Figure 6: Comparison of recall (left) and precision (right) scores of indirect (top) and direct (bottom) links in LAST, BART, and SO fine-tuned BART models.</figcaption></figure>

<table><caption>Table 4: STAC test set ground-truth tree and non-tree statistics. “Single-in” and “multi-in” means EDU with single or multiple incoming arcs.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">#Doc</th><th colspan="2">#EDUs</th><th colspan="2">#Arcs</th></tr><tr><th>Single-in</th><th>Multi-in</th><th>Proj.</th><th>N-proj.</th></tr></thead><tbody><tr><td>(1) Non-Tree</td><td>48</td><td>706</td><td>79</td><td>575</td><td>170</td></tr><tr><td>(2) Tree</td><td>61</td><td>444</td><td>0</td><td>348</td><td>35</td></tr><tr><td><strong>- Proj. tree</strong></td><td><strong>48</strong></td><td>314</td><td>0</td><td>266</td><td>0</td></tr></tbody></table>

For documents with less than 23 EDUs, all fine-tuned models perform better than LAST, with BART fine-tuned on STAC reaching the best result. We note that PLMs exhibit an increased capability to predict distant arcs in longer documents. However, in the range of [23, 30], the PLMs are inclined to predict a greater number of false positive distant arcs, leading to under-performance compared to the LAST baseline. As a result, we see that longer documents (≥ 23) are indeed more difficult to predict, with even the performance of our best model (BART+STAC) strongly decreasing.

## 6.3 Projective Trees Examination

Given the fact that our method only extracts projective tree structures, we now conduct an additional analysis, exclusively examining the subset of STAC containing projective trees, on which our method could in theory achieve perfect accuracy.

Table 4 gives some statistics for this subset (“proj. tree”). For the 48 projective tree examples, the document length decreases from an average of 11 to 7 EDUs, however, still contains ≈ 40% indirect arcs, keeping the task difficulty comparable. The scores for the extracted structures are presented in Table 5. As shown, all three unsupervised models outperform LAST. The best model is still BART fine-tuned on STAC, followed by the inter-domain

<table><caption>Table 5: Micro-F<sub>1</sub> scores on STAC projective tree subset with BART and SO fine-tuned BART models.</caption><thead><tr><th>Train on →<br>Test with ↓</th><th>BART<br>F<sub>1</sub></th><th>+ SO-DD<br>F<sub>1</sub></th><th>+ SO-STAC<br>F<sub>1</sub></th></tr></thead><tbody><tr><td>LAST BSL</td><td>62.0</td><td>62.0</td><td>62.0</td></tr><tr><td>H<sub>ora</sub></td><td>64.8</td><td>67.4</td><td>68.6</td></tr><tr><td>Unsup H<sub>g</sub></td><td><u>62.5</u></td><td>62.5</td><td>62.1</td></tr><tr><td>Unsup H<sub>l</sub></td><td>62.1</td><td><u>62.9</u></td><td><u>63.3</u></td></tr><tr><td>Semi-sup 10</td><td>54.6<sub>0.058</sub></td><td>59.2<sub>0.047</sub></td><td>61.6<sub>0.056</sub></td></tr><tr><td>Semi-sup 30</td><td>60.3<sub>0.047</sub></td><td>60.3<sub>0.044</sub></td><td>65.6<sub>0.043</sub></td></tr><tr><td>Semi-sup 50</td><td><strong>64.8</strong><sub>0.000</sub></td><td><strong>66.3</strong><sub>0.023</sub></td><td><strong>68.1</strong><sub>0.014</sub></td></tr></tbody></table>

<table><caption>Table 6: Statistics for gold and extracted projective trees in BART and fine-tuned BART models.</caption><thead><tr><th></th><th>Avgbranch</th><th>Avg. height</th><th>%leaf</th><th>Norm. arc</th></tr></thead><tbody><tr><td>GT</td><td>1.67</td><td>3.96</td><td>0.46</td><td>0.43</td></tr><tr><td>BART</td><td>1.20</td><td>5.31</td><td>0.31</td><td>0.34</td></tr><tr><td>+SO-DD</td><td>1.32<sub>0.014</sub></td><td>5.31<sub>0.146</sub></td><td>0.32<sub>0.019</sub></td><td>0.37<sub>0.003</sub></td></tr><tr><td>+SO-STAC</td><td>1.27<sub>0.076</sub></td><td>5.28<sub>0.052</sub></td><td>0.32<sub>0.011</sub></td><td>0.35<sub>0.015</sub></td></tr></tbody></table>

fine-tuned +SO-DD and BART models. Using the semi-supervised approach, we see further improvement with the F<sub>1</sub> score reaching 68% (+6% than LAST). Degradation for direct and indirect edges' precision and recall scores see Appendix C.

Following Ferracane et al. (2019), we analyze key properties of the 48 gold trees compared to our extracted structures using the semi-supervised method. To test the stability of the derived trees, we use three different seeds to generate the shuffled datasets to fine-tune BART. Table 6 presents the averaged scores and the standard deviation of the trees. In essence, while the extracted trees are generally “thinner” and “taller” than gold trees and contain slightly less branches, they are well aligned with gold discourse structures and do not contain “vacuous” trees, where all nodes are linked to one of the first two EDUs. Further qualitative analysis of inferred structures is presented in Appendix D. Tellingly, on two STAC examples our model succeeds in predicting > 82% of projective arcs, some of which span across 4 EDUs. This is encouraging, providing anecdotal evidence that our method is suitable to extract reasonable discourse structures.

## 6.4 Performance with Predicted EDUs

Following previous work, all our experiments have started with gold-standard EDU annotations. However, this would not be possible in a deployed dis-
<table><caption>Table 7: EDU segmentation results on STAC test set using DisCoDisCo model (Gessler et al., 2021), which is re-trained on 50 random dialogues from the validation set. Scores are averaged over three runs.</caption><thead><tr><th>Gold #</th><th>Predicted #</th><th>Precision %</th><th>Recall %</th><th>F<sub>1</sub> %</th></tr></thead><tbody><tr><td>1155</td><td>1081</td><td>96.0</td><td>93.4</td><td>94.8</td></tr></tbody></table>

<table><caption>Table 8: Gold EDUs and predicted EDUs parsing results with BART+SO-STAC model. Scores for predicted EDUs are averaged over three runs.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">LAST</th><th colspan="3">Unsupervised</th><th colspan="3">Semi-supervised</th></tr><tr><th>H<sub>g</sub></th><th>H<sub>l</sub></th><th>H<sub>ora</sub></th><th>semi-10</th><th>semi-30</th><th>semi-50</th></tr></thead><tbody><tr><th>Gold</th><td>56.8</td><td>56.7</td><td>57.2</td><td>59.5</td><td>57.4<sub>0.004</sub></td><td>57.7<sub>0.005</sub></td><td><strong>59.3<sub>0.007</sub></strong></td></tr><tr><th>Pred</th><td>48.9</td><td>50.8</td><td>51.1</td><td>52.6</td><td>50.6<sub>0.020</sub></td><td>52.1<sub>0.007</sub></td><td><strong>52.2<sub>0.004</sub></strong></td></tr></tbody></table>

course parser for dialogues. To assess the performance of such system, we conduct additional experiments in which we first perform EDU segmentation and then feed the predicted EDUs to our methods.

To perform EDU segmentation, we employ the DisCoDisCo model (Gessler et al., 2021), pretrained on a random sample of 50 dialogues from the STAC validation set. We repeat this process three times to accommodate instability. Our results, as shown in Table 7, align with those previously reported in Gessler et al. (2021) (94.9), with an F-score of 94.8. In the pre-training phase, we utilize all 12 hand-crafted features<fnref n="5" />, and opt for treebanked data for enhanced performance (94.9 compared to 91.9 for plain text data). The treebanked data is obtained using the Stanza Toolkit (Qi et al., 2020).

For evaluation, we adapt the discourse analysis pipeline proposed by Joty et al. (2015). The results are shown in Table 8, comparing the predicted and gold EDUs. The best head (i.e., H<sub>ora</sub>) performance decreases by ≈ 7 points, from 59.5 to 52.6, as well as unsupervised and semi-supervised results. Despite the drop, our unsupervised and semi-supervised models still outperform the LAST baseline. A similar loss of ≈ 6 points is also observed for RST-style parsing in monologues, as reported in Nguyen et al. (2021).

## 7 Conclusion

In this study, we explore approaches to build naked discourse structures from PLMs attention matrices to tackle the extreme data sparsity issue in dialogues. We show sentence ordering to be the best

fine-tuning task and our unsupervised and semi-supervised methods for selecting the best attention head outperform a strong baseline, delivering substantial gains especially on tree structures. Interestingly, discourse is consistently captured in deeper PLMs layers, and more accurately for shorter links.

In the near future, we intend to explore graph-like structures from attention matrices, for instance, by extending treelike structures with additional arcs of high DAS score and applying linguistically motivated constraints, as in Perret et al. (2016). We would also like to expand shuffling strategies for sentence ordering and to explore other auxiliary tasks. In the long term, our goal is to infer full discourse structures by incorporating the prediction of rhetorical relation types, all while remaining within unsupervised or semi-supervised settings.

## Limitations

Similarly to previous work, we have focused on generating only projective tree structures. This not only covers the large majority of the links (≈ 94%), but it can also provide the backbone for accurately inferring the remaining non-projective links in future work. We focus on the naked structure, as it is a significant first step and a requirement to further predict relations for discourse parsing.

We decided to run our experiments on the only existing high quality corpus, i.e., STAC. In essence, we traded-off generalizability for soundness of the results. A second corpus we considered, Molweni, had to be excluded due to serious quality issues.

Lastly, since we work with large language models and investigate every single attention head, computational efficiency is a concern. We used a 4-core GPU machine with the highest VRAM at 11MiB. The calculation for one discourse tree on one head was approximately 0.75 seconds (in STAC the averaged dialogue length is 11 EDUs), which quickly summed up to 4.5 hours with only 100 data points for 192 candidate trees in one LM. When dealing with much longer documents, for example AMI and conversational section in GUM (in average > 200 utterances/dialogue), our estimation shows that one dialogue takes up to ≈ 2 minutes, which means 6.5 hours for 192 candidate trees. Even though we use parallel computation, the exhaustive “head” computation results in a tremendous increase in time and running storage. One possibility is to investigate only those “discourse-rich” heads, mainly in the deeper layers, for future work.

<footnote n="5">⁵Such as POS tag, UD deprel, sentence length, etc..</footnote>
## Ethical Considerations

We carefully select the dialogue corpora used in this paper to control for potential biases, hate-speech and inappropriate language by using human annotated corpora and professionally curated resources. Further, we consider the privacy of dialogue partners in the selected datasets by replacing names with generic user tokens.

Since we are investigating the nature of the discourse structures captured in large PLMs, our work can be seen as making these models more transparent. This will hopefully contribute to avoid unintended negative effects, when the growing number of NLP applications relying on PLMs are deployed in practical settings.

In terms of environmental cost, the experiments described in the paper make use of RTX 2080 Ti GPUs for tree extraction and A100 GPUs for BART fine-tuning. We used up to 4 GPUs for the parallel computation. The experiments on corpus STAC took up to 1.2 hours for one language model, and we tested a dozen models. We note that while our work is based on exhaustive research on all the attention heads in PLMs to obtain valuable insights, future work will able to focus more on discourse-rich heads, which can help to avoid the quadratic growth of computation time for longer documents.

## Acknowledgements

The authors thank the anonymous reviewers for their insightful comments and suggestions. This work was supported by the PIA project "Lorraine Université d'Excellence", ANR-15-IDEX-04-LUE, as well as the CPER LCHN (Contrat de Plan État-Région - Langues, Connaissances et Humanités Numériques). It was partially supported by the ANR (ANR-19-PI3A-0004) through the AI Interdisciplinary Institute, ANITI, as a part of France's "Investing for the Future — PIA3" program, and through the project AnDiAMO (ANR-21-CE23-0020). It was also supported by the Language & Speech Innovation Lab of Cloud BU, Huawei Technologies Co., Ltd and the Natural Sciences and Engineering Research Council of Canada (NSERC). Nous remercions le Conseil de recherches en sciences naturelles et en génie du Canada (CRSNG) de son soutien. Experiments presented were carried out in clusters on the Grid'5000 testbed. We would like to thank the Grid'5000 community (https://www.grid5000.fr/).

## References

Stergos Afantenos, Nicholas Asher, Farah Benamara, Anais Cadilhac, Cedric Dégremont, Pascal Denis, Markus Guhe, Simon Keizer, Alex Lascarides, Oliver Lemon, et al. 2012. Modelling strategic conversation: model, annotation design and corpus. In *Proceedings of the 16th Workshop on the Semantics and Pragmatics of Dialogue (Seinedial), Paris*.

Stergos Afantenos, Eric Kow, Nicholas Asher, and Jérémy Perret. 2015. Discourse parsing for multi-party chat dialogues. In *Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing*, pages 928–937, Lisbon, Portugal. Association for Computational Linguistics.

Nicholas Asher, Nicholas Michael Asher, and Alex Lascarides. 2003. *Logics of conversation*. Cambridge University Press.

Nicholas Asher, Julie Hunter, Mathieu Morey, Benamara Farah, and Stergos Afantenos. 2016. Discourse structure and dialogue acts in multiparty dialogue: the STAC corpus. In *Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)*, pages 2721–2727, Portorož, Slovenia. European Language Resources Association (ELRA).

Sonia Badene, Kate Thompson, Jean-Pierre Lorré, and Nicholas Asher. 2019a. Data programming for learning discourse structure. In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 640–645, Florence, Italy. Association for Computational Linguistics.

Sonia Badene, Kate Thompson, Jean-Pierre Lorré, and Nicholas Asher. 2019b. Weak supervision for learning discourse structure. In *EMNLP*.

Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. *Computational Linguistics*, 34(1):1–34.

Lynn Carlson, Daniel Marcu, and Mary Ellen Okurovsky. 2001. Building a discourse-tagged corpus in the framework of Rhetorical Structure Theory. In *Proceedings of the Second SIGdial Workshop on Discourse and Dialogue*.

Somnath Basu Roy Chowdhury, Faeze Brahman, and Snigdha Chaturvedi. 2021. Is everything in order? a simple way to order sentences. In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pages 10769–10779.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jason Eisner. 1996. Three new probabilistic models for dependency parsing: An exploration. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics.

Xiachong Feng, Xiaocheng Feng, and Bing Qin. 2021. A survey on dialogue summarization: Recent advances and new frontiers. arXiv preprint arXiv:2107.03175.

Elisa Ferracane, Greg Durrett, Junyi Jessy Li, and Katrin Erk. 2019. Evaluating discourse in structured text representations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 646–653, Florence, Italy. Association for Computational Linguistics.

Luke Gessler, Shabnam Behzad, Yang Janet Liu, Siyao Peng, Yilun Zhu, and Amir Zeldes. 2021. Discodisco at the disrpt2021 shared task: A system for discourse segmentation, classification, and connective detection. In Proceedings of the 2nd Shared Task on Discourse Relation Parsing and Treebanking (DISRPT 2021), pages 51–62.

Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79, Hong Kong, China. Association for Computational Linguistics.

Yuchen He, Zhuosheng Zhang, and Hai Zhao. 2021. Multi-tasking dialogue comprehension with discourse parsing. In Proceedings of the 35th Pacific Asia Conference on Language, Information and Computation, pages 551–561, Shanghai, China. Association for Computational Linguistics.

John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138, Minneapolis, Minnesota. Association for Computational Linguistics.

Patrick Huber and Giuseppe Carenini. 2019. Predicting discourse structure using distant supervision from sentiment. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2306–2316, Hong Kong, China. Association for Computational Linguistics.

Patrick Huber and Giuseppe Carenini. 2020. MEGA RST discourse treebanks with structure and nuclearity from scalable distant sentiment supervision. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7442–7457, Online. Association for Computational Linguistics.

Patrick Huber and Giuseppe Carenini. 2022. Towards understanding large-scale discourse structures in pretrained and fine-tuned language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics.

Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651–3657, Florence, Italy. Association for Computational Linguistics.

Qi Jia, Yizhu Liu, Siyu Ren, Kenny Zhu, and Haifeng Tang. 2020. Multi-turn response selection using dialogue dependency relations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1911–1920.

Kailang Jiang, Giuseppe Carenini, and Raymond Ng. 2016. Training data enrichment for infrequent discourse relations. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2603–2614, Osaka, Japan. The COLING 2016 Organizing Committee.

Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438.

Shafiq Joty, Giuseppe Carenini, and Raymond T Ng. 2015. Codra: A novel discriminative framework for rhetorical analysis. Computational Linguistics, 41(3):385–435.

Taeuk Kim, Jihun Choi, Daniel Edmiston, and Sang-goo Lee. 2019. Are pre-trained language models aware of phrases? simple but strong baselines for grammar induction. In International Conference on Learning Representations.

Naoki Kobayashi, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2021. Improving neural rst parsing model with silver agreement subtrees. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1600–1612.

Naoki Kobayashi, Tsutomu Hirao, Kengo Nakamura, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2019. Split or merge: Which is better for unsupervised RST parsing? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5797–5802, Hong Kong, China. Association for Computational Linguistics.

Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2021. Discourse probing of pretrained language models. In Proceedings of the 2021 Conference of the North
*American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 3849–3864.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. *BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension*. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 7871–7880, Online. Association for Computational Linguistics.

Jiaqi Li, Ming Liu, Min-Yen Kan, Zihao Zheng, Zekun Wang, Wenqiang Lei, Ting Liu, and Bing Qin. 2020. *Molweni: A challenge multiparty dialogues-based machine reading comprehension dataset with discourse structure*. In *Proceedings of the 28th International Conference on Computational Linguistics*. pages 2642–2652, Barcelona, Spain (Online). International Committee on Computational Linguistics.

Jiaqi Li, Ming Liu, Zihao Zheng, Heng Zhang, Bing Qin, Min-Yen Kan, and Ting Liu. 2021. *Dadgraph: A discourse-aware dialogue graph neural network for multiparty dialogue machine reading comprehension*. *arXiv preprint arXiv:2104.12377*.

Manling Li, Lingyu Zhang, Heng Ji, and Richard J. Radke. 2019. *Keep meeting summaries on topic: Abstractive multi-modal meeting summarization*. In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 2190–2196, Florence, Italy. Association for Computational Linguistics.

Sujian Li, Liang Wang, Ziqiang Cao, and Wenjie Li. 2014. *Text-level discourse dependency parsing*. In *Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 25–35, Baltimore, Maryland. Association for Computational Linguistics.

Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. *DailyDialog: A manually labelled multi-turn dialogue dataset*. In *Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, pages 986–995, Taipei, Taiwan. Asian Federation of Natural Language Processing.

Yang Liu and Mirella Lapata. 2018. *Learning structured text representations. Transactions of the Association for Computational Linguistics*, 6:63–75.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. *Roberta: A robustly optimized bert pretraining approach*. *arXiv preprint arXiv:1907.11692*.

Zhengyuan Liu and Nancy Chen. 2021. *Improving multi-party dialogue discourse parsing via domain integration*. In *Proceedings of the 2nd Workshop on Computational Approaches to Discourse*, pages 122–127, Punta Cana, Dominican Republic and Online. Association for Computational Linguistics.

Annie Louis, Aravind K Joshi, and Ani Nenkova. 2010. *Discourse indicators for content selection in summarization*.

Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. *The Ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems*. In *Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue*, pages 285–294, Prague, Czech Republic. Association for Computational Linguistics.

William C Mann and Sandra A Thompson. 1988. *Rhetorical structure theory: Toward a functional theory of text organization*. *Text-interdisciplinary Journal for the Study of Discourse*, 8(3):243–281.

David Mareček and Rudolf Rosa. 2019. *From balustrades to pierre vinken: Looking for syntax in transformer self-attentions*. In *Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP*, pages 263–275, Florence, Italy. Association for Computational Linguistics.

Philippe Muller, Stergos Afantenos, Pascal Denis, and Nicholas Asher. 2012. *Constrained decoding for text-level discourse parsing*. In *Proceedings of COLING 2012*, pages 1883–1900, Mumbai, India. The COLING 2012 Organizing Committee.

Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gulçehre, and Bing Xiang. 2016. *Abstractive text summarization using sequence-to-sequence RNNs and beyond*. In *Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning*, pages 280–290, Berlin, Germany. Association for Computational Linguistics.

Thanh-Tung Nguyen, Xuan-Phi Nguyen, Shafiq Joty, and Xiaoli Li. 2021. *Rst parsing from scratch*. In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 1613–1625.

Noriki Nishida and Yuji Matsumoto. 2022. *Out-of-domain discourse dependency parsing via bootstrapping: An empirical analysis on its effectiveness and limitation*. *Transactions of the Association for Computational Linguistics*, 10:127–144.

Jérémy Perret, Stergos Afantenos, Nicholas Asher, and Mathieu Morey. 2016. *Integer linear programming for discourse parsing*. In *Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 99–109, San Diego, California. Association for Computational Linguistics.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations*, pages 101–108.

Kechen Qin, Lu Wang, and Joseph Kim. 2017. Joint modeling of content and discourse relations in dialogues. In *Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 974–984, Vancouver, Canada. Association for Computational Linguistics.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. *OpenAI blog*, 1(8):9.

Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In *Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*, pages 784–789, Melbourne, Australia. Association for Computational Linguistics.

Harvey Sacks, Emanuel A Schegloff, and Gail Jefferson. 1978. A simplest systematics for the organization of turn taking for conversation. In *Studies in the organization of conversational interaction*, pages 7–55. Elsevier.

Emanuel A Schegloff. 2007. *Sequence organization in interaction: A primer in conversation analysis I*, volume 1. Cambridge university press.

Zhouxing Shi and Minlie Huang. 2019. A deep sequential model for discourse parsing on multi-party dialogues. In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 33, pages 7007–7014.

Ante Wang, Linfeng Song, Hui Jiang, Shaopeng Lai, Junfeng Yao, Min Zhang, and Jinsong Su. 2021. A structure self-aware model for discourse parsing on multi-party dialogues. In *Proceedings of the Thirtieth International Conference on International Joint Conferences on Artificial Intelligence*.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierre Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*, pages 38–45, Online. Association for Computational Linguistics.

Wen Xiao, Patrick Huber, and Giuseppe Carenini. 2020. Do we really need that many parameters in transformer for extractive summarization? discourse can help ! In *Proceedings of the First Workshop on Computational Approaches to Discourse*, pages 124–134, Online. Association for Computational Linguistics.

Wen Xiao, Patrick Huber, and Giuseppe Carenini. 2021. Predicting discourse trees from transformer-based neural summarizers. In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 4139–4152, Online. Association for Computational Linguistics.

Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Discourse-aware neural extractive text summarization. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 5021–5031.

Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT: Large-scale generative pre-training for conversational response generation. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations*, pages 270–278, Online. Association for Computational Linguistics.

Ming Zhong, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2022. Dialoglm: Pre-trained model for long dialogue understanding and summarization. In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 36, pages 11765–11773.

Zining Zhu, Chuer Pan, Mohamed Abdalla, and Frank Rudzicz. 2020. Examining the rhetorical capacities of neural language models. In *Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP*, pages 16–32, Online. Association for Computational Linguistics.
## A Semi-sup. Layer-Wise Results

We consider both **layer-wise** attention matrices - averaging 16 attention heads for every layer which gives 12 candidate layers -, and **head-wise** attention matrices - taking each attention matrix individually which results in 192 candidate matrices. Here we show results completed with layer-wise matrices for the whole test set and treelike examples in Table 9 and Table 10.

## B Molweni Corpus Quality Investigation

Molweni (Li et al., 2020) is a corpus derived from Ubuntu Chat Corpus (Lowe et al., 2015). It contains 10,000 short dialogues between 8 to 15 utterances, annotated in SDRT framework.

Considering the complexity of Ubuntu chat logs (multiple speakers, entangled discussion with various topics), we first conduct an examination of the corpus. Disappointingly, we found heavy repetition within sequential documents and inconsistency in discourse annotation among the same utterances. We thus decide not to include it in this work.

<table><caption>Table 9: Micro-F<sub>1</sub> scores on STAC test set with BART and fine-tuned models. H = “head”, L = “layer”. Best semi-supervised score is in bold. Subscription is std. deviation.</caption><thead><tr><th>Train on →<br>Test with ↓</th><th>BART<br>F<sub>1</sub></th><th>+ SO-DD<br>F<sub>1</sub></th><th>+ SO-STAC<br>F<sub>1</sub></th></tr></thead><tbody><tr><td>Gold H</td><td>57.6</td><td>58.2</td><td>59.5</td></tr><tr><td>Semi-sup-10 1L</td><td>55.8<sub>0.008</sub></td><td>55.7<sub>0.010</sub></td><td>55.6<sub>0.009</sub></td></tr><tr><td>Semi-sup-30 1L</td><td>55.8<sub>0.006</sub></td><td>56.5<sub>0.004</sub></td><td>56.3<sub>0.004</sub></td></tr><tr><td>Semi-sup-50 1L</td><td>56.2<sub>0.002</sub></td><td>56.4<sub>0.007</sub></td><td>56.4<sub>0.001</sub></td></tr><tr><td>Semi-sup-10 1H</td><td>57.0<sub>0.012</sub></td><td>57.2<sub>0.012</sub></td><td>57.1<sub>0.026</sub></td></tr><tr><td>Semi-sup-30 1H</td><td>57.3<sub>0.005</sub></td><td>57.3<sub>0.013</sub></td><td>59.2<sub>0.009</sub></td></tr><tr><td>Semi-sup-50 1H</td><td><strong>57.4</strong><sub>0.004</sub></td><td><strong>57.7</strong><sub>0.005</sub></td><td><strong>59.3</strong><sub>0.007</sub></td></tr></tbody></table>

<table><caption>Table 10: Micro-F<sub>1</sub> scores on STAC projective tree subset with BART and SO fine-tuned BART models.</caption><thead><tr><th>Train on →<br>Test with ↓</th><th>BART<br>F<sub>1</sub></th><th>+ SO-DD<br>F<sub>1</sub></th><th>+ SO-STAC<br>F<sub>1</sub></th></tr></thead><tbody><tr><td>Gold H</td><td>64.8</td><td>67.4</td><td>68.6</td></tr><tr><td>Semi-sup-10 1L</td><td>59.4<sub>0.028</sub></td><td>60.6<sub>0.029</sub></td><td>58.3<sub>0.018</sub></td></tr><tr><td>Semi-sup-30 1L</td><td>62.1<sub>0.002</sub></td><td>61.8<sub>0.012</sub></td><td>59.8<sub>0.009</sub></td></tr><tr><td>Semi-sup-50 1L</td><td>62.1<sub>0.000</sub></td><td>62.3<sub>0.003</sub></td><td>59.9<sub>0.006</sub></td></tr><tr><td>Semi-sup-10 1H</td><td>54.6<sub>0.058</sub></td><td>59.2<sub>0.047</sub></td><td>61.6<sub>0.056</sub></td></tr><tr><td>Semi-sup-30 1H</td><td>60.3<sub>0.047</sub></td><td>60.3<sub>0.044</sub></td><td>65.6<sub>0.043</sub></td></tr><tr><td>Semi-sup-50 1H</td><td><strong>64.8</strong><sub>0.000</sub></td><td><strong>66.3</strong><sub>0.023</sub></td><td><strong>68.1</strong><sub>0.014</sub></td></tr></tbody></table>

<table><caption>Table 11: Quantitative resume of link and relation inconsistency in Molweni test set. “Theor =arc”: number of arcs between the same utterances, *a priori* should be linked in the same way; “Theor =rel”: number of relations between the linked utterances.</caption><thead><tr><th>Clus<br>ID</th><th>Doc<br>ID</th><th>#Theor<br>=arc</th><th>#Err<br>arc</th><th>#Theor<br>=rel</th><th>#Err<br>rel</th></tr></thead><tbody><tr><td>1</td><td>{1, 2, 3}</td><td>18</td><td>2</td><td>16</td><td>2</td></tr><tr><td>2</td><td>{7, 8, 9}</td><td>18</td><td>0</td><td>18</td><td>7</td></tr><tr><td>3</td><td>{10, 11, 12, 13, 14}</td><td>80</td><td>4</td><td>76</td><td>25</td></tr><tr><td>...</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>105</td><td>500</td><td>4787</td><td>284</td><td>4503</td><td>606</td></tr><tr><td>-</td><td>-</td><td>100%</td><td>5.9%</td><td>100%</td><td>13.5%</td></tr></tbody></table>

**Clusters:** Among 500 dialogues in discourse augmented test set, we found 105 “clusters”. One cluster groups all the documents with only one or two different utterances. For instance, document id 10 and 11 are in the same cluster since only the second utterance is different (Figure 10). A similar situation is attested in the documents {1, 2, 3}, {7, 8, 9}, {19, 20, 21}, to name a few.

**Annotation Inconsistency:** A closer examination of the annotation in similar examples reveals inconsistency for both discourse links and rhetorical relations. Precisely, we investigate every *document pair* (two documents in the same cluster) in all 105 clusters in the test set. A visualization of inconsistency for documents 10 and 11 is shown in Figure 10: apart from EDU<sub>2</sub>, we expect the same links and relations among other EDUs. However, we observe one link inconsistency (in red) and two relation inconsistencies (in blue). In total, we find 6% of link errors (#Err arc) within the same EDUs and 14% of relation errors (#Err rel) in the test set<sup>6</sup>. The scores are shown in Table 11.

The Ubuntu Chat Corpus contains long dialogues with entangled discussion. A pre-processing had been made to generate shorter dialogues. While these slightly different short dialogues could be interesting for other dialogue studies in the field. Our focus on the discourse structure request more various data points and most importantly, the coherent discourse annotation.

## C Precision and Recall Scores for Direct and Indirect Arcs in STAC Tree Set

To compare the performance of the whole test set and tree-structured subset, we present the recall and

<footnote n="6"><sup>6</sup>For validation and train sets we find similar error rates.</footnote>
<figure><img src="image_9.png" /><figcaption>Figure 7: Recall and precision metrics in whole test set (darker color) vs. projective tree subset (brighter color), with BART model.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Figure 8: Recall and precision metrics in whole test set (darker color) vs. projective tree subset (brighter color), with BART+SO-DD model.</figcaption></figure>

precision scores of BART (Fig. 7), BART+SO-DD (Fig. 8), and BART+SO-STAC (Fig. 9) separately.

## D Qualitative Analysis in STAC

We show a few concrete tree examples: 3 well predicted (Figure 11, 12, 13), 3 badly predicted (Figure 14, 15, 16), and 2 random examples (Figure 17, 18). Some patterns observed from badly predicted structures: (1) chain-style prediction: as shown in Figure 15 and 18 where only adjacent EDUs are linked together; (2) inaccurate indirect arc prediction: especially for long documents such as the one in Figure 16.

## E Results with other PLMs

We test with RoBERTa (Liu et al., 2019), DialoGPT (Zhang et al., 2020), and DialogLED (DialogLM with Longformer) (Zhong et al., 2022) to see how different language models encode discourse information. As shown in Table 12, the most discourse-rich head in RoBERTa slightly under-

<figure><img src="image_11.png" /><figcaption>Figure 9: Recall and precision metrics in whole test set (darker color) vs. projective tree subset (brighter color), with model BART+SO-STAC.</figcaption></figure>

<table><caption>Table 12: Micro-F<sub>1</sub> on STAC with other PLMs. Best score (except H<sub>ora</sub>) in each row is underlined.</caption><thead><tr><th rowspan="2">Model</th><th colspan="5">Unsup</th><th colspan="3">Semi-sup</th></tr><tr><th>H<sub>ora</sub></th><th>H<sub>g</sub></th><th>H<sub>l</sub></th><th>Semi10</th><th>Semi30</th><th>Semi50</th></tr></thead><tbody><tr><td>BART</td><td>57.6</td><td>56.6</td><td>56.4</td><td>57.0<sub>0.012</sub></td><td>57.3<sub>0.005</sub></td><td>57.4<sub>0.004</sub></td></tr><tr><td>+ SO-DD</td><td>58.2</td><td>56.8</td><td>57.1</td><td>57.2<sub>0.012</sub></td><td>57.3<sub>0.013</sub></td><td>57.7<sub>0.005</sub></td></tr><tr><td>+ SO-STAC</td><td>59.5</td><td>56.7</td><td>57.2</td><td>57.1<sub>0.026</sub></td><td>59.2<sub>0.009</sub></td><td><strong>59.3</strong><sub>0.007</sub></td></tr><tr><td>RoBERTa</td><td>57.4</td><td>56.8</td><td>56.8</td><td>55.6<sub>0.013</sub></td><td>56.8<sub>0.002</sub></td><td>56.9<sub>0.003</sub></td></tr><tr><td>DialoGPT</td><td>56.2</td><td>42.7</td><td>36.2</td><td>52.9<sub>0.043</sub></td><td>55.1<sub>0.017</sub></td><td>56.2<sub>0.000</sub></td></tr><tr><td>DialogLED</td><td>57.2</td><td>56.8</td><td>56.7</td><td>54.6<sub>0.026</sub></td><td>54.7<sub>0.061</sub></td><td>56.6<sub>0.019</sub></td></tr><tr><td>+ SO-DD</td><td>57.7</td><td>56.4</td><td>56.6</td><td>55.0<sub>0.028</sub></td><td>56.1<sub>0.024</sub></td><td>57.3<sub>0.009</sub></td></tr><tr><td>+ SO-STAC</td><td>58.4</td><td>56.8</td><td>57.1</td><td>57.7<sub>0.001</sub></td><td>58.2<sub>0.005</sub></td><td>57.7<sub>0.001</sub></td></tr></tbody></table>

perform BART (−0.2%), so does the DialogLED (−0.4%) and DialoGPT (−1.4%). Sentence ordering fine-tuned DialogLED model outperforms the original one, proving that our proposed SO task can help encoding the discourse information.

## F Huggingface Models

Table 13 shows the models and the sources we obtained from Huggingface library (Wolf et al., 2020).

<table><caption>Table 13: Huggingface models and URLs.</caption><thead><tr><th>Model</th><th></th></tr></thead><tbody><tr><td>BART-large</td><td>https://huggingface.co/facebook/bart-large</td></tr><tr><td>BART-large-cnn</td><td>https://huggingface.co/facebook/bart-large-cnn</td></tr><tr><td>BART-large-samsum</td><td>https://huggingface.co/linydub/bart-large-samsum</td></tr><tr><td>BART-large-finetuned-squad2</td><td>https://huggingface.co/phiyodr/bart-large-finetuned-squad2</td></tr><tr><td>RoBERTa-large</td><td>https://huggingface.co/roberta-large</td></tr><tr><td>DialoGPT-small</td><td>https://huggingface.co/microsoft/DialoGPT-small</td></tr><tr><td>DialogLED-large-5120</td><td>https://huggingface.co/MingZhong/DialogLED-large-5120</td></tr></tbody></table>
<figure><img src="image_12.png" /><figcaption>Figure 10: Similar documents in the same cluster. Circled EDUs are different. In red: inconsistent discourse arcs; in blue: inconsistent rhetorical relation.</figcaption></figure>

test id 10:

[e1] matthew99857: so do i need additional hardware to fix it ?

[e2] vocx: can't you disable the raid from the bios ? check your motherboard manual.

[e3] ikonia: just use the disk as an individual disk

[e4] sugi: vocxi: oh i am sorry. i misunderstood you. thank i will try it now

[e5] vocx: you need to word better your answers, seems like nobody in getting you today.

[e6] sugi: vocx: iso 9660 cd-rom filesystem data udf filesystem data ( unknown version, id 'nsr01' )

[e7] ikonia: looks like that should work as a loop back file system

[e8] sugi: -mount -o loop but instead of .iso .mdf ? or the .mds file ?

[e9] ikonia: try it, linux see 's it as a "image" so it may work

[e10] sugi: vocx: wow it worked, i feel retard for nto

test id: 11

[e1] matthew99857: so do i need additional hardware to fix it ?

[e2] ikonia: no you need to stop using raid

[e3] ikonia: just use the disk as an individual disk

[e4] sugi: vocxi: oh i am sorry. i misunderstood you. thank i will try it now

[e5] vocx: you need to word better your answers, seems like nobody in getting you today.

[e6] sugi: vocx: iso 9660 cd-rom filesystem data udf filesystem data ( unknown version, id 'nsr01' )

[e7] ikonia: looks like that should work as a loop back file system

[e8] sugi: -mount -o loop but instead of .iso .mdf ? or the .mds file ?

[e9] ikonia: try it, linux see 's it as a "image" so it may work

[e10] sugi: vocx: wow it worked, i feel retard for nto

<figure><img src="image_13.png" /><figcaption>Figure 11: Well predicted example: *pilot02-4*. #EDUs: 11. UAS: 90%. In red: FP arcs; in blue: FN arcs.</figcaption></figure>

[e1] Cat: anyone would give me clay? [e2] Thomas: none here [e3] william: no [e4] Cat: I have one wood to exchange [e5] Cat: any takers? [e6] william: no [e7] Cat: for sheep, wheat or clary [e8] Thomas: can I buy a sheep for two ore? [e9] william: have none [e10] Thomas: kk [e11] Cat: no sheep
<figure><img src="image_14.png" /><figcaption>Figure 12: Well predicted example: pilot02-18. #EDUS: 19. UAS: 88.9%. In red: FP arcs; in blue: FN arcs. [e1] william: hi Markus. [e2] william: how many people are we waiting for? [e3] Thomas: think it's 1 more [e4] william: ok [e5] Markus: yes, one more [e6] Markus: seems there's a hickup logging into the game ... [e7] Thomas: that's ok, I not on a schedule [e8] Thomas: *I'm [e9] Markus: I guess you two had no problems joining the game? [e10] william: nope [e11] Markus: Ah great! [e12] Markus: So, one of you can now start the game. [e13] Markus: Have fun! [e14] william: the arrow is pointing at me [e15] william: but i cant press roll [e16] william: oh sorry [e17] Thomas: u can place a settlement [e18] Thomas: first [e19] Thomas: u roll later</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>Figure 13: Well predicted example: s1-league3-game3. #EDUS: 7. UAS: 83.3%. In red: FP arcs; in blue: FN arcs. [e1] Gaeilgeoir: ? [e2] yiin: build road [e3] inca: think we're meant to negotiate trades in the chat before offering [e4] yiin: oop [e5] yiin: ok then [e6] inca: part of the guys' experiment [e7] yiin: oh i see</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Figure 14: Badly predicted example: s2-leagueM-game4. #EDUS: 5. UAS: 20%. In red: FP arcs; in blue: FN arcs. [e1] dmm: i can give a sheep or wood for a wheat. [e2] dmm: any takers? [e3] inca: sheep would be good. [e4] CheshireCatGrin: Not here. [e5] dmm: okay.</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>Figure 15: Badly predicted example: s1-league3-game3. #EDUS: 5. UAS: 25%. In red: FP arcs; in blue: FN arcs. [e1] nareik15: anyone have ore. [e2] nareik15: I have some wood to trade. [e3] yiin: no sorry. [e4] inca: nope, sorry. [e5] Gaeilgeoir: no, sorry.</figcaption></figure>
<figure><img src="image_18.png" /><figcaption>Figure 16: Badly predicted example: s1-league4-game2. #EDUS: 21. UAS: 30%. In red: FP arcs; in blue: FN arcs. [e1] Shawnus: need wheat [e2] Shawnus: want..clay? [e3] ztime: you odo? [e4] ztime: yer.. [e5] ztime: I need clay.. [e6] ztime: can give wheat [e7] Shawnus: k [e8] Shawnus: this might be where i lose my road card a? [e9] ztime: er.. [e10] ztime: I think the trade is wrong? [e11] ztime: did you want wheat? [e12] Shawnus: yes [e13] Shawnus: for clay [e14] ztime: it said you wanted clay... [e15] somdechn: We all want wheat man [e16] somdechn: and clay... [e17] ztime: ok [e18] ztime: thanks.. [e19] Shawnus: haha [e20] Shawnus: thanks [e21] somdechn: That happens in the real game as well.</figcaption></figure>

<figure><img src="image_19.png" /><figcaption>Figure 17: Random example: s2-league4-game2. #EDUS: 14. UAS: 53.9%. In red: FP arcs; in blue: FN arcs. [e1] ztime: 7!!!! [e2] somdechn: Yeah right... [e3] ztime: what... is this a fix? [e4] Shawnus: hahaha [e5] ztime: ok anyone want wheat? [e6] Shawnus: nope [e7] Shawnus: just someone to roll 9's.. [e8] somdechn: Yes [e9] somdechn: I can give you wood. [e10] ztime: was that yes to a trade somdech? [e11] ztime: OK.. cool.. for 1 wheat? [e12] somdechn: and an ore.. :) [e13] ztime: err.. don't have ore.. [e14] ztime: thanks..</figcaption></figure>

<figure><img src="image_20.png" /><figcaption>Figure 18: Random example: s1-league3-game3. #EDUS: 11. UAS: 50%. In red: FP arcs; in blue: FN arcs. [e1] nareik15: anyone have wood to trade. I have sheep [e2] yiin: no [e3] Gaeilgeoir: Sorry, [e4] Gaeilgeoir: I need wood too [e5] Gaeilgeoir: I have wheat [e6] Gaeilgeoir: if you want [e7] inca: do you have wheat kieran? [e8] inca: if so [e9] inca: i can trade wood [e10] nareik15: sorry, [e11] nareik15: plenty of sheep though :)</figcaption></figure>