Venue: EMNLP 2020

# Multi-View Sequence-to-Sequence Models with Conversational Structure

for Abstractive Dialogue Summarization

Jiaao Chen

School of Interactive Computing

Georgia Institute of Technology

jiaachen@gatech.edu &Diyi Yang

School of Interactive Computing

Georgia Institute of Technology

dyang888@gatech.edu

###### Abstract

Text summarization is one of the most challenging and interesting problems in NLP. Although much attention has been paid to summarizing structured text like news reports or encyclopedia articles, summarizing conversations--an essential part of human/machine interaction where most important pieces of information are scattered across various utterances of different speakers--remains relatively under-investigated. This work proposes a multi-view sequence-to-sequence model by first extracting conversational structures of unstructured daily chats from different views to represent conversations and then utilizing a multi-view decoder to incorporate different views to generate dialogue summaries. Experiments on a large-scale dialogue summarization corpus demonstrated that our methods significantly outperformed previous state-of-the-art models via both automatic evaluations and human judgment. We also discussed specific challenges that current approaches faced with this task. We have publicly released our code at [https://github.com/GT-SALT/Multi-View-Seq2Seq](https://github.com/GT-SALT/Multi-View-Seq2Seq).

## 1 Introduction

We live in an information age where communications between human and human/machine are increasing exponentially in the form of textual dialogues between users and users-agents Kester (2004). It is challenging and time-consuming to review all the content before starting any conversations especially when the chatting history becomes very long Gao et al. (2020). How to process and organize those interaction activities into concise and structured data, i.e. conversation summarization, becomes technically and socially important.

Most existing research efforts on text summarization have been focused on single-speaker documents like news reports Nallapati et al. (2016); See et al. (2017), scientific publications Nikolov et al. (2018) or encyclopedia articles Liu* et al. (2018), where structured text is usually used to elaborate a core idea in the third-person point of view, and the information flow is very clear through paragraphs or sections. Different from these structured documents, conversations are often informal, verbose and repetitive, sprinkled with false-starts, back channeling, reconfilrations, hesitations, speaker interruptions Sacks et al. (1978) and the salient information is scattered in the whole chat, making current summarization models hard to focus on many informative utterances. Take the conversation in Table 1 as an example, turns, informal words, abbreviations, and emoticons all introduce new forms of challenges to the task of summarization. This calls for the design and development of new methods for dialogue summarization instead of directly applying current document summarization models.

There has been some recent research on conversation summarization such as directly deploying existing document summarization models Gliwa et al. (2019) and exploring multi-sentence compression Shang et al. (2018), however, most of them haven't utilized specific conversational structures, which refer to the way utterances are organized in order to make the conversation meaningful, enjoyable and understandable Sacks et al. (1978), in dialogues - a key factor that differentiates dialogues from structured documents. As a way of using language socially of "doing things with words" together with other persons, the conversation has its own dynamic structures that organize utterances in certain orders to make the conversation meaningful, enjoyable, and understandable Sacks et al. (1978). Although there are a few exceptions such as utilizing topic segmentation Liu et al. (2019b); Li et al. (2019), dialogue acts Goo and Chen (2018) or key point sequence Liu et al. (2019a), they either need

extensive expert annotations of discourse acts[14, 20], or only encode conversations based on their topics [35], which fails to capture rich conversation structures in dialogues.

Even one single conversation can be viewed from different perspectives, resulting in multiple conversational or discourse patterns. For instance, in Table 1, based on what topics were discussed (**topic view**) [6, 38, 39], it can be segmented into _greetings_, _today's plan_, _plan for tomorrow_, _plan for Saturday_ and _pick up time_; from a conversation progression perspective (**stage view**) [7, 20, 38], the same dialogue can be categorized into _openings_, _intention_, _discussion_, and _conclusion_. From a coarse perspective (**global view**), conversations can be treated as a whole, or each utterance can serve as one segment (**discrete view**). Models that only utilized a fixed topic view of the conversation [13, 38] may fail to capture its comprehensive and nuanced conversational structures, and any amount of information loss introduced by the conversation encoder may lead to larger error cascade in the decoding stage. To fill these gaps, we propose to combine those multiple, diverse views of conversations in order to generate more precise summaries.

To sum up, our contributions are: (1) we propose to utilize rich conversational structures, i.e., structured views (topic view and stage view) and the generic views (global view and discrete view) for abstractive conversation summarization. (2) We design a multi-view sequence-to-sequence model that consists of a conversation encoder to encode different views and a multi-view decoder with multi-view attention to generate dialogue summaries. (3) We perform experiments on a large-scale conversation summarization dataset, SAMSum [21], and demonstrate the effectiveness of our proposed methods. (4) We conduct thorough error analyses and discuss specific challenges that current approaches faced with this task.

## 2 Related Work

Document SummarizationDocument summarization has received extensive research attention, especially for abstractive summarization. For instance, rush2015unsupervised introduced to use sequence-to-sequence models for abstractive text summarization. Seeetal2017unsupervised proposed a pointer-generator network to allow copying words from the source text to handle the OOV issue and avoid generating repeated content. paulus2018unsupervised,chen2018unsupervised further utilized reinforcement learning to select the correct content needed by summarization. Large-scale pre-trained language models [38, 39, 20] have also been introduced to further improve the summarization performance. Other line of work explored long-document summarization by utilizing discourse structures in text [10], introducing hierarchical models [14] or modifying attention mechanisms [1]. There are also recent studies looking at the faithfulness in

\begin{table}
\begin{tabular}{|c||l||p{142.3pt}||p{142.3pt}|} \hline
**Conversation** & **Topic View** & **Stage View** \\ \hline James: & Hey! I have been thinking about you : ) & Greetings & Openings \\ \hline Hannah: & Oh, that’s nice ; ) & Today’s plan & Intention \\ \hline James: & H miss u. I was hoping to see you & & \\ \hline Hannah: & Have to get up early for work tomorrow & & \\ \hline James: & What about tomorrow? & Plan for tomorrow & \\ \hline Hannah: & To be honest I have plans for tomorrow evening & & Discussion \\ \hline James: & Oh ok. What about Sat then? & Plan for Saturday & \\ \hline Hannah: & Yeah. Sure I am available on Sat & & \\ \hline James: & I’ll pick you up at 8? & Pick up time & \\ \hline Hannah: & Sounds good. See you then. & & Conclusion \\ \hline
**Summary** & James misses Hannah. They agree for James to pick Hannah up on Saturday at 8. & \\ \hline  & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\ \end{tabular}
\end{table}
Table 1: Example conversation from SAMSum [21] with its topic view and stage view (extracted by our methods), and the human annotated summary.

 

document summarization Cao et al. (2018); Zhu et al. (2020a), in order to enhance the information consistency between summaries and the input.

Dialogue SummarizationWhen it comes to the summarization of dialogues, Shang et al. (2018) proposed a simple multi-sentence compression technique to summarize meetings. Zhao et al. (2019); Zhu et al. (2020b) introduced turn-based hierarchical models that encoded each turn of utterance first and then used the aggregated representation to generate summaries. A few studies have also paid attention to utilizing conversational analysis for generating dialogue summaries, such as leveraging dialogue acts Goo and Chen (2018), key point sequence Liu et al. (2019) or topics Liu et al. (2019); Li et al. (2019). However, they either needed a large amount of human annotation for dialogue acts, key points or visual focus Goo and Chen (2018); Liu et al. (2019); Li et al. (2019), or only utilized topical information in conversations Li et al. (2019); Liu et al. (2019).

These prior work also largely ignored diverse conversational structures in dialogues, for instance, reply relations among participants Mayfield et al. (2012); Zhu et al. (2019), dialogue acts Ritter et al. (2010); Paul (2012), and conversation stages Althoff et al. (2016). Models that only utilized a fixed topic view of the conversation Galley et al. (2003); Joty et al. (2010) may fail to capture its comprehensive and nuanced conversational structures, and any amount of information loss introduced by the conversation encoder may lead to larger error cascade in the decoding stage. To fill these gaps, we propose to leverage diverse conversational structures including topic segments, conversational stages, dialogue overview, and utterances to design a multi-view model for dialogue summarization.

## 3 Method

Conversations can be interpreted from different views and every single view enables the model to focus a specific aspect of the conversation. To take advantages of those rich conversation views, we design a Multi-view Sequence-to-Sequence Model (see Figure 1) that firstly extracts different views of conversations (Section 3.1) and then encodes them to generate summaries (Section 3.2).

### Conversation View Extraction

Conversation summarization models may easily stray among all sorts of information across various speakers and utterances especially when conversations become long. Naturally, if informative structures in the form of small blocks can be explicitly extracted from long conversations, models may be able to understand them better in a more organized way. Thus, we first extract different views of structures from conversations.

Topic ViewAlthough conversations are often less structured than documents, they are mostly organized around topics in a coarse-grained structure Honneth et al. (1988). For instance, a telephone chat could possess a pattern of "greetings \(\rightarrow\) invitation \(\rightarrow\) party details \(\rightarrow\) rejection" from a topical perspective. Such explicit view and topic flow could help models interpret conversations more precisely and generate summaries that cover important topics. Here we combine the classic topic segment

Figure 1: Model architecture. Different views of conversations are first extracted automatically, and then encoded through the conversation encoder (a) and combined in the multi-view decoder to generate summaries (b). In the conversation encoder, each view (consists of blocks) is encoded separately and the block’s representations \(S_{i}\) are encoded through LSTM to represent the view. In the multi-view decoder, the model decides attention weights over different views and then attend to each token in different views through the multi-view attention.

 

algorithm, C99 (Choi, 2000) that segments conversations based on inter-sentence similarities, with recent advanced sentence representations Sentence-BERT (Reimers and Gurevych, 2019), to extract the topic view. Specifically, each utterance \(u_{i}\) in a conversation \(\mathbf{C}=\{u_{1},u_{2},...,u_{m}\}\) is first encoded into hidden vectors via Sentence-BERT. Then the conversation \(\mathbf{C}\) is divided into blocks \(\mathbf{C}_{topic}=\{\mathbf{b}_{1},...,\mathbf{b}_{n}\}\) through C99, where \(\mathbf{b}_{i}\) is one block that contains several consecutive utterances, such as the topic view described in Table 1.

Stage ViewAs a way of doing things with words socially together with other people, conversation organizes utterances in certain orders to make it meaningful, enjoyable, and understandable. (Sacks et al., 1978; Althoff et al., 2016) For example, counseling conversations are found to follow a common pattern of "introductions\(\rightarrow\)problem exploration\(\rightarrow\)problem solving\(\rightarrow\)wrap up" (Althoff et al., 2016). Such conversation stage view provides high-level sketches about the functions or goals of different parts in conversations, which could help models focus on the stages with key information.

We follow Althoff et al. (2016) to extract stages through a Hidden Markov Model (HMM). We impose a fixed ordering on the stages and only allow transitions from the current stage to the next one. The observations in the HMM model are the encoded representations \(h_{i}\) from Sentence-BERT. We set the number of hidden stages as 4. Similar to the topic view extraction, we segment the conversations into blocks \(\mathbf{C}_{stage}=\{\mathbf{b}_{1},...,\mathbf{b}_{n}\}\), where \(\mathbf{s}_{i}\) is one block that contains several consecutive utterances. We interpret the inferred stages qualitatively and further visualize the top 6 frequent words appearing in each stage in Table 2. We found that conversations around daily chats usually start with openings, introduce the goals/focus of the conversation followed by discussions of the details, and finally conclude with certain endings. Table 1 shows an example of the stage view.

Global View and Discrete ViewIn addition to the aforementioned two structured views, conversations can also be naturally viewed from a relatively coarse perspective, i.e., a global view that concatenates all utterances into one giant block Gliwa et al. (2019), and a discrete view that separates each utterance into a distinct block Liu and Chen (2019); Gliwa et al. (2019).

### Multi-view Sequence-to-Sequence Model

We extend generic sequence-to-sequence models to encode and combine different conversation views. To better utilize semantic information in recent pre-trained models, we implement our base encoders and decoders with a transformer based pre-trained model, BART Lewis et al. (2019). Note that our multi-view sequence-to-sequence model is agnostic to BART with which it is initialized.

Conversation EncoderGiven a conversation under a specific view \(k\) with \(n\) blocks: \(\mathbf{C}_{k}=\{\mathbf{b}^{k}_{1},...,\mathbf{b}^{k}_{n}\}\), each token \(x^{k}_{i,j}\) in a block \(\mathbf{b}^{kj}_{j}=\{x^{k}_{0,j},x^{k}_{1,j},...,x^{k}_{m,j}\}\) is first encoded through the conversation encoder \(\mathbf{E}\) , e.g., BART encoder as shown in Figure 1(a), into hidden representations:

\[\{h^{k}_{0,j},h^{k}_{1,j},...,h^{k}_{m,j}\}=\mathbf{E}(\{x^{k}_{0,j},x^{k}_{1 ,j},...,x^{k}_{m,j}\}) \tag{1}\]

Note that we add special tokens \(x^{k}_{0,j}\) at the beginning of each block and use these tokens' representations to describe each block, i.e., \(S^{k}_{j}=h^{k}_{0,j}\).

To depict different views using hidden vectors, we aggregate the information from all blocks in one conversation through LSTM layers Hochreiter and Schmidhuber (1997):

\[S^{k}_{j}=\text{LSTM}(h^{j,k}_{0},S^{k}_{j-1}),j\in[1,n] \tag{2}\]

\begin{table}
\begin{tabular}{|c|p{56.9pt}|p{34.1pt}|} \hline
**Stage** & **Interpretation** & **Top Freq Words** \\ \hline
1 & Openings & hey, hi, good, yeah,going, time \\ \hline
2 & Intentions & need, like, think, get, want, really \\ \hline
3 & Discussions & will, know, time, come,tomorrow, meet \\ \hline
4 & Conclusions & thanks, ok, see, great, thank, sure \\ \hline \end{tabular}
\end{table}
Table 2: The top 6 frequent words appearing in each stage and the interpretations for different stages.

 

We use the last hidden state \(S_{n}^{k}\) to represent the current view \(k\), denoted as \(V_{k}\).

Multi-view DecoderDifferent views could provide different types of conversational aspects for models to learn and further determine which set of utterances should deserve more attention in order to generate better dialogue summaries. As a result, the ability to strategically combine different views is essential. To this end, we propose a transformer based multi-view decoder to integrate encoded representations from different views and generate summaries as shown in Figure 1(b).

The input to the decoder contains \(l-1\) previously generated tokens \(t_{1},...,t_{l-1}\). Via our multi-view decoder **D**, the \(l\)-th token is predicted via:

\[\{y_{1},...,y_{l-1}\}=\textbf{D}(\{t_{1},...,t_{l-1}\},\mathbf{E} (C)) \tag{3}\] \[P(\tilde{t}_{l}|t_{<l},\mathbf{C})=\text{Softmax}(W_{p}y_{l-1}) \tag{4}\]

Here, \(W_{p}\) is a parameter to be learned.

Different from generic transformer decoder, we introduce a multi-view attention layer in each transformer block. Multi-view attention layer first decides the importance \(\alpha_{k}\) of each view \(V_{k}\) through:

\[u_{k} =\tanh{(WC_{k}+b)} \tag{5}\] \[\alpha_{k} =\frac{\exp{\left(u_{k}^{\top}v\right)}}{\sum_{i}\exp{\left(u_{ i}^{\top}v\right)}} \tag{6}\]

where \(v\) is a randomly initialized context vector; \(W\) and \(b\) are parameters. To avoid the attention weights being too similar to each other as views are actually encoded from a similar context, we utilize a sharpening function over \(\alpha_{k}\) with a temperature T: \(\tilde{\alpha}_{k}=\alpha_{k}^{\frac{1}{T}}/\!\sum_{i}\alpha_{i}^{\frac{1}{T}}\). When \(T\to 0\), the attention weights will behave like a one-hot vector.

Then the multi-head attention is performed over conversation tokens \(h_{i,j}^{k}\) from different views \(k\) and form \(A^{k}\) separately. The attended results are further combined based on the view-attention weights \(\tilde{\alpha}_{k}\) and continue forward passing:

\[\tilde{A}=\sum_{k}\tilde{\alpha}_{k}A^{k} \tag{7}\]

TrainingWe minimize the cross entropy loss during training:

\[L=-\sum\log P(\tilde{t}_{l}|t_{<l},\mathbf{C}) \tag{8}\]

Specifically, we apply the teacher forcing strategy: at training time, the inputs are previous tokens from the ground truth; at test time, the inputs are previous tokens predicted by the decoder.

## 4 Experiments

### Dataset and Baselines

We evaluate our model on a large-scale dialogue summary dataset SAMSum Gliwa et al. (2019) that has 14732 dialogues with human-written summaries. The data statistics are shown in Table 3. SAMSum contains messenger-like conversations about daily topics, such as chit-chats, arranging meetings, discussing events, etc. We compare our Multi-view Sequence-to-Sequence Model (Multi-view BART) with several baseline models:

* **Pointer Generator**See et al. (2017): Following Gliwa et al. (2019), we added separators between each utterance (**discrete view**) and used it as input for pointer generator model.
* **DynamicConv + GPT-2/News**Wu et al. (2019): We followed Gliwa et al. (2019) to use GPT-2 to initialize token embeddings Radford et al. (2019). We also added news summarization corpus CNN/DM Nallapati et al. (2016) as extra training data.
* **Fast Abs RL Enhanced**Chen and Bansal (2018) first selects salient sentences and then rewrites them abstractively via sentence-level policy gradient methods. We combined it with the **global view**Gliwa et al. (2019).
* **BART + Generic views**Lewis et al. (2019) utilized BART, a denoising autoencoder for pretraining sequence-to-sequence models, together with generic views (**global view** and **discrete view**). We used the BART-large model with its default settings 1. Footnote 1: [https://github.com/pytorch/fairseq](https://github.com/pytorch/fairseq)

### Model Settings2
Footnote 2: More details are shown in Section A in the Appendix.

We loaded the pre-trained "bert-base-nli-stsb-mean-tokens"3 for sentence-BERT to get representations for each utterance. For extracting the topic view via C99, we set the window size 4 and std coefficient 1. For extracting the stage view, we set the number of hidden states 4 in HMM. These hyper-parameters were set with a grid search. The **BART + Structured views** (stage and topic views) used the same set of parameters as **BART + Generic views**. For 

- global view, was combined with two structured views (stage and topic view) separately; (2) the best two structured views are also combined (topic + stage). The settings for BART encoder/decoder kept identical as baselines. We used a one-layer LSTM for encoding sections. The learning rate for section encoder and multi-view attention was set 3e-3. The temperature \(T\) was 0.2. The beam search size during inference for all the models was 4.

### Results

**Quantitative Results** We evaluated models with the standard metric ROUGE Score (with stemming) Lin and Och [2004], and reported ROUGE-1, ROUGE-2 and ROUGE-L4. Results on the test set for different models were shown in Table 4. Compared to Pointer Generator, using reinforcement learning to select important sentences first (Fast Abs RL Enhanced) slightly increased F scores. Adding pre-trained embeddings or extra documents training data to lightweight convolution models, (DynamicConv + GPT-2/News) lead to even better ROUGE scores. When using pre-trained transformer based model BART with generic views, all ROUGE scores improved significantly, and BART + Global outperformed BART + Discrete especially in terms of ROUGE-L F scores. Segmenting conversations into blocks from structured views (stage view and topic view) further boosted the performance, suggesting that our extracted conversation structures help conversational encoders to capture nuanced and informative aspects of dialogs.

Footnote 4: Here we followed BART and used [https://github.com/pltdy/rouge](https://github.com/pltdy/rouge). Note that different tools may generate different ROUGE scores.

We did not see any performance boost when combining the generic global view with either topic or conversational stage views, partially due to that the coarse granularity of global view does not complement structured views well. In contrast, utilizing both structured views (topic view + stage view) further increased ROUGE scores consistently, indicating the effectiveness of synthesizing informative conversation blocks introduced by both views.

We visualized the attention weight distributions

Figure 3: Relations between ROUGE scores and the number of participants/turns in conversations.

 

for the stage view and topic view in our best model (see Appendix) and found contributions of topic views are slightly more prominent compared to stage views. This also communicated that the two different structured views can complement each other well though sharing the same dialogue content. Note that the gains from Multi-view BART _(Topic + Stage_)_ are mainly from the precision scores while recall scores are kept comparable, suggesting that our proposed model produced fewer irrelevant tokens while preserving necessary information in its generated summary.

Impact of Participants and TurnsWe visualized the impact of two essential components in conversations--the number of participants and turns--on rouge scores via our best-performing model Multi-view BART with topic view + stage view in Figure 3. As the number of participants/turns increases, ROUGE scores decrease, indicating that the difficulty of conversation summarization increased with more participants involved in conversations and more utterances.

Qualitative/Human EvaluationWe also conducted human annotations to evaluate the extracted dialogue summaries, in addition to ROUGE scores. Similar to Gliwa et al. (2019), we asked human annotators on Amazon Mechanical Turk 5 to rate each summary (200 randomly sampled summaries in total) on the scale of [-2, 0, 2], where -2 means that a summary was poor, extracted irrelevant information or did not make sense at all, 2 means it was understandable and gave a concise overview of the text, and 0 refers to that the summary only extracted only a part of relevant information, or made some mistakes. The score for each summary was averaged among three different annotators. The Intra-class Correlation was 0.583, indicating moderate agreement Koo and Li (2016).

Footnote 5: [https://www.mturk.com/](https://www.mturk.com/)

As shown in Figure 4, consistent with ROUGE scores in Table 4, our multi-view model achieved the highest human annotation scores, significantly higher (via a student t-test) than either generic (discrete or global) view or structured (stage or topic) view, which further proved the effectiveness of combing different views.

## 5 Model Analysis and Discussion

So far, we have achieved a reasonable summarization performance. To further study why dialog summarization is challenging and how future research could advance this direction, we take a closer look at this dialogue summarization dataset (SAMSum), model generation errors, as well as certain challenges that existing approaches are struggling with.

### Challenges in Dialog Summarization

We conduct a thorough examination of the challenges in conversation summarization and organized them into 7 categories as below:

1. **Informal language use** Many conversations especially in online contexts such as Twitter/Reddit Jackson and Moulinier (2007), contain typos, word abbreviations, slang or emoticons/emojis, making it hard to be represented and summarized.
2. **Multiple participants** As shown in Figure 3, conversations with more speakers are harder to be summarized since it may require models to accurately differentiate both language styles and content from different speakers, similar to the multiple characters issue in story summarization Zhang et al. (2019).
3. **Multiple turns** Similar to long document summarization Xiao and Carenini (2019), conversations with many utterances contain more information to be processed, thus harder to be summarized.
4. **(Referral and coreference** People usually refer to each other, mention others' names or use coreference in their messages, which introduces extra difficulty to dialogue summarization, also a challenge also exists in reading comprehension Chen et al. (2016) and document summarization Falke et al. (2017).
5. **Repetition and interruption** Information is generally scattered through the whole conversation, and speakers may interrupt each other,

Figure 4: Human evaluation results. The mean score for each model is also shown in the box plot.

 

reconfirm, back channeling or repeat themselves, a unique discourse challenge for dialogue summarization.
6. **Negations and rhetorical questions** As a long-standing problem in NLP field Li et al. (2016), negation related issues are even more frequent in conversations, as there are more question-answer exchanges between speakers.
7. **Role and language change** Conversations usually involve more than one speaker, and the role of a speaker may shift from a questioner to an answerer, requiring the summarization model to dynamically deal with speaker roles and the associated language (e.g., first personal pronouns)

We randomly sampled 100 examples6 from our test set and classified them using the above challenge taxonomy. A conversation might have more than one category labels, and if it had none of the aforementioned challenges, we labeled it as **(0) Generic**. Usually, the one marked as Generic were shorter or had a simple structure.

Footnote 6: The full analyzed set of examples are shown in Appendix.

Table 5 presents the percentage of each type of challenge and per-category performances from our best model (Multi-view BART with Topic view + Stage view). We observed that: (i) Referral & coreference (33%) and Role & language change (30%) were the two most frequent challenges that dialogue summarization task faced. (2) As expected, Generic conversations were relatively easier summarize. (3) Our best model performed relatively worse when it came to Repetition & interruption, Multiple turns, and Referral & coreference, calling for more intelligent summarization methods to tackle those challenges.

### Error Analysis7
Footnote 7: Error analysis for baselines are displayed in the Appendix.

We examined summaries generated by our best-performing model compared to ground-truth summaries, and observed several major error types:

1. **Missing information**: content mentioned in references is missing in generated summaries.
2. **Redundancy**: content occurred in generated summaries was not mentioned by references.
3. **Wrong references**: generated summaries contain information that is not faithful to the original dialogue, and associate one's actions/locations with a wrong speaker.
4. **Incorrect reasoning**: generated summaries reasoned relations in dialogues incorrectly, thus came to wrong conclusions.
5. **Improper gendered pronouns**: summaries used improper gendered pronouns (e.g., the misuse of gendered pronouns).

We annotated the same set of 100 randomly sampled summaries via the above error type taxonomy. A summary might have more than one category labels and we categorized a summary as **(0) Other** if it did not belong to any error types.

Table 6 presents the breakdown of error types and per-category ROUGE scores. We found that: (i) missing information (37%) was the most frequent error type, indicating that current summarization models struggled with identifying key information. (ii) Incorrect reasoning had a percentage of 24% with the worst ROUGE-2; despite of being a minor type 6%, improper gendered pronouns seemed to severely decrease both ROUGE-1 and ROUGE-2. (iii) The relatively low ROUGE scores associated with incorrect reasoning and wrong references urged better summarization models in dealing with faithfulness in dialogue summarization.

\begin{table}
\begin{tabular}{|r|c|l|} \hline
**Errors** & **\%** & **ROUGE-1/2/L** \\ \hline  & Other & **24** / **0.611 / 0.363 / 0.584** \\ \hline Missing information & 37 & 0.448 / 0.236 / 0.445 \\ Redundancy & 13 & 0.442 / 0.231 / 0.441 \\ Wrong references & 27 & 0.460 / 0.232 / 0.454 \\ Incorrect reasoning & 24 & 0.447 / 0.187 / 0.411 \\ Impro proper gendered pronouns & 6 & 0.421 / 0.212 / 0.428 \\ \hline \end{tabular}
\end{table}
Table 6: The common error types of our model compared to golden reference on 100 sampled conversations, and the ROUGE scores per error type.

\begin{table}
\begin{tabular}{|r|c|l|} \hline
**Challenge** & **\%** & **ROUGE-1/2/L** \\ \hline  & Generic & **24** / **0.613 / 0.384 / 0.579** \\ \hline  & 25 & 0.471 / 0.241 / 0.459 \\ Multiple participants & 10 & 0.473 / 0.243 / 0.461 \\ Multiple turns & 23 & 0.432 / 0.213 / 0.432 \\ Referral \& coreference & 33 & 0.445 / 0.206 / 0.430 \\ Repetition \& interruption & 18 & 0.423 / 0.180 / 0.415 \\ Negations \& rhetorical & 20 & 0.458 / 0.227 / 0.431 \\ Role \& language change & 30 & 0.469 / 0.211 / 0.450 \\ \hline \end{tabular}
\end{table}
Table 5: The breakdown of challenges in dialogue summarization based on our analyses of 100 sampled conversations, and the ROUGE scores per challenge 



### Relation between Challenges and Errors

To figure out relations between challenges and errors made by our models, i.e., how different types of errors correlate with different types of challenges, we visualized the co-occurrence heat map in Figure 5. We found that: (i) Our model generated good summary for generic, simple conversations. (ii) All kinds of challenges had high correlations with, or could lead to the missing information error. (iii) Wrong references were highly associated with referral & coreference; this was as expected since co-references in conversations would naturally increase the difficulty for models to associate correct speakers with correct actions. (iv) High correlations between role & language change, referral & coreference and incorrect reasoning indicated that interactions between multiple participants with frequent co-references might easily lead current summarization models to reason incorrectly.

## 6 Conclusion

In this work, we proposed a multi-view sequence-to-sequence model that leveraged multiple conversational structures (topic view and stage view) and generic views (global view and discrete view) to generate summaries for conversations. In order to strategically combine these different views for better summary generations, we propose a multi-view sequence-to-sequence model. Experiments conducted demonstrated the effectiveness of our proposed models in terms of both quantitative and qualitative evaluations. Via thorough error analyses, we concluded a set of challenges that current models struggled with, which can further facilitate future research on conversation summarization. Due to the lack of annotations, we only adopted simple unsupervised segmentation methods to extract different views. In the future, we plan to annotate some of the data, explore supervised segmentation models Li et al. (2018) and introduce more conversation structures like dialogue acts Oya and Carenini (2014); Joty and Hoque (2016) into abstractive dialogue summarization.

## Acknowledgment

We would like to thank the anonymous reviewers for their helpful comments, and the members of Georgia Tech SALT group for their feedback. We acknowledge the support of NVIDIA Corporation with the donation of GPU used for this research.

## References

* Althoff et al. (2016) Tim Althoff, Kevin Clark, and Jere Leskovec. 2016. Large-scale analysis of counseling conversations: An application of natural language processing to mental health. _Transactions of the Association for Computational Linguistics_, 4:463-476.
* Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer.
* Cao et al. (2018) Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In _AAAI_.
* Chen et al. (2016) Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A thorough examination of the CNN/daily mail reading comprehension task. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2358-2367, Berlin, Germany. Association for Computational Linguistics.
* Chen and Bansal (2018) Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 675-686, Melbourne, Australia. Association for Computational Linguistics.
* Choi (2000) Freddy Y. Y. Choi. 2000. Advances in domain independent linear text segmentation. In _1st Meeting of the North American Chapter of the Association for Computational Linguistics_.
* Cohan et al. (2018) Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 615-621, New Orleans, Louisiana. Association for Computational Linguistics.

Figure 5: Relations between difficulties in conversations and errors made by our model.



Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.
* Falke et al. (2017) Tobias Falke, Christian M Meyer, and Iryna Gurevych. 2017. Concept-map-based multi-document summarization using concept coreference resolution and global importance optimization. In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 801-811.
* Galley et al. (2003) Michel Galley, Kathleen R. McKeown, Eric Fosler-Lussier, and Hongyan Jing. 2003. Discourse segmentation of multi-party conversation. In _Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics_, pages 562-569, Sapporo, Japan. Association for Computational Linguistics.
* Gao et al. (2020) Shen Gao, Xiuying Chen, Zhaochun Ren, Dongyan Zhao, and Rui Yan. 2020. From standard summarization to new tasks and beyond: Summarization with manifold information.
* Gliwa et al. (2019) Bogdan Gliwa, Ivona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_, pages 70-79, Hong Kong, China. Association for Computational Linguistics.
* Goo and Chen (2018) Chih-Wen Goo and Yun-Nung Chen. 2018. Abstractive dialogue summarization with sentence-gated modeling optimized by dialogue acts. _2018 IEEE Spoken Language Technology Workshop (SLT)_.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Comput._, 9(8):1735-1780.
* Honnett et al. (1988) Axel Honnett, Hans Joas, et al. 1988. _Social action and human nature_. CUP Archive.
* Jackson and Moulinier (2007) Peter Jackson and Isabelle Moulinier. 2007. _Natural language processing for online applications: Text retrieval, extraction and categorization_, volume 5. John Benjamins Publishing.
* Joty et al. (2010) Shafiq Joty, Giuseppe Carenini, Gabriel Murray, and Raymond T. Ng. 2010. Exploiting conversation structure in unsupervised topic segmentation for emails. In _Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing_, pages 388-398, Cambridge, MA. Association for Computational Linguistics.
* Joty and Hoque (2016) Shafiq Joty and Enamul Hoque. 2016. Speech act modeling of written asynchronous conversations with task-specific embeddings and conditional structured models. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1746-1756.
* Kester (2004) Grant H Kester. 2004. _Conversation pieces: Community and communication in modern art_. Univ of California Press.
* Koo and Li (2016) Terry K Koo and Mae Y Li. 2016. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. _Journal of chiropractic medicine_, 15(2):155-163.
* Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.
* Li et al. (2018) Jing Li, Aixin Sun, and Shafiq R Joty. 2018. Segbot: A generic neural text segmentation model with pointer network. In _IJCAI_, pages 4166-4172.
* Li et al. (2016) Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in NLP. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 681-691, San Diego, California. Association for Computational Linguistics.
* Li et al. (2019) Manling Li, Lingyu Zhang, Heng Ji, and Richard J. Radke. 2019. Keep meeting summaries on topic: Abstractive multi-modal meeting summarization. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2190-2196, Florence, Italy. Association for Computational Linguistics.
* Lin and Och (2004) Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In _Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics_, page 605. Association for Computational Linguistics.
* Liu et al. (2019a) Chunyi Liu, Peng Wang, Jiang Xu, Zang Li, and Jieping Ye. 2019a. Automatic dialogue summary generation for customer service. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, KDD'19, page 1957-1965, New York, NY, USA. Association for Computing Machinery.
* Liu* et al. (2018) Peter J. Liu*, Mohammad Saleh*, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. In _International Conference on Learning Representations_.
* Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ 

Zhengyuan Liu and Nancy Chen. 2019. Reading turn by turn: Hierarchical attention architecture for spoken dialogue comprehension. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5460-5466, Florence, Italy. Association for Computational Linguistics.
* Liu et al. (2019) Zhengyuan Liu, Angela Ng, Sheldon Lee, Ai Ti Aw, and Nancy F. Chen. 2019b. Topic-aware pointer-generator networks for summarizing spoken conversations. _2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_.
* Mayfield et al. (2012) Elijah Mayfield, David Adamson, and Carolyn Penstein Rose. 2012. Hierarchical conversation structure prediction in multi-party chat. In _Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue_, pages 60-69, Seoul, South Korea. Association for Computational Linguistics.
* Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Guljlecherre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
* Nikolov et al. (2018) Nikola I. Nikolov, Michael Pfeiffer, and Richard H. R. Hahnloser. 2018. Data-driven summarization of scientific articles. _CoRR_, abs/1804.08875.
* Oya and Carenini (2014) Tatsuro Oya and Giuseppe Carenini. 2014. Extractive summarization and dialogue act modeling on email threads: An integrated probabilistic approach. In _Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)_, pages 133-140.
* Paul (2012) Michael J. Paul. 2012. Mixed membership Markov models for unsupervised conversation modeling. In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_, pages 94-104, Jeju Island, Korea. Association for Computational Linguistics.
* Paulus et al. (2018) Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In _International Conference on Learning Representations_.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Illya Sutskever. 2019. Language models are unsupervised multitask learners. _OpenAI Blog_, 1(8):9.
* Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bertnetworks. _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_.
* Ritter et al. (2010) Alan Ritter, Colin Cherry, and Bill Dolan. 2010. Unsupervised modeling of twitter conversations. In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_, pages 172-180, Los Angeles, California. Association for Computational Linguistics.
* Rush et al. (2015) Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 379-389, Lisbon, Portugal. Association for Computational Linguistics.
* Sacks et al. (1978) Harvey Sacks, Emanuel A Schegloff, and Gail Jefferson. 1978. A simplest systematics for the organization of turn taking for conversation. In _Studies in the organization of conversational interaction_, pages 7-55. Elsevier.
* See et al. (2017)bigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
* Shang et al. (2018) Guokan Shang, Wensi Ding, Zekun Zhang, Antoine Tixier, Polykarpos Meladianos, Michalis Vazirgiannis, and Jean-Pierre Lorne. 2018. Unsupervised abstractive meeting summarization with multisentence compression and budgeted submodular maximization. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 664-674, Melbourne, Australia. Association for Computational Linguistics.
* Wu et al. (2019) Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. 2019. Pay less attention with lightweight and dynamic convolutions. In _International Conference on Learning Representations_.
* Xiao and Carenini (2019) Wen Xiao and Giuseppe Carenini. 2019. Extractive summarization of long documents by combining global and local context. In _EMNLP/IJCNLP_.
* Zhang et al. (2019) Weiwei Zhang, Jackie Chi Kit Cheung, and Joel Oren. 2019. Generating character descriptions for automatic summarization of fiction. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 33, pages 7476-7483.
* Zhao et al. (2019) Zhou Zhao, Haojie Pan, Changjie Fan, Yan Liu, Linlin Li, Min Yang, and Deng Cai. 2019. Abstractive meeting summarization via hierarchical adaptive segmental network learning. In _The World Wide Web Conference_, WWW '19, page 3455-3461, New 

York, NY, USA. Association for Computing Machinery.
* Zhu et al. (2020a) Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng, Xuedong Huang, and Meng Jiang. 2020a. Boosting factual correctness of abstractive summarization.
* Zhu et al. (2020b) Chenguang Zhu, Ruochen Xu, Michael Zeng, and Xuedong Huang. 2020b. End-to-end abstractive summarization for meetings.
* Zhu et al. (2019) Henghui Zhu, Feng Nan, Zhiguo Wang, Ramesh Nallapati, and Bing Xiang. 2019. Who did they respond to? conversation structure modeling using masked hierarchical transformer.

## Appendix A Model Settings

We load the pre-trained "bert-base-nli-stsb-mean-tokens"8 for sentence-BERT to get representations for each utterance. When extracting the **topic view**, we set the window size 4 and std coefficient 1 in C99. When extracting the **stage view**, we set the number of hidden states 4 in HMM. These hyper-parameters were set after a grid search with evaluating randomly sampled segmented results by human. The **BART + Structured views** (stage and topic views) followed the same parameters as **BART + Generic views**.

Footnote 8: [https://github.com/UKPLab/sentence-transformers](https://github.com/UKPLab/sentence-transformers)

For **Multi-View BART**, we selected different views to combine: (1) generic view + structured view: best generic view, global view, was combined with two structured views (stage and topic view); (2) structured view + structured view: best two single views are combined (topic + stage). The settings for BART encoder/decoder kept the same as baseline. We used a one layer LSTM for encoding sections. The learning rate for section encoder and multi-view attention was set 3e-3. The temperature \(T\) was 0.2. The beam search size during inference for all the models was 4.

Experiments were performed on two Tesla P100 (16GB memory).

## Appendix B View Attention Visualization

We visualized the attention weights distribution for the stage view and topic view in our best multi-view model to explore the importance of stage verses topic in Figure 6.We found that the topic views were more prominent than the stage views, consistent with the performances of _BART + topic view_ and _BART + stage_ view. This indicated that having discourse structures about topics might be more important while both topic and stage could improve the conversation summarization. This also communicated that the two different structured views can complement each other well though sharing the same dialogue content.

We displayed two examples in Table 8 with the golden references, each single view's generated summaries and the combined views' generated summaries. The combined view could balance the advantages of each single view and generated more precise summaries. And the attention weights the model learned were also consistent with single view's performances.

## Appendix C Supplementary Examples for Model Analysis and Discussion

For the analysis in the **Model Analysis and Discussion** section in our paper, we randomly sampled 100 examples from the test set of the SAMSum

Figure 6: Attention weights distribution for stage view and topic view in the multi-view model.

 

dataset which can be downloaded here 9. Table 7 provides a full index list of the samples.

Footnote 8: [https://arxiv.org/abs/1911.12237](https://arxiv.org/abs/1911.12237)

Table 9 shows the error analysis for _BART-Discrete, BART-Global, BART-Stage, BART-Topic_ and _BART-Multi-view_ models. It can be observed that, (i) without any explicit structures, discrete-view and global-view models generated summaries with more _redundancies_ compared to golden reference summaries, as models may easily lost focus on massive information; (ii) once we introduced certain conversation structures such as topic-view and stage-view, models behaved better in terms of _redundancy_ and _incorrect reasoning_, which indicated that the structured views could help models to better understand the conversations; (iii) our multi-view models which combined both stage-view and topic-view made the least number of errors compared to all single view models, suggesting the effectiveness of combining different views for conversation summarization.

\begin{table}
\begin{tabular}{|c|p{56.9pt}||l|} \hline  & James misses Hannah. They agree for James to pick Hannah up on Saturday at 8. & Petra is very sleepy at work today, Andy finds the day boring, and Ezgi is working. \\ \hline \hline Stage & Hannah has to get up early for wo-rk tomorrow. James will pick her up at 8 on Saturday. & Petra needs to sleep, because she’s sleepy. Ezgi is working. [0.37/0.16/0.38] \\ \cline{2-4}  & Jones and Hannah will see each other on Saturday at 8. & Nobody is working at the office today. Ezgi is working. Petra is sleepy and wants to sleep. [0.53/0.19/0.53] \\ \hline Stage + Topic & James will pick Hannah up on Saturday at 8 pm. & Petra is sleepy and needs to sleep. Ezgi is working at the office. [0.60/0.21/0.43] \\ \hline Attention Weight & [0.52, 0.48] & [0.45. 0.55] \\ \hline \end{tabular}
\end{table}
Table 8: Some generated summary examples compared to references. [Rouge-1/Rouge-2/Rouge-L] is shown after each summary, and [stage weight/topic weight] is displayed in the last row.

\begin{table}
\begin{tabular}{|r|c|c|c|c|c|} \hline
**Errors** & **Discrete** & **Global** & **Stage** & **Topic** & **Multi-view** \\ \hline Other & 16 & 19 & 21 & 22 & 24 \\ \hline Missing information & 40 & 46 & 45 & 42 & 37 \\ Redundancy & 33 & 44 & 18 & 25 & 13 \\ Wrong references & 32 & 33 & 26 & 30 & 27 \\ Incorrect reasoning & 27 & 28 & 22 & 28 & 24 \\ Improper gendered pronouns & 5 & 6 & 6 & 6 \\ \hline \end{tabular}
\end{table}
Table 9: Common error types of different models compared to golden reference on 100 sampled conversations.