SlowGuess/NAACL_2021_OCR huggingface dataset:
Jiaao Chen
School of Interactive Computing Georgia Institute of Technology
jiaochen@gatech.edu
Abstractive conversation summarization has received much attention recently. However, these generated summaries often suffer from insufficient, redundant, or incorrect content, largely due to the unstructured and complex characteristics of human-human interactions. To this end, we propose to explicitly model the rich structures in conversations for more precise and accurate conversation summarization, by first incorporating discourse relations between utterances and action triples ("WHO-DOING-WHAT") in utterances through structured graphs to better encode conversations, and then designing a multi-granularity decoder to generate summaries by combining all levels of information. Experiments show that our proposed models outperform state-of-the-art methods and generalize well in other domains in terms of both automatic evaluations and human judgments. We have publicly released our code at https://github.com/GT-SALT/Structure-Aware-BART.
Online interaction has become an indispensable component of everyday life and people are increasingly using textual conversations to exchange ideas, make plans, and share information. However, it is time-consuming to recap and grasp all the core content within every complex conversation (Gao et al., 2020; Feng et al., 2020). As a result, how to organize massive everyday interactions into natural, concise, and informative text, i.e., abstractive conversation summarization, starts to gain importance.
Significant progress has been made on abstractive summarization for structured document via pointer generator (See et al., 2017), reinforcement methods (Paulus et al., 2018; Huang et al., 2020a) and pre-trained models (Liu and Lapata, 2019; Lewis et al., 2020; Zhang et al., 2019). Despite the huge success, it is challenging to directly apply document models to summarize conversations, due to
(3) Helen: I was calling you before ....
(b) Action Graph
Figure 1: An example of discourse relation graph (a) and action graph (b) from one conversation in SAMSum (Gliwa et al., 2019). The annotated summary is Simon was on the phone before, so he didn't here Helen calling. Simon will fetch Helen some tissues.
(4) Simon: Sorry I was on the phone. I didn't hear you .... Tell me Comment
(5) Helen: The toilet paper is finished, could you fetch me some tissues, please? QAP
(6) Simon: Hahaha sure, no worries!
(a) Discourse Relation Graph
a set of inherent differences between conversations and documents (Gliwa et al., 2019). First, speaker interruptions like repetitions, false-starts, and hesitations are frequent in conversations (Sacks et al., 1978), and key information resides in different portions of a conversation. These unstructured properties pose challenges for models to focus on salient contents that are necessary for generating both abstractive and informative summaries. Second, there is more than one speaker in conversations and people interact with each other in different language styles (Zhu et al., 2020b). The complex interactions among multiple speakers make it harder for mod
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1380-1391 June 6-11, 2021. ©2021 Association for Computational Linguistics
els to identify and associate speakers with correct actions so as to generate factual summaries.
In order to summarize the unstructured and complex conversations, a growing body of research has been conducted, such as transferring document summarization methods to conversation settings (Shang et al., 2018; Gliwa et al., 2019), adopting hierarchical models (Zhao et al., 2019; Zhu et al., 2020b), or incorporating conversation structures like topic segmentation (Liu et al., 2019b; Li et al., 2019; Chen and Yang, 2020), dialogue acts (Goo and Chen, 2018), and conversation stages (Chen and Yang, 2020). However, current approaches still face challenges in terms of succinctness and faithfulness, as most prior studies (i) fail to explicitly model dependencies between utterances which can help identify salient portions of conversations (Bui et al., 2009), and (ii) lack structured representations (Huang et al., 2020a) to learn the associations between speakers, actions and events. We argue that these rich linguistic structures associated with conversations are key components towards generating abstractive and factual conversation summaries.
To this end, we present a structure-aware sequence-to-sequence model, in which we equip abstractive conversation summarization models with rich conversation structures through two types of graphs: discourse relation graph and action graph. Discourse relation graphs are constructed based on dependency-based discourse relations (Kirschner et al., 2012; Stone et al., 2013; Asher et al., 2016; Qin et al., 2017) between intertwined utterances, where each Elementary Discourse Unit (EDU) is one single utterance and they are linked through 16 different types of relations (Asher et al., 2016). As shown in Figure 1(a), highly related utterances are linked based on discourse relations like Question Answer Pairs, Comment and Explanation. Explicitly modeling these utterances relations in conversations can aid models in recognizing key content for succinct and informative summarization. Action graphs are constructed as the "WHO-DOING-WHAT" triplets in conversations which express socially situated identities and activities (Gee, 2014). For instance, in Figure 1(b), the action graph provides explicit information between Simon, fetch, and tissues for the utterance it is Simon who will fetch the tissues, making models less likely to generate summaries with wrong references (e.g., Helen will fetch the tissues).
To sum up, our contributions are: (1) We pro
pose to utilize discourse relation graphs and action graphs to better encode conversations for conversation summarization. (2) We design structure-aware sequence-to-sequence models to combine these structured graphs and generate summaries with the help of a novel multi-granularity decoder. (3) We demonstrate the effectiveness of our proposed methods through experiments on a large-scale conversation summarization dataset, SAM-Sum (Gliwa et al., 2019). (4) We further show that our structure-aware models can generalize well in new domains such as debate summarization.
Document Summarization Compared to extractive document summarization (Gupta and Lehal, 2010; Narayan et al., 2018; Liu and Lapata, 2019), abstractive document summarization is generally considered more challenging and has received more attention. Various methods have been designed to tackle abstractive document summarization like sequence-to-sequence models (Rush et al., 2015), pointer generators (See et al., 2017), reinforcement learning methods (Paulus et al., 2018; Huang et al., 2020a) and pre-trained models (Lewis et al., 2020; Zhang et al., 2019). To generate faithful abstractive document summaries (Maynez et al., 2020), graph-based models were introduced recently such as extracting entity types (Fernandes et al., 2018; Fan et al., 2019), leveraging knowledge graphs (Huang et al., 2020a; Zhu et al., 2020a) or designing extra fact correction modules (Dong et al., 2020). Inspired by these graph-based methods, we also construct action graphs for generating more factual conversation summaries.
Conversation Summarization Extractive dialogue summarization (Murray et al., 2005) has been studied extensively via statistical machine learning methods such as skip-chain CRFs (Galley, 2006), SVM with LDA models (Wang and Cardie, 2013), and multi-sentence compression algorithms (Shang et al., 2018). Such methods struggled with generating succinct, fluent, and natural summaries, especially when the key information needs to be aggregated from multiple first-person point-of-view utterances (Song et al., 2020). Abstractive conversation summarization overcomes these issues by designing hierarchical models (Zhao et al., 2019; Zhu et al., 2020b), incorporating commonsense knowledge (Feng et al., 2020), or leveraging conversational structures like dialogue acts (Goo and Chen,
(a) Utterance and Graph Encoder
(b) Multi-granularity Decoder
Figure 2: Model architecture. Each utterance is encoded via transformer encoder; discourse relation graphs and action graphs are encoded through Graph Attention Networks (a). The multi-granularity decoder (b) then generates summaries based on all levels of encoded information including utterances, action graphs, and discourse graphs.
2018), key point sequences (Liu et al., 2019a), topic segments (Liu et al., 2019b; Li et al., 2019) and stage developments (Chen and Yang, 2020). Some recent research has also utilized discourse relations as input features in classifiers to detect important content in conversations (Murray et al., 2006; Bui et al., 2009; Qin et al., 2017). However, current models still have not explicitly utilized the dependencies between different utterances, making models hard to leverage long-range dependencies and utilize these salient utterances. Moreover, less attention has been paid to identify the actions of different speakers and how they interact with or refer to each other, leading to unfaithful summarization with incorrect references or wrong reasoning (Gliwa et al., 2019). To fill these gaps, we propose to explicitly model actions within utterances, and relations between utterances in conversations in a structured way, by using discourse relation graphs and action graphs and further combining these through relational graph encoders and multi-granularity decoders for abstractive conversation summarization.
To generate abstractive and factual summaries from unstructured conversations, we propose to model structural signals in conversations by first constructing discourse relation graphs and action graphs (Section 3.1), and then encoding the graphs together with conversations (Section 3.2) as well as incorporating these different levels of information in the decoding stage through a multi-granularity
decoder (Section 3.3) to summarize given conversations. The overall architecture is shown in Figure 2.
3.1 Structured Graph Construction
This section describes how to construct the discourse relation graphs and action graphs. Formally, for a given conversation  $\mathbf{C} = \{\mathbf{u}_0,\dots ,\mathbf{u}_m\}$  with  $m$  utterances, we construct discourse relation graph  $\mathcal{G}^D = (\mathbf{V}^D,\mathbf{E}^D)$ , where  $\mathbf{V}^D$  is the set of nodes representing Elementary Discourse Units (EDUs), and  $\mathbf{E}^D$  is the adjacent matrix that describes the relations between EDUs, and action graph  $\mathcal{G}^A = (\mathbf{V}^A,\mathbf{E}^A)$ , where  $\mathbf{V}^A$  is the set of nodes representing "WHO", "DOING" and "WHAT" arguments, and  $\mathbf{E}^A$  is the adjacent matrix to link "WHO-DOING-WHAT" triples.
Discourse Relation Graph Utterances from different speakers do not occur in isolation; instead, they are related within the context of discourse (Murray et al., 2006; Qin et al., 2017), which has been shown effective for dialogue understanding like identifying the decisions in multi-party dialogues (Bui et al., 2009) and detecting salient content in email conversations (McKeown et al., 2007). Although current attention-based neural models are supposed to, or might implicitly, learn certain relations between utterances, they often struggle to focus on many informative utterances (Chen and Yang, 2020; Song et al., 2020) and fail to address long-range dependencies (Xu et al., 2020), especially when there are frequent interruptions. As a result, explicitly incorporating the discourse relations will help neural summarization models better
encode the unstructured conversations and concentrate on the most salient utterances to generate more informative and less redundant summaries.
To do so, we view each utterance as an EDU and use the discourse relation types defined in Asher et al. (2016). We first pre-train a discourse parsing model (Shi and Huang, 2019) on a human-annotated multiparty dialogue corpus (Asher et al., 2016), with 0.775 F1 score on link predictions and 0.557 F1 score on relation classifications, which are comparable to the state-of-the-art results (Shi and Huang, 2019). We then utilize this pre-trained parser to predict the discourse relations within conversations in our SAMSum corpus (Gliwa et al., 2019).
After predictions, there are 138,554 edges identified in total and 8.48 edges per conversation. The distribution of these predicted discourse relation types is: Comment (19.3%), Clarification Question (15.2%), Elaboration (2.3%), Acknowledgement (8.4%), Continuation (10.1%), Explanation (2.8%), Conditional (0.2%), Question Answer Pair (21.5%), Alternation (0.3%), Q-Elab (2.5%), Result (5.5%), Background (0.4%), Narration (0.4%), Correction (0.4%), Parallel (0.9%), and Contrast (1.0%). Then for each conversation, we construct a discourse relation graph  $\mathcal{G}^D = (\mathbf{V}^D,\mathbf{E}^D)$ , where  $\mathbf{V}^D[k]$  represents the  $k$ -th utterance.  $\mathbf{E}^D[i][j] = r$  if there is a link from the  $i$ -th utterance to the  $j$ -th one with discourse relation  $r$ .
Action Graph The "who-doing-what" triples from utterances can provide explicit visualizations of speakers and their actions, the key to understanding concrete details happened in conversations (Moser, 2001; Gee, 2014; Sacks et al., 1978). Simply relying on neural models to identify this information from conversations often fail to produce factual characterizations of concrete details happened (Cao et al., 2018; Huang et al., 2020a). To this end, we extract "WHO-DOING-WHAT" triples from utterances and construct action graphs for conversation summarization (Chen et al., 2019; Huang et al., 2020b,a). Specifically, we first transform the first-person point-of-view utterances to its third-person point-of-view forms based on simple rules: (i) substituting first/second-person pronouns with the names of current speaker or surrounding speakers and (ii) replacing third-person pronouns based on coreference clusters in conversations detected by the Stanford CoreNLP (Manning et al., 2014). For example, an utterance "I'll bring it to you to
morrow" from Amanda to Jerry will be transformed into "Amanda'll bring cakes to Jerry tomorrow". Then we extract "WHO-DOING-WHAT" (subject-predicate-object) triples from transformed conversations using the open information extraction (OpenIE) systems  ${}^{1}$  (Angeli et al.,2015). We then construct the Action Graph  ${\mathcal{G}}^{A} = \left( {{\mathbf{V}}^{A},{\mathbf{E}}^{A}}\right)$  from the extracted triples by taking arguments ("WHO", "DOING", or "WHAT") as nodes in  ${\mathbf{V}}^{A}$  ,and connect them with edge  ${\mathbf{E}}^{A}\left\lbrack  i\right\rbrack  \left\lbrack  j\right\rbrack   = 1$  if they are adjacent in one "WHO-DOING-WHAT" triple.
Given a conversation and its corresponding discourse relation graph and action graph, we utilize an utterance encoder and two graph encoders, to obtain its hidden representations shown in Figure 2(a).
3.2.1 Utterance Encoder
We initialize our utterance encoder  $F_{U}(.)$  with a pre-trained encoder, i.e., BART-base (Lewis et al., 2020), and encode tokens  $\{x_{i,0},\dots,x_{i,l}\}$  in an utterance  $\mathbf{u}_i$  into its hidden representation:
$$
\left\{h _ {i, 0} ^ {U}, \dots , h _ {i, l} ^ {U} \right\} = F _ {U} \left(\left\{x _ {i, 0}, \dots , x _ {i, l} \right\}\right) \tag {1}
$$
Here we add a special token  $x_{i,0} = \langle \mathrm{s} \rangle$  at the beginning of each utterance to represent it.
3.2.2 Graph Encoder
Node Initialization For discourse relation graph, we employ the output embeddings of the special tokens  $x_{i,0}$  from the utterance encoder, i.e.,  $h_{i,0}^{U}$ , to initialize the  $i$ -th node  $v_{i}^{D}$  in  $\mathcal{G}^{D}$ . We use a one-hot embedding layer to encode the relations  $\mathbf{E}^{D}[i][j] = e_{i,j}^{D}$  between utterance  $i$  and  $j$ . For action graph, we first utilize  $F_{U}(.)$  to encode each token in nodes  $v_{i}^{A}$  and then average their output embeddings as their initial representations.
Structured Graph Attention Network Based on Graph Attention Network (Velicković et al., 2018), we utilize these relations between nodes to encode each node  $v_{i}^{D}$  in  $\mathcal{G}^D$  or  $v_{i}^{A}$  in  $\mathcal{G}^A$  through:
$$
\begin{array}{l} \alpha_ {i j} = \frac {\exp \left(\sigma \left(\mathbf {a} ^ {T} \left[ \mathbf {W} v _ {i} \| \mathbf {W} v _ {j} \| \mathbf {W} _ {\mathbf {e}} e _ {i , j} ]\right)\right)\right)}{\sum_ {k \in \mathcal {N} _ {i}} \exp \left(\sigma \left(\mathbf {a} ^ {T} \left[ \mathbf {W} v _ {i} \| \mathbf {W} v _ {k} \| \mathbf {W} _ {\mathbf {e}} e _ {i , k} ]\right)\right)\right)} \\ h _ {i} = \sigma (\sum_ {j \in \mathcal {N} _ {i}} \alpha_ {i j} \mathbf {W} v _ {j}) \\ \end{array}
$$
<https://github.com/philipperemy/Stanford-OpenIE-Python>
Dataset	Split	# Conv	# Participants	# Turns	# Discourse Edges	# Action Triples
SAMSum	Train	14732	2.40	11.17	8.47	6.72
Val	818	2.39	10.83	8.34	6.48
Test	819	2.36	11.25	8.63	6.81
ADSC	Full	45	2.00	7.51	6.51	37.20
Table 1: Statistics of the used datasets, including the total number of conversations (# Conv), the average number of participants, turns, discourse edges and action triples per conversation.
$\mathbf{W}, \mathbf{W}_{\mathrm{e}}$  and a are trainable parameters.  $[\cdot, \|\cdot]$  denotes the concatenation of two vectors.  $\sigma$  is the activation function,  $\mathcal{N}_i$  is the set containing node  $i$ 's neighbours in  $\mathcal{G}$ .
Through two graph encoders  $F_{D}(:,.)$  and  $F_{A}(.,.)$ , we then obtain the hidden representations of these nodes as:
$$
\left\{h _ {0} ^ {D}, \dots , h _ {m} ^ {D} \right\} = F _ {D} \left(\left\{v _ {0} ^ {D}, \dots , v _ {m} ^ {D} \right\}, \mathbf {E} ^ {D}\right) \tag {2}
$$
$$
\left\{h _ {0} ^ {A}, \dots , h _ {n} ^ {A} \right\} = F _ {A} \left(\left\{x _ {0} ^ {A}, \dots , x _ {n} ^ {A} \right\}, \mathbf {E} ^ {A}\right) \tag {3}
$$
3.3 Multi-Granularity Decoder
Different levels of encoded representations are then aggregated via our multi-granularity decoder to generate summaries as shown in Figure 2(b). With  $s - 1$  previously generated tokens  $y_{1},\ldots ,y_{s - 1}$ , our decoder  $G(.)$  predicts the  $l$ -th token via:
$$
\hat {y} = G \left(y _ {1: s - 1}, F _ {U} (\mathbf {C}), F _ {D} \left(\mathcal {G} ^ {D}\right), F _ {A} \left(\mathcal {G} ^ {A}\right)\right) \tag {4}
$$
$$
P \left(\tilde {y} _ {s} \mid y _ {<   s}, \mathbf {C}, \mathcal {G} ^ {D}, \mathcal {G} ^ {A}\right) = \operatorname {S o f t m a x} \left(W _ {p} \hat {y}\right) \tag {5}
$$
To better incorporate the information in constructed graphs, different from the traditional pretrained BART model (Lewis et al., 2020), we improve the BART transformer decoder with two extra cross attentions (Discourse Attention and Action Attention) added to each decoder layer, which attends to the encoded node representations in discourse relation graphs and action graphs.
In each decoder layer, after performing the original cross attentions over every token in utterances  $\{h_{i,0:l}^{U}\}$  and getting the utterance-attended representation  $x^{U}$ , multi-granularity decoder then conducts cross attentions over nodes  $\{h_{0:m}^{D}\}$  and  $\{h_{0:n}^{A}\}$  that are encoded from graph encoders in parallel, to obtain the discourse-attended representation  $x^{D}$  and action-attended representation  $x^{A}$ . These two attended vectors are then combined into a structure-aware representation  $x^{S}$ , through a feed-forward network for further forward passing in the decoder.
To alleviate the negative impact of randomly initialized graph encoders and cross attentions over graphs on pre-trained BART decoders at
early stages and accelerate the learning of newly-introduced modules during training, we apply ReZero (Bachlechner et al., 2020) to the residual connection after attending to graphs in each decoder layer:
$$
\tilde {x} ^ {S} = x ^ {U} + \alpha x ^ {S} \tag {6}
$$
where  $\alpha$  is one trainable parameter instead of a fixed value 1, which modulates updates from cross attentions over graphs.
Training During training, we seek to minimize the cross entropy and use the teacher-forcing strategy (Bengio et al., 2015):
$$
\mathcal {L} = - \sum \log P \left(\tilde {y} _ {l} \mid y _ {<   l}, \mathbf {C}, \mathcal {G} ^ {D}, \mathcal {G} ^ {A}\right) \tag {7}
$$
We trained and evaluated our models on a conversation summarization dataset SAMSum (Gliwa et al., 2019) covering messenger-like conversations about daily topics, such as arranging meetings and discussing events. We also showed the generalizability of our models on the Argumentative Dialogue Summary Corpus (ADSC) (Misra et al., 2015), a debate summarization corpus. The data statistics of two datasets were shown in Table 1, with the discourse relation types distributions in the Appendix.
We compare our methods with several baselines:
- Pointer Generator (See et al., 2017): We followed the settings in Gliwa et al. (2019) and used special tokens to separate each utterance.
- Transformer (Vaswani et al., 2017): We trained transformer seq2seq models following the OpenNMT (Klein et al., 2017).
D-HGN (Feng et al., 2020) incorporated commonsense knowledge from ConceptNet (Liu and Singh, 2004) for dialogue summarization.
Model	ROUGE-1	ROUGE-2	ROUGE-L
F	P	R	F	P	R	F	P	R
Pointer Generator (See et al., 2017)	40.08	-	-	15.28	-	-	36.63	-	-
Transformer (Vaswani et al., 2017)	37.27	-	-	10.76	-	-	32.73	-	-
D-HGN (Feng et al., 2020)	42.03	-	-	18.07	-	-	39.56	-	-
Multi-view Seq2Seq (Chen and Yang, 2020)	45.56	52.13	44.68	22.30	25.58	22.03	44.70	50.82	43.29
BART (Lewis et al., 2020)	45.15	49.58	45.97	21.66	23.95	22.16	44.46	48.92	44.26
S-BART w. Discourse †	45.89	51.34	45.87	22.50	25.26	22.33	44.83	49.93	44.17
S-BART w. Action †	45.67	50.25	46.44	22.39	24.70	22.96	44.86	49.29	44.75
S-BART w. Discourse&Action †	46.07	51.13	46.24	22.60	25.11	22.81	45.00	49.82	44.47
Table 2: ROUGE-1, ROUGE-2 and ROUGE-L scores for different models on the SAMSum Corpus test set. Results are averaged over three random runs.  $\dagger$  means our methods. We performed Pitman's permutation test (Dror et al., 2018) and found that  $S$ -BART w. Discourse& Action significantly outperformed the base BART ( $p < 0.05$ ).
Model	ROUGE-1	ROUGE-2	ROUGE-L
F	P	R	F	P	R	F	P	R
BART (Lewis et al., 2020)	20.90	51.71	13.53	5.04	12.46	3.24	21.23	56.29	13.54
S-BART w. Discourse †	22.42	54.13	14.60	5.58	13.83	3.61	22.16	51.88	14.45
S-BART w. Action †	30.91	85.42	19.12	20.64	56.31	12.78	35.30	85.51	22.58
S-BART w. Discourse&Action †	34.74	84.99	22.20	23.86	58.08	15.24	38.69	83.81	25.51
- BART (Lewis et al., 2020): We utilized BART  ${}^{2}$  ,and separated utterances by a special token.
- Multi-View Seq2Seq (Chen and Yang, 2020) utilized topic and stage views on top of BART for summarizing conversations. Here we implemented it based on BART-base models.
We used the BART-base model to initialize our sequence-to-sequence model for training in all experiments. For parameters in the original BART encoder/decoder, we followed the default settings and set the learning rate 3e-5 with 120 warm-up steps. For graph encoders, we set the number of hidden dimensions as 768, the number of attention heads as 2, the number of layers as 2, and the dropout rate as 0.2. For graph cross attentions added to BART decoder layers, we set the number of attention heads as 2. The weights  $\alpha$  in ReZero residual connections were initialized with 1. The learning rate for parameters in newly added modules was 3e-4 with 60 warm-up steps. All experiments were performed on GeForce RTX 2080Ti (11GB memory).
4.4 Results on In-Domain Corpus
Automatic Evaluation We evaluated all the models with the widely used automatic metric,
Table 3: ROUGE-1, ROUGE-2 and ROUGE-L scores on the out-of-domain ADSC corpus using different models trained on SAMSum Corpus.  $\dagger$  means our methods.
Models	Fac.	Suc.	Inf.
Ground Truth	4.29	4.40	4.06
BART	3.90	4.13	3.74
S-BART w. Discourse	4.11	4.42	3.98
S-BART w. Action	4.17	4.29	3.95
S-BART w. Discourse&Action	4.19	4.41	3.91