SlowGuess/NAACL_2022_OCR huggingface dataset:
TANET: Thread-Aware Pretraining for Abstractive Conversational Summarization
Ze Yang $^{1}$ , Liran Wang $^{1}$ , Zhoujin Tian $^{1}$ , Wei Wu $^{2}$ , Zhoujun Li $^{1*}$
<sup>1</sup>State Key Lab of Software Development Environment, Beihang University, Beijing, China
$^{2}$ Meituan, Beijing, China
{tobey, wanglr, eikitbar, lizj}@buaa.edu.cn
wuwei19850318@gmail.com
Although pre-trained language models (PLMs) have achieved great success and become a milestone in NLP, abstractive conversational summarization remains a challenging but less studied task. The difficulty lies in two aspects. One is the lack of large-scale conversational summary data. Another is that applying the existing pre-trained models to this task is tricky because of the structural dependence within the conversation and its informal expression, etc. In this work, we first build a large-scale (11M) pretraining dataset called RCSUM, based on the multi-person discussions in the Reddit community. We then present TANET, a thread-aware Transformer-based network. Unlike the existing pre-trained models that treat a conversation as a sequence of sentences, we argue that the inherent contextual dependency among the utterances plays an essential role in understanding the entire conversation and thus propose two new techniques to incorporate the structural information into our model. The first is thread-aware attention which is computed by taking into account the contextual dependency within utterances. Second, we apply thread prediction loss to predict the relations between utterances. We evaluate our model on four datasets of real conversations, covering types of meeting transcripts, customer-service records, and forum threads. Experimental results demonstrate that TANET achieves a new state-of-the-art in terms of both automatic evaluation and human judgment.
Text summarization is a long-standing challenging task in artificial intelligence, aiming to condense a piece of text to a shorter version, retaining the critical information. There are various promising applications of conversational summarization in the real world, emphasizing the need to build auto summarization systems. For example, online
Figure 1: An abbreviated example from RCSUM. It contains a total of 14k comments and more than 210k words in this post. The title and the lead comment are selected as the pseudo summary of this thread.
People who are  $40+$  and happy with their life, what is your advice to people in their 20s?
91.9k
Don't fall for the trap that your life needs to be one long narrative that you should be building. Life is .... Don't try to make your
life into a novel, make it a book of poems. 11.9k 12 months ago
I needed this. Thank you. I always compare my life to a movie, and it stresses the hell out of me because reality is never like it, but I guess this is life after all. 804 12 months ago
…
Why do you say it's a trap? I think both can be viable ways of experiencing life. Of course, we only get to experience one, so who is to say? This reminds me of Eastern vs Western music structure .... 10 12 months ago
It is a trap in the sense that believing that it is the only way to have a successful life sets you up for disappointment, because many people do not have a strong conviction towards one path, making them feel like they are wasting time when they should be getting started on a long career path, and it is very difficult for people to actually have that much control over their lives without immense privilege and great luck... 10 12 months ago
customer-service staff can improve work efficiency by recording the customer demands and current solutions after each communication. In the industry, meeting summaries are also generally required in order to track the progress of projects. The automatic doctor-patient interaction summary can save doctors much time from filling out medical records. Therefore, conversational summarization has been a potential field in summarization and has received increasing attention.
Benefiting from the availability of large-scale high-quality data, abstractive document summarization has been extensively explored in the past years (Rush et al., 2015; See et al., 2017; Chen and Bansal, 2018). Recently, the pretraining methods further extend the success (Lewis et al., 2020; Zhang et al., 2020). In contrast, abstractive conversational summarization is a more challenging but less studied task. The reason mainly lies in: (1) compared with news, there are no large-scale
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2594 - 2607
publicly available labeled datasets for abstractive conversational summarization; (2) conversations are usually informal, verbose, and repetitive, sprinkled with false starts, backchanneling, reconfirmations, hesitations, and speaker interruptions (Sacks et al., 1978), which makes the whole session difficult to understand; (3) unlike the linear relationship in the one-speaker document, there are always multiple speakers in a conversation, and the inherent contextual relationships are structured; (4) conversations in some scenarios could be much longer than a document. For instance, in CNN/Daily Mail dataset (Hermann et al., 2015), the average number of words in a document is 781, while the average length of the transcripts in ICSI, a widely explored meeting corpus, is 10, 189. These challenges encourage us to explore conversation-oriented summarization methods.
To overcome the challenges, we study pretraining for abstractive conversational summarization in this work. To tackle the bottleneck of insufficient data, we first build a large-scale (11M) corpus for conversational summarization called RCSUM, based on the multi-person discussions crawled from the Reddit website. For the absence of the summary of discussions, we propose two heuristic strategies to select the title and the lead comment of a thread as its summary-like sentences. Figure 1 shows an abbreviated example in RCSUM. For the model architecture, we present TANET, a Thread-Aware NETwork for abstractive conversational summarization. As conversations are usually lengthy, we adopt the hierarchical encoders, which consist of a token encoder and an utterance encoder. Unlike the existing pre-trained models that treat a conversation as a sequence of sentences, we argue that the inherent contextual dependency among the utterances plays an essential role in understanding the entire conversation and thus propose two new techniques to incorporate the structural information into TANET. First, we replace the self-attention layers in the utterance encoder with the thread-aware relative attention. Second, we propose a new pretraining task, the thread prediction, to further enhance the representations by predicting the relations across a small set of utterances.
We evaluate TANET on four datasets of conversational summarization, covering domains of meeting transcripts (Carletta et al., 2005; Janin et al., 2003), customer-service records (Yuan and Yu, 2019), and forum threads (Tarnpradab et al., 2017).
Experimental results indicate that TANET achieves new state-of-the-art on all datasets in terms of both automatic evaluation and human judgement.
In summary, our contributions in this work are three-fold: (1) We build a large-scale pretraining corpus based on real conversations for abstractive conversational summarization. (2) TANET is the first pre-trained abstractive conversational summarization model with inherent structure modeling. (3) The effectiveness of TANET is demonstrated on four downstream datasets of conversational summarization, covering types of meeting transcripts, customer-service records, and forum threads.
In general, the abstractive conversational summarization task could be formalized as follows. Denote the dataset as  $\mathcal{D} = \{(C_i,S_i)\}_{i = 1}^N$ , where  $\forall i$ ,  $(C_i,S_i)$  is a conversation-summary pair and  $N$  is the size of  $\mathcal{D}$ . The conversation  $C_i = \{(u_{i,j},a_{i,j})\}_{j = 1}^{n_i}$  consists of  $n_i$  rounds of utterances  $\{u_{i,j}\}_{j = 1}^{n_i}$  and their associated attributes  $\{a_{i,j}\}_{j = 1}^{n_i}$ . For example, each meeting transcript in the AMI dataset comprises multiple turns, where each turn is an utterance of a participant who has a specific role in the project, such as manager or designer. Our goal is to learn a generation probability  $P(S|C)$ , so that given a new conversation input  $C$ , we can generate a summary  $S$ .
Since there is always limited availability of  $\mathcal{D}$  to support accurately learning for  $P(S|C)$ , we propose to build a large-scale summarization-like corpus  $\mathcal{D}_p = \{(C_k, S_k)\}_{k=1}^M (M \gg N)$  by leveraging massive accessible conversation data.  $S_k$  represents the pseudo summary of the conversation  $C_k$ . In this way, we first pretrain our model on  $\mathcal{D}_p$  and then finetune it on the respective dataset  $\mathcal{D}$  of each downstream task.
3 Reddit for Conversational Summarization
Existing conversational summarization corpora (Carletta et al., 2005; Janin et al., 2003; Tarnpradab et al., 2017; Yuan and Yu, 2019; Gliwa et al., 2019) have a low number of conversations, which prevents research community from engaging into this problem. Different from (Zhu et al., 2020) that using news documents to simulate multi-person conversations for pretraining, it is reasonable to hypothesize that leveraging real conversation data could lead to better downstream performance. In
this work, to benefit from the large-scale conversation corpus, we mined and processed a large-scale dataset from Reddit  $^{1}$  for Conversational Summarization called RCSUM. Figure 1 shows an example in the dataset. To our best knowledge, RCSUM is the first large-scale pretraining corpus with real conversations for abstractive conversational summarization.
We collected the posts on the Reddit site from 2019 to 2020. A post is composed of a title and its corresponding discussions which usually consist of multiple threads. The comments in a thread can naturally expand into a tree structure. Remarkably, each comment has rich attributes, including the user information, creation timestamp and the accumulated score $^2$ , etc. With the large-scale real multi-person conversation data, the key is how to construct a summary-like instance for a thread. We consider two strategies to select sentences that appear to dominate the thread: (1) Title. The discussions of each post are all developed upon the topic of the title, so we select the title as a part of the pseudo summary; (2) Lead Comment. Despite the topic given by the title, the lead comment (i.e., the first comment of a thread) also well influences the future direction of what is discussed in this thread. We concatenate them as the pseudo summary of the discussions in a thread. Lead comment's original position is replaced by a special token [MASK].
To clean up RCSUM, we adopted a series of heuristics including: (1) We removed any threads where the number of comments less than 10; (2) We discarded any not-safe-for-work posts, such as posts containing adult or violent content; (3) We replaced all URLs with a special token [URL]; (4) We removed all markup and any other non-text content such as “*, ∼, [,]”; (5) We removed any threads whose title or lead comment scored less than 0; (6) We removed any posts which contain quarantine, picture, or video, etc. After then, the dataset has 11, 200, 981 instances.
In this section, we present TANET, a thread-aware pretrained model which incorporates the inherent dependencies between utterances to enable improved conversation's representations for summary generation. Below, we first introduce the model
architecture, thread-aware attention, and then introduce our pretraining objectives. Finally, we move on to the application of downstream tasks.
Encoder. We employ hierarchical encoders, a token encoder and an utterance encoder, to represent the input conversation. This design mainly comes from two considerations: (1) The conversations in actual applications are lengthy (e.g., The Reddit post in Figure 1 has more than  $210\mathrm{k}$  words, and a meeting transcript usually consists of thousands of tokens.), thus it may not be feasible to simply apply the canonical transformer structure. (2) Hierarchical architecture is more suitable for the conversational tasks to carry out modeling of utterances and interactive structure of the conversation.
Let  $C = (u_0,\dots ,u_{|C|})$  denote an conversation instance in the pretraining corpus  $\mathcal{D}_p$ .  $u_{i} = (\langle \mathsf{bos}\rangle ,w_{i,1},\dots ,w_{i,|u_i|})$  is the token sequence of  $i$ -th utterance after tokenization, where  $\langle \mathsf{bos}\rangle$  is a special token in vocabulary  $\nu$  to represent the beginning of a turn. The token encoder takes each sequence  $u_{i}$  as the input and first converts it into input vectors  $\mathbf{H}_i^{\mathcal{T},0}\in \mathbb{R}^{|\boldsymbol {u}_i|\times d_h}$ . For each token, its input vector is constructed by summing up the corresponding token embedding and the sine-cosine positional embedding (Vaswani et al., 2017). Then,  $N$  identical layers are nested over  $\mathbf{H}_i^{\mathcal{T},0}$  to produce the contextual representations by:
$$
\mathbf {H} _ {i} ^ {\mathcal {T}, N} = \operatorname {T r a n s f o r m e r} ^ {\mathcal {T}} \left(\mathbf {H} _ {i} ^ {\mathcal {T}, 0}\right) \tag {1}
$$
Each layer consists of two sub-layers, a self-attention sub-layer followed by a position-wise feed-forward sub-layer and uses residual connections around each of them. We adopt the prelayer normalization following several recent works (Baevski and Auli, 2019; Child et al., 2019; Wang et al., 2019; Xiong et al., 2020), which place the layer normalization inside the residual connection. That is, given input  $x$ , the output of each sub-layer is  $x + \mathrm{Sublayer}(\mathrm{LayerNorm}(x))$ . The utterance encoder also has  $N$  identical transformer layers in structure, which processes the information at turn level. All utterances are arranged in the order of their timestamps, and we employ the sine-cosine positional embedding to model the chronological order. Let  $\mathbf{H}^{\mathcal{U},0} = (h_{u_0},\dots ,h_{u_{|C|}})$  denotes the sequence of representations of utterances. For the  $i$ -th turn  $u_i$ , the embedding of  $\langle \mathsf{bos} \rangle$  is chosen as its representation, i.e.,  $h_{u_i} = \mathbf{H}_{i,0}^{T,N}$ . Different from
<https://www.reddit.com
2The score is the number of upvotes minus the number of downvotes.
the token encoder, we propose the Thread-Aware Attention sub-layer to replace the self-attention sublayer to encode the tree-structure information into our model.
Thread-Aware Attention. Each sub-layer consists of  $h$  attention heads, and the results from each head are concatenated together and projected to form the output of the sub-layer. Formally, given the input  $\mathbf{H}^{\mathcal{U},0}$ , the  $k$ -th head computes a new sequence  $z_{k} = (z_{k,0},\dots ,z_{k,|C|})$  by:
$$
z _ {k, i} = \sum_ {j = 1} ^ {| x |} \alpha_ {i j} \left(h _ {u _ {j}} W _ {k} ^ {V}\right), \tag {2}
$$
$$
\alpha_ {i j} = \frac {\exp e _ {i j}}{\sum_ {t = 1} ^ {| x |} \exp e _ {i t}}
$$
where  $z_{k,i}\in \mathbb{R}^{d_z},d_z = d_h / h.e_{ij}$  is the attention weight from  $h_{u_j}$  to  $h_{u_i}$ . Inspired by the relative position encoding (RPE) works (Shaw et al., 2018; Huang et al., 2020), we consider the interactions of queries, keys, and relative positions simultaneously to fully utilize the structural information of a conversation:
$$
e _ {i j} = \frac {\left(h _ {u _ {i}} W _ {k} ^ {Q} + r _ {i j}\right) \left(h _ {u _ {j}} W _ {k} ^ {K} + r _ {i j}\right) ^ {\top} - r _ {i j} r _ {i j} ^ {\top}}{\sqrt {d _ {z}}} \tag {3}
$$
where  $W_{k}^{Q}, W_{k}^{K}, W_{k}^{V} \in \mathbb{R}^{d_{h} \times d_{z}}$  are parameter matrices.  $\sqrt{d_{z}}$  is a scaling factor for stable training. The key to this mechanism is that  $r_{i,j} \in \mathbb{R}^{d_{z}}$  encodes the relation from utterance  $u_{j}$  to  $u_{i}$ , which is defined as:
$$
r _ {i, j} = \left\{ \begin{array}{l l} w _ {\text {c l i p} (\text {d e p t h} (u _ {i}) - \text {d e p t h} (u _ {j}), k)}, & 1) \\ w _ {*}, & 2) \end{array} \right. \tag {4}
$$
As illustrated in Figure 2, the relation between two utterances has two situations: 1) one is a parent or child utterance of the other, that is, they belong to the same path, e.g.  $u_{1}$  and  $u_{4}$ ; 2) otherwise, e.g.  $u_{2}$  and  $u_{3}$ . We totally define  $2k + 2$  learnable thread-aware position embeddings  $\{w_{*}, w_{-k}, \dots, w_{k}\}$ . where  $\mathrm{clip}(x, k) = \max(-k, \min(k, x))$ , and the function  $\mathrm{depth}(u_{i})$  returns the distance between utterance  $u_{i}$  and the first utterance  $u_{0}$  in the thread, e.g.  $\mathrm{depth}(u_{4}) = 2$ . The output of the utterance encoder is  $\mathbf{H}^{\mathcal{U}, N} \in \mathbb{R}^{|C| \times d_{h}}$ .
Decoder. The decoder is a  $N$ -layer transformer to generate the summary  $S$ . At the training stage, the decoder takes the right-shifted token sequence
Figure 2: Illustration of Thread-Aware attention. The left is the tree structure of a conversation thread, and  $u_{i}$  represents the  $i$ -th utterance. The thread-aware attention weights across the utterances are on the right.
5	w3	w2	w*	w*	w1	w0
4	w2	w1	w*	w*	w0	w-1
3	w2	w1	w*	w0	w*	w*
2	w1	w*	w0	w*	w*	w*
1	w1	w0	w*	w-1	w-1	w-2
0	w0	w-1	w-1	w-2	w-2	w-3
0	1	2	3	4	5
of  $S$  as input. In each layer, the self-attention sublayer leverages a lower triangular mask to prevent positions from attending to their future positions. Then, the cross-attention sub-layers attend with the outputs from the hierarchical encoder. In particular, we make an encoder-wise residual connection around the utterance encoder to propagate the token-level information directly to the decoder. We found that this can improve the model's capability to reproduce the words involved in the conversation. Denote the output of the decoder as  $\mathbf{H}^{\mathcal{D},N}\in \mathbb{R}^{|S|\times d_h}$ . When predicting the  $i$ -th token  $s_i$ , we reuse the embedding matrix of the vocabulary  $\mathcal{E}_{\mathcal{V}}\in \mathbb{R}^{|\mathcal{V}|\times d_h}$  to project  $\mathbf{H}_{i - 1}^{\mathcal{D},N}$  into a probability distribution:
$$
P (S _ {i} | S _ {<   i}, C) = \operatorname {S o f t m a x} (\mathbf {H} _ {i - 1} ^ {\mathcal {D}, N} \mathcal {E} _ {\mathcal {V}} ^ {\top}) \quad (5)
$$
4.2 Pretraining Objectives
In this section, we describe the pretraining objectives used for pretraining TANET. In addition to the causal language modeling, we newly introduce another thread-aware pretraining task to predict the contextual relation between utterances.
Causal Language Modeling. Following many previous works (Lewis et al., 2020; Zhang et al., 2020), we apply the causal language modeling objective, which seeks to minimize the cross-entropy loss:
$$
\mathcal {L} _ {C L M} (\theta) = - \frac {1}{| S |} \sum_ {i = 1} ^ {| S |} \log P \left(S _ {i} \mid S _ {<   i}, C\right) \tag {6}
$$
Thread Prediction. To enhance the representation of the thread structure in a conversation, we introduce a new pretraining task of thread prediction. The motivation is to encourage the model
to learn thread-aware representations that encode the information of which comments this one was written based on, that is, its historical comments. Specifically, we randomly sample  $20\%$  utterances  $C_s$  from  $C$  and then let the model predict their historical comments. Formally, the pretraining objective is calculated as:
$$
\begin{array}{l} \mathcal {L} _ {\text {T h r e a d P r e d}} = - \sum_ {a _ {i j} \in \mathcal {A}} \left(\delta \left(a _ {i j}\right) \log p _ {a _ {i j}} \right. \tag {7} \\ \left. + \left(1 - \delta \left(a _ {i j}\right)\right) \log \left(1 - p _ {a _ {i j}}\right)\right) \\ \end{array}
$$
where  $\mathcal{A} = C_s\times C\cup C\times C_s$  is the set of comment pair candidates for prediction.  $\delta (a_{ij})$  returns 1 if  $u_{j}$  and  $u_{i}$  belong to one thread and  $u_{j}$  is history of  $u_{i}$ , otherwise 0.  $p_{ij}$  is the probability of  $u_{j}$  being the historical comment of  $u_{i}$  and is computed by:
$$
p _ {a _ {i j}} = \operatorname {S i g m o i d} \left(\left(\mathcal {H} _ {i, 0} ^ {\mathcal {T}, N} W _ {a}\right) \left(\mathcal {H} _ {j, 0} ^ {\mathcal {T}, N} W _ {b}\right) ^ {\top}\right) \tag {8}
$$
$W_{a}, W_{b} \in \mathbb{R}^{d_{h} \times d_{h}}$  are two parameter matrices.
4.3 Application on Downstream Tasks
After pretraining on RCSUM, we finetune our model on the downstream tasks. Different tasks will have some differences in data annotation that requires us to adapt it flexibly. For example, the interdependencies among the utterances in a meeting are not labeled in AMI and ICSI, so we treat them as a sequence arranged by time. Besides, some additional information is essential for the generation of summary, which can be prompted to the model by modifying the input utterances. For example, the name and role of each participant are useful for meeting summarization in AMI and ICSI. Without changing the model structure, we inform TANET of the information by replacing the original utterance with template “{participant} of role {role} said: {utterance}”.
We evaluate TANET and all baseline models on four benchmark datasets of long and real-life conversations, covering domains of meeting transcripts, customer-service records, and threads in web forum. Table 1 summarizes the statistics of the four datasets.
AMI (Carletta et al., 2005) is a multi-modal dataset consisting of 100 hours of meeting recordings with rich annotations. Following Shang et al. (2018);
Dataset	AMI	ICSI	MultiWOZ	FORUM
Domain	Meeting	Meeting	Customer Service	Forum Thread
# Speakers	4	6.2	2	6.8
# Conversations	137	59	10,438	689
# Conv. words	4,757	10,189	180.7	825.0
# Summ. words	322	534	91.9	190.6
# Turns	289	464	13.7	10.5
Table 1: Statistics of the conversational summarization datasets. The number of conversation words, summary words, turns and speakers are all averaged across all conversations in the dataset.
Zhu et al. (2020), we select 137 meetings of scenario where the participants play different roles in a design team. Each meeting is labeled with transcripts produced by automatic speech recognition (ASR) and an abstractive summary written by a human annotator. Furthermore, each dialogue is also associated with additional information, including its speaker id with role, dialogue act. We use the same data split of  $100 / 17 / 20$  as training/validation/test sets.
ICSI (Janin et al., 2003) is another widely-used meeting corpus consisting of about 70 hours of meeting audio recordings with orthographic transcription and other manual annotations. We follow the pre-processing pipeline from Zhu et al. (2020) and split the training/validation/test sets of size  $43/10/6$ , respectively. Each meeting also contains a manually labeled abstractive summary and the associated role information for each participant.
MultiWOZ (Yuan and Yu, 2019) is an abstractive dialog summarization dataset based on the MultiWOZ corpus (Budzianowski et al., 2018; Ramadan et al., 2018; Eric et al., 2019; Zang et al., 2020), which is a fully-labeled collection of human-human written conversations spanning over multiple domains and topics. The dataset is built on various customer-service records in the corpus, such as booking restaurants, hotels, taxis. We use the summary annotation provided by Yuan and Yu (2019), and the same data split of 8438/1000/1000 as training/validation/test sets.
FORUM (Tarnpradab et al., 2017) contains 700 human-annotated forum threads. Each thread contains a human-annotated abstractive summary and multiple posts written by several different users. These threads are collected from tripadvisor.com and ubuntuforums.org. Bhatia et al. (2014) annotated 100 threads from TripAdvisor with human-written summaries, and Tarnpradab et al. (2017) further extend the summary annotation with 600 more threads. In our experiments, we divide
the dataset into  $500 / 100 / 89$  examples for training/validation/test sets.