SlowGuess/NAACL_2022_OCR huggingface dataset:
ROUGE (Lin, 2004) is a standard metric for summarization task. Following Zhu et al. (2020), we use ROUGE-1, ROUGE-2, and ROUGE-SU4 to evaluate all meeting summarization models. The models on MultiWOZ and FORUM are evaluated by ROUGE-1, ROUGE-2 and ROUGE-L (Tarn-pradab et al., 2017; Yuan and Yu, 2019). We obtain the scores by the rouge-metric package<sup>3</sup>.
5.3 Evaluation Results and Discussions
5.3.1 Meeting Summarization
We compare TANET with a variety of models from previous literature: Random (Riedhammer et al., 2008), the template-based model Template (Oya et al., 2014), two ranking systems TextRank (Mihalcea and Tarau, 2004) and ClusterRank (Garg et al., 2009), the unsupervised method UNS (Shang et al., 2018), Extractive Oracle, which concatenates top sentences with the highest ROUGE-1 scores with the golden summary, the document summarization model PGNet (See et al., 2017), Copy from Train, which randomly copies a summary from the training set as the prediction, the multimodal model MM (Li et al., 2019), and the hierarchical Network HMNet (Zhu et al., 2020). Besides the baselines above, BART (Lewis et al., 2020) and PEGASUS (Zhang et al., 2020), two state-of-the-art pre-trained models on document summarization, and Longformer-Encoder-Decoder (LED) (an, 2020) are also included in comparison to have a thorough understanding towards our model. We concatenate all turns of a transcript into a sequence and then truncate it to meet the length constraints of the model input.  $\mathsf{LED}_{\text{large}}$  is initialized from  $\mathsf{BART}_{\text{large}}$  and able to process 16k tokens. Please refer to the Appendix for more implementation details.
Table 2 reports the automatic evaluation results on datasets AMI and ICSI. We can see that, except for ROUGE-1 on AMI, TANET outperforms all baseline models in all metrics. MM is a multimodal model which requires additional annotation of topic segmentation (TopicSeg) and multi-modal features derived from the visual focus of attention (VFOA) collected by cameras. In practice, the
Models	R-1	R-2	R-SU4
AMI
Random (Riedhammer et al., 2008)	35.13	6.26	13.17
Template (Oya et al., 2014)	31.50	6.80	11.40
TextRank (Mihalcea and Tarau, 2004)	35.25	6.90	13.62
ClusterRank (Garg et al., 2009)	35.14	6.46	13.35
UNS (Shang et al., 2018)	37.86	7.84	14.71
Extractive Oracle	39.49	9.65	13.20
PGNet (See et al., 2017)	40.77	14.87	18.68
Copy from Train	43.24	12.15	14.01
MM+TopicSeg (Li et al., 2019)†	51.53	12.23	-
MM+TopicSeg+VFOA (Li et al., 2019)†	53.29	13.51	-
HMNet (Zhu et al., 2020)	53.02	18.57	24.85
Our re-implementation
LEDlarge (an, 2020)	53.10	19.83	24.95
BARTbase (Lewis et al., 2020)	50.26	18.18	17.83
PEGASUSlarge (Zhang et al., 2020)	47.05	16.64	16.03
TANET (ours)	53.26	20.73*	25.98*
ICSI
Random (Riedhammer et al., 2008)	29.28	3.78	10.29
TextRank (Mihalcea and Tarau, 2004)	29.70	4.09	10.64
ClusterRank (Garg et al., 2009)	27.64	3.68	9.77
UNS (Shang et al., 2018)	31.60	4.83	11.35
Extractive Oracle	34.66	8.00	10.49
PGNet (See et al., 2017)	32.00	7.70	12.46
Copy from Train	34.65	5.55	10.65
HMNet (Zhu et al., 2020)	46.28	10.60	19.12
Our re-implementation
LEDlarge (an, 2020)	43.13	11.76	19.08
BARTbase (Lewis et al., 2020)	42.01	9.96	11.72
PEGASUSlarge (Zhang et al., 2020)	42.44	9.15	11.10
TANET (ours)	47.21*	12.35*	19.27
Table 2: Automatic evaluation results on datasets AMI and ICSI. Numbers in bold indicate the best performing models on the corresponding metrics. Numbers marked with “*” mean that the improvement over the best baseline is statistically significant (t-test with  $p$ -value  $< 0.05$ ). Models marked with “†” require additional human annotations of topic segmentation and visual signals from cameras.
visual information is rarely available, such as online chat, so the application scenarios of MM are very limited. In comparison, TANET is completely based on meeting transcripts from ASR systems, so it has better scalability. Comparable performance is achieved in ROUGE-1 on AMI, but it is significantly higher in ROUGE-2 by 7.2 points. In particular, TANET outperforms HMNet, indicating that pretraining on large-scale conversation data while incorporating the inherent structural information can lead to better performances on downstream tasks. Moreover, TANET significantly outperforms BART and PEGASUS on both AMI and ICSI. Although the two baselines own strong capabilities to summary a document, the tricky part is that a meeting transcript is very long and cannot be fully fed into the models. For example, the average number of words in ICSI is 10, 189, which far exceeds
3https://pypi.org/project/ rouge-metric/
Models	R-1	R-2	R-L
PGNet (See et al., 2017)	62.89	48.61	59.30
Transformer (Vaswani et al., 2017)	63.12	50.63	61.04
SPNet (Yuan and Yu, 2019)	90.97	84.14	85.00
Our re-implementation
LEDlarge (an, 2020)	91.41	79.93	83.63
HMNet (Zhu et al., 2020)	66.33	50.49	64.52
BARTbase (Lewis et al., 2020)	81.47	70.24	73.14
PEGASUSlarge (Zhang et al., 2020)	93.51	88.09	84.73
TANET (ours)	93.25	88.60*	85.67*
the maximum input length 512 tokens of BART and PEGASUS. As a result, most of the content in a meeting transcript are discarded, which will inevitably limit the performances of the two models. LED can input all sentences, but it is still difficult to fully understand the conversation, which further demonstrate the effectiveness of the pretraining on the corpus RCSUM.
5.3.2 Customer-service Records Summarization
To demonstrate the effectiveness of TANET on customer-service records summarization, following models are selected as baselines from previous literature: the pointer-generator network PGNet (See et al., 2017), Transformer (Vaswani et al., 2017) and SPNet which incorporates three types of semantic scaffolds - speaker role, semantic slot and dialogue domain for summarization (Yuan and Yu, 2019). Besides, we include HMNet (Zhu et al., 2020) as a baseline and implement it using the official code  $^{4}$ . We also apply Longformer-Encoder-Decoder (LED) (an, 2020), BART (Lewis et al., 2020) and PEGASUS (Zhang et al., 2020) in this task by concatenating all utterances in a conversation as a document.
Table 3 reports the automatic evaluation results on MultiWOZ. We can observe that TANET achieves new state-of-the-art performance on ROUGE-2 and ROUGE-L, which demonstrate the effectiveness of pretraining on large-scale conversation data. Different from the results of the meeting summarization given in Table 2, PEGASUSlarge achieves close performance to TANET and even the best in ROUGE-1, showing its great general
Table 3: Automatic evaluation results on MultiWOZ. Numbers in bold indicate the best performing models on the corresponding metrics. Numbers marked with “*” mean that the improvement over the best baseline is statistically significant (t-test with  $p$ -value  $< 0.05$ ).
Models	R-1	R-2	R-L
ILP (Berg-Kirkpatrick et al., 2011)	29.3	9.9	-
Sum-Basic (Vanderwende et al., 2007)	33.1	10.4	-
KL-Sum	35.5	12.3	-
Lex-Rank (Erkan and Radev, 2011)	38.7	14.2	-
MEAD (Radev et al., 2004)	38.5	15.4	-
SVM (Chang and Lin, 2011)	24.7	10.0	-
LogReg (Fan et al., 2008)	29.4	7.8	-
HAN (Tarnpradab et al., 2017)	37.8	14.7	-
Our re-implementation
LEDlarge (an, 2020)	42.39	22.78	30.48
HMNet (Zhu et al., 2020)	41.30	17.12	31.76
BARTbase (Lewis et al., 2020)	42.91	22.32	30.35
PEGASUSlarge (Zhang et al., 2020)	42.92	20.50	29.16
TANET (ours)	45.20*	25.61*	33.59*
Table 4: Automatic evaluation results on FORUM. Numbers in bold indicate the best performing models on the corresponding metrics. Numbers marked with \*\* mean that the improvement over the best baseline is statistically significant (t-test with  $p$  -value  $<  0.05$
ization ability in this task. This is because: (1) the "documents" can be fully fed into the model without content loss. The average length of the dialogue in MultiWOZ is 180.7 words, which do not exceed the model's maximum input length 512. (2) each conversation takes place between two speakers (i.e. a customer and a staff), so the structure of a dialogue can be viewed as a sequence, which is similar to the sentences in a document. Compared with LED, HMNet and BART, the un-pretrained model SPNet obtains surprisingly better scores. This motivates us to combine richer conversation-related information, such as speaker role, dialogue act, semantic slot, and dialogue domain, to further improve model's summarization capabilities in the future.
5.3.3 Forum Threads Summarization
In this task, TANET is compared against a range of baselines, including following unsupervised methods: (1) ILP (Berg-Kirkpatrick et al., 2011), a baseline integer linear programming framework; (2) Sum-Basic (Vanderwende et al., 2007), a model that assumes words occurring frequently in a document cluster have a higher chance of being included in the summary; (3) KL-Sum, an approach that selects the sentences decreasing the KL divergence as the summary; (4) Lex-Rank (Erkan and Radev, 2011), a graph-based model based on eigenvector centrality; (5) MEAD (Radev et al., 2004), a centroid-based approach which scores sentences based on length, centroid, and position; and supervised extractive systems, including (1) SVM
<sup>4</sup>https://github.com/microsoft/HMNet
Models	AMI	FORUM
R-1	R-2	R-SU4	R-1	R-2	R-L
TANET	53.26	20.73	25.98	45.20	25.61	33.59
- Pretraining	46.43	16.85	18.42	39.67	15.37	26.45
- Thread-Aware Attention	51.33	18.90	23.71	41.85	22.41	30.79
- Thread Prediction	51.94	19.30	24.75	41.70	21.33	31.80
- Encoder-wise Residual	51.86	20.02	24.59	44.93	24.78	33.30
(Chang and Lin, 2011), the support vector machine; (2) LogReg (Fan et al., 2008), the logistic regression; (3) HAN (Tarnpradab et al., 2017), a hierarchical attention network with redundancy removal process; Besides, we also apply the cross-domain pre-trained model HMNet (Zhu et al., 2020) in this task and implement Longformer-Encoder-Decoder (LED) (an, 2020), BART (Lewis et al., 2020) and PEGASUS (Zhang et al., 2020) in a similar way to the adaptation in the above two tasks.
Table 4 reports the automatic evaluation results on the dataset FORUM. TANET outperforms all baseline models in terms of all metrics, and the improvements are statistically significant (t-test with  $p$ -value  $< 0.05$ ), which further demonstrate the effectiveness of our method. In this task, the gains over the pre-trained baselines are relatively high, due to (1) the consistency of conversation domain between the pretraining stage and downstream finetuning. The conversation in FORUM and our pretraining corpus RCSUM are both forum threads. Note that, although the data in FORUM is collected from TripAdvisor (tripadvisor.com) and UbuntuForums (ubuntuforums.org.), the subjects are also included in some specific sub-reddits on the Reddit website. In contrast, LED, HMNet, BART and PEGASUS are all pre-trained with document-like text, so there will be a domain gap in thread understanding; (2) structure modeling. The tree-like reply relationship in a thread plays a vital role in understanding the entire thread, but the baselines can only process it linearly, which poses challenges for the model to fully understand the context and generate accurate summaries.
To understand the impact of our pretraining strategies on model performance, we compare the full TANET with the following variants: (1) - Pretraining: the pretraining stage is removed; (2) - Thread-Aware Attention: the Thread-Aware Attention sublayers in the utterance encoder degenerate into standard self-attention sublayers; (3) - Thread Prediction:  $\mathcal{L}_{\text{ThreadPred}}$  is removed; and (4) -
Table 5: Ablation study on AMI and FORUM.
Models	AMI	FORUM
Read.	Conc.	Kappa	Read.	Conc.	Kappa
HMNet	1.67	1.40	0.60	1.76	1.57	0.61
Bartbase	1.58	1.02	0.72	1.82	1.63	0.69
TANet	1.70	1.53	0.63	1.84	1.70	0.60
Table 6: Human evaluation results on AMI and FO-RUM. "Read," "Conc." are abbreviations for readability and conciseness, respectively.
Encoder-wise Residual: the encoder-wise residual around the utterance encoder is removed. Table 5 reports the evaluation results on AMI and FORUM. We can conclude that (1) the pretraining on RCSUM helps to significantly improve the performance, as removing it results in dramatic performance drop on both AMI and FORUM; (2) both Thread-Aware attention and Thread Prediction objective are useful, indicating that the structure of thread is essential to facilitate the understanding of conversation, especially for the threads with tree structure; (3) the encoder-wise residual is meaningful, as removing it causes performance drop.
5.3.5 Human Evaluation
As the human annotation for this task is very time-consuming and labor-intensive, we also conduct human evaluation on the test sets of AMI and FORUM to verify whether the improvements on automatic evaluation is in line with the human perceived quality. We recruit 3 well-educated native speakers as annotators and compare TANET with BART<sub>base</sub> and HMNet on 2 aspects - readability and conciseness. The former measures how fluent a generated summary is, while the later measures how well the summary sums up the main ideas of a conversation. For each sample, we show its conversation, reference summary, as well as summaries generated by models (the order is shuffled to hide their sources) to the annotators and ask them to judge the quality and assign a score in {0,1,2} (indicating "bad", "fair", and "good") to each summary for each aspect. Table 6 reports the evaluation results. We can observe (1) the three models are comparable on readability on both datasets; (2) TANET outperforms the others on conciseness, which is consistent with the automatic evaluation results in Table 2 and Table 4; (3) BART<sub>base</sub> does not perform well on conciseness on AMI, as most utterances of a conversation are discarded due to the input length
5 Ablation results on AMI and FORUM could provide more insights, whereas ICSI is similar to AMI, and MultiWOZ is less challenging.
constraint 512. All kappa values are no less than 0.6, indicating substantial agreement among the annotators. For reference, we present case study in the Appendix.
With the recent success of seq2seq models, the research focus of conversational summarization has been transferred from the extractive methods to abstractive models. Various semantic patterns have been applied to these abstractive approaches, such as dialogue acts (Goo and Chen, 2018), auxiliary key point sequences (Liu et al., 2019a), topic segments (Li et al., 2019; Liu et al., 2019b), conversational stages and dialogue overview (Chen and Yang, 2020), discourse relations (Murray et al., 2006; Bui et al., 2009; Qin et al., 2017). At the same time, some work is devoted to providing high-quality datasets to promote the development of this research direction (Carletta et al., 2005; Janin et al., 2003; Tarnpradab et al., 2017; Yuan and Yu, 2019; Gliwa et al., 2019). However, these corpora have a low number of conversations, which hinders the progress of abstractive summarization (an, 2021). Recently, large neural models pre-trained on huge corpora have led to strong improvements on numerous natural language understanding and generation tasks (Devlin et al., 2019; Radford et al., 2019; Lewis et al., 2020; Zhang et al., 2020). Encouraging by the promising progress of pretraining, Zhu et al. (2020) first introduce a hierarchical structure and propose pretraining on cross-domain data for meeting summarization. The pretraining data is collected from the news domain. Regarding one document as the utterances from one participant, multiple documents are combined and reshuffled to simulate a multi-person meeting. However, there are two disadvantages - the first is the style inconsistency between conversation and news, and the second is that there is no contextual relationship between the two documents, so the participants have no communication actually. In this work, we build a large-scale corpus based on real conversations. Besides, we further incorporate the structure information of thread in the model.
In this work, we introduce TANET, a thread-aware pre-trained model for abstractive conversational summarization. TANET employs the thread-aware attention and a new pretraining objective to fully
leverage the structure information of conversation. Furthermore, we build a large-scale pretraining corpus based on the discussions on Reddit. Experiments on four downstream tasks demonstrate the effectiveness of TANET.
Iz Beltagy an. 2020. Longformer: The long-document transformer. ArXiv preprint, abs/2004.05150.
Xiachong Feng an. 2021. A survey on dialogue summarization: Recent advances and new frontiers. ArXiv preprint, abs/2107.03175.
Alexei Baevski and Michael Auli. 2019. Adaptive input representations for neural language modeling. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Taylor Berg-Kirkpatrick, Dan Gillick, and Dan Klein. 2011. Jointly learning to extract and compress. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 481-490, Portland, Oregon, USA. Association for Computational Linguistics.
Sumit Bhatia, Prakhar Biyani, and Prasenjit Mitra. 2014. Summarizing online forum discussions - can dialog acts of individual messages help? In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2127-2131, Doha, Qatar. Association for Computational Linguistics.
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
Trung Bui, Matthew Frampton, John Dowding, and Stanley Peters. 2009. Extracting decisions from multi-party dialogue using directed graphical models and semantic similarity. In Proceedings of the
SIGDIAL 2009 Conference, pages 235-243, London, UK. Association for Computational Linguistics.
Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, Guillaume Lathoud, Mike Lincoln, Agnes Lisowska, Iain McCowan, Wilfried Post, Dennis Reidsma, and Pierre Wellner. 2005. The ami meeting corpus: A pre-announcement. In Proceedings of the Second International Conference on Machine Learning for Multimodal Interaction, MLMI'05, Berlin, Heidelberg. Springer-Verlag.
Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A library for support vector machines. ACM Trans. Intell. Syst. Technol., 2(3):27:1-27:27.
Jiaao Chen and Diyi Yang. 2020. Multi-view sequence-to-sequence models with conversational structure for abstractive dialogue summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4106-4118, Online. Association for Computational Linguistics.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675-686, Melbourne, Australia. Association for Computational Linguistics.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. ArXiv preprint, abs/1904.10509.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyag Gao, and Dilek Hakkani-Tur. 2019. Multiwoz 2.1: Multi-domain dialogue state corrections and state tracking baselines. ArXiv preprint, abs/1907.01669.
Günes Erkan and Dragomir R. Radev. 2011. Lexrank: Graph-based lexical centrality as salience in text summarization. CoRR, abs/1109.2128.
Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008. LIBLINEAR: A library for large linear classification. J. Mach. Learn. Res., 9:1871-1874.
Nikhil Garg, Benoit Favre, Korbinian Riedhammer, and Dilek Hakkani-Tür. 2009. Clusterrank: a graph based method for meeting summarization. In IN-TERSPEECH 2009, 10th Annual Conference of the International Speech Communication Association, Brighton, United Kingdom, September 6-10, 2009, pages 1499-1502. ISCA.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70-79, Hong Kong, China. Association for Computational Linguistics.
Chih-Wen Goo and Yun-Nung Chen. 2018. Abstractive dialogue summarization with sentence-gated modeling optimized by dialogue acts. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 735-742.
Karl Moritz Hermann, Tomás Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 1693-1701.
Zhiheng Huang, Davis Liang, Peng Xu, and Bing Xiang. 2020. Improve transformer models with better relative position embeddings. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 3327-3335, Online. Association for Computational Linguistics.
A. Janin, D. Baron, J. Edwards, D. Ellis, D. Gelbart, N. Morgan, B. Peskin, T. Pfau, E. Shriberg, A. Stolcke, and C. Wooters. 2003. The icsi meeting corpus. In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings (ICASSP '03), volume 1, pages I-I.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Manling Li, Lingyu Zhang, Heng Ji, and Richard J. Radke. 2019. Keep meeting summaries on topic: Abstractive multi-modal meeting summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2190-2196, Florence, Italy. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summariza