SlowGuess/NAACL_2022_OCR huggingface dataset:
Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6866-6880, Online. Association for Computational Linguistics.
Xiachong Feng, Xiaocheng Feng, and Bing Qin. 2021a. A survey on dialogue summarization: Recent advances and new frontiers. arXiv preprint arXiv:2107.03175.
Xiachong Feng, Xiaocheng Feng, Libo Qin, Bing Qin, and Ting Liu. 2021b. Language model as an annotator: Exploring DialogGPT for dialogue summarization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1479-1491, Online. Association for Computational Linguistics.
Prakhar Ganesh and Saket Dingliwal. 2019. Restructuring conversations using discourse relations for zero-shot abstractive dialogue summarization. arXiv preprint arXiv:1902.01615.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. Samsum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70-79.
Chih-Wen Goo and Yun-Nung Chen. 2018. Abstractive dialogue summarization with sentence-gated modeling optimized by dialogue acts. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 735-742. IEEE.
Matthew Henderson, Paweł Budzianowski, Iñigo Casanueva, Sam Coope, Daniela Gerz, Girish Kumar, Nikola Mrkšić, Georgios Spithourakis, Pei-Hao Su, Ivan Vulić, and Tsung-Hsien Wen. 2019. A repository of conversational datasets. In Proceedings of the First Workshop on NLP for Conversational AI, pages 1-10.
Yuejie Lei, Fujia Zheng, Yuanmeng Yan, Keqing He, and Weiran Xu. 2021. A finer-grain universal dialogue semantic structures based model for abstractive dialogue summarization. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 1354–1364, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Junpeng Liu, Yanyan Zou, Hainan Zhang, Hongshen Chen, Zhuoye Ding, Caixia Yuan, and Xiaojie Wang. 2021a. Topic-aware contrastive learning for abstractive dialogue summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1229-1243, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Zhengyuan Liu, Ke Shi, and Nancy Chen. 2021b. Coreference-aware dialogue summarization. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 509-519, Singapore and Online. Association for Computational Linguistics.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807.
Shashi Narayan, Yao Zhao, Joshua Maynez, Goncalo Simoes, and Ryan McDonald. 2021. Planning with entity chains for abstractive summarization. arXiv preprint arXiv:2104.07606.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083.
Chien-Sheng Wu, Linqing Liu, Wenhao Liu, Pontus Stenetorp, and Caiming Xiong. 2021. Controllable abstractive dialogue summarization with sketch supervision. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5108-5122, Online. Association for Computational Linguistics.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020a. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 11328-11339. PMLR.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020b. Dialogpt: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270-278.
Lulu Zhao, Weihao Zeng, Weiran Xu, and Jun Guo. 2021. Give the truth: Incorporate semantic slot into
abstractive dialogue summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2435-2446, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. 2018. Neural document summarization by jointly learning to score and select sentences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 654-663, Melbourne, Australia. Association for Computational Linguistics.
Chenguang Zhu, Ruochen Xu, Michael Zeng, and Xuedong Huang. 2020. A hierarchical network for abstractive meeting summarization with cross-domain pretraining. arXiv preprint arXiv:2004.02016.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19-27.
Dialogue summarization and pretrained language models are discussed as follows.
Dialogue Summarization: A growing number of works have been proposed for dialogue summarization in recent years. In this work, we mainly refer to the chat summarization defined in (Feng et al., 2021a). Previous works widely explore dialogue features explicitly and input them as known labels to enhance the dialogue understanding ability of summarization models. Features, including dialogue acts (Goo and Chen, 2018), topic transitions (Chen and Yang, 2020), discourse dependencies (Chen and Yang, 2021), coreference relations (Liu et al., 2021b), argument graphs (Fabbri et al., 2021), semantic structures or slots (Lei et al., 2021; Zhao et al., 2021), etc. are carefully designed and collected by transferring tools pre-trained on other corpus or unsupervised methods with multiple hyper-parameters. These work also modify the basic transformer-based models with additional encoders (Chen and Yang, 2020) or attention layers (Chen and Yang, 2021; Liu et al., 2021b; Lei et al., 2021; Zhao et al., 2021) to utilize the injected features. Liu et al. (2021a) propose a contrastive learning approach for dialogue summarization with multiple training objectives. They also introduce a number of hyper-parameters for contrastive dataset construction and balancing among those objectives.
Pretrained Language Models: Previous pretrained seq-to-seq models can be divided into two categories by training data formats. One is models pretrained on narrative text, such as BART (Lewis et al., 2020), PEGASUS (Zhang et al., 2020a), and T5 (Raffel et al., 2020). They use training data from Wikipedia, BookCorpus (Zhu et al., 2015) and C4 (Raffel et al., 2020). These models show great potentials for tasks such as translation and story ending generation. The other is models pretrained on dialogue, such as DialogoGPT (Zhang et al., 2020b) and PLATO (Bao et al., 2020). Their training data are general-domain dialogues, such as Reddit (Henderson et al., 2019) and Twitter (Cho et al., 2014). These models work for dialogue response selection and generation tasks. All of the above models are trained to exploit language features within the same data format, with pre-training tasks such as masked token/sentence prediction and utterance permutation. Pretraining with cross-format data hasn't been researched so far. As a first step, we focus on narrowing the gap by learn
ing to rephrase unidirectionally from dialogue to narratives.
We use BART $^7$  as our basic language model. For both post-training and fine-tuning, the speakers and utterances of each dialogue are concatenated into a single sequence and truncated to the first 1024 tokens. The learning rate is set to  $3e-5$  with weight decay equaling 0.01. The number of warmup steps is 500 and dropout is 0.1. The model is tested on the corresponding validation set after each training epoch and the early-stop is activated if there is no improvement in the Rouge-2 F1 score. The early-stop and maximum training epochs are set to 3 and 10. During inference, i.e., validation and testing, the beam size is set to 4 with length penalty equaling 1.0 and no-repeat-n-gram size equaling 3. The minimum and maximum lengths are set to the corresponding lengths of the reference summaries based on statistics of each dataset, allowing for freelength text generation. Besides, for the inference on the validation set during the post-training stage, we also set the first 3 tokens as the known prefix. This constant number enables a fair comparison of performances on validation sets under different experimental settings. All of our experiments are done on an RTX 2080Ti with 11G GPU memory. We run experiments three times and show the best results following (Feng et al., 2021b).
C Other Types of Paraphrase Datasets
To make the input and output carry the same amount of information, one way is to fix  $D$  as input and convert utterances into indirect speech as the output. Ganesh and Dingliwal (2019) restructured dialogue into text with complicated rules which are not released and difficult to transfer among datasets under different scenarios. Thus, we only use simple rules to convert all of the utterances into  $[r_t$  says, " $u_t$ "] and concatenated as the output. We call this dataset as DialIndirect.
Another way is fixing  $S$  as output and removing the redundant utterances in  $D$  to get the rephrasing input. We take advantage of the idea of oracle extraction for news summarization (Zhou et al., 2018) and regard the combination of dialogue utterances with the highest Rouge scores computed with  $S$  as the input. Considering that utterances are
$^{7}$ https://huggingface.co/facebook/bart-large
Datasets	Input	Output
DialIndirect	U1~8	Katarina says,“Hello, I got ... we work together” Jill says, “Hi :) ...... nice and sunny”
ExtSum	U3, U6	Katarina ...... a flat from Liz. She will ...... after 6 pm.
ExtSumM	U3~6	Katarina ...... a flat from Liz. She will ...... after 6 pm.
ExtSent/ExtSentM	U3	Katarina ...... a flat from Liz.
U6	Katarina will ...... after 6 pm.
DSum	U1~8	Katarina ...... a flat from Liz. She will ...... after 6 pm.
DialSent	U1~8	Katarina ...... a flat from Liz.
U1~8	Katarina will ...... after 6 pm.
Table 8: An illustration of post-training pairs generated from the example in Figure 1. ExtSent and ExtSentM get the same training pairs in this case.
Datasets	Train/Val	IW	OW	CR
SAMSum
DialIndirect	14,731/818	124.10	157.41	1.31
ExtSum	14,731/818	31.23	23.44	0.94
ExtSumM	14,731/818	66.09	23.44	0.69
EntSent	29,757/1,654	31.05	11.93	0.68
ExtSentM	29,757/1,654	46.45	11.93	0.60
DSum	14,731/818	124.10	23.44	0.25
DialSent	29,757/1,654	149.93	11.93	0.13
DialSumm
DialIndirect	12,460/500	187.52	215.30	1.16
ExtSum	12,460/500	44.43	30.02	0.84
ExtSumM	12,460/500	94.32	31.02	0.61
EntSent	22,407/840	39.27	17.78	0.65
ExtSentM	22,407/840	61.17	17.78	0.56
DSum	12,460/500	187.52	31.02	0.18
DialSent	22,407/840	214.00	17.78	0.10
highly dependent, we modify the original extraction algorithm by extracting all of the utterances lying between the extracted ones, different from the window-sized snippet selection in (Liu et al., 2021a). Datasets with or without this modification are called ExtSum and ExtSumM respectively.
A summary  $S$  is divided into sentences to construct more rephrase pairs. Similar extraction operations can be done between  $D$  and  $p$ , and we get ExtSent and ExtSentM datasets.
An example of the paraphrase pair generated from the dialogue-summary pair in Figure 1 is shown in Table 8. The statistics of post-training datasets derived from SAMSum and DialSumm are shown in Table 9. We compare the performances between different rephrasing approaches with these datasets of our two-stage approach with the finetuning-only BART. The results are in Table 10.
Table 9: Statistics of constructed datasets. IW and OW refer to the number of words in the input and output of corresponding dataset. DSum and DialSent are in-list for easier comparison.
Models	Rouge-1	Rouge-2	Rouge-L
SAMSum
BART	52.06	27.45	48.89
DialIndirect	53.08	28.51	50.25
ExtSum	53.20	28.26	49.80
ExtSumM	52.20	27.91	49.74
EntSent	51.82	27.43	49.19
ExtSentM	51.66	27.27	48.96
DSum	52.52	27.51	49.03
DialSent	53.54	28.91	50.21
DialSumm
BART	53.01	29.18	51.34
DialIndirect	52.54	29.13	51.68
ExtSum	51.83	27.92	50.33
ExtSumM	52.29	27.72	50.09
EntSent	51.41	27.81	49.65
ExtSentM	52.46	28.86	51.36
DSum	53.27	28.64	51.69
DialSent	54.73	30.47	53.46
Table 10: Comparisons among different post-training approaches and fine-tuning-only BART baseline on dialogue summarization.
DialIndirect performs incredibly well on SAMSum. However, if we use the converted dialogue as input and directly fine-tune the original BART, the results are only 50.91/28.51/50.25 for Rouge-1/2/L. It shows that when accompanied with the posttraining stage, the model can learn relationships between speakers and utterances, and boundaries of utterances better than a direct transformation of dialogue inputs. This rule-based transformation falls on DialSumm compared with BART baseline. More complicated rules may lead to better results, but such labored work is not what we are after.
The extraction-based methods fall behind the others. The modification to the algorithm tends to bring more noises than useful information to the input as the results drop mostly. Besides, splitting the summary into sentences doesn't improve the results here. In a word, such hard extractions hurt the intricate discourse and coreference relations among utterances and are not suitable for cross-format data construction.
DialSent with PGG task outperforms other methods and BART consistently across datasets, while DSum with PGG performs almost the same as BART. If we use DialSent data to augment the original DSum during fine-tuning, the results on SAMSum are 44.61/22.81/44.15 for Rouge-1/2/L respectively showing that the data in both datasets is not compatible. Thus, our approach is different from data augmentation. Overall, post-training with cross-format rephrasing intuition does help with dialogue summarization,
We show more cases as follows.
Dialogue	Kate: Hey, do you know if our medical insurance covers hospital costs?
Greg: Hm, it depends
Mel: What happened dear?
Kate: I broke my arm and they're sending me to the hospital :
Greg: Call Linda or ask someone at the reception, they should be able to tell you what kind of package you have
Kate: thnx
Reference	Kate broke her arm and she's going to the hospital. She'd like to know whether her medical insurance covers hospital costs. Greg suggests her to call Linda or ask someone at the reception about it.
BART	Kate broke her arm and they're sending her to the hospital. Greg doesn't know if their medical insurance covers hospital costs. (53.33/37.93/53.19)
Multi-view	Kate broke her arm and they're sending her to the hospital. Greg will call Linda or ask someone at the reception to find out if their insurance covers hospital costs. (67.64/51.52/56.15)
DialoBART	Kate broke her arm and they're sending her to the hospital. Greg advises her to call Linda or ask someone at the reception. (65.57/50.85/67.62)
DialSent-PGG	Kate broke her arm and they're sending her to the hospital. Greg advises her to call Linda or ask someone at the reception if their insurance covers hospital costs. (71.64/55.38/62.39)
The case in Table 11 is a dialogue happened between three speakers from SAMSum. The labeled dialogues, which are directly extracted from Multi-view's and DialoBART's released datasets are shown in Table 12. "  $\mid$  " label for Multi-view refers to the topic transitions and stage transitions for the same dialogue respectively. We can see that topic segments by Multi-view BART are reasonable. However, such linear segmentation is not quite suitable for this dialogue since the first and third topics are the same. "  $\mid$  " in DialoBART just refers to the end of each utterance. DialoBART failed to label any topic transitions or redundant utterances.
Compared to the reference summary, the summary generated by BART lost the information about Greg's suggestion, and DialoBART lost the information about "medical insurance" even though it recognized "medical insurance" as a keyword. Multi-view did incorrect reasoning on who will call Linda. Our model generated a more condensed summary covering the same key points as the reference with the original dialogue as input.
Another case from DialSumm between two speakers is in Table 13. BART recognized "him" in the second utterance as "#Person1#" incorrectly. DialoBART regarded the man as "#Person1#"s
Table 11: A case from SAMSum. Names are in bold and unfaithful contents are in italic. Rouge-1/2/L scores(%) are in parentheses.
Multi-view Topic	Kate: Hey, do you know if our medical insurance covers hospital costs? Greg: Hm, it depends | Mel: What happened dear? Kate: I broke my arm and they're sending me to the hospital :/ | Greg: Call Linda or ask someone at the reception, they should be able to tell you what kind of package you have Kate: thnx |
Multi-view Stage	| Kate: Hey, do you know if our medical insurance covers hospital costs? Greg: Hm, it depends Mel: What happened dear? | Kate: I broke my arm and they're sending me to the hospital :/ | Greg: Call Linda or ask someone at the reception, they should be able to tell you what kind of package you have Kate: thnx
DialoBART	Kate : Hey , do you know if our medical insurance covers hospital costs ? | Greg : Hm , it depends | Mel : What happened dear ? | Kate : I broke my arm and they're sending me to the hospital | Greg : Call Linda or ask someone at the reception , they should be able to tell you what kind of package you have | Kate : thnx #KEY# Mel Kate Greg Hey do you know if our medical insurance covers hospital costs happened dear Linda reception package
friends” which isn’t mentioned in the original dialogue. Our model, DialSent-PGG generates a more accurate summary.
Table 12: Modified inputs by Multi-view and Dialog-BART.
Dialogue	#Person1#: Like a cat on hot bricks, as you might say. I don 't believe you are listening at all.
#Person2#: Sorry, I just worried about him. You know, he should be here an hour ago.
#Person1#: Don 't worry him, he has been grown up and I think he can take himself very well.
#Person2#: But he still does not come back.
#Person1#: Maybe he is on the way home now.
Reference-1	#Person2# is worried about one man, and #Per-son1# thinks that that man might be on the way home now.
Reference-2	#Person2# is worried about a man, but #Person1# thinks it would be fine.
Reference-3	#Person2# is worried about a man but #Person1# is not.
BART	#Person2# is worried about #Person1# because he hasn't come back from work. (43.48/28.57/50.01)
DialoBART	#Person2# is worried about #Person1#s friend who hasn't come back. (45.45/30.00/51.87)
DialSent-PGG	#Person2# is worried about a boy who hasn't come back.(47.62/42.11/53.90)
Table 13: A case from DialSumm.