SlowGuess/NAACL_2022_OCR huggingface dataset:
To evaluate our models, we utilize both quantitative metrics and human evaluation in our experiment. In detail, we use ROUGE-1, ROUGE-2 and ROUGE-L as quantitative metrics, which is widely used in NLP and summary tasks (Liu et al., 2021a,b; Chen and Yang, 2020). For human evaluation, we randomly select 100 dialogue-summary pairs from the test set of SAMSum and DialSum, respectively. Five annotators(all CS majors students studying NLP) are demanded to give the comparison between our model and baseline models. They are not told which summaries are derived from the baseline model and which summaries are derived from our model. They are required to evaluate the generated summary from three aspects: whether the generation is fluent, whether it has omitted content,
SAMSum Dataset
Model	R-1	R-2	R-L
Lead3	31.4	8.7	29.4
PTGen	40.1	15.3	36.6
DynamicConv+GPT-2	41.8	16.4	37.6
FastAbs-RL	42.0	18.1	39.2
DynamicCov+News	45.4	20.7	41.5
Multiview BART	52.2	27.4	49.9
BART(large)	50.9	25.0	47.1
ReWriteSum(ours)	54.2	27.1	50.1
DialSum Dataset
Model	R-1	R-2	R-L
Lead3	27.5	6.8	27.3
LONGEST	24.1	6.2	22.7
BART(base)	33.7	13.8	30.9
BART(large)	34.1	13.7	31.2
ReWriteSum(ours)	35.1	14.6	32.1
Table 2: Metric-based evaluations of ReWriteSum and baselines on SAMSum and DialSum. R-1, R-2, R-L denote ROUGE-1, ROUGE-2, ROUGE-L, respectively.
SAMSum Dataset
Model	ReWriteSum vs.
win(%)	loss(%)	tie(%)
Multi-view	48.5	6.9	44.6
BART	52.6	5.1	42.3
DialSum Dataset
Model	ReWriteSum vs.
win(%)	loss(%)	tie(%)
Multi-view	42.3	8.1	49.6
BART	46.8	7.3	45.9
Table 3: Human evaluations on SAMSum and DialSum.
and whether it has factual inconsistent errors. The evaluation results are represented as win, loss and tie, respectively indicating that the quality of generated summary by ReWriteSum is better, weaker or equal to baselines.
In this section, we demonstrate our experiment results on SAMSum and DialSum datasets.
4.2.1 Metric-based Evaluation
The quantitative evaluation results on SAMSum and DialSum datasets are shown in Table 2. For SAMSum dataset, we refer to (Gliwa et al., 2019) to show the results of Lead3, PTGen, DynamicConv+GPT-2/News, and FastAbsRL. From the results, we can see that the pretrained models, such as BART and Multiview BART, outperform the traditional summarization models, showing the effectiveness of pre-training
$^{2}$ https://huggingface.co/facebook/bart-large
Dialogue Example 1	Dialogue Example 2
Mia: could anybody help me to buy a flight ticket? ... Mia: I don’t have a credit card at the moment. ... Tom: You can use mine help Mia to buy a flight ticket! Mia: Should I send you the link to buy a flight ticket? Tom: Just send me the flight, company and your personal data that I may need to buy a flight ticket. Mia: Great, so nice of you, thanks Tom.	Maria: Who’s gonna be at IMF lecture tomorrow? ... Alexander: On Saturday Alexander already meet for another. So my option is Friday afternoon or tomorrow. Sarah: Tomorrow and on Friday Sarah available ... Sarah: So can we meet tomorrow evening? 17:15? Alexander: It is fine by me. Lawrence: Lawrence will be late, but you can start without me.
BART Prediction: Mia doesn’t have a credit card at the moment. Tom will use his card to buy a flight ticket for himself. Tom needs the flight, company and personal data.	BART Prediction: Alexander, Martha, Sarah, Lawrence and Sarah will meet tomorrow evening at 17:15 to discuss the IMF lecture.
ReWriteSum Prediction: Mia doesn’t have a credit card at the moment. Tom will use his card to buy a flight ticket for her.	ReWriteSum Prediction: Maria, Sarah, Alexander, and Martha will meet tomorrow evening at 17:15 to discuss the IMF lecture. Lawrence will be late.
Table 4: Generated summaries from different models on SAMSum. Red words show the inconsistent content. Green words show the factul content. Blue words show the supplemented part by our model. Orange words show the name replacement by our model.
language model for dialogue summarization task. Our ReWriteSum model performs the best. Take the ROUGE-1 and ROUGE-L score for example, our ReWriteSum model obtains 54.2 and 50.1, respectively, which obviously outperforms Multiview BART model, i.e., 52.2 and 49.9.
From the results on DialSum in Table 2, we can see that our model also obtains the best performance. Take the ROUGE-1 and ROUGE-L score for example, our ReWriteSum obtains 35.1 and 32.1, respectively, which obviously outperforms BART(large), i.e., 34.1 and 31.2. However, the performance increment on DialSum is not significant as comparison on SAMSum. The reason is that utterances in DialSum are relatively more complete and the interactive speakers are fewer than SAMSum. According to statistics, there are only 13 sentences with more than 4 speakers in DialSum, which leads to relatively few errors caused by multi-speakers. We have conducted the significant test, and the result shows that the improvements of our model are significant on both datasets, i.e., p-value  $< 0.01$ .
In conclusion, our ReWriteSum model has the ability to generate a more complete and accurate summary than baselines.
4.2.2 Human Evaluation
Human evaluation results are shown in Table 3. The percentages of win, loss and tie, as compared with the baselines, are given to evaluate the fluency, completeness and consistency of generated summary by ReWriteSum. From the results, we can see that the proportion of evaluators who think our model better is the largest, surpassing other models. Take SAMSum dataset for example, ReWriteSum model obtains preference gains (win subtract loss)  $41.6\%$ ,  $47.5\%$ , respectively.
4.2.3 Case Study
To further understand our proposed model, we give some generated cases in Table 4. According to the result, we can notice that ReWriteSum model performs better than baseline models. Take example1 in Table 4 as an example, BART model generates that "Tom will buy a flight ticket for himself", but in the dialogue content, the dialogue fact is "Tom will buy a flight ticket for Mia'. The reason is that the dialogue content tends to be omitted in daily dialogues. From example1, we can see that, in the entire dialogue, only Mia mentions "help me to buy a flight ticket" at the beginning, and this sentence is omitted in the subsequent utterances, which makes BART unable to correctly understand who the "ticket" will be bought for. When we rewrite the incomplete dialogue (in blue font), "help Mia to buy a flight ticket" is added to the end of some utterances, so that our model can generate a more accurate and logical consistent summary.
From example2, due to the complex references in this dialogue, BART misunderstood "Lawrence will be late" as "Lawrence will meet tomorrow evening at 17:15". When co-reference data augmentation is carried out, it strengthens the connection between "you" and "Alexander, Martha, Sarah" in the sentence "Lawrence: I will be late, but you can start without me", so as to avoid this personal inconsistency error.
In order to confirm whether the improvement is related to incomplete utterance rewriting(IUR) and co-reference data augmentation(CDA), a further analysis is conducted, containing ablation study, the impact of participants, and the error analysis.
Model	R-1	R-2	R-L
ReWriteSum	54.2	27.1	50.1
- w/o CDA	51.1	25.1	47.5
- w/o IUR	52.3	25.1	48.1
Figure 3: Rouge-L scores of ReWriteSum and BART with different number of speakers.
To confirm the effectiveness of our IUR and CDA module, we conduct ablation experiments on SAMSum dataset. The results are shown in Table 6. ReWriteSum w/o IUR means that ReWriteSum model removes IUR module and only with CDA to generate summaries. From the results, we can see that when only CDA is applied, the ROUGE score is still larger than the baseline, but smaller than the ReWriteSum model. ReWriteSum w/o CDA means that our ReWriteSum model removes the CDA module and only with IUR to generate summaries. We can see that the ROUGE has decreased as compared with our ReWriteSum model, but it is still higher than baseline models. Therefore, we think that both incomplete utterance rewriting and co-reference data augmentation have positive effects for dialogue summarization.
Not only that, we also notice that the ROUGE score using IUR alone is higher than using CDA alone, indicating that IUR contributes more to the dialogue summarization task.
4.3.2 Impact of Participants
We conduct an experimental analysis with different number of participants, by calculating the ROUGE score for baselines and our ReWriteSum model on SAMSum. From the Figure 3, we can see that with the increase of participants, the rouge score of our model decreases more slowly, because: (1) with the increase of participants, the omitted information will also increase, but our incomplete utterance rewriting module has the ability to reduce the impact of too much omitted information in the summary; (2) our co-reference data augmen
Table 5: Ablation experiment results on SAMSum.
Model	Missing Information	Wrong Reference	Incorrect Reasoning
BART(large)	36	24	19
ReWriteSum	14	6	8
- w/o CDA	17	19	9
- w/o IUR	29	7	11
Table 6: Percentage of typical errors in summaries generated by BART(large) and our ReWriteSum model.
tation module can reduce the impact of complex referencing caused by too many participants.
4.3.3 Error Analysis
To further study the impact of IUR and CDA on the quality of generated summaries, we count the following 3 kinds of errors that appear in the summaries generated by the baseline model and our model: Missing Information: content information that appears in gold summaries is missing from generated summaries. Wrong Reference: content in the generated summaries, such as the person's actions or name, does not match what is described in the source dialogue. Incorrect reasoning: the conclusions drawn by the generated summaries are inconsistent with the facts in the source dialogue.
We randomly select 100 dialogues and their generations from SAMSum and count the error categories, as shown in Table 5. In terms of missing information, our model outperforms the baseline model because IUR can effectively prevent the model from missing information. According to the wrong reference numbers, our model performs better than the baselines because CDA can enhance the model's understanding of referential information. Errors occur in incorrect reasoning are also reduced as our model complements default information and enhances understanding of referential information.
In this work, we propose a new dialogue summarization model, namely ReWriteSum, which leverages incomplete utterance rewriting and co-reference data augmentation mechanism to generate summaries for dialogue. Our motivation comes from the fact that there are a lot of ellipsis and demonstrative pronouns in the dialogue, which seriously affects the quality of dialogue summary generation. Our core idea is to utilize the incomplete utterance rewriting module to complete the ellipsis information in the dialogue content and enhance the personal entities with the co-reference data augmentation mechanism. We conduct exper
iments on both SAMSum and DialSum datasets, and the results on both quantitative and qualitative analysis verify the effectiveness of our proposed model. Therefore, we obtain the conclusion that the incomplete utterance rewriting and co-reference data augmentation are effective for improving the quality of generation for dialogue summarization.
The abstractive summarization dialogue system proposed in this work can be applied to dialogue scenarios. It can quickly process a lengthy dialogue into a short content containing the core idea of the dialogue. Such features can be applied to meetings, customer service, and medical scenarios to facilitate people's life. The datasets SAMSum and DialSum used in this work are publishable and for research purposes only. There may be some biased content in the datasets, which should be viewed carefully.
Hainan Zhang and Yanquan Zhou are the corresponding authors. We would like to thank anonymous reviewers for their thoughtful comments and suggestions.
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Jiaao Chen and Diyi Yang. 2020. Multi-view sequence-to-sequence models with conversational structure for abstractive dialogue summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4106-4118.
Jiaao Chen and Diyi Yang. 2021. Structure-aware abstractive conversation summarization via discourse and action graphs. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1380-1391.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675-686.
Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021. Dialogsum: A real-life scenario dialogue summarization dataset. In *Findings of the Association*
for Computational Linguistics: ACL-IJCNLP 2021, pages 5062-5074.
Ahmed Elgohary, Denis Peskov, and Jordan Boyd-Graber. 2019. Can you unpack that? learning to rewrite questions-in-context. Can You Unpack That? Learning to Rewrite Questions-in-Context.
Xiachong Feng, Xiaocheng Feng, Bing Qin, Xinwei Geng, and Ting Liu. 2020. Dialogue discourse-aware graph convolutional networks for abstractive meeting summarization.
Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. 2018. Bottom-up abstractive summarization. In EMNLP.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. Samsum corpus: A human-annotated dialogue dataset for abstractive summarization. EMNLP-IJCNLP 2019, page 70.
Chih-Wen Goo and Yun-Nung Chen. 2018. Abstractive dialogue summarization with sentence-gated modeling optimized by dialogue acts. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 735-742. IEEE.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In *ICLR (Poster)*.
Wojciech Krysciński, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 540-551.
Wojciech Krysciński, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332-9346.
Vineet Kumar and Sachindra Joshi. 2016. Non-sentential question resolution using sequence to sequence learning. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2022-2031.
Vineet Kumar and Sachindra Joshi. 2017. Incomplete follow-up question resolution using retrieval based sequence to sequence learning. In Proceedings of
the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 705-714.
Yuejie Lei, Yuanmeng Yan, Zhiyuan Zeng, Keqing He, Ximing Zhang, and Weiran Xu. 2021. Hierarchical speaker-aware sequence-to-sequence model for dialogue summarization. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7823-7827. IEEE.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880.
Manling Li, Lingyu Zhang, Richard J Radke, and Heng Ji. 2019. Keep meeting summaries on topic: Abstractive multi-modal meeting summarization. In 57th Conference of the Association for Computational Linguistics.
Chunyi Liu, Peng Wang, Jiang Xu, Zang Li, and Jieping Ye. 2019a. Automatic dialogue summary generation for customer service. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1957-1965.
Junpeng Liu, Yanyan Zou, Hainan Zhang, Hongshen Chen, Zhuoye Ding, Caixia Yuan, and Xiaojie Wang. 2021a. Topic-aware contrastive learning for abstractive dialogue summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1229-1243.
Qian Liu, Bei Chen, Jian-Guang Lou, Ge Jin, and Dongmei Zhang. 2019b. Fanda: A novel approach to perform follow-up query analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6770-6777.
Qian Liu, Bei Chen, Jian-Guang Lou, Bin Zhou, and Dongmei Zhang. 2020. Incomplete utterance rewriting as semantic segmentation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2846-2857.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730-3740.
Zhengyuan Liu, Angela Ng, Sheldon Lee, Ai Ti Aw, and Nancy F Chen. 2019c. Topic-aware pointer-generator networks for summarizing spoken conversations. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 814-821. IEEE.
Zhengyuan Liu, Ke Shi, and Nancy Chen. 2021b. Coreference-aware dialogue summarization. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 509-519.
Zhufeng Pan, Kun Bai, Yan Wang, Lianqiang Zhou, and Xiaojiang Liu. 2019. Improving open-domain dialogue systems via multi-turn incomplete utterance restoration. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1824-1833.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Jun Quan, Deyi Xiong, Bonnie Webber, and Changjian Hu. 2019. Gecor: An end-to-end generative ellipsis and co-reference resolution model for task-oriented dialogue. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4547-4557.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241. Springer.
Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In EMNLP.
Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 45(11):2673-2681.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083.
Guokan Shang, Wensi Ding, Zekun Zhang, Antoine Tixier, Polykarpos Meladianos, Michalis Vazirgiannis, and Jean-Pierre Lorre. 2018. Unsupervised abstractive meeting summarization with multi-sentence compression and budgeted submodular maximization. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 664-674.
Hui Su, Xiaoyu Shen, Rongzhi Zhang, Fei Sun, Pengwei Hu, Cheng Niu, and Jie Zhou. 2019. Improving multi-turn dialogue modelling with utterance rewriter. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 22-31.
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. 2018. Pay less attention with lightweight and dynamic convolutions. In International Conference on Learning Representations.
Yusen Zhang, Ansong Ni, Tao Yu, Rui Zhang, Chenguang Zhu, Budhaditya Deb, Asli Celikyilmaz, Ahmed Hassan, and Dragomir Radev. 2021. An exploratory study on long dialogue summarization: What works and what's next. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 4426-4433.
Zhou Zhao, Haojie Pan, Changjie Fan, Yan Liu, Linlin Li, Min Yang, and Deng Cai. 2019. Abstractive meeting summarization via hierarchical adaptive segmental network learning. In The world wide web conference, pages 3455-3461.