SlowGuess/NAACL_2022_OCR huggingface dataset:
Qi Jia $^{1}$ , Yizhu Liu $^{1}$ , Haifeng Tang $^{2}$ , Kenny Q. Zhu $^{3*}$
$^{1}$ Shanghai Jiao Tong University, Shanghai, China
<sup>2</sup>China Merchants Bank Credit Card Center, Shanghai, China
$^{1}\{Jia_{-}q_{i},$  liuyizhu  $\} @sjtu.edu.cn$
$^{2}$ thfeng@cmbchina.com
$^{3}$ kzhu@cs.sjtu.edu.cn
Previous dialogue summarization techniques adapt large language models pretrained on the narrative text by injecting dialogue-specific features into the models. These features either require additional knowledge to recognize or make the resulting models harder to tune. To bridge the format gap between dialogues and narrative summaries in dialogue summarization tasks, we propose to post-train pretrained language models (PLMs) to rephrase from dialogue to narratives. After that, the model is fine-tuned for dialogue summarization as usual. Comprehensive experiments show that our approach significantly improves vanilla PLMs on dialogue summarization and outperforms other SOTA models by the summary quality and implementation costs.
Dialogue summarization is a specialized summarization task that takes a series of utterances from multiple speakers in the first person as input, and outputs fluent and concise summaries in third persons as shown in Figure 1. Different from previous monologue inputs such as news (Narayan et al., 2018) and scientific publications (Cohan et al., 2018), dialogues are always less well-organized. They usually contain complicated reference relations, inconsecutive inter-utterance dependencies, informal expressions, and so on, making dialogue summarization a more challenging task.
The most obvious characteristic of this task is the difference in the format and language styles between dialogue and its narrative summary. Liu, Shi and Chen (2021b) mentioned that coreference resolution models trained on general narrative text underperforms by about  $10\%$  on dialogue corpus, demonstrating the inherent gap between dialogue
Figure 1: An example from SAMSum dataset.
and narrative text. As a result, popular PLMs such as BART (Lewis et al., 2020) and PEGASUS (Zhang et al., 2020a) which excel on news summarization perform mediocrely on dialogue summarization.
To narrow this gap, previous work on dialogue summarization mainly resort to injecting dialogue features into PLMs to enhance dialogue understanding. These features include dialogue acts (Goo and Chen, 2018), topic transitions (Chen and Yang, 2020), coreference relations (Liu et al., 2021b), discourse graphs (Chen and Yang, 2021), etc, leading to the rule-based conversion from dialogues to plain text (Ganesh and Dingliwal, 2019). However, they suffer from three weaknesses. First, collecting or extracting these features becomes an additional step in the summarization pipeline, complicating the inference procedure at runtime. Second, oracle feature labels are hard to collect and errors can propagate from wrong labels to poor summaries. Third, additional layers or more encoders are required to incorporate features into PLMs, increasing the GPU memory footprint both during training and inference.
A more natural way to bridge this gap is to give the model more dialogue-narrative pairs to train on.
<sup>1</sup>Our code and results are publicly available at https:
//github.com/JiaQiSJTU/DialSent-PGG.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1660 - 1669
Due to the scarcity of dialogue summarization data, one approach (Zhu et al., 2020) is to convert other text summarization pairs into dialogue to summary pairs via some template, but such work requires additional data<sup>2</sup>.
In this paper, we propose an alternative approach that doesn't use any more data than the original dialogue summarization dataset. We convert each existing data pair into many "pseudo-paraphrase" pairs between a dialogue and a narrative sentence. Then we post-train a pre-trained seq2seq language model using a prefix-guided generation (PGG) task on the augmented paraphrase dataset. After that, the post-trained model is further fine-tuned as usual for dialogue summarization. To this end, no human efforts on crafting complicated rules or hyper-parameter tuning, or additional memory costs, as well as additional training data, is required. In sum, our contributions are:
- We propose a novel and effective post-training process to close the format and linguistic style gap between dialogues and narrative texts (§ 2).
- PGG with pseudo-paraphrase pairs requires no extra training data or labeling tools for features extractions (§ 3.2).
- Extensive experiments show that the proposed approach compares favorably with current SOTA models using less human efforts and computational costs (§ 3.3).
The training of a dialogue summarization model is divided into two stages: post-training and fine-tuning. The model can be any seq-to-seq PLMs and it remains unchanged except for the parameters which are updated stage by stage. We will elaborate on the post-training stage in the rest of this section.
2.1 Pseudo-paraphrase Dataset Construction
We construct rephrasing datasets from the dialogue summarization dataset itself. The original dialogue summarization dataset (DSum) is made up of dialogue-summary  $(D - S)$  pairs. Each dialogue  $D$  is a sequence of utterances and can be concatenated into a whole sequence:
$$
D = \left\{U _ {1}, U _ {2}, \dots , U _ {T} \right\} = \left\{x _ {1}, \dots , x _ {n} \right\} \tag {1}
$$
Each turn  $U_{t}$  is in the form of  $[r_t: u_t]$ , where  $r$  is a speaker and  $u$  is the actual utterance.
Our goal is to create more dialogue to narration kind of paraphrasing pairs. The most intuitive approach is to divide  $S$  into sentences, and pair each sentence to  $D$ . We call such pairs "pseudoparaphrases" because the output sentence (which we call  $p$ ) isn't exactly the paraphrase of the whole input, but rather part of the input.
However, doing this poses two challenges: 1)  $S$  is a coherent piece of text, and its sentences may depend on each other, so a single sentence  $p$  out of it may not stand by itself; 2) one  $D$  will be paired with several different  $p$ , and it is hard for the model to distinguish the meaning of these pairs.
Datasets	Input	Output
DSum	U1~8	Katarina wants to rent a flat from Liz. She will come visit it today after 6 pm.
DialSent	U1~8	Katarina wants to rent a flat from Liz.
U1~8	Katarina will come visit it today after 6 pm.
Table 1: Example pseudo-paraphrase pairs generated from the example in Figure 1. One pair in DSum becomes two pairs in DialSent. The prefix tokens determined by linguistic features, NOUN and ROOT, are underlined and italic respectively.
To solve 1) we apply coreference resolution<sup>3</sup> on  $S$  and convert every personal pronoun in it to the full reference first, before splitting the summary  $S$  into sentences. Sentence with fewer than 3 words (e.g., "Ally agree") are discarded since it carries too little information. The set of data pairs thus created is called (DialSent). An example is in Table 1.
To tackle 2), one obvious thought is to further split  $D$  into sets of sentences in which each set corresponds to a sentence  $p$  in the summary. However, our extensive experiments (see Appendix C) showed that none of the straight-forward heuristics work well to establish such alignments. This is mainly due to the fact that dialogue utterances are highly dependent. Thus, splitting operations are not optimal. Instead of changing  $D$ , we decide to use the pseudo-paraphrases directly but introduce a prefix-guided generation task to guide the model learning to extract relevant information from  $D$ .
2.2 Prefix-guided Generation Task
Summarization for dialogues focuses on analyzing "who-did-what" storylines (Chen and Yang, 2021) and the beginning of each summary sentence are
2More related work is in Appendix A.
3We use https://spacy.io/.
usually different speakers or the same speaker doing different things. As a result, using the prefix made up of "who" or "who-did" can help to select the related information from dialogues or plan the content to be generated.
In other words, we take the inspiration from content planning (Narayan et al., 2021; Wu et al., 2021). When training, the first few tokens of  $p$  are provided as prefix to the decoder. This prefix serves as an information selection hint to the model so it is easier to learn why that particular  $p$  should be generated. The losses are calculated between the generated tokens and reference tokens after the prefix as shown in Figure 2.
Figure 2: A illustration of our approach. BOS and EOS stand for begin and end of the sequence.
Let  $p = \{s_1, \ldots, s_l\}$ . Our prefix-guided training task is a vanilla auto-regressive generation task minimizing the negative log-likelihood of  $p$ :
$$
L = - \frac {1}{l - a} \sum_ {t = a} ^ {l} \log P \left(s _ {t} \mid s _ {<   t}, H ^ {d}\right) \tag {2}
$$
where  $a$  is the number of prefix tokens.  $H^{d}$  is the output hidden vectors of the encoder with input  $D$ .
There are various ways to determine the prefix length  $a$ . We can take a fixed length, a random length or a prefix up to a certain linguistic feature such as NOUN, VERB or ROOT. The exact linguistic feature to use is a dataset-dependent hyperparameter and can be tuned by the validation set. Examples of prefix tokens is marked in Table 1.
We first present the experimental setups, then conduct an ablation study to determine the proper prefix in PGG training, before our main results. More implementation details are in Appendix B.
We implement our experiments on SAMSum (Gliwa et al., 2019) and DialSumm (Chen et al., 2021), whose statistics are listed in Table 2.
Datasets	Variation	Train/Val/Test	IW	OW	CR
SAMSum	DSum	14,731/818/819	124.10	23.44	0.25
DialSent	29,757/1,654	149.93	11.93	0.13
DialSumm	DSum	12,460/500/500	187.52	31.02	0.18
DialSent	22,407/840	214.00	17.78	0.10
Table 2: Statistics of dialogue summarization datasets. IW, OW and CR represent the number of input words, the number of output words and compression ratio (OW/IW) respectively.
We compare our method with these baselines. Lead-3 and Longest-3 are simple rule-based baselines that extract the first or the longest 3 utterances in a dialogue as the summary respectively. PGN (See et al., 2017), Fast-Abs (Chen and Bansal, 2018), and PEGASUS (Zhang et al., 2020a) are well-known models for text summarization. BART (Lewis et al., 2020) is a general PLM and performs well after fine-tuning. CODS (Wu et al., 2021), Multi-view (Chen and Yang, 2020) and DialoBART (Feng et al., 2021b) are the SOTA models designed for dialogue summarization.
We evaluate both automatically and by human. For automatic evaluation, we use Rouge-1, 2, and L (Lin, 2004) F1-scores  ${}^{4}$  . Following Feng et al. (2021b),we adopt the same Rouge evaluation tool and compute between reference summaries and generated summaries. For DialSumm, we use maximum rouge scores among references for each sample. For human evaluation, we three proficient English speakers to evaluate 100 random samples from SAMSum. Each original dialogue and its reference summary are shown with generated summaries in a random order simultaneously. Showing summaries from different approaches together helps humans do comparisons between them. Following Chen and Yang (2020) and Liu et al. (2021b), each summary is scored on the scale of  $\left\lbrack  {2,0, - 2}\right\rbrack$  ,where 2 means concise and informative, 0 means acceptable with minor errors, and -2 means unacceptable. The final scores are averaged among annotators. We also ask human annotators to label the error types in the summary. We consider the following 4 error types: Missing important contents, Redundant content, Coreference mismatches, and Reasoning error. Rea and Cor concentrate on comparisons to the dialogue, and the rest two focus on comparisons to the reference. We determine the error for each case by majority voting, and count the errors of each model.
4https://pypi.org/project/py-rouge/
Models	Rouge-1	Rouge-2	Rouge-L
SAMSum
DSum-VG	51.48	27.27	49.45
DSum-PGG	52.52	27.51	49.03
DialSent-VG	52.16	27.79	49.41
DialSent-PGG	53.54	28.91	50.21
DialSumm
DSum-VG	53.15	28.86	51.48
DSum-PGG	53.27	28.64	51.69
DialSent-VG	52.99	29.14	51.40
DialSent-PGG	54.73	30.47	53.46
3.2 Ablations Study
We conduct ablations to verify the effectiveness of post-training on DialSent with PGG, including posttraining on DSum with PGG task (DSum-PGG), DSum with vanilla generation task (DSum-VG), and DialSent with vanilla generation task (DialSent-VG) in Table 3. The results of DSum-VG drop, indicating that fine-tuning for BART on DSum with early-stop is enough. Post-training with the same data and task leads to overfitting. DialSent-PGG performs best for two reasons. Compared with DialSent-VG, the prefix solves one-to-many mappings between a dialogue and summary sentences, so that the same dialogue can lead to different generations. On the other hand, the prefix can manipulate the selection within a short sentence but is not strong enough to direct content in multiple sentences. Thus, DialSent-PGG learns more cross-format paraphrasing ability and performs better.
Table 3: Ablations on DialSent with PGG task.
Models	Rouge-1	Rouge-2	Rouge-L
SAMSum
w/o	52.16	27.79	49.41
const	51.71	27.34	49.25
random	52.32	27.99	49.68
Ling-Noun	53.54	28.91	50.21
DialSumm
w/o	52.99	29.14	51.40
const	53.29	29.57	52.10
random	53.82	29.88	52.43
Ling-Root	54.73	30.47	53.46
We try several choices of prefix length: (1) W/O: without any prefix. (2) Const: Constant length set to 2 and 3 for SAMSum and DialSumm respectively, since a person's name is  $1.69 \pm 0.69$  tokens long on average. (3) Random: set by uniform sampling from a range of numbers. We set the range to  $1 \sim 3$  and  $2 \sim 4$  for the two datasets respectively. (4) Ling: using the validation set, we
determined that Noun and Root are the best choice for the two datasets, respectively. In this way, the number of prefix tokens for SAMSum and DialSum are  $1.90 \pm 1.10$  and  $3.55 \pm 1.24$ .
In Table 4, Ling performs the best among these variants. The actual linguistic feature to use may vary from dataset to dataset though. The remaining experiments will be conducted using PGG-Ling.
Table 4: Ablations on prefix designs for PGG.
Models	Rouge-1	Rouge-2	Rouge-L
SAMSum
Lead-3	31.41	8.68	30.38
Longest-3	32.46	10.27	29.92
PGN	40.08	15.28	36.63
Fast-Abs	41.95	18.06	39.23
PEGASUS	50.50	27.23	49.32
BART†	52.06	27.45	48.89
CODS	52.65	27.84	50.79
Multi-view	53.42	27.98	49.97
DialoBART	53.70	28.79	50.81
DialSent-PGG†	53.54	28.91	50.21
DialSumm
Lead-3	31.15	10.08	30.68
Longest-3	27.00	9.41	25.31
BART†	53.01	29.18	51.34
DialoBART†	53.26	29.58	52.01
DialSent-PGG†	54.73	30.47	53.46
Table 5: Dialogue summarization results compared with baselines.  $\dagger$  represents the models implemented by ourselves. Underlined scores are statistically significantly better than BART with  $p < 0.05$  based on t-test.
3.3 Comparison to SOTA Models
Automatic Evaluation: Our model DialSent-PGG performs competitively against other models on SAMSum and significantly better than the peers on DialSumm. It improves 1.5 on Rouge scores over BART for both datasets, while DialBART achieves less gains on DialSumm. Based on Table 1, DialSumm is a more difficult dataset with lower compression ratios. Our model performs better on samples with lower CR, i.e. more compressed samples, as shown in Figure 3, thus differences between DialSent-PGG and DialBART are more obvious on DialSumm. A simple case study is shown in Table 6. Multi-view faces the repetition problem as it takes the dialogue as input twice with two encoders. DialoBART has reasoning errors because it regards "William" as a keyword. DialSent-PGG instead generates a concise and correct summary. More cases are in Appendix D.
Human Evaluation: The overall human scores on BART, Multi-view, DialogBART and DialSent-PGG are 0.35, 0.40, 0.43 and 0.55 respectively. The Fleiss Kappa among three annotators is  $0.39^{6}$ .
${}^{5}$  DialSumm normalizes speaker names into " "#Person1#" resulting in more tokens.
6Fleiss Kappa between 0.4 and 0.6 is considered moderate.
Figure 3: Comparison for models on samples with different CR. X-axis represents the ranges for  $\mathrm{CR}(\%)$ . Y-axis is the Rouge-2 F1(\%).
Dialogue	William: are you still angry?
Emilia: YES
William: :()
Multi-view	Emilia is still angry and still angry.
DialoBART	William and Emilia are still angry.
DialSent-PGG	Emilia is still angry.
The latter three models all improve BART, with DialSent-PGG topping the ranks.
Figure 4: Error analysis on SAMSum.
For error analysis, the Fleiss Kappa for Mis, Red, Cor and Rea are 0.55, 0.10, 0.26, 0.42 respectively. The agreement on Red is lower because identifying unimportant information is hard. The agreement on Cor is fair due to undistinguishable errors. For example, mismatching of a person and an event among multiple utterances can be either a Cor or a Rea. Besides, Red always leads to Mis. So, we divide the error types into two groups and merge them with "OR" logical operation within each group. The Fleiss Kappa for Mis|Red and Cor|Rea are 0.45 and 0.46. We show error types with the agreement larger than 0.40 in Figure 4.
Multi-view performs better on content selection and DialSent-PGG performs better on reasoning and coreference understanding, while DialoBART lies in between. Fewer errors on Rea and CorlRea reflect that our approach successfully narrows the understanding gap. Because references are not the only good summary, high missing content doesn't mean that the generated summary is unacceptable. As a result, the model with fewer CorlRea errors receives higher overall score.
Implementation Costs: We compare the im
presentation costs between our approach and two state-of-the-art models, i.e. Multi-view and DialogBART, in Table 7. Although explicitly injecting features for dialogue understanding is effective, labels for these features are hard to collect and implementation costs for these approaches on a new dataset are high. Multi-view and DialogBART proposed doing labeling automatically with unsupervised algorithms or language models. However, these labeling approaches bring extra hyper-parameters which are different between datasets and need to be found by trial and error. If we use the same keywords extraction ratio, similarity threshold and topic segmentation ratio from SAMSum directly, the results on DialSumm are only 50.61/26.67/49.06 (Rouge-1/2/L). We searched for the best combination of hyper-parameters following their paper and did 14 trials, while applying our approach on DialSumm only need 4 trials.
On the other hand, injecting features increases the requirement of GPU memory. With the same training parameters(max tokens=1024, batch size=1, gradient checkpointing=False), Multiview with double-encoder design encounters an out-of-memory error on RTX 2080Ti with 11G GPU memory. DialoBART occupies around 10.36G since it lengthens the dialogue with additional annotations. DialSent-PGG only occupies 9.87G during post-training for recording the length of the prefix, and 9.65G during fine-tuning which is the same as vanilla BART. In a word, our approach costs less for implementation.
Table 6: A case from SAMSum. Errors are in italic.
Models	Mem	#HP	#Tri	#St
Multi-view	OOM	5	-	-
DialoBART	10.36G	3	14	38.61k
DialSent-PGG	9.87G/9.65G	1	4	19.32k
Table 7: The upper-bound of GPU memory footprint (Mem), newly introduced hyper-parameter counts (#HP), the number of trails (#Tri) and total training steps (#St) for implementing different models.
We propose to post-train dialogue summarization models to enhance their cross-format rephrase ability by prefix-guided generation training on dialogue-sentence pseudo-paraphrases, and get promising results. Creating self-supervised tasks for cross-format post-training and incorporating compatible features for downstream fine-tuning are plausible future directions.
Siqi Bao, Huang He, Fan Wang, Hua Wu, and Haifeng Wang. 2020. Plato: Pre-trained dialogue generation model with discrete latent variable. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 85-96.
Jiaao Chen and Diyi Yang. 2020. Multi-view sequence-to-sequence models with conversational structure for abstractive dialogue summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4106-4118.
Jiaao Chen and Diyi Yang. 2021. Structure-aware abstractive conversation summarization via discourse and action graphs. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1380-1391.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675-686.
Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021. DialogSum: A real-life scenario dialogue summarization dataset. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5062-5074, Online. Association for Computational Linguistics.
Kyunghyun Cho, Bart Van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 615-621.
Alexander Fabbri, Faiaz Rahman, Imad Rizvi, Borui Wang, Haoran Li, Yashar Mehdad, and Dragomir Radev. 2021. ConvoSumm: Conversation summarization benchmark and improved abstractive summarization with argument mining. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International