SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 3: ROUGE scores of the zero-shot setting for TODSum. Results are averaged over three random runs. "DS" denotes the dialogue states. Values in the second row denote the size of train-valid/test set.  $(p < 0.05$  under t-test)
Models	Academic	Committee	Product
1,069 / 195 / 312	981 / 178 / 417	614 / 115 / 847
R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L
Lead-3	14.86	2.68	12.62	23.18	7.71	19.33	18.61	4.20	15.55
Oracle	32.44	8.78	27.92	54.92	35.03	51.85	40.23	15.29	35.83
BertExt	19.36	2.72	16.90	31.08	11.32	27.46	23.43	4.93	20.63
PGN	19.38	2.65	17.06	20.14	3.30	18.22	17.59	2.07	15.21
Transformer	18.72	2.53	16.97	20.01	3.16	17.88	17.14	2.02	15.14
BertAbs	20.36	2.45	17.84	20.93	3.23	18.26	17.32	1.86	15.30
BART	20.53	3.63	18.09	26.65	8.28	24.84	23.14	5.12	20.87
BART w. QR	22.45	4.11	20.14	27.93	9.73	25.87	24.89	5.93	22.39
Prefix-tuning	21.90	4.01	18.71	27.70	9.37	25.62	25.77	7.26	23.06
DOP	24.43	5.84	21.49	30.28	11.63	27.33	29.85	9.41	26.88
Table 4: ROUGE scores of the zero-shot setting for QMSum. "QR" denotes the queries. Same as TODSum, values in the second row denote the size of train-valid/test set. All results are averaged over three random runs.  $(p <   0.05$  under t-test)
trainable number of parameters of the DOP model is only  $15\%$  of the BART w. DS. For all experiments, we set the number of training epochs to 30. At the decoding phase, we use a beam size of 6 and max generated tokens of 125. The decoding takes 1.3 seconds per batch with the size of 2.
Results on TODSum Table 3 presents the results of the zero-shot setting for TODSum dataset, where each of the five domains is regarded as the target domain respectively. The division of dataset in the second row intuitively shows that the amount of data in  $D_{s}$  is small and limited. We conduct experiments based on some common extractive models and some strong abstractive baselines. We also add a lightweight fine-tuning summarizer for comparison. As observed, for most ROUGEs, Prefix-tuning performs worse than BART and BART w. DS. It is because the dialogue text is long and complex, and using only  $20\%$  parameters of fine-tuning can not well understand the domain knowledge and identify the key contents in dialogues. Compared to Prefix-tuning of the same magnitude parameters, our model improves by  $7\%$ ,  $3\%$ ,  $7\%$  for train domain,  $5\%$ ,  $5\%$ ,  $3\%$  for taxi domain,  $4\%$ ,  $8\%$ ,
$4\%$  for restaurant domain,  $5\%$ ,  $6\%$ ,  $5\%$  for hotel domain, and  $8\%$ ,  $8\%$ ,  $9\%$  for attraction domain. This shows that the prefix module initialized by domain words and the discrete prompts composed of dialogue states play important roles. Besides, our model still surpasses BART w. DS, a full-parameter fine-tuning based model, which further illustrates that our model efficiently disentangles the knowledge of the source and target domains. Note that attraction domain gets the highest ROUGEs and the increased margins are also the largest. This may be due to the high overlaps between the attraction and source domains. All the results suggest that with limited data, the performance of our model still reaches state-of-the-art.
Results on QMSum Table 4 displays the results on zero-shot out-domain tests in three domains of QMSum dataset. As seen from the second row, in addition to the limited data, the source domain size may be even less than the target domain size, i.e., product domain. The trend of overall performance is consistent with that of TODSum dataset  $^{5}$ , where the improvement in product domain is the most obvious and there are  $5\%$ ,  $4\%$  and  $5\%$  increased for R-1, R-2, and R-L, respectively. However, all the ROUGEs are low, which is because there are no obvious domain words, leading to serious domain entanglement. Besides, due to the longer meeting text, it is hard to capture the key contents in dialogues, so as to the poor generalization in the target domain. Generally, these results show that the multi-domain setting in meeting summarization task is apparently necessary and meaningful. Meeting transcripts cover various domains, making the
<sup>5</sup>Note that ROUGEs of Oracle are very high in QMSum, which is because most parts of the golden summaries are directly copied from the original dialogue. It is determined by the characteristics of the QMSum and the results are consistent with Zhong et al. (2021).
(a) Number of domain words (b) Quality of domain words
Figure 3: Effect of different numbers and different qualities of domain words for train domain in TODSum. The dotted lines in (b) respectively represent the F1 sources of ROUGE-1, ROUGE-2, and ROUGE-L of the Prefix-tuning model.
adaptation of models particularly difficult.
6.1 Effect of Domain Words
Number of Domain Words We set different numbers of domain words, i.e., prefix length, to test the performance of our model DOP for train domain in TODSum dataset. As shown in Figure 3 (a), among these setting candidates, there is a threshold (140) that allows the ROUGEs to reach the peak. When choosing a fewer setting, the model does not perform well due to insufficient number of parameters, which is improved as the number increases. When being more than this threshold, a drop in performance occurs. One reason is that too long a sequence adds a large burden to BART, and the other one is that it introduces excessive noise. However, the change in the number of the domain words does not have a great impact on the performance (only  $2\sim 3\%$  fluctuation), which also reflects the effectiveness of domain-oriented prefix module and the robustness of our model.
Quality of Domain Words For train domain in TODSum, we randomly replace a certain percentage of the domain words with words that are not related to the source domain. As Figure 3 (b) shows, when more noise is introduced, the model suffers more interference and its performance decreases. However, it performs better than Prefix-tuning. Only when the proportion of noise reaches  $100\%$ , the performance of our model is even worse than that of Prefix-tuning. This is because we especially use completely irrelevant words for initialization and fitting, which introduces more noise than simple random initialization and affects the performance of DOP. From this point of view, introducing high-quality domain words is good for
Model	R-1	R-2	R-L
DOP (ours)	52.51	25.45	47.78
w/o DW	48.87	23.81	44.52
w/o DS	47.59	23.25	43.41
w/o DW & DS	45.92	22.70	41.06
Table 5: F1 scores of ablation study on train domain of TODSum dataset. "DW" denotes domain words and "DS" denotes dialogue states.
Model	R-1	R-2	R-L
DOP (ours)	30.28	11.63	27.33
w/o DW	29.10	10.59	26.53
w/o QR	28.47	9.60	26.38
w/o DW & QR	27.70	9.37	25.62
Table 6: F1 scores of ablation study on committee domain of QMSum dataset. "DW" denotes domain words and "QR" denotes queries.
domain disentanglement and the quality of domain words is critical to summary generation.
We perform ablation experiments on train domain of TODSum dataset and committee domain of QM-Sum dataset, as shown in Tables 5 and 6. We can observe that the removal of domain-oriented initialization in the prefix module will make the ROUGEs decrease significantly. Especially for TODSum, R-1, R-2, and R-L drop by  $4\%$ ,  $2\%$ , and  $3\%$ , which shows the importance of domain word information for inducing the relevant knowledge while adapting to a new domain. In addition, after we remove the discrete prompts, i.e. dialogue state and query, the performance of the models becomes worse, but still outperforms the results of Prefix-tuning. It demonstrates that discrete prompts help the model pay attention to the key elements in the dialogue and improve the generalization of the model. Notably, our model achieves summary generation only by optimizing the domain-oriented prefix module, where domain words are available in all datasets. Since the DS and QR features happen to exist in the two datasets, we take advantage of them together with dialogue texts. When removing both DW and DS/QR at the same time, the model is equivalent to Prefix-tuning and the results are consistent.
6.3 Effect of Prefix Module in Encoder and Decoder
Since both the encoder and the decoder in our DOP introduce the prefix module, we verify their effects in the train and committee domains respectively.
Domain	Model	R-1	R-2	R-L
Train	DOP	52.51	25.45	47.14
w/o enc_prefix	50.69	23.58	45.98
w/o dec_prefix	45.51	22.15	40.67
Committee	DOP	30.28	11.63	27.33
w/o enc_prefix	29.20	11.34	26.37
w/o dec_prefix	29.11	11.29	26.30
Table 7: Effects of prefix module in encoder and decoder. We removed the prefix module from the encoder and decoder respectively to verify its effectiveness.
Figure 4: Effect of different few-shot data size for train domain in TODSum dataset. "0" denotes the zero-shot setting.
As shown in Table 7, when the encoder prefix module or decoder prefix module is removed, the performance of the model decreases, which shows that both are necessary and effective. In addition, we find that it is interesting that removing the prefix on the encoder side has a smaller impact on the model than removing the decoder side, especially in TODSum (about  $5\%$  on R-1). A reasonable explanation is that the prefix modules in encoder and decoder are responsible for different tasks. The prefix module on the encoder side assists the model to understand the dialogue, while the prefix module on the decoder side assists in model generation. Therefore, for summary generation, the prefix module in decoder is more helpful to the model.
6.4 Effect of Training Data
Performance in Few-shot Settings For TODSum, we fix the size of source domain data, adding a certain amount of target (train) domain data for training, as shown in Figure 4. As the size of target domain data increases, the performance of both BART w. DS and DOP present a steady improvement trend and that of our DOP model is consistently better than BART w. DS, which is as expected. Besides, there is a substantial improvement from 50 to 100. This phenomenon shows that adding target knowledge can help the model learn about information of target domain and after adding a certain amount will help the model more efficiently.
Effect of Source Domain Data Size We keep
Figure 5: Effect of different source domain data size for train domain in TODSum dataset. "origin (2332)" denotes the same setting as the main results.
(a) TODSum
(b) QMSum
Figure 6: Relationships between length of encoder/decoder input and the best prefix length.
the zero-shot setting unchanged and adjust the size of source domain data for training to observe changes in the performance of the two models for train domain in TODSum. As shown in Figure 5, the smaller of data size, the greater the difference between the performance of the DOP and BART w. DS, that is, the performance of BART w. DS is getting worse, while the DOP maintains excellent performance steadily. This demonstrates that our DOP model is insensitive to the data scale and robustness to a certain extent. This also confirms that in the main experiment, our model can be outstanding in very limited and uneven data.
6.5 Prefix Length vs. Input Length
Through experiments, we explore an interesting thing, that is, the prefix length (number of domain words) that makes the model perform best may be related to the input length. Based on this assumption, we collect the source input length, target input length, and their corresponding optimal prefix length from two datasets, as shown in Figure 6. We conclude a general rule that the longer inputs may prefer the shorter prefix. This phenomenon may serve as a research point in the future.
7.1 Human Evaluation
We further conduct a manual evaluation to assess the models. We randomly select 50 samples from
Model	Flu.	Inf.	Fac.	Dom.	Red.
Ground Truth	4.95	4.56	4.28	4.71	4.33
BART	4.19	4.21	3.55	3.19	3.53
BART w. DS	4.36	4.34	4.09	3.35	3.62
Prefix-tuning	4.23	4.29	3.67	3.28	4.09
DOP	4.68	4.42	4.10	4.07	4.13
Table 8: Human evaluation on Fluency (Flu.), Informativeness (Inf.), Factual Consistency (Fac.), Domain Relevance (Dom.), and Redundancy (Red.) for TOD-Sum dataset.
each target domain dataset of TODSum (Zhao et al., 2021b) and ask 10 professional linguistic evaluators to score the ground truth and summaries generated by 4 models according to 5 metrics: fluency, informativeness, factual consistency, domain relevance, and redundancy. Each metric is rated by 3 evaluators from 1 (worst) to 5 (best) and the scores for each summary are averaged. Note that the intraclass agreement score is 0.605.
As shown in Table 8, the fluency scores of all models are high, which is because that abstractive models fine-tuned on contextualized language backbones can generate fluent sentences (Lewis et al., 2020). For factual consistency, both DOP and BART w. DS achieve better performance than Prefix-tuning, which suggests that the dialogue state information guides the model to focus more on the key information, such as slots and intents. Besides, the DOP outperforms all baselines in the domain relevance metric. This demonstrates that the domain-oriented prefix module plays a crucial role in enhancing the ability of the model to identify domain-related features and disentangle the knowledge of the source and target domains. Surprisingly, the scores about the redundancy of Prefix-tuning and DOP are higher than that of BART and BART w. DS. This is because the model can efficiently extract key contents from a limited amount of data without relying on large-scale pre-trainings.
8 Challenges
Through the analysis of cases in Appendix C, we summarize two challenges of low-resource domain adaption for abstractive dialogue summarization task:
1. Confusion between domains with high similarity: We found that in domains with high-overlap vocabularies, i.e., restaurant and hotel, train and taxi, the model generates some domain-confusing sentences. Taken hotel-restaurant pair as an example, when restaurant is as the target domain,
a sentence like "book a restaurant room that can accommodate 3 people, ..." is generated, which is more likely to exist in the hotel domain. Note that this challenge does not affect the accuracy of key elements, but the language style is not appropriate.
2. Information dispersion: Because of the long input sequence, it is difficult for the models to pay attention to all aspects of the long dialogue and there will be problems with attention deviations on the key elements of dialogues, especially for this lightweight and small parameter training paradigm.
In this paper, we present a domain-oriented prefix-tuning model to handle the domain adaptation for dialogue summarization based on an efficient and generalizable fine-tuning method. The domain word initialized prefix module disentangles the target domain knowledge from the source domain and the discrete prompts enhance the generalization ability of the model. The experiments in zero-shot and few-shot settings show that our methods have made great progress on two datasets.
We thank all anonymous reviewers. This work was partially supported by National Key R&D Program of China No. 2019YFF0303300 and Subject II No. 2019YFF0303302, DOCOMO Beijing Communications Laboratories Co., Ltd, MoE-CMCC "Artificial Intelligence" Project No. MCM20190701, BUPT Excellent Ph.D. Students Foundation CX2021204.
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
J. Carletta, S. Ashby, S. Bourban, Mike Flynn, Maël Guillemot, Thomas Hain, J. Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, G. Lathoud, M. Lincoln, Agnes Lisowska Masson, I. McCowan, W. Post, D. Reidsma, and P. Wellner. 2005. The ami meeting corpus: A pre-announcement. In MLMI.
Jiaao Chen and Diyi Yang. 2020. Multi-view sequence-to-sequence models with conversational structure for
abstractive dialogue summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4106-4118, Online. Association for Computational Linguistics.
Jiaao Chen and Diyi Yang. 2021. Structure-aware abstractive conversation summarization via discourse and action graphs. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online. Association for Computational Linguistics.
Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2021. GSum: A general framework for guided neural abstractive summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4830-4842, Online. Association for Computational Linguistics.
Alexander Fabbri, Simeng Han, Haoyuan Li, Haoran Li, Marjan Ghazvininejad, Shafiq Joty, Dragomir Radev, and Yashar Mehdad. 2021. Improving zero and few-shot abstractive summarization with intermediate fine-tuning and data augmentation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 704-717, Online. Association for Computational Linguistics.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79, Hong Kong, China. Association for Computational Linguistics.
Karl Moritz Hermann, Tomás Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NIPS.
Matthew Hoffman, Francis R. Bach, and David M. Blei. 2010. Online learning for latent dirichlet allocation. In J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems 23, pages 856-864. Curran Associates, Inc.
Xinyu Hua and Lu Wang. 2017. A pilot study of domain adaptation effect for neural abstractive summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 100-106, Copenhagen, Denmark. Association for Computational Linguistics.
A. Janin, D. Baron, J. Edwards, D. Ellis, D. Gelbart, N. Morgan, B. Peskin, T. Pfau, E. Shriberg, A. Stolcke, and C. Wooters. 2003. The icsi meeting corpus. In 2003 IEEE International Conference on Acoustics,
Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03), volume 1, pages I-I.
M. Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 605-612.
Haitao Lin, Liquan Ma, Junnan Zhu, Lu Xiang, Yu Zhou, Jiajun Zhang, and Chengqing Zong. 2021. Csd: A fine-grained chinese dataset for customer service dialogue summarization.
Chunyi Liu, P. Wang, Jiang Xu, Zang Li, and Jieping Ye. 2019. Automatic dialogue summary generation for customer service. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.
Junpeng Liu, Yanyan Zou, Hainan Zhang, Hongshen Chen, Zhuoye Ding, Caixia Yuan, and Xiaojie Wang. 2021. Topic-aware contrastive learning for abstractive dialogue summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1229-1243, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730-3740, Hong Kong, China. Association for Computational Linguistics.
Zhengyuan Liu and Nancy Chen. 2021. Controllable neural dialogue summarization with personal named entity planning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 92-106, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.
Ahmed Magooda and Diane Litman. 2020. Abstractive summarization for low resource data using domain transfer and data synthesis.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarrunner: A recurrent neural network based sequence model for extractive summarization of documents. In *Thirty-First AAAI Conference on Artificial Intelligence*.
Xingchao Peng, Zijun Huang, Xineng Sun, and Kate Saenko. 2019. Domain agnostic learning with disentangled representations. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5102-5112. PMLR.
Timo Schick and Hinrich Schütze. 2021. Few-shot text generation with pattern-exploiting training.
Roy Schwartz, Jesse Dodge, Noah Smith, and Oren Etzioni. 2020. Green ai. Communications of the ACM, 63:54 - 63.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, page 6000-6010, Red Hook, NY, USA. Curran Associates Inc.
Danqing Wang, Pengfei Liu, Ming Zhong, Jie Fu, Xipeng Qiu, and Xuanjing Huang. 2019. Exploring domain shift in extractive text summarization.
Ziyi Yang, Chenguang Zhu, Robert Gmyr, Michael Zeng, Xuedong Huang, and Eric Darve. 2020. TED: A pretrained unsupervised summarization model with theme modeling and denoising. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 1865–1874, Online. Association for Computational Linguistics.
Tiezheng Yu, Zihan Liu, and Pascale Fung. 2021. AdaptSum: Towards low-resource domain adaptation for abstractive summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5892-5904, Online. Association for Computational Linguistics.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with ex-tracted gap-sentences for abstractive summarization. In In International Conference on Machine Learning, pages 11328-11339, Online.
Lulu Zhao, Weiran Xu, and Jun Guo. 2020. Improving abstractive dialogue summarization with graph structures and topic words. In Proceedings of the