SlowGuess/NAACL_2022_OCR huggingface dataset:
28th International Conference on Computational Linguistics, pages 437-449, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Lulu Zhao, Weihao Zeng, Weiran Xu, and Jun Guo. 2021a. Give the truth: Incorporate semantic slot into abstractive dialogue summarization. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 2435-2446, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Lulu Zhao, Fujia Zheng, Keqing He, Weihao Zeng, Yuejie Lei, Huixing Jiang, Wei Wu, Weiran Xu, Jun Guo, and Fanyu Meng. 2021b. Todsum: Task-oriented dialogue summarization with state tracking. arXiv preprint arXiv:2110.12680.
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. QMSum: A new benchmark for query-based multi-domain meeting summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5905-5921, Online. Association for Computational Linguistics.
Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. 2021. MediaSum: A large-scale media interview dataset for dialogue summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5927-5934, Online. Association for Computational Linguistics.
Yicheng Zou, Jun Lin, Lujun Zhao, Yangyang Kang, Zhuoren Jiang, Changlong Sun, Qi Zhang, Xuanjing Huang, and Xiaozhong Liu. 2021a. Unsupervised summarization for chat logs with topic-oriented ranking and context-aware auto-encoders. In AAAI.
Yicheng Zou, Lujun Zhao, Yangyang Kang, Jun Lin, Minlong Peng, Zhuoren Jiang, Changlong Sun, Qi Zhang, Xuanjing Huang, and Xiaozhong Liu. 2021b. Topic-oriented spoken dialogue summarization for customer service with saliency-aware topic modeling. In AAAI.
Yicheng Zou, Bolin Zhu, Xingwu Hu, Tao Gui, and Qi Zhang. 2021c. Low-resource dialogue summarization with domain-agnostic multi-source pretraining. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 80–91, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
We describe baselines in detail as follows.
Lead-3: This method is commonly used in news summarization task, which treats the first three sentences of the document as the summary.
Oracle: The method is used to obtain an oracle through a greedy way similar to Nallapati et al. (2017), which treats the sentences that maximize the ROUGE-2 as the summary.
BertExt: Proposed by Liu and Lapata (2019), this model is extractive and its parameters are initialized with BERT.
PGN: Proposed by See et al. (2017), this model adopts the pointer mechanism to deal with the issue of Out-Of-Vocabulary in the summary generation process.
Transformer: Proposed by Vaswani et al. (2017), this model captures long-distance information through the self-attention mechanism.
BertAbs: Proposed by Liu and Lapata (2019), this model is abstractive, which encoder is initialized with BERT and is trained with a different optimizer than decoder.
BART: Proposed by Lewis et al. (2020), the model is a state-of-the-art abstractive summarization model pre-trained with a denoising autoencoding objective.
BART w. DS/QR: Proposed by Zhao et al. (2021b), the model is a general summarization framework, with two encoders that share the underlying parameters and a decoder, which can fuse the input text and dialogue state/query.
Prefix-tuning: Proposed by Li and Liang (2021), this model introduces a prefix matrix on the basis of fixed pre-training BART parameters and allows the prefix matrix to learn task information through training, which optimizes the summarization performance in the small parameters and few-shot scenarios.
QMSum: This model is proposed by Zhong et al. (2021), which is a two-stage locate-thensummarize solution on query-based meeting summarization task.
A.2 Evaluation Metrics
We use the ROUGE (Lin, 2004; Lin and Och, 2004) $^6$  metrics to quantitatively evaluate the performance of our model. Rouge (Recall-Oriented Understudy for Gisting Evaluation) is metrics for evaluating summarization. It calculates by comparing the generated summary with references to
Models	Trainable Parameters
BertExt	120.51M
PGN	27.64M
Transformer	257M
BertAbs	180.22M
BART-large	400M
BART w. DS/QR	406M
Prefix-tuning	81.82M
DOP-tuning	61.52M
Table 9: Trainable parameter scales of different models, where "DS" means the dialogue state in TODSum and "QR" means the query in QMSum.
obtain the corresponding score to measure the similarity between them.
B Parameter Scale of Models
We show the amount of trainable parameters for our DOP model and other baseline models in Table 9. Among the full-parameter fine-tuning methods, except for the relatively simple PGN model, other models have reached the scale of hundreds of megabytes, which will take up a lot of time and space in model training and storage. Prefix-tuning and DOP-tuning greatly reduce the storage space of the model, improving the efficiency of the model. Compared with prefix-tuning, our method achieves better results with fewer parameters.
Figure 7 shows two examples from the TODSum and QMSum respectively. For example one of train domain in TODSum, BART w. DS generates some incorrect and redundant information related to the taxi domain and hotel domain. To make matters worse, for train domain, it loses the intent of the user about booking tickets and wrongly generates the key information, i.e., the departure location. The Prefix-tuning still confuses the knowledge of train and taxi domains, that is, the booking intent in the train domain is wrongly predicted as the user wants to know something about "cars". Moreover, the quality of the generated key information is not high, i.e., the wrong departure location "Seachage" and the missing time "Monday". For example two of academic domain in QMSum, both BART w. QR and Prefix-tuning predict too many details of the dialogue, which makes the summary redundant. Besides, Prefix-tuning generates the wrong speaker "Professor B", which leads to the summary being inconsistent.
Compared to the above two models, our method
$^{6}$ https://pypi.org/project/rouge/
solves some difficult issues in low-resource domain adaptation for dialogue summarization. By initializing prefix matrix with domain words, our model achieves domain disentanglement and the prediction of domain-related information is basically accurate. Through discrete prompts, our model has the ability to generalize to new domains and the accuracy of prediction about domain-independent key information is greatly improved.
D Domain Words
We present some domain words for each domain in Figure 8. In order to facilitate reading, we only show some of the domain words, that is, we select the first 20 words for each domain of TODSum, and the first 30 words for each field of QMSum as display.
For TODSum, we can see that there are relatively many common domain words, which are more concentrated on location words, or some information, such as "phone", "postcode" and so on. In addition, there are many common domain words that only appear in restaurant & hotel or train & taxi. For example, price-related descriptions are usually mentioned when booking a restaurant or hotel, and "destination", "depart", "from", "to" are usually mentioned when booking train tickets or taking a taxi. Special domain words can better distinguish different domains. There will be more special domain words in attraction, such as "entrance", "college", "nightclub", etc., which will not appear in other domains. Besides, users will mention the star rating when booking a hotel, and want to know the food type when booking a restaurant seat. When booking train tickets, they usually plan to travel, and when taking a taxi, they want to know the color of the car.
Compared with TODSum, QMSum has many more special field words, because the three fields contained in QMSum are more different. For product, participants will discuss various features of products such as TVs, LCDs, etc., such as screens, buttons, colors, and functions. For academic, participants generally discuss models, experimental data, or some errors. And for committee, participants generally discuss student education or national government issues. Common domain words have only some general words, such as "different", "system", etc., and only a few special common domain words exist in product & academic, such as "design", "bit" and other technology-related words.
Figure 7: Case study for two examples from TODSum and QMSum datasets. We present the dialogue, its corresponding dialogue state/query, ground truth, BART w. DS prediction, Prefix-tuning prediction, and prediction of our DOP model.
Figure 8: Domain words in each domain of two datasets. We use different signs to mark the special domain words, common domain words, and common domain words in specific domains separately.