Venue: NAACL 2022

Domain-Oriented Prefix-Tuning: Towards Efficient and Generalizable Fine-tuning for Zero-Shot Dialogue Summarization

Lulu Zhao\({}^{1*}\), Fujia Zheng\({}^{1}\), Weihao Zeng\({}^{1}\), Keqing He\({}^{2}\), Weiran Xu\({}^{1}\)+, Huixing Jiang\({}^{2}\), Wei Wu\({}^{2}\), Yanan Wu\({}^{1}\)

###### Abstract

The most advanced abstractive dialogue summarizes lack generalization ability on new domains and the existing researches for domain adaptation in summarization generally rely on large-scale pre-trainings. To explore the lightweight fine-tuning methods for domain adaptation of dialogue summarization, in this paper, we propose an efficient and generalizable **D**omain-**O**riented **P**reifx-tuning model, which utilizes a domain word initialized prefix module to alleviate domain entanglement and adopts discrete prompts to guide the model to focus on key contents of dialogues and enhance model generalization. We conduct zero-shot experiments and build domain adaptation benchmarks on two multi-domain dialogue summarization datasets, TODSum and QMSum. Adequate experiments and qualitative analysis prove the effectiveness of our methods.

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4848 - 4862

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4848 - 4862

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4848 - 4862

## 1 Introduction

Abstractive dialogue summarization task aims to distill the most critical information in a conversation to produce a concise version, involving chitch [35, 7], meeting[9], customer service [12, 13], and task-oriented dialogue scenarios [14]. Compared to the single-speaker texts, summarizing a dialogue presents a unique set of challenges, such as unstructured expressions and information sparsity issues. Recently, large-scale generative pre-trained models [16, 12] have promoted the development of abstractive dialogue summarization but they all require extensive human-annotated golden summaries, which makes them not scalable to new domains where only few/no labeled data is available. Considering that real-world applications often face the problem of data in the new domain, it is vital to develop low-resource dialogue summarization models for the target domain by leveraging limited annotated data of source domains.

Therefore, we try to explore the efficient domain adaptation of dialogue summarization models from the source domain \(D_{s}\) to the target domain \(D_{t}\), where \(D_{s}\) only has limited annotated summaries and \(D_{t}\) has few/no labeled data. There exist some domain adaptation approaches that focus on continual pre-trainings using some large domain/task-related corpora. Yu et al. yu2021transfer added multiple pre-training stages both on source domains and target domains. Further, Zou et al. zou2021transfer decomposed the pre-training into three procedures, i.e., the pre-training of encoder, decoder, and the combined encoder-decoder model. Fabbri et al. fabbri2021transfer constructed pseudo summaries based on external Wikipedia data to simulate characteristics of target dataset. Note that all these methods re

Figure 1: An example of TODSum dataset [14] with dialogue state and an example of QMSum dataset [9] with query. Note that dialogue state and query are existing characteristics of these two datasets, respectively.

 

quire time-consuming pre-trainings or large-scale external corpora. They only focus on the heavy pre-training stage rather than the lightweight fine-tuning, which makes it labor-expensive and environmentally unfriendly Schwartz et al. (2020) to practical applications.

Different from existing works that adopt general pre-trainings on the large-scale external corpora, we focus on exploring efficient fine-tuning methods specifically targeted at domain adaptation for the dialogue summarization task. We consider the following key principles while designing our methods: (1) **Efficiency**: We do not use any external data or pre-training and aim at leveraging efficient fine-tuning mechanisms based on existing summarization models. (2) **Domain Disentanglement**: Traditional models often memorize excessive knowledge from \(D_{s}\) and generate wrong summaries containing specific domain words in \(D_{s}\). We aim to disentangle shared domain knowledge from \(D_{s}\). (3) **Generalization**: Models often learn specific features of the source domain, making it difficult to generalize in a new domain Peng et al. (2019). For example, models learn the surface language style specific to \(D_{s}\) rather than adapting the way of saliency estimation and summary generation to \(D_{t}\). We encourage the summarizer to only focus on generic key contents rather than domain-specific attributes.

To be consistent with above principles, we propose a lightweight and efficient **D**omain-**O**riented **P**refix-tuning method, DOP, for domain adaptation of dialogue summarization. For efficiency, we focus on fine-tuning summarization models instead of performing pre-trainings like existing works, which reduces expensive and time-consuming computation. For domain disentanglement, we design a domain-oriented prefix module, which contains a novel prompt initialization mechanism. Concretely, we use domain words extracted by unsupervised LDA Hoffman et al. (2010) to initialize continuous prompt vectors and fit the outputs of MLP and pre-computed BART to obtain initial parameters and representations of the prefix module. We also add a domain-oriented prefix sequence of key-value pairs to augment the classical attention layer, which is independently applied to all Transformer layers of pre-trained models to elicit the knowledge interactively and achieve overall optimization. In this case, different domain words from \(D_{s}\) and \(D_{t}\) can induce relevant domain knowledge while adapting to a new domain. For generalization, we construct discrete prompts using dialogue states or queries, as shown in Figure 1, to guide the model to focus on key contents in dialogues and enhance generalization capability on unseen domains. Considering there is no unified and practical benchmark for domain adaptation of dialogue summarization 1, we build domain adaptation benchmarks based on two existing multi-domain summarization datasets TODSum Zhao et al. (2021) and QMSum Zhong et al. (2021). Extensive experiments demonstrate the benefits of our methods both in zero-shot and few-shot settings for domain adaptation.

Footnote 1: Existing work mostly takes news datasets like CNN/DailyMail Hermann et al. (2015) as source domains and dialogue datasets like SAMSum Gliwa et al. (2019) as target domains. We argue the setting doesn’t fit in with industrial scenarios transferring from one business to another, like from attraction consultation to hotel reservation.

Our contributions are threefold: (1) To the best of our knowledge, we are the first to explore fine-tuning methods for domain adaptation of dialogue summarization task, and establish two practical and comprehensive benchmarks for TODSum and QMSum datasets. (2) We propose a lightweight and efficient Domain-Oriented Prefix-tuning model, with domain word initialized prefix and discrete prompts, to elicit knowledge from large-scale pretrained models interactively. (3) We conduct sufficient experiments and qualitative analysis to prove the effectiveness of our methods and discuss current challenges of domain adaptation for dialogue summarization.

## 2 Related Work

**Abstractive Dialogue Summarization** Dialogue summarization has drawn much attention recently. For chit-chat scenarios, researchers improved the performance on SAMSum dataset Gliwa et al. (2019) via topic word information Zhao et al. (2020); Liu et al. (2021), conversational structures Chen and Yang (2020, 2021), personal named entity planning Liu and Chen (2021), and semantic slots Zhao et al. (2021). Liu et al. (2019), Zou et al. (2021, 2021), and Lin et al. (2021) proposed customer-service dialogue summarization datasets under diverse business scenarios. Besides, meeting transcripts, such as AMI Carletta et al. (2005), ICSI Janin et al. (2003), MediaSumm Zhu et al. (2021), and QMSum Zhong et al. (2021), were also studied to promote dialogue summarization technologies. Zhao et al. (2021) further proposed a task-oriented dialogue summarization 

tion dataset ,TODSum, with dialogue state knowledge. Although great progress has been made in dialogue summarization, few people pay attention to the issue of domain adaptation in dialogue summarization. In this paper, we explore this issue in two multi-domain dialogue summarization datasets, i,e, TODSum and QMSum.

**Domain Adaptation in Summarization**Hua and Wang (2017) and Wang et al. (2019) adopted the document categories in news publications to build a multi-domain summarization dataset and investigated the domain shift for extractive summarization task. Yang et al. (2020), Zhang et al. (2020), Magooda and Litman (2020), and Fabbri et al. (2021) regarded diverse summarization datasets as different domains and conducted the assessment of multi-domain settings. Furthermore, various stages of pre-trainings were added to narrow the gap between the pre-training in news domain and the fine-tuning in dialogue domain Yu et al. (2021); Zou et al. (2021). However, these methods focus on the heavy pre-training stage rather than the lightweight fine-tuning, which is time-consuming and relies on large-scale external corpora. Therefore, we try to explore the fine-tuning methods for domain adaptation of dialogue summarization task.

**Prompts in Summarization** With the arrival of GPT-3, prompt learning has become a nascent field, which conducts task-specific adaptation of large language models (LMs) via prepending an instruction. Schick and Schutze (2021) explored the fixed-prompt LM tuning for few-shot text summarization with manually crafted templates. Zhao et al. (2021) and Dou et al. (2021) further adopted the prompt+LM tuning strategy on text summarization task, where learnable prefix prompts are different types of guidance signals. Li and Liang (2021) investigated fixed-LM prompt tuning, where learnable prefix tokens are prepended to the input while parameters in pre-trained models are frozen. Following Li and Liang (2021), we design the domain information to initialize the continuous prefix module, and use discrete prompts and dialogue texts to optimize prefix parameters, which greatly reduces the size of parameters and is suitable for low-resource scenarios.

## 3 Problem Formulation

Domain adaptation of dialogue summarization aims to generate the summary \({\bf y}\) conditioned on the source dialogue \({\bf x}_{d}\), where the training and test are in different domains. We add the domain word prefix \({\bf x}_{dw}\) and discrete prompt \({\bf x}_{dp}\) as additional input which we will describe details later. Note that we only update the prefix-related parameters and fix the parameters of BART. We train the model using the source data and test using the target data.

## 4 Methodology

As Figure 2 shows, our model is on the basis of the framework of BART, including a domain-oriented prefix module, a prompt encoder, and a decoder.

### Domain-oriented Prefix

To alleviate domain entanglement, we present a domain-oriented prefix module to obtain the shared

Figure 2: Overview of the Domain-Oriented Prefix-tuning model. The input sequence of domain-oriented prefix module includes domain words from four source domains, i.e., _attraction, hotel, taxi, train_ and the domain words of _restaurant_ domain are used as the prefix sequence of the target domain during the test. The input sequence of prompt encoder is the dialogue state from TODSum dataset and its original dialogue text. The input sequence of decoder is the golden summary. Here, we use the pre-computed outputs of BART as the target of the fitting process to initialize the parameters of MLP. Note that the grey pre-computed BART module does not participate in the training of the model.

 

knowledge of the source domain \(D_{s}\) and the target domain \(D_{t}\). It is designed as follows:

**Initialization** We extract some keywords from dialogue texts in each domain by LDA Hoffman et al. (2010) and concatenate them all together as a domain word (prefix) sequence \(\mathbf{x}_{dw}\)2. Randomly initialized embeddings of the domain word sequence compose a learnable matrix \(M_{\theta}\in\mathbb{R}^{|\mathbf{x}_{deu}|\times d_{m}}\).

Footnote 2: We represent some of domain words for each domain in Appendix D.

ParametrizationWe use an MLP to encode the domain-oriented prefix module, which stably elicits knowledge from the large pre-trained model in the prefix-tuning process. Specifically, we first input the domain word sequence to the MLP and the pre-computed BART respectively, then re-train the MLP by fitting its outputs with the decoder hidden states of the pre-computed BART using MSE loss. In this fitting process, we only iteratively update MLP parameters \(\varphi\in\mathcal{R}^{d_{m}\times d_{n}}\) and keep the pre-computed BART fixed. Finally, we get the initialization parameters of MLP and use this pretrained MLP to map the initialized embeddings of prefix representations for each Transformer layer both in prompt encoder and decoder:

\[M_{\theta}^{{}^{\prime}}[i,:]=\mathrm{MLP}_{\varphi}(M_{\theta}[i,:]) \tag{1}\]

where \(i\in\mathbf{x}_{dw}\) and \(M_{\theta}^{{}^{\prime}}\in\mathbb{R}^{|\mathbf{X}_{dw}|\times d_{n}}\). Note that this continuous prefix is applied for every layer of the large-scale pre-trained model independently.

### Prompt Encoder

Discrete PromptsWe utilize some discrete prompts to emphasize key elements in dialogues and enhance the model generalization to new domains. Here, discrete prompts are dialogue states of TODSum dataset or queries of QMSum. Considering that the original form of dialogue states is _book (people=5; day= Monday)_ which is not compatible with BART encoder, we convert this structured information into a serialized sequence, i.e., _book, people is 5, day is Monday_, to improve the stability of training. Note that we do not make any changes to the query of QMSum dataset because it is already a serialized representation.

For prompt encoder, we firstly concatenate the discrete prompt sequence \(\mathbf{x}_{dp}\) and dialogue text sequence \(\mathbf{x}_{d}\) as the input sequence of encoder, i.e., \(\mathbf{x}_{enc}=[\mathbf{x}_{dp};\mathbf{x}_{d}]\). Then, the \(\mathbf{x}_{enc}\) is fed into the prompt encoder based on the BART encoder, containing multiple Transformer layers. Note that we modify the self-attention mechanism by adding a domain-oriented prefix sequence of key-value pairs, which learns the knowledge from the pre-trained model through interactions with the dialogue text to carry out the overall task. For the typical \(l_{e}\)-th Transformer layer in encoder, the query (\(\mathbf{Q}_{lc}\)), key (\(\mathbf{K}_{l_{e}}\)), and value (\(\mathbf{V}_{l_{e}}\)) matrices are computed through linear transformations on the hidden states of \(\mathbf{x}_{enc}\). Here, we further augment the \(\mathbf{K}_{l_{e}}\) and \(\mathbf{V}_{l_{e}}\):

\[\mathbf{K}_{l_{e}}^{{}^{\prime}}=[\mathbf{P}_{l_{e}k};\mathbf{K}_{l_{e}}],~{} \mathbf{V}_{l_{e}}^{{}^{\prime}}=[\mathbf{P}_{l_{e}v};\mathbf{V}_{l_{e}}] \tag{2}\]

where \(\mathbf{P}_{l_{e}k}\), \(\mathbf{P}_{l_{e}v}\) are computed through linear transformations on \(M_{\theta}^{{}^{\prime}}\). \(\mathbf{K}_{l_{e}}^{{}^{\prime}},\mathbf{V}_{l_{e}}^{{}^{\prime}}\in\mathbb{ R}^{(|\mathbf{x}_{dw}|+|\mathbf{x}_{dp}|+|\mathbf{x}_{d}|)\times d_{n}}\) and \(l_{e}\)\(\in L\). The augmented self-attention layer is finally calculated as follows:

\[\mathbf{A}_{self}=\operatorname*{softmax}(\mathbf{Q}_{l_{e}}\mathbf{K}_{l_{ e}}^{{}^{\prime}T})\mathbf{V}_{l_{e}}^{{}^{\prime}} \tag{3}\]

### Decoder

We also prepend the prefix module for decoder, where the cross-attention and masked-attention mechanisms are augmented in a similar way. The cross-attention between the \(l_{e}\)-th layer of prompt encoder and the \(l_{d}\)-th layer of the decoder is designed as:

\[\mathbf{A}_{cross}=\operatorname*{softmax}(\mathbf{Q}_{l_{d}}\mathbf{K}_{l_{ e}}^{{}^{\prime}T})\mathbf{V}_{l_{e}}^{{}^{\prime}} \tag{4}\]

where \(\mathbf{Q}_{l_{d}}\) is computed through a linear transformation on the hidden states of the summary text \(\mathbf{x}_{s}\) and \(l_{d}\)\(\in L\). Besides, the implementation of masked-attention layer is the same as the self-attention layer in the prompt encoder.

### Training Strategy

In the domain-oriented prefix module, the parameter set of all linear transformations is symbolized as \(\alpha\). For training strategy in DOP, we perform gradient updates on the following log-likelihood objective:

\[\max_{\alpha,\theta,\varphi}\log p_{\alpha,\theta,\varphi,\phi}(\mathbf{y}| \mathbf{x})=\sum_{i\in\mathbf{y}}\log p_{\alpha,\theta,\varphi,\phi}(y_{i}| y_{<i}) \tag{5}\]

where the BART parameters \(\phi\) are fixed. The prefix parameters \(\alpha\), \(\theta\), and \(\varphi\) are the only trainable parameters. During the training, we use the domain words from source domains as the prefix sequence. When the training is completed, we save all parameters of the domain-oriented prefix module and 

drop the pre-computed BART module. During the test, the domain words from the target domain are mapped to the representations of prefixes only via the reserved \({\rm MLP}_{\varphi}(\cdot)\), where the features of the source domain \(D_{s}\) can be transferred to the target domain \(D_{t}\).

## 5 Experimental Setup

### Datasets

We evaluate our model on two multi-domain dialogue summarization datasets and the details of statistics are shown in Table 1 and Table 2:

**TODSum**: This dataset is proposed by Zhao et al. (2021b), which is a task-oriented dialogue summarization dataset based on the classic dialogue dataset MultiWOZ Budzianowski et al. (2018). According to the domain information, the dataset can be divided into five domains: _restaurant_, _hotel_, _attraction_, _taxi_, and _train_. Considering that parts of dialogues in TODSum contain multiple domains, in this paper, we firstly select all single-domain dialogues from TODSum as the dataset used for this experiment. Then we integrate all these samples in any four of five domains as source domains \(D_{s}\) and the other one is regarded as target domain \(D_{t}\), where 200 samples extracted from \(D_{s}\) are as the validation set, the remaining as the training set, and samples of \(D_{t}\) are the test set.

**QMSum**: This dataset is proposed by Zhong et al. (2021), which contains hundreds of meeting transcriptions and includes three domains, i.e., _academic_, _committee_, _product_. In addition, each sample can be divided into several dialogue fragments according to some queries and the answer for the corresponding query is its golden summary. Such _(query-dialogue-answer)_ pairs are usually used for query-based meeting summarization tasks. In this paper, we separately integrate the training and validation sets, and test set in any two of three domains as the training data and validation data, i.e., the data of source domain \(D_{s}\). All data of the other domain is used as the test data, i.e., the data of target domain \(D_{t}\).

### Baselines and Evaluation Metrics

We compare our methods with several baselines. The extractive baselines are included: (1) Lead-3; (2) Oracle; (3) BertExt Liu and Lapata (2019). Some abstractive methods are also added for comparison: (1) PGN See et al. (2017); (2) Transformer Vaswani et al. (2017); (3) BertAbs Liu and Lapata (2019); (4) BART Lewis et al. (2020); (5) BART w. DS/QR Zhao et al. (2021b); (6) Prefixtuning Li and Liang (2021). For QMSum, we also introduce its benchmark Zhong et al. (2021). Since this method feeds the extracted spans into BART, we integrate the results of this method with the results of BART. We use ROUGEs Lin (2004); Lin and Och (2004) to quantitatively evaluate the performance of our methods. Our codes are publicly available 3. We give the baselines and evaluation metrics in Appendix A.1 and Appendix A.2.

Footnote 3: [https://github.com/Zeng-WH/DOP-Tuning](https://github.com/Zeng-WH/DOP-Tuning).

### Training Details

Our implementation is based on the Hugging Face Transformer models4. \({\rm BART}_{\rm LARGE}\) is used as a backbone and the source dialogue sequence is truncated to 1024 BPE tokens. For domain-oriented prefix module, the MLP maps 1024 dimension into 24576 dimension, which is calculated by \(2*{\rm number~{}of~{}decoder~{}layers}*1024\) and the numbers of domain words (prefix length) are set to 140 and 200 for TODSUM and QMSum datasets. Following the settings in Li and Liang (2021), we use an AdamW optimizer Loshchilov and Hutter (2019) and a linear learning rate scheduler with initial rate of \(5\cdot 10^{-5}\), and the batch size is set to 5. Our model is trained on RTX 2080 Ti machines, taking only 5 minutes per epoch on TODSum dataset and 3 min per epoch on QSum dataset. However, BART w. DS takes 13 minutes and 8 minutes per epoch on TODSum and QMSum datasets. The reason for the shorter training time o

\begin{table}
\begin{tabular}{l|c c|c|c} \hline
**Domains** & **Size** & **Dialog_len** & **Summ_len** & **DS len** \\ \hline
**Train** & 345 & 120.67 & 24.93 & 18.29 \\
**Taxi** & 435 & 80.24 & 29.04 & 15.80 \\
**Restaurant** & 1,311 & 105.42 & 23.04 & 14.30 \\
**Hotel** & 636 & 145.16 & 30.06 & 21.38 \\
**Attraction** & 150 & 95.48 & 22.27 & 7.92 \\
**All** & 2,877 & 111.71 & 25.68 & 16.24 \\ \hline \end{tabular}
\end{table}
Table 1: Details of TODSum. “Diàlog.len” denotes the average length of dialogues, ”Summ.len” denotes the average length of summaries, and “DS.len” denotes the average length of serialized dialogue states.

\begin{table}
\begin{tabular}{l|c|c|c|c} \hline
**Domains** & **Size** & **Dialog_len** & **Summ_len** & **QR_len** \\ \hline
**Academic** & 312 & 1,155.78 & 46.48 & 8.56 \\
**Committee** & 417 & 757.68 & 76.00 & 14.54 \\
**Product** & 847 & 971.65 & 63.96 & 13.36 \\
**All** & 1,576 & 951.49 & 63.68 & 12.73 \\ \hline \end{tabular}
\end{table}
Table 2: Details of QMSum. “QR.len” denotes the average length of queries.

 

trainable number of parameters of the DOP model is only 15% of the BART w. DS. For all experiments, we set the number of training epochs to 30. At the decoding phase, we use a beam size of 6 and max generated tokens of 125. The decoding takes 1.3 seconds per batch with the size of 2.

### Main Results

Results on TODSum Table 3 presents the results of the zero-shot setting for TODSum dataset, where each of the five domains is regarded as the target domain respectively. The division of dataset in the second row intuitively shows that the amount of data in \(D_{s}\) is small and limited. We conduct experiments based on some common extractive models and some strong abstractive baselines. We also add a lightweight fine-tuning summarizer for comparison. As observed, for most ROUGEs, Prefix-tuning performs worse than BART and BART w. DS. It is because the dialogue text is long and complex, and using only 20% parameters of fine-tuning can not well understand the domain knowledge and identify the key contents in dialogues. Compared to Prefix-tuning of the same magnitude parameters, our model improves by 7%, 3%, 7% for _train_ domain, 5%, 5%, 3% for _taxi_ domain, 4%, 8%, 4% for _restaurant_ domain, 5%, 6%, 5% for _hotel_ domain, and 8%, 8%, 9% for _attraction_ domain. This shows that the prefix module initialized by domain words and the discrete prompts composed of dialogue states play important roles. Besides, our model still surpasses BART w. DS, a full-parameter fine-tuning based model, which further illustrates that our model efficiently disentangles the knowledge of the source and target domains. Note that _attraction_ domain gets the highest ROUGEs and the increased margins are also the largest. This may be due to the high overlaps between the _attraction_ and source domains. All the results suggest that with limited data, the performance of our model still reaches state-of-the-art.

Results on QMSumTable 4 displays the results on zero-shot out-domain tests in three domains of QMSum dataset. As seen from the second row, in addition to the limited data, the source domain size may be even less than the target domain size, i.e., _product_ domain. The trend of overall performance is consistent with that of TODSum dataset 5, where the improvement in _product_ domain is the most obvious and there are 5%, 4% and 5% increased for R-1, R-2, and R-L, respectively. However, all the ROUGEs are low, which is because there are no obvious domain words, leading to serious domain entanglement. Besides, due to the longer meeting text, it is hard to capture the key contents in dialogues, so as to the poor generalization in the target domain. Generally, these results show that the multi-domain setting in meeting summarization task is apparently necessary and meaningful. Meeting transcripts cover various domains, making the

\begin{table}
\begin{tabular}{l|c c|c c|c c|c c|c c} \hline  & \multicolumn{2}{c|}{**Academic**} & \multicolumn{2}{c|}{**Committee**} & \multicolumn{2}{c}{**Product**} \\ \cline{2-11}
**Models** & \multicolumn{2}{c|}{**1.00/ 195 / 312**} & \multicolumn{2}{c|}{**981 / 178 / 47**} & \multicolumn{2}{c}{**641 / 115 / 847**} & \multicolumn{2}{c}{**641 / 115 / 847**} \\ \cline{2-11}  & **R-1** & **B-2** & **R-1** & **R-1** & **R-2** & **R-1** & **R-1** & **R-2** & **R-1** \\ \hline
* 15.55 &

* 15.21 &
* 17.51 &

**Prefix-tuning** & 21.90 & 40.1 & 18.71 & 27.70 & 37.35 & 26.52 & 25.77 & 27.26 & 20.66 \\
**DOF** & **24.53** & **5.84** & **21.49** & **30.28** & **11.63** & **27.33** & **29.85** & **9.41** & **26.88** \\ \hline \end{tabular}
\end{table}
Table 4: ROUGE scores of the zero-shot setting for QMSumm. “QR” denotes the queries. Same as TODSum, values in the second row denote the size of train/valid/test set. All results are averaged over three random runs. (\(p<0.05\) under t-test)

\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c c} \hline \multirow{2}{*}{**Models**} & \multicolumn{2}{c|}{**Train**} & \multicolumn{2}{c|}{**TASI**} & \multicolumn{2}{c|}{**Restaurant**} & \multicolumn{2}{c|}{**Hotel**} & \multicolumn{2}{c}{**Attraction**} \\ \cline{2-11}  & **2,332 / 200 / 345** & **2,242 / 200 / 435** & **1,366 / 200 / 1,311** & **2,041 / 200 / 636** & **2,527 / 200 / 150** \\ \cline{2-11}  & **R-1** & **R-2** & **R-1** & **R-1** & **R-2** & **R-1** & **R-2** & **R-1** & **R-2** & **R-1** & **R-2** & **R-1** & **R-2** & **R-L** \\ \hline
**Lead-3** & 20.36 & 2.78 & 16.07 & 24.20 & 7.34 & 20.75 & 28.27 & 6.10 & 23.49 & 23.86 & 4.58 & 18.80 & 22.76 & 5.28 & 19.66 \\
**Oracle** & 39.06 & 10.04 & 32.87 & 38.96 & 14.06 & 33.43 & 45.79 & 15.57 & 38.42 & 39.65 & 11.28 & 32.56 & 41.90 & 14.18 & 38.79 \\
**BertExt** & 39.19 & 9.71 & 33.24 & 38.49 & 13.57 & 33.36 & 40.64 & 12.34 & 34.43 & 35.96 & 9.71 & 30.10 & 36.25 & 11.19 & 31.41 \\ \hline
**PGN** & 32.50 & 14.07 & 29.33 & 32.48 & 7.79 & 29.82 & 33.63 & 10.78 & 31.47 & 32.18 & 9.36 & 30.93 & 32.66 & 9.95 & 30.29 \\
**Transformer** & 33.47 & 19.08 & 30.28 & 33.35 & 8.71 & 30.57 & 34.49 & 11.43 & 31.99 & 33.05 & 10.62 & 31.63 & 33.18 & 10.74 & 30.91 \\
**BertAbs** & 42.89 & 16.57 & 37.32 & 36.43 & 14.69 & 23.15 & 42.10 & 18.61 & 38.87 & 38.03 & 13.34 & 33.22 & 32.61 & 14.81 & 34.67 \\
**BART** & 46.82 & 18.42 & 42.06 & 39.98 & 15.79 & 34.41 & 47.02 & 22.62 & 44.93 & 40.84 & 14.20 & 36.83 & 43.67 & 20.23 & 41.44 \\
**BART w. DS** & 49.02 & 23.80 & 44.59 & 43.59 & 19.56 & 38.65 & 49.25 & 23.57 & 45.23 & 43.97 & 17.02 & 39.31 & 47.55 & 22.62 & 45.16 \\
**Prefix-tuning** & 45.92 & 22.70 & 41.06 & 41.89 & 19.47 & 39.62 & 47.19 & 24.20 & 42.99 & 43.41 & 18.75 & 36.75 & 44.48 & 22.43 & 40 

adaptation of models particularly difficult.

## 6 Qualitative Analysis

### Effect of Domain Words

**Number of Domain Words** We set different numbers of domain words, i.e., prefix length, to test the performance of our model DOP for _train_ domain in TODSum dataset. As shown in Figure 3 (a), among these setting candidates, there is a threshold (140) that allows the ROUGEs to reach the peak. When choosing a fewer setting, the model does not perform well due to insufficient number of parameters, which is improved as the number increases. When being more than this threshold, a drop in performance occurs. One reason is that too long a sequence adds a large burden to BART, and the other one is that it introduces excessive noise. However, the change in the number of the domain words does not have a great impact on the performance (only 2\(\sim\)3% fluctuation), which also reflects the effectiveness of domain-oriented prefix module and the robustness of our model.

**Quality of Domain Words** For _train_ domain in TODSum, we randomly replace a certain percentage of the domain words with words that are not related to the source domain. As Figure 3 (b) shows, when more noise is introduced, the model suffers more interference and its performance decreases. However, it performs better than Prefixtuning. Only when the proportion of noise reaches 100%, the performance of our model is even worse than that of Prefix-tuning. This is because we especially use completely irrelevant words for initialization and fitting, which introduces more noise than simple random initialization and affects the performance of DOP. From this point of view, introducing high-quality domain words is good for domain disentanglement and the quality of domain words is critical to summary generation.

### Ablation Study

We perform ablation experiments on _train_ domain of TODsum dataset and _committee_ domain of QMSum dataset, as shown in Tables 5 and 6. We can observe that the removal of domain-oriented initialization in the prefix module will make the ROUGEs decrease significantly. Especially for TODSum, R-1, R-2, and R-L drop by 4%, 2%, and 3%, which shows the importance of domain word information for inducing the relevant knowledge while adapting to a new domain. In addition, after we remove the discrete prompts, i.e. dialogue state and query, the performance of the models becomes worse, but still outperforms the results of Prefix-tuning. It demonstrates that discrete prompts help the model pay attention to the key elements in the dialogue and improve the generalization of the model. Notably, our model achieves summary generation only by optimizing the domain-oriented prefix module, where domain words are available in all datasets. Since the DS and QR features happen to exist in the two datasets, we take advantage of them together with dialogue texts. When removing both DW and DS/QR at the same time, the model is equivalent to Prefix-tuning and the results are consistent.

### Effect of Prefix Module in Encoder and Decoder

Since both the encoder and the decoder in our DOP introduce the prefix module, we verify their effects in the _train_ and _committee_ domains respectively.

\begin{table}
\begin{tabular}{l|c c c} \hline
**Model** & **R-1** & **R-2** & **R-L** \\ \hline
**DOP (ours)** & 30.28 & 11.63 & 27.33 \\
**w/o DW** & 29.10 & 10.59 & 26.53 \\
**w/o QR** & 28.47 & 9.60 & 26.38 \\
**w/o DW \& QR** & 27.70 & 9.37 & 25.62 \\ \hline \end{tabular}
\end{table}
Table 6: F1 scores of ablation study on _committee_ domain of QMSum dataset. “DW” denotes domain words and “QR” denotes queries.

Figure 3: Effect of different numbers and different qualities of domain words for _train_ domain in TODSum. The dotted lines in (b) respectively represent the F1 sources of ROUGE-1, ROUGE-2, and ROUGE-L of the Prefix-tuning model.

 

As shown in Table 7, when the encoder prefix module or decoder prefix module is removed, the performance of the model decreases, which shows that both are necessary and effective. In addition, we find that it is interesting that removing the prefix on the encoder side has a smaller impact on the model than removing the decoder side, especially in TODSum (about 5% on R-1). A reasonable explanation is that the prefix modules in encoder and decoder are responsible for different tasks. The prefix module on the encoder side assists the model to understand the dialogue, while the prefix module on the decoder side assists in model generation. Therefore, for summary generation, the prefix module in decoder is more helpful to the model.

### Effect of Training Data

**Performance in Few-shot Settings** For TODSum, we fix the size of source domain data, adding a certain amount of target (_train_) domain data for training, as shown in Figure 4. As the size of target domain data increases, the performance of both BART w. DS and DOP present a steady improvement trend and that of our DOP model is consistently better than BART w. DS, which is as expected. Besides, there is a substantial improvement from 50 to 100. This phenomenon shows that adding target knowledge can help the model learn about information of target domain and after adding a certain amount will help the model more efficiently.

**Effect of Source Domain Data Size** We keep the zero-shot setting unchanged and adjust the size of source domain data for training to observe changes in the performance of the two models for _train_ domain in TODSum. As shown in Figure 5, the smaller of data size, the greater the difference between the performance of the DOP and BART w. DS, that is, the performance of BART w. DS is getting worse, while the DOP maintains excellent performance steadily. This demonstrates that our DOP model is insensitive to the data scale and robustness to a certain extent. This also confirms that in the main experiment, our model can be outstanding in very limited and uneven data.

### Prefix Length vs. Input Length

Through experiments, we explore an interesting thing, that is, the prefix length (number of domain words) that makes the model perform best may be related to the input length. Based on this assumption, we collect the source input length, target input length, and their corresponding optimal prefix length from two datasets, as shown in Figure 6. We conclude a general rule that the longer inputs may prefer the shorter prefix. This phenomenon may serve as a research point in the future.

## 7 Discussion

### Human Evaluation

We further conduct a manual evaluation to assess the models. We randomly select 50 samples from

\begin{table}
\begin{tabular}{c|l|r r r r} \hline
**Domain** & **Model** & **R-1** & **R-2** & **R-L** \\ \hline  & **DOP** & 52.51 & 25.45 & 47.14 \\
**Train** & **w/o enc.prefix** & 50.69 & 23.58 & 45.98 \\  & **w/o dec.prefix** & 45.51 & 22.15 & 40.67 \\ \hline  & **DOP** & 30.28 & 11.63 & 27.33 \\
**Committee** & **w/o enc.prefix** & 29.20 & 11.34 & 26.37 \\  & **w/o dec.prefix** & 29.11 & 11.29 & 26.30 \\ \hline \end{tabular}
\end{table}
Table 7: Effects of prefix module in encoder and decoder. We removed the prefix module from the encoder and decoder respectively to verify its effectiveness.

Figure 4: Effect of different few-shot data size for _train_ domain in TODSum dataset. ”0” denotes the zero-shot setting.

Figure 5: Effect of different source domain data size for _train_ domain in TODSum dataset. ”origin (2332)” denotes the same setting as the main results.

Figure 6: Relationships between length of encoder/decoder input and the best prefix length.

 

each target domain dataset of TODSum Zhao et al. (2021) and ask 10 professional linguistic evaluators to score the ground truth and summaries generated by 4 models according to 5 metrics: fluency, informativeness, factual consistency, domain relevance, and redundancy. Each metric is rated by 3 evaluators from 1 (worst) to 5 (best) and the scores for each summary are averaged. Note that the intra-class agreement score is 0.605.

As shown in Table 8, the fluency scores of all models are high, which is because that abstractive models fine-tuned on contextualized language backbones can generate fluent sentences Lewis et al. (2020). For factual consistency, both DOP and BART w. DS achieve better performance than Prefix-tuning, which suggests that the dialogue state information guides the model to focus more on the key information, such as slots and intents. Besides, the DOP outperforms all baselines in the domain relevance metric. This demonstrates that the domain-oriented prefix module plays a crucial role in enhancing the ability of the model to identify domain-related features and disentangle the knowledge of the source and target domains. Surprisingly, the scores about the redundancy of Prefix-tuning and DOP are higher than that of BART and BART w. DS. This is because the model can efficiently extract key contents from a limited amount of data without relying on large-scale pre-trainings.

## 8 Challenges

Through the analysis of cases in Appendix C, we summarize two challenges of low-resource domain adaption for abstractive dialogue summarization task:

**1. Confusion between domains with high similarity:** We found that in domains with high-overlap vocabularies, i.e., _restaurant_ and _hotel_, _train_ and _taxi_, the model generates some domain-confusing sentences. Taken _hotel-restaurant_ pair as an example, when _restaurant_ is as the target domain, a sentence like "book a restaurant room that can accommodate 3 people, ..." is generated, which is more likely to exist in the _hotel_ domain. Note that this challenge does not affect the accuracy of key elements, but the language style is not appropriate.

**2. Information dispersion:** Because of the long input sequence, it is difficult for the models to pay attention to all aspects of the long dialogue and there will be problems with attention deviations on the key elements of dialogues, especially for this lightweight and small parameter training paradigm.

## 9 Conclusion

In this paper, we present a domain-oriented prefix-tuning model to handle the domain adaptation for dialogue summarization based on an efficient and generalizable fine-tuning method. The domain word initialized prefix module disentangles the target domain knowledge from the source domain and the discrete prompts enhance the generalization ability of the model. The experiments in zero-shot and few-shot settings show that our methods have made great progress on two datasets.

### Acknowledgement

We thank all anonymous reviewers. This work was partially supported by National Key R&D Program of China No. 2019YFF0303300 and Subject II No. 2019YFF0303302, DOCOMO Beijing Communications Laboratories Co., Ltd, MoE-CMCC "Artificial Intelligence" Project No. MCM20190701, BUPT Excellent Ph.D. Students Foundation CX2021204.

## References

* a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
* Carletta et al. (2005) J. Carletta, S. Ashby, S. Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, J. Kadlec, Vasilis Karasikos, Wessel Kraaij, Melissa Kronenthal, G. Lathoud, M. Lincoln, Agnes Lisowska Masson, I. McCowan, W. Post, D. Reidsma, and P. Wellner. 2005. The ami meeting corpus: A pre-announcement. In _MLMI_.
* Chen and Yang (2020) Jiaao Chen and Diyi Yang. 2020. Multi-view sequence-to-sequence models with conversational structure for

\begin{table}
\begin{tabular}{l|c c c c c} \hline
**Model** & **Flu.** & **Inf.** & **Fac.** & **Dom.** & **Red.** \\ \hline
**Ground Truth** & 4.95 & 4.56 & 4.28 & 4.71 & 4.33 \\
**BART** & 4.19 & 4.21 & 3.55 & 3.19 & 3.53 \\
**BART w. DS** & 4.36 & 4.34 & 4.09 & 3.35 & 3.62 \\
**Prefix-tuning** & 4.23 & 4.29 & 3.67 & 3.28 & 4.09 \\
**DOP** & 4.68 & 4.42 & 4.10 & 4.07 & 4.13 \\ \hline \end{tabular}
\end{table}
Table 8: Human evaluation on Fluency (Flu.), Informativeness (Inf.), Factual Consistency (Fac.), Domain Relevance (Dom.), and Redundancy (Red.) for TODSum datastet.

 

abstractive dialogue summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 4106-4118. External Links: Link Cited by: SS3.
* J. Chen and D. Yang (2021)Structure-aware abstractive conversation summarization via discourse and action graphs. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, pp. . External Links: Document Cited by: SS5.
* Z. Dou, P. Liu, H. Hayashi, Z. Jiang, and G. Neubig (2021)GSum: a general framework for guided neural abstractive summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, pp. 4830-4842. External Links: Link, Document Cited by: SS1, SS2.
* A. Fabbri, S. Han, H. Li, H. Li, M. Ghazvininejad, S. Joty, D. Radev, and Y. Mehdad (2021)Improving zero and few-shot abstractive summarization with intermediate fine-tuning and data augmentation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, pp. 704-717. External Links: Link, Document Cited by: SS1, SS2.
* B. Gliwa, I. Mochol, M. Biesek, and A. Wawer (2019)SAMSum corpus: a human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, Hong Kong, China, pp. 70-79. External Links: Link, Document Cited by: SS1, SS2.
* K. Moritz Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom (2015)Teaching machines to read and comprehend. In NIPS, Cited by: SS1, SS2.
* M. Hoffman, F. R. Bach, and D. M. Blei (2010)Online learning for latent dirichlet allocation. In Advances in Neural Information Processing Systems 23, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta (Eds.), Vol. , pp. 856-864. External Links: Link, Document Cited by: SS1, SS2.
* X. Hua and L. Wang (2017)A pilot study of domain adaptation effect for neural abstractive summarization. In Proceedings of the Workshop on New Frontiers in Summarization, Copenhagen, Denmark, pp. 100-106. External Links: Link, Document Cited by: SS1, SS2.
* A. Janin, D. Baron, J. Edwards, D. Ellis, D. Gelbart, N. Morgan, B. Peskin, T. Pfau, E. Shriberg, A. Stolcke, and C. Wooters (2003)The isci meeting corpus. In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03), Vol. 1, pp. I-I. External Links: ISBN 978-1-4503-3213-1, Link, Document Cited by: SS1, SS2.
* M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer (2020)BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL, Cited by: SS1, SS2.
* Xiang Lisa Li and Percry Liang (2021)Prefix-tuning: optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190. Cited by: SS1, SS2.
* Chin-Yew Lin (2004)Rouge: a package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81. External Links: Link, Document Cited by: SS1, SS2.
* C. Lin and F. Josef Och (2004)Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pp. 605-612. External Links: Link, Document Cited by: SS1, SS2.
* Haitao Lin, L. Ma, J. Zhu, L. Xiang, Y. Zhou, J. Zhang, and C. Zong (2021)CsdS: a fine-grained chinese dataset for customer service dialogue summarization. External Links: Link, Document Cited by: SS1, SS2.
* Chunyi Liu, P. Wang, J. Xu, Z. Li, and J. Ye (2019)Automatic dialogue summary generation for customer service. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Cited by: SS1, SS2.
* Junpeng Liu, Y. Zou, H. Zhang, H. Chen, Z. Ding, C. Yuan, and X. Wang (2021)Topic-aware contrastive learning for abstractive dialogue summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021, Punta Cana, Dominican Republic, pp. 1229-1243. External Links: Link, Document Cited by: SS1, SS2.
* Y. Liu and M. Lapata (2019)Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 3730-3740. External Links: Link, Document Cited by: SS1, SS2.
* Zhengyuan Liu and Nancy Chen (2021)Controllable neural dialogue summarization with personal named entity planning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 92-106. External Links: Link, Document Cited by: SS1, SS2.
* I. Loshchilov and Frank Hutter (2019)Decoupled weight decay regularization. In International Conference on Learning Representations, Cited by: SS1, SS2.
* Ahmed Magooda and Diane Litman (2020)Abstractive summarization for low resource data using domain transfer and data synthesis.

 

Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017: Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In Thirty-First AAAI Conference on Artificial Intelligence, Cited by: SS3.
* X. Peng, Z. Huang, X. Sun, and K. Saenko (2019)Domain agnostic learning with disentangled representations. In Proceedings of the 36th International Conference on Machine Learning, Volme 97 of Proceedings of Machine Learning Research, pp. 5102-5112. PMLR. Cited by: SS4.
* T. Schick and H. Schutze (2021)Few-shot text generation with pattern-exploiting training. External Links: Link Cited by: SS1, SS2.
* 63. External Links: ISSN 0899-1216, Document Cited by: SS1.
* A. See, P. J. Liu, and C. D. Manning (2017)Get to the point: summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada, pp. 1073-1083. External Links: ISBN 978-3-319-142-3, Document, Link Cited by: SS1.
* A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017)Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, Red Hook, NY, USA, pp. 6000-6010. External Links: ISBN 978-3-319-142-3, Document, Link Cited by: SS1.
* D. Wang, P. Liu, M. Zhong, J. Fu, X. Qiu, and X. Huang (2019)Exploring domain shift in extractive text summarization. External Links: Link Cited by: SS1.
* Z. Yang, C. Zhu, R. Gmyr, M. Zeng, X. Huang, and E. Darve (2020)TED: a pretrained unsupervised summarization model with theme modeling and denoising. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online, pp. 1865-1874. External Links: Link, Document Cited by: SS1.
* T. Yu, Z. Liu, and P. Fung (2021)AdaptSum: towards low-resource domain adaptation for abstractive summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, pp. 5892-5904. External Links: Link, Document Cited by: SS1.
* J. Zhang, Y. Zhao, M. Saleh, and P. Liu (2020)Pegasus: pre-training with ex- tracted gap-sentences for abstractive summarization. In In International Conference on Machine Learning, Online, Online, pp. 11328-11339. External Links: Link, Document Cited by: SS1.
* L. Zhao, W. Xu, and J. Guo (2020)Improving abstractive dialogue summarization with graph structures and topic words. In Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain, pp. 437-449. External Links: Link, Document Cited by: SS1.
* L. Zhao, W. Zeng, W. Xu, and J. Guo (2021a)Give the truth: incorporate semantic slot into abstractive dialogue summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021, Punta Cana, Dominican Republic, pp. 2435-2446. External Links: Link, Document Cited by: SS1.
* L. Zhao, F. Zheng, K. He, W. Zeng, Y. Lei, H. Jiang, W. Wu, W. Xu, J. Guo, and F. Meng (2021b)Todsum: task-oriented dialogue summarization with state tracking. arXiv preprint arXiv:2110.12680. Cited by: SS1.
* M. Zhong, D. Yin, T. Yu, A. Zaidi, M. Mutuma, R. Jha, A. Hassan Awadallah, A. Celikyilmaz, Y. Liu, X. Qiu, and Dragomir Radev (2021)QMSum: a new benchmark for query-based multi-domain meeting summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, pp. 5905-5921. External Links: Link, Document Cited by: SS1.
* C. Zhu, Y. Liu, J. Mei, and M. Zeng (2021)MediaSum: a large-scale media interview dataset for dialogue summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, pp. 5927-5934. External Links: Link, Document Cited by: SS1.
* Y. Zou, J. Lin, L. Zhao, Y. Kang, Z. Jiang, C. Sun, Q. Zhang, X. Huang, and X. Liu (2021a)Unsupervised summarization for chat logs with topic-oriented ranking and context-aware auto-encoders. In AAAI, Cited by: SS1.
* Y. Zou, L. Zhao, Y. Kang, J. Lin, M. Peng, Z. Jiang, C. Sun, Q. Zhang, X. Huang, and X. Liu (2021b)Topic-oriented spoken dialogue summarization for customer service with saliency-aware topic modeling. In AAAI, Cited by: SS1.
* Y. Zou, B. Zhu, X. Hu, T. Gui, and Q. Zhang (2021c)Low-resource dialogue summarization with domain-agnostic multi-source pretraining. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 80-91. External Links: Link, Document Cited by: SS1.

## Appendix A Experiment details

### Baselines

We describe baselines in detail as follows 



**Lead-3**: This method is commonly used in news summarization task, which treats the first three sentences of the document as the summary.

**Oracle**: The method is used to obtain an oracle through a greedy way similar to Nallapati et al. (2017), which treats the sentences that maximize the ROUGE-2 as the summary.

**BertExt**: Proposed by Liu and Lapata (2019), this model is extractive and its parameters are initialized with BERT.

**PGN**: Proposed by See et al. (2017), this model adopts the pointer mechanism to deal with the issue of Out-Of-Vocabulary in the summary generation process.

**Transformer**: Proposed by Vaswani et al. (2017), this model captures long-distance information through the self-attention mechanism.

**BertAbs**: Proposed by Liu and Lapata (2019), this model is abstractive, which encoder is initialized with BERT and is trained with a different optimizer than decoder.

**BART**: Proposed by Lewis et al. (2020), the model is a state-of-the-art abstractive summarization model pre-trained with a denoising autoencoding objective.

**BART w. DS/QR**: Proposed by Zhao et al. (2021b), the model is a general summarization framework, with two encoders that share the underlying parameters and a decoder, which can fuse the input text and dialogue state/query.

**Prefix-tuning**: Proposed by Li and Liang (2021), this model introduces a prefix matrix on the basis of fixed pre-training BART parameters and allows the prefix matrix to learn task information through training, which optimizes the summarization performance in the small parameters and few-shot scenarios.

**QMSum**: This model is proposed by Zhong et al. (2021), which is a two-stage locate-then-summarize solution on query-based meeting summarization task.

### Evaluation Metrics

We use the ROUGE Lin (2004); Lin and Och (2004)6 metrics to quantitatively evaluate the performance of our model. Rouge (Recall-Oriented Understudy for Gisting Evaluation) is metrics for evaluating summarization. It calculates by comparing the generated summary with references to obtain the corresponding score to measure the similarity between them.

Footnote 6: [https://pypi.org/project/rouge/](https://pypi.org/project/rouge/)

## Appendix B Parameter Scale of Models

We show the amount of **trainable** parameters for our DOP model and other baseline models in Table 9. Among the full-parameter fine-tuning methods, except for the relatively simple PGN model, other models have reached the scale of hundreds of megabytes, which will take up a lot of time and space in model training and storage. Prefix-tuning and DOP-tuning greatly reduce the storage space of the model, improving the efficiency of the model. Compared with prefix-tuning, our method achieves better results with fewer parameters.

## Appendix C Case Study

Figure 7 shows two examples from the TODSum and QMSum respectively. For example one of _train_ domain in TODSum, BART w. DS generates some incorrect and redundant information related to the _taxi_ domain and _hotel_ domain. To make matters worse, for _train_ domain, it loses the intent of the user about booking tickets and wrongly generates the key information, i.e., the departure location. The Prefix-tuning still confuses the knowledge of _train_ and _taxi_ domains, that is, the booking intent in the _train_ domain is wrongly predicted as the user wants to know something about "cars". Moreover, the quality of the generated key information is not high, i.e., the wrong departure location "Seachage" and the missing time "Monday". For example two of _academic_ domain in QMSum, both BART w. QR and Prefix-tuning predict too many details of the dialogue, which makes the summary redundant. Besides, Prefix-tuning generates the wrong speaker "Professor B", which leads to the summary being inconsistent.

Compared to the above two models, our method

\begin{table}
\begin{tabular}{|l|c|} \hline
**Models** & **Trainable Parameters** \\ \hline
**BertExt** & 120.51M \\ \hline
**PGN** & 27.64M \\
**Transformer** & 257M \\
**BertAbs** & 180.22M \\ \hline
**BART-large** & 400M \\
**BART w. DS/QR** & 406M \\ \hline
**Prefix-tuning** & 81.82M \\
**DOP-tuning** & 61.52M \\ \hline \end{tabular}
\end{table}
Table 9: Trainable parameter scales of different models, where “DS” means the dialogue state in TODSum and ”QR” means the query in QMSum.



solves some difficult issues in low-resource domain adaptation for dialogue summarization. By initializing prefix matrix with domain words, our model achieves domain disentanglement and the prediction of domain-related information is basically accurate. Through discrete prompts, our model has the ability to generalize to new domains and the accuracy of prediction about domain-independent key information is greatly improved.

## Appendix D Domain Words

We present some domain words for each domain in Figure 8. In order to facilitate reading, we only show some of the domain words, that is, we select the first 20 words for each domain of TODSum, and the first 30 words for each field of QMSum as display.

For TODSum, we can see that there are relatively many common domain words, which are more concentrated on location words, or some information, such as "phone", "postcode" and so on. In addition, there are many common domain words that only appear in _restaurant_& _hotel_ or _train_& _taxi_. For example, price-related descriptions are usually mentioned when booking a restaurant or hotel, and "destination", "depart", "from", "to" are usually mentioned when booking train tickets or taking a taxi. Special domain words can better distinguish different domains. There will be more special domain words in _attraction_, such as "entrance", "college", "nightclub", etc., which will not appear in other domains. Besides, users will mention the star rating when booking a hotel, and want to know the food type when booking a restaurant seat. When booking train tickets, they usually plan to travel, and when taking a taxi, they want to know the color of the car.

Compared with TODSum, QMSum has many more special field words, because the three fields contained in QMSum are more different. For _product_, participants will discuss various features of products such as IVs, LCDs, etc., such as screens, buttons, colors, and functions. For _academic_, participants generally discuss models, experimental data, or some errors. And for _committee_, participants generally discuss student education or national government issues. Common domain words have only some generral words, such as "different", "system", etc., and only a few special common domain words exist in _product_& _academic_, such as "design", "bit" and other technology-related words.

 

* [150] M. C. Bader, S. D. Krizhevsky, and A. Vedaldi. Deep residual learning for image recognition. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 238-246, 2017.
* [151] J. L. Friedman, R. G. Olschowski, and I. Zisserman. The k-means clustering algorithm: a survey. _Journal of Machine Learning Research_**9**(1):1-10, 2002.
* [152] P. F. Donahue, Y. Bengio, and E. Rabinovich. Going deeper into rectifiers by reducing internal covariate shift. In _Advances in Neural Information Processing Systems_, volume 1, pages 103-113, 2016.
* [153] H. Dong, X. Li, and T. Darrell. Fast convolutional networks for large scale visual recognition. In _NIPS'09 conference on computer vision and pattern recognition_, pages 139-140, 2009.

[MISSING_PAGE_POST]

 

* [14] M. C. B. Krizhevsky, A. Goyal, and R. Girshick, _Deep learning for image recognition_, vol. abs/cs. Pattern Recognition **2016**, no. 3, pp. 579-582, 2016.
* [15] D. J. Fleet, S. F. King, and L. W. Fergus, _Image classification with deep convolutional neural networks_, vol. abs/cs. Pattern Recognition **2016**, no. 1, pp. 1-10, 2016.
* [16] P. I. Goodfellow, I. Sutskever, and E. Hinton, _Advances in Neural Information Processing Systems_, vol. 2, no. 3, p. 1300-1313, 2016.
* [17] Y. Bengio, _The MIT press_. Cambridge: MIT Press, 2016.
* [18] H. Courville, _Deep Learning_. Springer, New York, 2015.
* [19] T. Darrell, _Fashion-specific models of natural language_. Oxford University Press, 2016.
* [20] V. Bengio, _Pattern recognition_. Prentice Hall, Upper Saddle River, NJ, USA, 2008.
* [21] N. DeVito, _A new approach to machine translation_. Addison Wesley, Reading, MA, 2008.

[MISSING_PAGE_POST]