SlowGuess/NAACL_2022_OCR huggingface dataset:
Lulu Zhao $^{1*}$ , Fujia Zheng $^{1*}$ , Weihao Zeng $^{1}$ , Keqing He $^{2}$ , Weiran Xu $^{1\dagger}$ , Huixing Jiang $^{2}$ , Wei Wu $^{2}$ , Yanan Wu $^{1}$
$^{1}$ Pattern Recognition & Intelligent System Laboratory
$^{2}$ Meituan Group, Beijing, China
{zhaoll, fujia_zheng, ZengWH, xuweiran}@bupt.edu.cn
{kqin}@bupt.cn,{jhx_bupt}@163.com,{wuwei19850318}@gmail.com
The most advanced abstractive dialogue summarizers lack generalization ability on new domains and the existing researches for domain adaptation in summarization generally rely on large-scale pre-trainings. To explore the lightweight fine-tuning methods for domain adaptation of dialogue summarization, in this paper, we propose an efficient and generalizable Domain-Oriented Prefix-tuning model, which utilizes a domain word initialized prefix module to alleviate domain entanglement and adopts discrete prompts to guide the model to focus on key contents of dialogues and enhance model generalization. We conduct zero-shot experiments and build domain adaptation benchmarks on two multi-domain dialogue summarization datasets, TODSum and QMSum. Adequate experiments and qualitative analysis prove the effectiveness of our methods.
Abstractive dialogue summarization task aims to distill the most critical information in a conversation to produce a concise version, involving chitchat (Gliwa et al., 2019; Chen and Yang, 2020), meeting(Zhong et al., 2021), customer service (Liu et al., 2019; Zou et al., 2021b), and task-oriented dialogue scenarios (Zhao et al., 2021b). Compared to the single-speaker texts, summarizing a dialogue presents a unique set of challenges, such as unstructured expressions and information sparsity issues. Recently, large-scale generative pre-trained models (Lewis et al., 2020; Liu and Lapata, 2019) have promoted the development of abstractive dialogue summarization but they all require extensive human-annotated golden summaries, which makes them not scalable to new domains where only few/no labeled data is available. Considering that real-world applications often face the problem
Dialogue	Dialogue
U: I am looking for a train. The train should arrive by 9:45 and should depart from Cambridge....A: What is your destination?U: I need to go to Norwich.A: How about TR2534 that leaves at 5:36?......	Professor A: if you found that nine was better than six that would be OK....Professor A: Yeah, just work with the models, yeah.PhD B: yeah, just look at the length of the models and just see what happens.......
Dialogue StateInform (destination=Norwich; arrive by=9:45; departure=Cambridge)	QueryWhat did the professor think about improving the back recognizer for the Aurora task?
SummaryThe user asks duration, arrive by and train id. The departure of the train is Cambridge and the destination is Norwich, and the train arrives at 9:45 on Friday.	SummaryThe Professor thought that the experiments were pretty simple, so computational power was not really a problem. He also thought that it would be okay to increase the states from six to nine.
Figure 1: An example of TODSum dataset (Zhao et al., 2021b) with dialogue state and an example of QMSum dataset (Zhong et al., 2021) with query. Note that dialogue state and query are existing characteristics of these two datasets, respectively.
of data in the new domain, it is vital to develop low-resource dialogue summarization models for the target domain by leveraging limited annotated data of source domains.
Therefore, we try to explore the efficient domain adaptation of dialogue summarization models from the source domain  $D_{s}$  to the target domain  $D_{t}$ , where  $D_{s}$  only has limited annotated summaries and  $D_{t}$  has few/no labeled data. There exist some domain adaptation approaches that focus on continual pre-trainings using some large domain/task-related corpora. Yu et al. (2021) added multiple pre-training stages both on source domains and target domains. Further, Zou et al. (2021c) decomposed the pre-training into three procedures, i.e., the pre-training of encoder, decoder, and the combined encoder-decoder model. Fabbri et al. (2021) constructed pseudo summaries based on external Wikipedia data to simulate characteristics of target dataset. Note that all these methods re
* The first two authors contribute equally.
$\dagger$  Weiran Xu is the corresponding author.
Human Language Technologies, pages 4848 - 4862
quire time-consuming pre-trainings or large-scale external corpora. They only focus on the heavy pre-training stage rather than the lightweight finetuning, which makes it labor-expensive and environmentally unfriendly (Schwartz et al., 2020) to practical applications.
Different from existing works that adopt general pre-trainings on the large-scale external corpora, we focus on exploring efficient fine-tuning methods specifically targeted at domain adaptation for the dialogue summarization task. We consider the following key principles while designing our methods: (1) Efficiency: We do not use any external data or pretraining and aim at leveraging efficient fine-tuning mechanisms based on existing summarization models. (2) Domain Disentanglement: Traditional models often memorize excessive knowledge from  $D_{s}$  and generate wrong summaries containing specific domain words in  $D_{s}$ . We aim to disentangle shared domain knowledge from  $D_{s}$ . (3) Generalization: Models often learn specific features of the source domain, making it difficult to generalize in a new domain (Peng et al., 2019). For example, models learn the surface language style specific to  $D_{s}$  rather than adapting the way of saliency estimation and summary generation to  $D_{t}$ . We encourage the summarizer to only focus on generic key contents rather than domain-specific attributes.
To be consistent with above principles, we propose a lightweight and efficient Domain-Oriented Prefix-tuning method, DOP, for domain adaptation of dialogue summarization. For efficiency, we focus on fine-tuning summarization models instead of performing pre-trainings like existing works, which reduces expensive and time-consuming computation. For domain disentanglement, we design a domain-oriented prefix module, which contains a novel prompt initialization mechanism. Concretely, we use domain words extracted by unsupervised LDA (Hoffman et al., 2010) to initialize continuous prompt vectors and fit the outputs of MLP and precomputed BART to obtain initial parameters and representations of the prefix module. We also add a domain-oriented prefix sequence of key-value pairs to augment the classical attention layer, which is independently applied to all Transformer layers of pre-trained models to elicit the knowledge interactively and achieve overall optimization. In this case, different domain words from  $D_{s}$  and  $D_{t}$  can induce relevant domain knowledge while adapting to a new domain. For generalization, we construct
discrete prompts using dialogue states or queries, as shown in Figure 1, to guide the model to focus on key contents in dialogues and enhance generalization capability on unseen domains. Considering there is no unified and practical benchmark for domain adaptation of dialogue summarization<sup>1</sup>, we build domain adaptation benchmarks based on two existing multi-domain summarization datasets TODSum (Zhao et al., 2021b) and QMSum (Zhong et al., 2021). Extensive experiments demonstrate the benefits of our methods both in zero-shot and few-shot settings for domain adaptation.
Our contributions are threefold: (1) To the best of our knowledge, we are the first to explore finetuning methods for domain adaptation of dialogue summarization task, and establish two practical and comprehensive benchmarks for TODSum and QM-Sum datasets. (2) We propose a lightweight and efficient Domain-Oriented Prefix-tuning model, with domain word initialized prefix and discrete prompts, to elicit knowledge from large-scale pretrained models interactively. (3) We conduct sufficient experiments and qualitative analysis to prove the effectiveness of our methods and discuss current challenges of domain adaptation for dialogue summarization.
Abstractive Dialogue Summarization Dialogue summarization has drawn much attention recently. For chit-chat scenarios, researchers improved the performance on SAMSum dataset (Gliwa et al., 2019) via topic word information (Zhao et al., 2020; Liu et al., 2021), conversational structures (Chen and Yang, 2020, 2021), personal named entity planning (Liu and Chen, 2021), and semantic slots (Zhao et al., 2021a). Liu et al. (2019), Zou et al. (2021a,b), and Lin et al. (2021) proposed customer-service dialogue summarization datasets under diverse business scenarios. Besides, meeting transcripts, such as AMI (Carletta et al., 2005), ICSI (Janin et al., 2003), MediaSum (Zhu et al., 2021), and QMSum (Zhong et al., 2021), were also studied to promote dialogue summarization technologies. Zhao et al. (2021b) further proposed a task-oriented dialogue summariza
$^{1}$ Existing work mostly takes news datasets like CNN/DailyMail (Hermann et al., 2015) as source domains and dialogue datasets like SAMSum (Gliwa et al., 2019) as target domains. We argue the setting doesn't fit in with industrial scenarios transferring from one business to another, like from attraction consultation to hotel reservation.
Figure 2: Overview of the Domain-Oriented Prefix-tuning model. The input sequence of domain-oriented prefix module includes domain words from four source domains, i.e., attraction, hotel, taxi, train and the domain words of restaurant domain are used as the prefix sequence of the target domain during the test. The input sequence of prompt encoder is the dialogue state from TODSum dataset and its original dialogue text. The input sequence of decoder is the golden summary. Here, we use the pre-computed outputs of BART as the target of the fitting process to initialize the parameters of MLP. Note that the grey pre-computed BART module does not participate in the training of the model.
tion dataset ,TODSum, with dialogue state knowledge. Although great progress has been made in dialogue summarization, few people pay attention to the issue of domain adaptation in dialogue summarization. In this paper, we explore this issue in two multi-domain dialogue summarization datasets, i.e, TODSum and QMSum.
Domain Adaptation in Summarization Hua and Wang (2017) and Wang et al. (2019) adopted the document categories in news publications to build a multi-domain summarization dataset and investigated the domain shift for extractive summarization task. Yang et al. (2020), Zhang et al. (2020), Magooda and Litman (2020), and Fabbri et al. (2021) regarded diverse summarization datasets as different domains and conducted the assessment of multi-domain settings. Furthermore, various stages of pre-trainings were added to narrow the gap between the pre-training in news domain and the fine-tuning in dialogue domain (Yu et al., 2021; Zou et al., 2021c). However, these methods focus on the heavy pre-training stage rather than the lightweight fine-tuning, which is time-consuming and relies on large-scale external corpora. Therefore, we try to explore the fine-tuning methods for domain adaptation of dialogue summarization task.
Prompts in Summarization With the arrival of GPT-3, prompt learning has become a nascent field, which conducts task-specific adaptation of large language models (LMs) via prepending an instruction. Schick and Schütze (2021) explored the fixed-prompt LM tuning for few-shot text summarization with manually crafted templates. Zhao
et al. (2021b) and Dou et al. (2021) further adopted the prompt+LM tuning strategy on text summarization task, where learnable prefix prompts are different types of guidance signals. Li and Liang (2021) investigated fixed-LM prompt tuning, where learnable prefix tokens are prepended to the input while parameters in pre-trained models are frozen. Following Li and Liang (2021), we design the domain information to initialize the continuous prefix module, and use discrete prompts and dialogue texts to optimize prefix parameters, which greatly reduces the size of parameters and is suitable for low-resource scenarios.
Domain adaptation of dialogue summarization aims to generate the summary  $\mathbf{y}$  conditioned on the source dialogue  $\mathbf{x}_d$ , where the training and test are in different domains. We add the domain word prefix  $\mathbf{x}_{dw}$  and discrete prompt  $\mathbf{x}_{dp}$  as additional input which we will describe details later. Note that we only update the prefix-related parameters and fix the parameters of BART. We train the model using the source data and test using the target data.
As Figure 2 shows, our model is on the basis of the framework of BART, including a domain-oriented prefix module, a prompt encoder, and a decoder.
4.1 Domain-oriented Prefix
To alleviate domain entanglement, we present a domain-oriented prefix module to obtain the shared
knowledge of the source domain  $D_{s}$  and the target domain  $D_{t}$ . It is designed as follows:
Initialization We extract some keywords from dialogue texts in each domain by LDA (Hoffman et al., 2010) and concatenate them all together as a domain word (prefix) sequence  $\mathbf{x}_{dw}^2$ . Randomly initialized embeddings of the domain word sequence compose a learnable matrix  $M_{\theta} \in \mathbb{R}^{|\mathbf{x}_{dw}| \times d_m}$ .
Parametrization We use an MLP to encode the domain-oriented prefix module, which stably elicits knowledge from the large pre-trained model in the prefix-tuning process. Specifically, we first input the domain word sequence to the MLP and the pre-computed BART respectively, then re-train the MLP by fitting its outputs with the decoder hidden states of the pre-computed BART using MSE loss. In this fitting process, we only iteratively update MLP parameters  $\varphi \in \mathcal{R}^{d_m\times d_n}$  and keep the pre-computed BART fixed. Finally, we get the initialization parameters of MLP and use this pretrained MLP to map the initialized embeddings of prefix representations for each Transformer layer both in prompt encoder and decoder:
$$
M _ {\theta} ^ {\prime} [ i,: ] = \operatorname {M L P} _ {\varphi} \left(M _ {\theta} [ i,: ]\right) \tag {1}
$$
where  $i\in \mathbf{x}_{dw}$  and  $M_{\theta}^{\prime}\in \mathbb{R}^{|\mathbf{x}_{dw}|\times d_n}$ . Note that this continuous prefix is applied for every layer of the large-scale pre-trained model independently.
4.2 Prompt Encoder
Discrete Prompts We utilize some discrete prompts to emphasize key elements in dialogues and enhance the model generalization to new domains. Here, discrete prompts are dialogue states of TODSum dataset or queries of QMSum. Considering that the original form of dialogue states is book (people=5; day=Monday) which is not compatible with BART encoder, we convert this structured information into a serialized sequence, i.e., book, people is 5, day is Monday, to improve the stability of training. Note that we do not make any changes to the query of QMSum dataset because it is already a serialized representation.
For prompt encoder, we firstly concatenate the discrete prompt sequence  $\mathbf{x}_{dp}$  and dialogue text sequence  $\mathbf{x}_d$  as the input sequence of encoder, i.e.,  $\mathbf{x}_{enc} = [\mathbf{x}_{dp};\mathbf{x}_d]$ . Then, the  $\mathbf{x}_{enc}$  is fed into the
prompt encoder based on the BART encoder, containing multiple Transformer layers. Note that we modify the self-attention mechanism by adding a domain-oriented prefix sequence of key-value pairs, which learns the knowledge from the pre-trained model through interactions with the dialogue text to carry out the overall task. For the typical  $l_{e}$ -th Transformer layer in encoder, the query  $(\mathbf{Q}_{l_e})$ , key  $(\mathbf{K}_{l_e})$ , and value  $(\mathbf{V}_{l_e})$  matrices are computed through linear transformations on the hidden states of  $\mathbf{x}_{enc}$ . Here, we further augment the  $\mathbf{K}_{l_e}$  and  $\mathbf{V}_{l_e}$ :
$$
\mathbf {K} _ {l _ {e}} ^ {\prime} = [ \mathbf {P} _ {l _ {e}, k}; \mathbf {K} _ {l _ {e}} ], \mathbf {V} _ {l _ {e}} ^ {\prime} = [ \mathbf {P} _ {l _ {e}, v}; \mathbf {V} _ {l _ {e}} ] \quad (2)
$$
where  $\mathbf{P}_{l_e,k}$ ,  $\mathbf{P}_{l_e,v}$  are computed through linear transformations on  $M_{\theta}^{\prime}$ .  $\mathbf{K}_{l_e}^{\prime}, \mathbf{V}_{l_e}^{\prime} \in \mathbb{R}^{(|\mathbf{x}_{dw}| + |\mathbf{x}_{dp}| + |\mathbf{x}_d|) \times d_n}$  and  $l_e \in L$ . The augmented self-attention layer is finally calculated as follows:
$$
\mathbf {A} _ {\text {s e l f}} = \operatorname {s o f t m a x} \left(\mathbf {Q} _ {l _ {e}} \mathbf {K} _ {l _ {e}} ^ {' T}\right) \mathbf {V} _ {l _ {e}} ^ {\prime} \tag {3}
$$
4.3 Decoder
We also prepend the prefix module for decoder, where the cross-attention and masked-attention mechanisms are augmented in a similar way. The cross-attention between the  $l_{e}$ -th layer of prompt encoder and the  $l_{d}$ -th layer of the decoder is designed as:
$$
\mathbf {A} _ {c r o s s} = \operatorname {s o f t m a x} \left(\mathbf {Q} _ {l _ {d}} \mathbf {K} _ {l _ {e}} ^ {' T}\right) \mathbf {V} _ {l _ {e}} ^ {'} \tag {4}
$$
where  $\mathbf{Q}_{l_d}$  is computed through a linear transformation on the hidden states of the summary text  $\mathbf{x}_s$  and  $l_{d}\in L$ . Besides, the implementation of masked-attention layer is the same as the self-attention layer in the prompt encoder.
4.4 Training Strategy
In the domain-oriented prefix module, the parameter set of all linear transformations is symbolized as  $\alpha$ . For training strategy in DOP, we perform gradient updates on the following log-likelihood objective:
$$
\max  _ {\alpha , \theta , \varphi} \log p _ {\alpha , \theta , \varphi , \phi} (\mathbf {y} | \mathbf {x}) = \sum_ {i \in \mathbf {y}} \log p _ {\alpha , \theta , \varphi , \phi} (y _ {i} | y _ {<   i}) \tag {5}
$$
where the BART parameters  $\phi$  are fixed. The prefix parameters  $\alpha$ ,  $\theta$ , and  $\varphi$  are the only trainable parameters. During the training, we use the domain words from source domains as the prefix sequence. When the training is completed, we save all parameters of the domain-oriented prefix module and
2We represent some of domain words for each domain in Appendix D.
Domains	Size	Dialog.len	Summ.len	DS.len
Train	345	120.67	24.93	18.29
Taxi	435	80.24	29.04	15.80
Restaurant	1,311	105.42	23.04	14.30
Hotel	636	145.16	30.06	21.38
Attraction	150	95.48	22.27	7.92
All	2,877	111.71	25.68	16.24
Table 1: Details of TODSum. "Dialog.len" denotes the average length of dialogues, "Summ.len" denotes the average length of summaries, and "DS.len" denotes the average length of serialized dialogue states.
Domains	Size	Dialog.len	Summ.len	QR.len
Academic Committee	312	1,155.78	46.48	8.56
417	757.68	76.00	14.54
Product All	847	971.65	63.96	13.36
1,576	951.49	63.68	12.73
Table 2: Details of QMSum. "QR.len" denotes the average length of queries.
drop the pre-computed BART module. During the test, the domain words from the target domain are mapped to the representations of prefixes only via the reserved  $\mathrm{MLP}_{\varphi}(\cdot)$ , where the features of the source domain  $D_{s}$  can be transferred to the target domain  $D_{t}$ .
We evaluate our model on two multi-domain dialogue summarization datasets and the details of statistics are shown in Table 1 and Table 2:
TODSum: This dataset is proposed by Zhao et al. (2021b), which is a task-oriented dialogue summarization dataset based on the classic dialogue dataset MultiWOZ (Budzianowski et al., 2018). According to the domain information, the dataset can be divided into five domains: restaurant, hotel, attraction, taxi, and train. Considering that parts of dialogues in TODSum contain multiple domains, in this paper, we firstly select all single-domain dialogues from TODSum as the dataset used for this experiment. Then we integrate all these samples in any four of five domains as source domains  $D_{s}$  and the other one is regarded as target domain  $D_{t}$ , where 200 samples extracted from  $D_{s}$  are as the validation set, the remaining as the training set, and samples of  $D_{t}$  are the test set.
QMSum: This dataset is proposed by Zhong et al. (2021), which contains hundreds of meeting transcriptions and includes three domains, i.e., academic, committee, product. In addition, each sample can be divided into several dialogue fragments according to some queries and the answer for the
corresponding query is its golden summary. Such (query-dialogue-answer) pairs are usually used for query-based meeting summarization tasks. In this paper, we separately integrate the training and validation sets, and test set in any two of three domains as the training data and validation data, i.e., the data of source domain  $D_{s}$ . All data of the other domain is used as the test data, i.e., the data of target domain  $D_{t}$ .
5.2 Baselines and Evaluation Metrics
We compare our methods with several baselines. The extractive baselines are included: (1) Lead3; (2) Oracle; (3) BertExt (Liu and Lapata, 2019). Some abstractive methods are also added for comparison: (1) PGN (See et al., 2017); (2) Transformer (Vaswani et al., 2017); (3) BertAbs (Liu and Lapata, 2019); (4) BART (Lewis et al., 2020); (5) BART w. DS/QR (Zhao et al., 2021b); (6) Prefixtuning (Li and Liang, 2021). For QMSum, we also introduce its benchmark (Zhong et al., 2021). Since this method feeds the extracted spans into BART, we integrate the results of this method with the results of BART. We use ROUGEs (Lin, 2004; Lin and Och, 2004) to quantitatively evaluate the performance of our methods. Our codes are publicly available<sup>3</sup>. We give the baselines and evaluation metrics in Appendix A.1 and Appendix A.2.
Our implementation is based on the Hugging Face Transformer models<sup>4</sup>. BART<sub>LARGE</sub> is used as a backbone and the source dialogue sequence is truncated to 1024 BPE tokens. For domain-oriented prefix module, the MLP maps 1024 dimension into 24576 dimension, which is calculated by  $2 *$  number of decoder layers  $*1024$  and the numbers of domain words (prefix length) are set to 140 and 200 for TODSUM and QMSum datasets. Following the settings in Li and Liang (2021), we use an AdamW optimizer (Loshchilov and Hutter, 2019) and a linear learning rate scheduler with initial rate of  $5 \cdot 10^{-5}$ , and the batch size is set to 5. Our model is trained on RTX 2080 Ti machines, taking only 5 minutes per epoch on TODSum dataset and 3 min per epoch on QSum dataset. However, BART w. DS takes 13 minutes and 8 minutes per epoch on TODSum and QMSum datasets. The reason for the shorter training time of our model is that the
<sup>3</sup>https://github.com/Zeng-WH/DOP-Tuning.
Models	Train	Taxi	Restaurant	Hotel	Attraction
2,332 / 200 / 345	2,242 / 200 / 435	1,366 / 200 / 1,311	2,041 / 200 / 636	2,527 / 200 / 150
R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L
Lead-3	20.36	2.78	16.07	24.20	7.34	20.75	28.27	6.10	23.49	23.86	4.58	18.80	22.76	5.28	19.66
Oracle	39.06	10.04	32.87	38.96	14.06	33.43	45.79	15.57	38.42	39.65	11.28	32.56	41.90	14.18	38.79
BertExt	39.19	9.71	33.24	38.49	13.57	33.36	40.64	12.34	34.43	35.96	9.71	30.10	36.25	11.19	31.41
PGN	32.50	10.47	29.33	32.48	7.79	29.82	33.63	10.78	31.47	32.18	9.36	30.93	32.66	9.95	30.29
Transformer	33.47	10.98	30.28	33.35	8.71	30.57	34.49	11.43	31.99	33.05	10.62	31.63	33.18	10.74	30.91
BertAbs	42.89	16.57	37.32	36.43	14.69	32.15	42.10	18.61	38.87	38.03	13.34	33.22	36.21	14.81	34.67
BART	46.82	18.42	42.06	39.98	15.79	34.41	47.02	22.62	44.93	40.84	14.20	36.83	43.67	20.23	41.44
BART w. DS	49.02	23.80	44.59	43.59	19.56	38.65	49.25	23.57	45.23	43.97	17.02	39.31	47.55	22.62	45.16
Prefix-tuning	45.92	22.70	41.06	41.89	19.47	39.62	47.19	24.20	42.99	43.41	18.75	36.75	44.48	22.43	40.94
DOP (ours)	52.51	25.45	47.78	47.14	24.37	42.75	51.28	32.68	47.44	48.44	24.58	41.45	52.90	30.51	49.48