Multi-Stage Pre-training Enhanced by ChatGPT for Multi-Scenario Multi-Domain Dialogue Summarization
Authors: Weixiao Zhou
Gengyao Li
Xianfu Cheng
Xinnian Liang
Junnan Zhu
Feifei Zhai
Zhoujun Li
Venue: EMNLP 2023 Findings (2023)
Keywords: Multi-Scenario Multi-Domain Dialogue Summarization
Multi-Stage Pre-training
ChatGPT

Abstract: Dialogue summarization involves a wide range of scenarios and domains. However, existing methods generally only apply to specific scenarios or domains. In this study, we propose a new pre-trained model specifically designed for multi-scenario multi-domain dialogue summarization. It adopts a multi-stage pre-training strategy to reduce the gap between the pre-training objective and fine-tuning objective. Specifically, we first conduct domain-aware pre-training using large-scale multi-scenario multi-domain dialogue data to enhance the adaptability of our pre-trained model. Then, we conduct task-oriented pre-training using large-scale multi-scenario multi-domain "dialogue-summary" parallel data annotated by ChatGPT to enhance the dialogue summarization ability of our pre-trained model. Experimental results on three dialogue summarization datasets from different scenarios and domains indicate that our pre-trained model significantly outperforms previous state-of-the-art models in full fine-tuning, zero-shot, and few-shot settings.

Multi-Stage Pre-training Enhanced by ChatGPT for Multi-Scenario Multi-Domain Dialogue Summarization
Weixiao Zhou
Gengyao Li
Xianfu Cheng" Xinnian Liang"
Junnan Zhu
Feifei Zhai" Zhoujun Li
Staae Key Lab of Software Development Environment, Beihang University
State Key Lab of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS
$School of Artificial Intelligence, University of Chinese Academy of Sciences Fanyu Al Research, Zhongke Fanyu Technology Co., , Ltd (wxzhou, buaacxf, xnliang, lizj Jebuaa. edu. cn junnan. mirian..hu@nllr... ia..a.... ta.ac.c... ac.
Abstract
dialogue summarization. Existing dialogue sum- marization pre-trained model (Zhong al., 2022) achieves good performance on long dialogue sum- marization. However, it still has the following limitations: (1) It is only pre-trained on dialogue corpora that include two domains (i.e., Interview and TV show), making it difficult to apply to dia- logue summarization in wide range of scenarios and domains. (2) It utilizes window-based de- noising task the pre-training objective, which presents a significant gap with the fine-tuning ob- jective. Simultancously, existing state-of-the-ar (SOTA) models generally improve dialogue sum- marization by modeling dialogue interactions (Lin et al., 2022; Tang et al., 2022), incorporating extra information (e.g.. topics and roles) (Wang et al., 2022c; Kim al., 2022), and rewriting dialogues (Xu al., 2022; Fang et al., 2022). Although these methods have some effect, they still have limited applicability to downstream datasets in different scenarios and domains and are often difficult to apply within the current pre-training paradigm due to complex model architectures.
Dialogue summarization involves wide range of scenarios and domains. However, existing methods generally only apply to specific sce- narios domains. In this study, we propose new pre-trained model specifically designed for seetii-ccenario multi-domain dialogue summa- rization. It adopts a multi-stage pre-training strategy to reduce the gap between the pre- training objective and fine-tuning objective. Specifically, we first conduct domain-aware pre-training using large-scale multi-scenario multi-domain dialogue data enhance the adaptability of our pre-trained model. Then, we conduct task-oriented pre-training using large- scale multi-scenario multi -otmin "dialogue- summary" parallel data annotated by ChatGPT enhance the dialogue summarization abil- ity of our pre-trained model. Experimental re- sults on three dialogue summarization datasets from different scenarios and domains indicate that our pre-traine pre-trained model significantly outper- forms previous state-of-the- models in full fine-tuning, zero-shot, and few-shot settings
1 Introduction
To address the limitations of previous works, in this study, our goal is to propose task- lask-specific pre- trained model for dialogue summarization, which has extremely small gap between the pre-training objective and the fine-tuning objective, enabling it to excellently adapt to downstream datasets from a wide range of scenarios and domains in full fine- tuning, few-shot, and zero-shot settings.
Dialogue summarization is the task of generating summary from dialogue (Xu al., 2022). Specif- ically, open-domain dialogue summarization in- volves various scenarios (e.g., Online-Chii (Gliwa et al., 2019) and Daie (Chen et al., 2021)), while customer service dialogue summarization in- volves different domains (e.g.. Tweet (Feigenblat et al., 2021) and E-commerce (Lin et al., 2022)). Recently,
Motivated by the above goal, we consider three key components in the implementation of our pre- trained model: model architecture, pre-training corpus, and pre-training strategy. For model ar- chitecture, our pre-trained model is based on the standard Transformer (Vaswani al., 2017) encoder-decoder architecture and initialized with BART (Lewis al., 2020). To capture the un- derlying role interactions during the dialogue pro- cess, we incorporate additional speaker embed-
pre-trained models have achieved significant success in dialogue sum- marization tasks (Lewis al., 2020; Bao al., 2020; Beltagy al., 2020). Furthermore, sev- eral task-specific pre-trained models (Zhang et al., 2020a; Zhong et al., 2022) have further improved
Our corpus and enhanced pre-trained models can be found at https: //tit//ittttt// com/ zhouwe ao/MP4 *Corresponding Author

---

Summaries
Umiraace Natkh!
Dowells-tiond
Preatnining
ChatGPT
Utieaaee Pensstation
Ballggie
sPt
Dialagie
Luersee Iniliig
Tuniag
MaaeWau 2.1
L350
Mutial
Tatroaid
TeplosiChar
Teken Biledee
Denskr
Maal WOr
Riniee SVilTTV Tvlk
HP Encader
Irame
PresdriA
Summary
Duily-Life
Engedetz Utdagaas
Dedagge
Sdere
Taken Makiar
and Eneeder
Malti Sccnese Dialogse
Malli-hamata Bialggss
Castomer Service Dtalague
Molti Seearioo Diolegie Muli-Damsia Didogge
ependlennts dilapa diauce wullie
(a) construction process LCMDD
Multi-stage pre-training and fine-tuning for MP4
Figure I: Our pre-training corpus LCM DS and pre-trained model MP4.
dings (Gu et al., 2020, 2021) into token representa- tions. For pre-training corpus, we collect open- domain dialogue datasets across multiple scenar- ios and 6 multi-domain customer service dialogue datasets. Furthermore, due to the development of Large Language Models (LLMs) (Zeng et al., 2022; Thoppilan al., 2022; Scao et al., 2022) and their excellent generative ability, obtaining high- parallel pre-training data has become possible. Therefore, we utilize ChatGPT (Ouyang et al., 2022) to annotate the collected multi-scenario multi-domain dialogues and obtain corresponding summaries. We refer to our pre-trainn pre-training corpus as LCM" DS (Large-scale ChatGPT-annotater Multi-scenario Multi-domain Multi-turn Dialogue Summarization) (see Figure (a)). For pre-training strategy, we conduct multi- stage pre-training reduce the gap between the pre-training objective and the fine-tuning objective. Specifically, we first conduct domain-awand pre- training using the dialogue data from LCM 'Ds to enhance the adaptability of pre-trained model to dialogues in multiple scenarios and domains. Then, we utilize the "dialogue sumtuueresmmmr""" parallel data from LC for task-triented pre-training hance the ability of pre-trained model summarize dialogues. We refer to our pre-trained model as MP4 (Multi-stage Pre- trained Model for Multi-scenario Multi-domain Dialogue Summarization) (see Figure (b)).
tomer service dialogue summarization dataset from specific domain (i.e., Tweet (Feigenblat et al., 2021)). The experimental results indicate that MP4 significantly outperforms previous SOTA models in full fine-tuning, zero-shot, and few-shot settings, demonstrating remarkable performance improve- ments.
Our contributions summarized follows:
quality "dialogue-summer""
We construct LCM which includes a large-scale collection multi-scenario multi- domain dialogues and their corresponding summaries annotated by ChatGPT.
We propose MP4, a multi-stage pre-traue pre-trained model for multi-scenario multi multi-domain dia- logue summarization.
Our pre-trained model achieves new state-of- the-art performance on three dialogue sum- marization datasets from different scenarios and domains full fine-tuning, zero-shot, and few-shot settings.
2 LCM Corpus
2.1 Dialogue Preparation
multi-scenario
Dialogue Collection. We collect 20 high-quality human-to-hume multi-turn dialogue datasets to construct the dialogue part of LCM includ- ing 14 open-domain dialogue datascts across mul- tiple scenarios (Rashkin al., 2019; Nie al., 2021; Dinan al., 2019; Komeili al., 2022: Gopalakrishnan al., 2019; Zhang et al., 2018: Smith et al., 2020; Dinan et al., 2018; Li et al., 2017; Cui et al., 2020: Zhou et al., 2018; Yu et al.,
We evaluate our pre-trained model on open- domain dialogue summarization datasets from two scenarios (i.c., Online- Chat (Gliwa et al., 2019) and Daily-Life (Chen et al., 2021)), as well as cus-

---

Dataset
Sce./Dom.
#Dillogue
#Tokens dial.
#Tokens/
#Comp.
#Coy.
#Dens.
& of novel n-grams
% of redundant n-grams bigram
inigram
bigram frigram
unigrom
trigram
SAMSum
ODDS-Online
16,368
5.99
0.71
34.24
1.04
0.20
DEALOGSUM
ODDS Daily
13,460
208.5
34,5
6.48
0.81
2.20
26.80
63.95
81.70
19.53
6.31
2.79
TWEETSUMM
CSDS Tweet
1,087
328.8
7.54
0.75
32.83
73.31
1.32
0.24
LcM"Ds
Multiple 206,768
211.3
51.9
4.03
0.73
188
34.54
74.19
86.17
21.87
3,00
0.66
Table 1: Comparison between LCM DS and existing dialogue summarization datasets. Sce. denotes scenario of Open- Wven-Domain Dialogue Summarization (ODDS). Dom. denotes domain of Customer Service Dialogue Summariza- tion (CSDS). # represents the average value. dial. represents dialogue. Summ. represents reference summary ChatGPT--annotan summary. Comp. indicates compression ratio. Cov. indicates coverage. Dens. indicates density,
2020; Yang and Choi, 2019; Sun et al., 2019) and 6 multi-domain customer service dialogue datasets (Lee et al., 2019; Rastogi et al., 2020; Byrne et al., 2019; Zang et al., 2020; Li et al., 2018; Feng et al., 2020). In total, there are (55,426 open-domain dia- logues containing over IM utterances and 101,342 customer service dialogues containing over 1.4M utterances. We provide the details of data statistics for all dialogue datasets in Appendix G.
the text "TI;dr:" at the end of each dialogue as prompt and inputting it into ChatGPT2 (in zero- shot setting) to obtain annotated summaries. We also investigate the performance of three different prompts for dialogue summarization in zero-shot setting, and the details can be found in Appendix A.
Role-Replaced Data Augmentation. In dia- logue summarization, there are multiple scenar- ios and domains involving different roles. To al- leviate this problem, we propose a simple yet ef- fective method that can be extended to dialogue summarization involving any role. Specifically, we directly replace the roles in the dialogues and summaries from LCM to obtain an augmented parallel corpus. In this study, we perform replace- ments for two common types of roles, including named coreference and customer service. The ex- ample we provide can be found in Appendix C.
Dialogue Pre-Processing and Cleaning. We conduct series of automated data pre-preeeee processsing and cleaning to further improve the quality of the dialogues. For pre-processing, we perform the fol- lowing steps: (1) Normalizing punctuations, spe- cial characters, and capitalization in each dialogue. (2) Following previous studies (Dinan et al., 2019; Chen al., 2021), we preprocess each dialogue into dual-turn dialogue format by merging consec- utive utterances from the same speaker. For clean- ing, we perform the following steps: (1) Remov- ing duplicate and highly similar dialogues using the Jaccard text similarity algorithm. (2) Delet- ing highly similar dialogues between the dialogue datasets and the evaluation datasets using the same algorithm as in (1), ensuring that they have no intersection. (3) Removing dialogues with less than 4 turns or 32 tokens.
2.3 Data Analysis
We empirically compare LCM Ds with existing di- alogue summarization datasets (Gliwa et al., 2019; Chen et al., 2021; Feigenblat et al., 2021) based on five metrics (Grusky et al., 2018; Fabbri et al.. 2021): Compression Ratio, Coverage, Density, Novelty, and Redundancy (see Table 1).
Compared to existing dialogue summarization datasets, L exhibits lower Compression Ra- tio, moderate Coverage, and lower Density, in- dicating that the summaries maintain a high de- gree of abstraction while covering the important content of the dialogue and retaining more details and information from the dialogue. Additionally, LCM² Ds shows higher Novelty and Redundancy, which is mainly caused by the lower Compres- sion Ratio. Furthermore, unlike existing small- scale, human-annotated, single-scenario, single- domain dialogue summarization datasets, LCM³DS is lcrge-scale, ChatGPT-annotatte multi-scenario, and multi-
Role Adding. In order to standardize the differ- ent speaker formats of original dialogues across various datasets, we collect a list containing over 4,000 real names. For each dialogue, we randomly selected several real names from the list to assign a role group (e.g., Danny and Alejandra), where the number and order of real names in each role group corresponds the speakers in the original dialogue.
2.2 Annotation
Prompt Format. We follow the previous study of InstructGPT (Ouyang et al., 2022) by inserting
We use gpt-3. 5-urbo--tu

---

Aey
iPoroor/ A
Apent
Alos
Choiveer
Inpur
<ear>
How
Token
Libeddings
Ernw
Ere
E
Podtion
Euibedingg
Spy
Epy
Eyn
E
Exe
Eppx
Eprr
Eppr
Speaker
Eutbekkings
Figure 2: Dialogue modeling of MP4.
3 Model
scenario multi-domain dialogues into structured written-language summaries.
3.1 Dialogue Modeling
3.2.1 Domain-Aware Pre-training
4M44 is based on the standard Transformer (Vaswani et al., 2017) encoder-decoder architecture. For multi-turn dialogue modeling, the input embed- ding of each token is the sum of the corresponding token, position, and speaker embeddings. Figure illustrates the dialogue modeling of MP4.
General-purpose pre-trained models (Lewis et al., 2020) are pre-trained on free-form text data with universal pre-training objectives, limiting their abil- ity in specific domains and tasks. Therefore, it is common practice to further train these models with the language modeling objective using text from the target domain to reduce negative impact (Zhang and Zhao, 2021; Whang et al., 2021). ). In this study, we conduct domain-aware pre- trainini stage on MP4 using the dialogue data from LCM Specif- ically, we achieve this by modeling series of dia- logue reconstruction pre-training objectives in- spired by BART (Lewis et al., 2020).
Input Structure. Given the dialogue context D, we first concatenate all roles R2 and utterances J in the dialogue context with two additional special tokens as a separate, consecutive token sequence:
X= = {Ri PRRceerrUUieee <eou>... Ren <eou>> where the special end-of-role token <eor> and end-of-utterance token <eou> are respectively appended to the end of cach role and utterance for multi dialogue separation. Then, add the start token and the end token 
 around the token sequence X the input for MP4.
Token Masking. For tokens of each utterance in the dialogue, 20% of them are randomly sampled and replaced with a special <mask> token.
Speaker Embeddings. To distinguish utterances in the dialogue context and capture underlying role interactions during the dialogue process, follow previous dialogue modeling studies (Gu et al., 2020, 2021) and add additional speaker embeddings to token representations. This process is performed alternately based on the role transitions and can be extended to dialogues with an unlimited number of roles. The speaker embeddings are combined with the initial token and position embeddings and then fed into the MP4 encoder-decoder framework.
Token Deletion. 20% of the tokens in the dia- logue utterances are randomly sampled and deleted.
Utterance Infilling. Several utterance spans are randomly sampled, and each span replaced with a single <mask> token. The length of each utterance span is drawn from the Poisson Distribution (A = 3). O-length spans correspond to the insertion of <mask> tokens.
Utterance Permutation. The order of all utter- ances in the dialogue turns is randomly shuffled. In contrast to previous studies (Zhong al., 2022; Wang al., 2022b), we did not shuffle the order of roles. Therefore, MP4 needs reconstruct the correct order of utterances and ensure the precise alignment between utterances and roles.
3.2 Multi-Stage Pre-training
We conduct multi-stage pre-training to reduce the gap between the pre-training objective and the fine- tuning objective. Domain-aware pre-training aims enhance the adaptability of MP4 to dialogues in multiple scenarios and domains, while task- oriented pre-training aims to enhance the ability MP4 to summarize unstructured spoken multi-
Utterance Masking. 20% of the utterances in the dialogue are selected and replaced with a special
<uttr-maske token. We did not perform random

---

selection but instead followed the method of PE- GASUS (Zhang al., 2020a) using greedy search to obtain the principal Gap-utterances. During the decoding process, MP4 needs to reconstruct the complete dialogue.
the non-pretrained model, which includes only spe- cialized input structure and speaker embeddings. MP4 (DAP) denotes domain-aware pre-training ap- plied to MP4 (VANILLA). MP4 (TOP) signifies task- oriented pre-training applied to MP4 (VANTILLA). MP4 (DAP-TOP) indicates multi-stage pre- tre-training applied to MP4 (VANILLA). More implementation details of pre-training are provided in Appendix D.
Multi-Tasi Learning. The model is trained with a maximum likelihood objective Le. Given the training sample D (z, v). Le is defined as
Downstream Datasets. We evaluate the perfor- mance of MP4 on open-domain dialogue summa- rization datasets from two scenarios (i.c., Online- Chat and Daily-Life), namely SAMSum (Gliwa et al., 2019) and DIALOGSUM (Chen et al., 2021), as well as a customer service dialogue summariza- tion dataset from specific domain (i.e., Tweet), namely TWEETSUMM (Feigenblat al., 2021). Table provides the statistics of the downstream datasets. More details are provided Appendix B.
b log (e(
Lo
=
(1)
where è is the model parameters, x is the noisy dialogue, and is the original dialogue.
During each iteration of the multi-task domain- aware pre-training stage, training samples are ran- domly selected from different pre-training tasks as mini-batches and used to calculate the cumulative loss and optimize the model parameters e
Comparison Methods. We compare MP4 with three types of baselines: extractive models, ab- stractive models, and previous SOTA models. The following presents the comparison methods.
3.2.2 Task-Oriented Pre-training
Several task-specific summarization pre-trained models (Zhang al., 2020a; Xiao et al., 2022; Zhong et al., 2022) reduce the gap with downstream datasets by modeling the task-oriented pre-training objective. Specifically, They typically select seg- ments (e.g., gap-sentences or window-basee ut- terances) of the original text (e.g., document or dialogue) optimization targets for the dccoder. Although they have some effects, however, there still exists a significant gap between the segments selected through unsupervised methods and abstrac- tive languae- summaries. In this study, we directly utilize the 'dialogue-summan"" summaryy""""" parallel data from D for task-orienn orientree pre-training stage. The learning objective is similar to Eq. (1), where the training sample D (z, y). with rep- resenting the original dialogue and representing the summary annotated by ChatGPT.
Extractive Models. Based on heuristic algo- rithms or graph-based algorithms, including LONGEST, Lead-3, and TextRank (Mihal- cea and Tarau, 2004).
Abstractive Models. Based on neural net- work sequence-to-ssqquece models, includ- ing PGNet (See et al., 2017), FastAbs-RL (Chen and Bansal, 2018), and Transformer (Vaswani al., 2017).
Previous SOTA Models. State-of-the-ar dialogue summarization models based on pre-trained models, including BART(DALL ALL) (Feng et al., 2021), Coref-ATTN (Liu et al., 2021), BART-ConFiT (Tang al., 2022). DialSent-PGG (Jia et al., 2022), BART- NARR (Xu et al., 2022), MV-BART (Chen and Yang. 2020), ReWriteSum (Fang et al., 2022), BART-SCL (Geng et al., 2022), UNILMv2 (Bao et al., 2020), BART (Lewis et al., 2020), LA-BART (Wang al., 2022a), and BART-MT (Bhattacharjee et al., 2022).
4 Experiments
4.1 Experimental Setup
Implementation Details. MP4 is initialized with BART-largee (Lewis et al., 2020). which is a de- noising soquence-to-squeenee
pre-trained Trans- former (Vaswani et al., 2017) model with 12 lay- ers and 16 attention heads. To facilitate perfor- mance comparison, we have implemented four types MP4 models. MP4 (VANILLA) represents
Evaluation Metrics. We evaluate the full fine-
tuning, zero-shot, and few-shot performance of all models using ROUGE 2 (i.e., R-1, -2, and -L), which are standard evaluation metrics.
"https://huggingfe ca/ wttp://ee.oaaaaaaaaeeeeeeee wttp:/ccccboo/k/ee bitteebboookbrtt-arrggg

---

Model
R-1
R-2
R-I.
Model
R-1
R-2
R-L
Extractive and Abstractive Models
Extractive and Abstractive Models
TextRank (Mihalcea and Tarau, 2004)
29.27
8.02
28.78
LONGEST
24.10
6.20
22.70
PGNNet (See et al., 2017)
37.27
14.42
34.36
Lead-3
27.50
6.80
27.30
FastAbs-RL (Chen and Bansal, 2018)
41.03
16.93
39.05
Transfommrr (Vaswami el al., 2017)
35,91
8.74
33.50
Transformer (Vaswani et al., 2017)
42.37
18.44
39.27
Previous SOTA Models
Previous SOTA Models
UKILMMV--inse (Bao et al., 2020)
47.04
21.13
45.0
BART(PALL) (Feng ct al., 2021)
53.70
28.79
50.81
BART-large (Lewis et al.. 2020)
47.28
21.18
44.83
Coref-ATTIN (1 au
2021)
53,93
BART-NARR (Xu
2022)
47.52
20.82
45.10
BART-ConFiT (Tang et al., 2022)
53.89
28.85
49.29
LA-BART (Wang ct al., 2022a)
47.28
21.09
45.11
DialSent-PGG (Jia ct 2022)
53.54
28.91
50.21
BART-MT (Bhattacharjee et al., 2022)
47.26
21.18
45.17
BART-NARR (Xu al., 2022)
Our Models
MV-BART (Feng al., 2022)
54.05
28.56
50.57
ReWriteSum (Fang el al., 2022)
54,20
27.10
50.10
BART-large
46.68
20.96
44.68
BART-SCI. (Geng al., 2022)
29.87
(VANILLA)
46.87
44.87
MP4 (DAP)
47.01
21.37
44.94
Our Models
MPd (TOP)
47.74
21.84
45.77
BART-large
53.32
28.78
50.63
MP4 (DAP-TOP)
48.01
21.72
45.92
(VANILLA)
53.41
29.23
50.97
MP4 (DAP)
53.82
29.55
51.21
Table 3: Full fine-tuning results DIALOGSUM test
MP4 (TOP)
54.56
30.33
$1.70
(DAP-TOP)
30.57
$1.79
set. We report the average of multiple-reference results.
Table 2: Full fine-tuning results SAMSum
Model
R-1
R-2
R-L
DistiBART (Feigenhlat et al., 2021)*
37.94
19.26
33.51
BaRT-large (ours)
44.77
4.2
Full Fine- Sine-Tuning Evaluation
MP4 (VANILLA)
45,56
22.31
44.84
To demonstrate the advantages of our pre-trained model with large amount of training samples, we train the model using the entire training set for full fine-tuning evaluation.
MP4 (DAP)
46.61
22.82
45 08
MPd (TOP)
46.84
23.63
45.74
MP4 (DAP-TOP)
46.93
23.60
45.82
Table 4: Full fine-tuning results on TWEETSUMM test
set. denotes results obtained from (Feigenblat al.. 2021).
Settings. We provide all the hyper-parameters used for finettnnig and inference in Appendix During the evaluation, for SAMSum, we followed (Liu and Lapata, 2019) by testing with the top-3 best checkpoints on the validation set and reporting the average ROUGE scores. For MAALOGSUM, we followed (Chen et al., 2021) by reporting the aver- age ROUGE scores between the inference output and multiple reference summaries. For TWEET- SUMM, due to limited research and the lack of detailed evaluation procedures the original paper (Feigenblat et al., 2021), we suggest the following evaluation method: (1) During training, use the reference summary with the highest COUGE-AVg score (i.c., the average value of R-1, R-2, and R-L) between the original dialogue and multiple refer- (2) During testing, calculate the average ROUGE scores between the inference output and multiple reference summaries.
state-of-the-art results on all metrics across three downstream datasets from different scenarios and domains, demonstrating significant performance improvements. Specifically, on SAMSum and DI- ALOGGSUM, MP4 (DAP-TOP) surpasses the previ- ous SOTA models by improving the R-2 score by 0.70 and 0.54 (29.87-30.55 and 21.18 12..722,,, respectively. The improvement on TWEETSUMM This indicates that
reaches 1.46 multi-stage pre-training can assist the model bet- ter adapt to downstream datasets from wide range of scenarios and domains. Additionally, compared BART-large, MP4 (VANILLA) demon- strates stronger performance on most metries, prov- ing the effectiveness of introducing dialogue mod- eling. Furthermore, compared to MP4 (VANILLA), the results of MP4 (DAP) indicate that domain- aware pre-training can enhance the adaptability of model to dialogues in multiple scenarios and do- mains. Moreover, MP4 (TOP) achieves significant performance improvements, highlighting the im- portance of equipping the model with the ability summarize
ence summaries for training.
Since most previous SOTA models have not been evaluated on wide range of dialogue summarization datasets, therefore, we present the full fine-tuning performance on the SAMSum, D1- ALOGGSU,, and TWEETSUMM test sets in Ta- bles 2, 3, and 4, respectively. Compured to previous SOTA models, our MP4 (DAP-TOP) achieves new
Results.
multi domti- dialogues during the pre-training stage.

---

Model
SAMSum
DIALOGSUM
TWEETSUMM
zevo-shor
few-shor (10)
zero-shor
fes-shot (10)
zero-shot
Jew-shot (10)
BaRT-large (ours)
27.94/8.56/266 56/26.88
39.47/16.51/88766
15.55/6.1025.32
36.75/12.9836.04
29/42/10.6822.18
46.14779.4644175
MP4 (DAP)
21.75/17.3444..2
27 27.80/6.61/27.4
36.89112.88336008
13.98/20.0842..1
MP4 (DAP-TOP)
42.41/17.15/399669
48.73/2363/45.66
38.76/14.59/37.18
40.18/16.22/99.11
38.92/13.1635511
45.09/20.20/43.03
Table 5: R-1/R-2/R-L results in zero-shot and sew-shot settings, For zero-shot setting, we report the results at the optimal summary length limits. For few-shot setting, we report the average results from random runs training samples (all models share the same seed set).
4.3 Zero- and Few-Shot Evaluation
Model
R-1
R-2
R-L
Many existing studics that apply pre-trained models to dialogue summarization require large amount
MP4 (VANILLA)
53.41
29.23
50.97
w/o speaker embeddings
53.29
29.02
50.85
MP4 (DAP)
53.82
29.55
51,21
of fine-tuning data, which is often impractical in
w/o token-level tasks
53.74
29.25
51,05
w/o utterance infiling
53.75
24.31
51.06
new scenarios or domains. In contrast, we expect
w/o utterance permulation
$3.69
29.38
51.08
model to quickly adapt new scenarios or domains without the need for large amount of fine-tuning data. To validate this hypothesis, we conduct evaluations in zero-shot (no training sam- ples) and few-shot (10 training samples) settings. Obtaining such a small number of samples is feasi- ble in practice for new scenarios or domains,
wfo utterance masking
53.53
29.30
50.94
(TOP)
54.56
30.33
51.70
w/o CSDS pre-training corpus
54.47
30.12
51.69
MP4 (DAP-TOP)
$4,60
30.57
$1.79
Table 6: Ablation study on SAMSum in full fine-tuning setting. The token-level tasks refer to Token Masking and Token Deletion.
Settings. We compare the performance of BART- large (Lewis al., 2020), MP4 (DAP), and MP4 (DAP-TOP) in zero-shot and few-shot settings. In Appendix F, we provide all the hyper-parameters used. Specifically, for zero-shot evaluation, since the models have not been trained on downstream datasets, we report the results of using the optimal summary length limits during inference. For few- shot evaluation, randomly sample 10 training samples for training. Additionally, to ensure that the results are not affected by sampling variability, we conduct the same experiment five times with different random seeds (shared among all models) and report the average results.
4.4 Ablation Study
To further validate the contributions of the fine- grained components in our pre-trained models, we conduct an ablation study on SAMSum in full fine- tuning setting. Table 6 shows the evaluation results.
Speaker Embeddings. As the results show, in- corporating additional speaker embeddings in di- alogue modeling can capture the underlying role interactions during the dialogue process and im- prove the performance of dialogue summarization.
Aomain-Aware Pre-training Objectives. As the results show, each domain-aware pre-training ob- jective brings performance improvements. It is worth noting that the utterance masking task has the greatest impact performance, indicating that completing principal Gap-utterances during dia- logue reconstruction is crucial for dialogue summa- rization.
Results. The results presented in Table 5 indi- cate that our pre-trained model achieves significant improvements compared to BART-large. Specif- ically, for zero-shot results, MP4 (DAP-TOP) in- creases the R-1 score by 14.47 13.26 (25.50->38.70 738.76), and 9.50 (29.42->38 +38.92) on SAMSum, DIALOGSUM, and TWEETSUMM, re- spectively. Moreover, the zero-shot performance of MP4 (DAP-TOP) surpassed the few-shot perfor- mance of BART-large on multiple datasets, demon- strating its powerful zero-shot capability. Addi- tionally, the few-shot results also highlight the ad- vantages of MP4 (DAP-TOP). indicating that our pre-trained model converges faster than other mod- els even with only 10 training samples.
Impact of CSDS Pre-training Corpus on ODDS. We remove the customer service "dialogue- summary" parallel data from LCM²DS to inves- tigate the impact of this portion of data on the per- formance of open-domain dialogue summarization. The results show that the model trained without this data exhibits slight decrease in performance. One possible reason is that few dialogues in SAMSum also involve customer service topics.

---

Model
Flu.
Conci.
Info.
Comp.
Dinan al., 2019), task-oriented dialogue systems
ChatGPT (zero-shot)
2.24
3.82
3.55
1.37
(Lee al., 2019; Rastogi et al., 2020; Byrne et al.,
BART-large (owrx)
2.77
2.19
2.48
3.13
2019; Zang et al., 2020; Li et al., 2018; Feng et al.,
MP4 (DAP-TOP)
2,53
2.02
2.14
3.08
2020), dialogue comprehension (Sun et al., 2019;
Ground Truth
2.46
1.97
1.83
2.42
Yang and Choi, 2019; Yu et al., 2020; Cui et al.,
Table 7: Human evaluation SAMSum test set.
2020; Nie et al., 2021; Rashkin et al., 2019), and dialogue summarization (Gliwa et al., 2019; Chen et al., 2021; Feigenblat et al., 2021).
5 Human Evaluation
PTMs for Dialogue Summarization. Recently, cemerrl-purpose, pre-trained models have achieved significant success in dialogue summarization tasks (Lewis et al., 2020; Raffel et al., 2020; Bao et al.. 2020; Beltagy et al., 2020). Furthermore, sev- eral task-specific pre-trained models (Zhang et al., 2020a; Zhong et al., 2022) have further improved dialogue summarization. Moreover, existing state- af- dialogue summarization models typically leverage pre-trained models and model the charac- teristics of dialogues to achieve better results, in- cluding modeling dialogue interactions (Lin et al., 2022: Tang al., 2022), incorporating extra in- formation (Wang et al., 2022c; Kim al., 2022), and dialogue rewriting (Xu et al., 2022; Fang et al., 2022). A Although these models are effective, they of- ten have complex model structures that are difficult to apply within the current pre-training paradigm.
We conduct human evaluation to further evalu- ate the performance of our pre-trained model and strong baselines under various paradigms, as well as the Ground Truth (i.e., MP4 (DAP-TOP), Chat- GPT, BaRT-large, Ground Truth). Specifically, randomly select 50 samples from the test set of SAMSum. Then, invite participants rank four candidate summaries according to four met- rics: fluency (Flu.), conciseness (Conci.), informa- tiveness (Info.), and comprehensiveness (Comp.). The top-ranking indicates the best performance on that metric.
Table shows the results of human evaluation (lower average rank is better). Our pre-trained model outperforms BART-large in all metrics but falls behind the Ground Truth. Specifically, Chat- GPT achieves the first rank in fluency and compre- hensiveness for the summaries generated in zero- shot setting, surpassing the Ground Truth. How- ever, it exhibits the weakest performance in con- ciseness and informativeness. The main reason for this is that ChatGPT tends to generate longer sum- maries that describe various aspects of the dialogue, including both important and minor details. More- over, the longer summaries also contribute to an improved overall impression to some extent.
Large Language Models. More recently, LLMs have attracted widespread attention due to their remarkable performance in various knowledge- intensive NL.P tasks (Zeng et al., 2022; Ouyang et al., 2022: Thoppilan et al., 2022; Scao et al., 2022). Through large-scale pre-training on massive text corpora (Brown et al., 2020; Wei et al., 2022b), LLMs possess powerful foundational capabilities. Instruction tuning (Raffel et al.. 2020; Wei et al.. 2022a; Chung et al., 2022) helps L.MMs in under- standing natural language task descriptions. while Reinforcement Learning with Human Feedback (RLHI') (Stiennon et al., 2020; Bai al., 2022) aligns generated text with human preferences.
6 Related Work
Dialogue Corpora. In general, dialogue data can be obtained from two main sources. One is massive- scale dialogue corpora crawled from web platforms such as Reddit (Baumgartner et al., 2020; Hender- son et al., 2019) and Twitter (Ritter et al., 2010), which are commonly used for pre-training open- domain chatbots (Zhang et al., 2020b; Adiwardana et al., 2020; Roller et al., 2021; Chen et al., 2022; Henderson et al., 2020). Another source is a col- lection of open-source dialogue datasets designed for specific tasks, including open-domain dialogue systems (Li et al., 2017: Zhou al., 2018: Dinan et al., 2018; Smith et al., 2020; Zhang et al., 2018; Gopalakrishnan et al., 2019; Komeili et al., 2022;
7 Conclusion
In this study, we propose MP4, a multi-stage pre- trained model for multi-scenario multi-domain dia- logue summarization. To conduct the pre-training, we construct large-scale CharGPT-annotated multi-scenario multi-domain multi-turn dialogue summarization corpus called LCM Extensive experimental results demonstrate that MP4 exhibits remarkable dialogue summarization capabilities.

---

Limitations
Ben Mann, and Jared Kaplan. 2022. Training a help- ful and harmless assistant with reinforcement learn- ing from human feedback.
Although have demonstrated the powerful per- formance of MP4 in multi-scenario multi-domain dialogue summarization, there are still some limi- tations that provide directions for future work: (1) Due to limitations in computational resources, we did not consider long dialogues when constructing Ds. Therefore, MP4 may be more suitable for short dialogue summarization. (2) MP4 is initial- ized with BART-large, which has only 0.4 billion parameters. future work, we will consider using larger base models.
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang. Nan Yang. Xiaodong Liu, Yu Wang. Jianfeng Gao, Song- hao Piao, Ming Zhou, and Hsiao- Hon. 2020. Uni Pseudo seauko-masedd language models for uni- fied language model pre-training. In Proceedings of the 37th International Conference on Machine Learn- ing, volume 119 of Proceedings of Machine Learning Research, pages 642 022-652. PMLR.
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. Proceedings of the Interna- rional AAAI Conference on Web and Social Media, 14(1):830-88) 88808893..
Ethics Statement
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The document transformer. arXiv preprint 55X555200..55/50.
All dialogue data used in this study are sourced from previously published works, and there are no copyright restrictions on their academic use, allow- ing free online access. Since we used ChatGPT for data annotation, the LCM DS corpus we con- structed is intended solely for academic research purposes. Additionally, MP4 is initialized with the weights from BaRge Therefore, MP4 may exhibit biases and harmful behaviors commonly observed in language models.
Saprativa Bhattacharjee, Kartik Shinde, Tirthankar Ghosal, and Asif Ekhal. 2022. A multi-task learn- ing approach for summarization of dialogues. Proceedings of the 15th International Conference Natural Language Generation: Generation Chal- lenges, pages 10-120, Waterville, Maine, USA and virtual meeting. Association for Computational Lin- guistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse. Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei, 2020. Language models are few-shot learners. In Ad- vances Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Acknowledgement
This work received support from both the National Natural Science Foundation of China (Grant Nos. 62276017, U1636211, 61672081) I) and the Fund of the State Key Laboratory of Software Development Enviromment (Grant No. 12X- We extend our sincere appreciation Yu Zhou and Rongping Chang for their valuable contributions to the revision of this paper.
Bill Bymee Karthik Krishnamoorthi, Chinnadhurai Sankar. Arvind Neelakantan. Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu-Young Kim, and Andy Cedilnik. 2019. Taskmaster-1: To- ward realistic and diverse dialog dataset. In Pro- ceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing JEMNLP-IJCNLL. pages 4516-4525. Hong Kong, China. Association for Computational Linguistics.
References
Daniel Adiwardana, Minh-Thame Sinh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan. Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoe V. Le. 2020. Towards a human open- domain chatbot.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodde Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah,
Jiaoo Chen and Diyi Yang. 2020. Multi-view sequence- ro-sequence models with conversational structure for abstractive dialogue summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4106- 4118, Online. Association for Computational Lin- guistics.

---

Wei Chen, Yeyun Gong, Song Wang. Bolun Yao. Weizhen Qi, Zhongyu Wei, Xiaowu Hu, Bartuer Zhou, Yi Mao, Weizhu Chen, Biao Cheng, and Nan Duan. 2022. DiD A pre-trained latent vari- able encoder-decoder model for dialog response gen- eration. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4852-4864, Dublin, Ireland. Association for Computational Linguisties.
Yue Fang, Hainan Zhang, Hongshen Chen, Zhuoye Ding, Bo Long, Yanyan Lan, and Yanquan Zhou. 2022. From spoken dialogue to formal summary: An utterance rewriting for dialogue summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies, pages
Seattle, United States. Association for Computational Linguistics.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstrac- tive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 675-686, Melbourne, Australia. Association for Computational Linguistics.
Guy Feigenblat, Chulaka Gunasekara, Benjamin Szna- jder, Sachindra Joshi, David Konopnicki. and Ranit Aharonov. 2021. TWEETSUMM marization dataset for customer service. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2021, pages 245-260,, Punta Cana, Do- minican Republic. Association for Computational Linguistics.
dialog sum-
Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021. DialogSum: A real-life scenario dialogue sum- marization dataset. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pages
Song Feng, Hui Wan, Chulaka Gunasekara, Siva Patel, Sachindra Joshi, and Luis Lastras. 2020. doc2dial: A goal- document- grounddd dialogue dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8118 8128, Online. Association for Computa- tional Linguisties.
Online. Association for Computa- tional Linguisties.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani. Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- cry, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Edd H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instruction-finetiee language models.
Xiachong Feng. Xiaocheng Feng, and Bing Qin. 2022. A survey on dialogue summarization: Recent ad- vances and new frontiers.
Xiachong Feng, Xiaocheng Feng. Libo Qin, Bing Qin. and Ting Liu. 2021. Language model as an annota- tor: Exploring DialoGPT for dialogue summarization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume I: Long Papers), pages 1479-1491, Online. Association for Computational Linguistics.
Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. 2020. MuTual: A dataset for multi-turn dia- logue reasoning, In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 1406-1416, Online. Association for Computational Linguistics.
Zhichao Geng, Ming Zhong, Zhangyue Yin, Xipeng Qiu, and Xuanjing Huang. 2022. Improving abstrac- tive dialogue summarization with speaker-aware pervised contrastive learning. In Proceedings of the 29th International Conference on Computational Lin- guistics, pages 6540-6546, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanck. Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexan- der Rudnicky, Jason Williams, Joelle Pineau, Mikhail Burtsev, and Jason Weston. 2018 The second con- versational intelligence challenge (convai2). In The NeurlPS 18 Competition, pages 187-208, Cham. Springer International Publishing.
Bogdan Gliwa, Twona Mochol, Maciej Biesek, and Alek- sander Wawer. 2019. SAMSum corpus: A human- annotated dialogue dataset for abstractive summa- rization. Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70-79, Hong Kong, China. Association for Computational Linguis- tics.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge-powere conversational agents. In International Conference on Learning Representations.
Karthik Gopalakrishnan. Behnam Hedayatnia, Qin- lang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkami-Tüür. 2019. Topical-Chat: Towards soowlede-Grrounde Open-Domain Conversations. In Proc. Interspeech 2019, pages 1891-
Alexander R. Fabbri, Wojciech Kryšeinsski, Bryan Mc- Cann, Caiming Xiong, Richard Socher, and Dragomir Radev, 2021. SummEval: Re-evaluating summariza- tion evaluation. Transactions of the Association for Computational Linguistics, 9:391 -009.

---

Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long Pu- pers), pages 708-719, New Orleans, Louisiana. As- sociation for Computational Linguistics.
et al. 2019. Multi-domain task-completion dialog challenge. Dialog system technology challenges. 8(9).
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdeirahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequeeee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Compulational Linguistics, pages 7871-7880 Online. Association for Computa- tional Linguistics.
Jia- Jia-Chen Gu Tianda Li, Quan Liu, Zhen-Hua Ling, Zhiming Su, Si Wei, and Xiaodan Zhu. 2020. Speaker-aware bert for multi-turn response selection retrieval- based chatbots.
Jia-Chen Gu, Chongyang Tao, Zhenhua Ling, Can Xu, Xiubo Geng, and Daxin Jiang. 2021. MPC-BERT: A pre-trained language model for multi-party con- versation understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1thh International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 3682-3692, Online, Association for Computational Linguistics.
Xiujun Li, Yu Wang, Siqi Sun, Sarah Panda, Jingjing Liu, and Jianfeng Gao. 2018. Microsoft dialogue challenge: Building end-to and-to-end task completon dia- logue systems. arXiv preprint arXiv: 1115.
Yanran Li. Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi mullti-tum turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Nat- ural Language Processing (Volume I: Long Papers), pages 986-995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Matthew Henderson, Pawet Budzianowski, Iñigo Casanueva, Sam Coope. Daniela Gerz, Girish Kumar, Nikola Mrkšić, Georgios Spithourakis, Pei-Hao Su, Ivan Vulié, and Tsung- Hsien Wen. 2019. A reposi- tory of conversational datasets. In Proceedings of the First Workshop on NLP for Conversational AI, pages 1-10, Florence, Italy. Association for Computational Linguistics.
Haitao Lin, Jumnan Zhu, Lu Xiang, Yu Zhou, Jiajun Zhang, and Chengqing Zong. 2022. Other roles mat- ter! enhancing role-oriented dialogue summarization via role interactions. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 2545- 2558, Dublin, Ireland. Association for Computational Linguisties.
Matthew Henderson, lñigo Casanueva, Nikola Mrkšić, Pei-Hao Su, 'sung-Hsien Wen, and Ivan Vulié. 2020. ConveRT: Efficient and accurate conversational repre- sentations from transformers. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2020, pages 2161-2174, Online. Association for Computa- tional Linguistics.
Yang Liu and Mirella Lapata. 2019. Text summariza- tion with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIPP-ICCNIP)) pages 3730-3740, Hong Kong, China. Association for Computational Linguistics.
Qi Jia, Yizhu Liu, Haifeng Tang, and Kenny Zhu, 2022. Post-training dialogue summarization using pseudo- paraphrasing. In Findings of the Association for Com- putational Linguisties: NAACL 2022. pages 1660- Seattle, United States. Association for Compu- tational Linguistics.
Zhengyuan Liu, Ke Shi, and Nancy Chen. 2021. Coreference- aware dialogue summarization. In Pro- ceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 509-519, Singapore and Online. Association for Computational Linguistics.
Seungone Kim, Se June Joo, Hyungjoo Chae, Chae- hyeong Kim, Seung-won Hwang, and Jinyoung Yeo. 2022. Mind the gap! injecting commonsense knowl- edge for abstractive dialogue summarization. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6285-6300, Gyeongju. Republic of Korea. International Com- mittee on Computational Linguistics.
Rada Mihalcea and Paul Tarau. 2004. TextRank: Bring- ing order into text. In Proceedings of the 2004 Con- ference on Empirical Methods in Natural Language Processing, pages 404411 Barcelona, Spain. Asso- ciation for Computational Linguistics.
Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2022, Internet-augmenteed dialogue generation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Pu- pers), pages 8460-8478, Dublin, Ireland. Association for Computational Linguistics.
Yixin Nie. Mary Williamson, Mohit Bansal, Douwe Kiela, and Jason Weston. 2021. I like fish, espe- cially dolphins: Addressing contradictions in dia- logue modeling. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference Natural Language Processing (Volume I: Long Papers), pages 1699-1713, Online, Association for Computational Linguistics.
S Lee, H Schulz, Atkinson, J Gao, K Suleman, L El Asti, M Adada, M Huang, S Sharma, W Tay,

---

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 37730-27744 Curran Associates, Inc.
put all together; Evaluating conversational agents' ability to blend skills. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2021-2030, Online, Association for Computational Linguistics.
Nisan Stiennon, Long Ouyang. Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learn- ing to summarize with human feedback. In Ad- vances in Neural Information Processing Systems, volume 33, pages 3008-3021. Curran Associates, Inc.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21(1).
Kai Sun, Dian Yu, Jianshu Chen. Dong Yu, Yejin Choi, and Claire Cardie. 2019. DREAM: A challenge data set and models for dialogue- reading compre- hension. Transactions of the Association for Compu- tational Linguistics, 7:217-231.
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open- domain conversation models; A new benchmark and dataset. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 5370-5381, Florence, Italy. Association for Computational Linguistics.
Xiangru Tang, Arjun Nair, Borui Wang. Bingyao Wang. Jai Desai, Aaron Wade, Haoran Li, Asli Celikyil- maz, Yashar Mehdad, and Dragomir Radev, 2022. CONFIT: Toward faithful dialogue summarization with Tiguistically-inforees
contrastive fine-tuning.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. To- wards scalable multi--omain conversational agents: The schema-guided dialogue dataset. Proceedings of the AAAI Conference Artificial Intelligence, 34(00)):689-886) 8696.
In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguisties: Human Language Technologies. pages 5657-5668, Seattle, United States. Association for Computational Linguistics.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoory Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applica- tions. arXiv preprint arXiv:2201.02255
Alan Ritter, Colin Cherry, and Bill Dolan. 2010. Un- supervised modeling of Twitter conversations. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the As- sociation for Computational Linguistics, pages 172- 180, Los Angeles, California, Association for Com- putational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Out, Eric Michael Smith, Y-Lan Boureau, and Jason We- open-domain
ston. 2021. Recipes for building chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguisties: Main Volume, pages 300 -20,, Online. Association for Computational Linguistics.
Bin Wang, Chen Zhang, Chengwei Wei, and Haizhou Li. 2022a. A focused study on sequence length for dialogue summarization.
Jiaan Wang, Fandong Meng, Ziyao Lu, Duo Zheng. Zhixu Li, Jianfeng Qu, and Jie Zhou. 2022b. Clid- Sum: A benchmark dataset for cross-line poss-lingual dialogue summarization. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 7716-7729, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.00 1.0111155/00 100
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 1073- 1083, Vancouver, Canada. Association for Computa- tional Linguistics.
Ye Wang. Xiaojun Wan, and Zhiping Cai. 2022c. Guid- ing abstractive dialogue summarization with content planning. In Findings of the Association for Com- putational Linguistics: EMNLP 2022, pages 3408 3413, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022a. Finetuned language
Eric Michael Smith, Mary Williamson, Kurt Shuster, Jason Weston, and Y-Lan Boureau. 2020. Can you

---

models are zero-shot leamner.. In International Con- ference on Learning Representations.
of Machine Learning Research, pages 11328-11339. PMLR.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. 2022b. Chain of thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston, 2018. Per- sonalizing dialogue agents: have dog. do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume Long Papers), pages 2213. Melbourne, Australia. Association for Computational Linguisties.
Taesun Whang. Dongyub Lee, Dongsuk Oh, Chan- hee Lee, Kijong Ham, Dong-hun Lec, and Sacbyeok Lee. 2021. Do response selection models really know what's next? utterance manipulation strate- gies for multi-turn response selection. Proceedings of the AAAI Conference Artificial Intelligence, 35(16); 4041-14049
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan, 2020b. DIALOGPT Large-- scale generative pre-training for conversational re- sponse generation. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270-278. Online. Association for Computational Linguistics.
Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. 2022. PRIMERA: grramid-based masked sentence pre-training for multi-document summariza- tion. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 5245-5263, Dublin, Ireland. Association for Computational Linguistics.
Zhuosheng Zhang and Hai Zhao. 2021. Structural pre- training for dialogue comprehension. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages Association for Computational Linguisties.
Ruochen Xu, Chenguang Zhu, and Michael Zeng. 2022. Narrate dialogues for better summarization. In Find- ings the Association for Computational Linguistics: EMNLP 2022, pages 3565-3575, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Online.
Ming Zhong. Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2022. Dialoglm: Pre-trained model for long dialogue understanding and summa- rization. Proceedings of the AAAI Conference on Artificial Intelligence, 16(10):11755-117)
Zhengzhe Yang and Jinho D. Choi. 2019. FriendsQA: Open-domain question answering on TV show tran- scripts. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 188-197, Stockholm, Sweden. Association for Computational Linguisties.
Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. 2018. A dataset for document grounded con- versations. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Process- ing, pages 008-713, Brussels, Belgium. Association for Computational Linguistics.
Dian Yu, Kai Sun, Claire Cardie, and Dong Yu. 2020. Dialogue-base based relation extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4927-4940, Online. Association for Computational Linguisties.
Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. MultiWOZ 2.2 A dialogue dataset with additional annotation corrections and state tracking baselines. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 109-117, Online. Association for Computa- tional Linguistics.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang. Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b An open bilingual pre-trained model. arXiv preprint arxiv:2210.02414.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter Liu. 2020a. PEGASUS: Pre-training with ex- tracted gap-sentences for abstractive summarization. Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings

---

A Prompts for Dialogue Summarization
A customer service dialogue summarization dataset, consisting of reconstructed dialogues extracted from the Kaggle Customer Sup- port On Twitter dataset. The Kaggle dataset is a large-scale dataset based on dialogues between consumers and customer support agents on Twit- ter.com. It covers wide range of topics and ser- vices provided by various companies, including airlines, retail, gaming, music, etc. The sizes of the training, validation, and test sets are 869, 108, and 110 respectively.
TWEETSUMM.
We utilize ChatGPT unsupervised summa- rizer to annotate dialogues in the SAMSum (Gliwa et al., 2019) test set using three different prompt formats in zero-shot setting:
Preceding Prompt. Insert the prompt "Sum- marize the following dialogue into a short summary:" before the dialogue.
InstructGPT Prompt. Following Instruct- GPT (Ouyang et al., 2022) insert the prompt "Tl:dr:" after the dialoguc.
C Example of Data Augmentation
Subsequent Prompt. Insert the prompt "Sum- marize the above dialogue into a short sum- mary:" after the dialogue.
Dialogee
Danny (iPersonid) Customer:
Are there pg-
playing aroud 6:00gm
70072?
Alejandex [PPersn22 Agent:
Sure! There's Batmun Superman 6ipm.. The reset are later the evening.
Are you interested in that?
Danny [PPerson Customer:
Prompt
Rouge-1
Rouge-2
Rouge-L
Yes, that's fine 6pm.
Alejandra [PPenso22) Agents
Preceding 37.90
15.19
35.89
Hoo many tickets?
InstructGPT
42.17
16.84
39.26
Danny [APerson1e] Customer:
Subsoquent 40.08
15.41
37.22
Alejandra [Persoo22 Agent: Gnea! You've got lickets to the 6pm showing of Baitmom Enjoy the show!
Table 8: Comparison of dialogue summarization per- formance for different prompts SAMSum test set, which contains 819 dialogues.
CharGPT-Anee P-Antuttteeeedd Summary
Danny neks for pa-15 movic playing at fimm tomorow sip code 70072. Alejandra suggest 6pm showing.
Supermom Danny
Role RelieRee HammaryyNamec C Coreference)
Table 8 shows the evaluation results of three differ- ent prompts. The InstructGPT Prompt achieves the best performance.
PPPrrsnn maks pg- 13 movic plaing at fyr tomorrow zip code 70072, #lPessn2e suggests Botman vs Superman and seils #iPersoniit lickets for the 6pmm showing.
Role-Replaced Sammary(Customer Service)
Details of Downstream Datasets
Customer for movie
tomomoow code Supemian sells Customer tickets
Agent suggests for the 4pm showing.
SAMSum. natural messenger dialogue sum- marization dataset containing dialogues created and written down by linguists fluent in English. The dataset includes various topics such chit-chats, gossiping about friends, arranging meetings, dis- cussing politics, consulting university assignments with colleagues, etc. The sizes of the training, val- idation, and test sets are 14,731, 818, and 819 re- spectively.
Figure 3: An example of role-replaced dialogue and summary, Red indicates role group added through role adding. yellow indicates named coreference, and blue indicates roles in customer service.
D Implementation Details of Pre-training
The following are the hyper-parameters used domain-aware pre-training.
DIALOGSUM. A large-scale dataset for dialogue summarization, which includes sice-to-face spoken dialogues covering a wide range of daily-life topics, including schooling, work, medication, shopping, leisure, and travel, etc. Most dialogues take place between friends, colleagues, and between service providers and customers. The sizes of the training, validation, and test sets are 12,460, 500, and 500 respectively.
gpus 8 steps 5000
batch_size 16
-Ir 3e - 05
warmup_steps 500
label_mmothhing 0.1
optimizer Adam
The following are the hyper-parameted used task-oriented pre-training.

---

gpus
SAMSum, DIALOGSUM, and TWEETSUMM is 60, 40, and 80 respectively. Moreover, other hyper- parameters used during inference remain consistent with Appendix E. For few-shot evaluation (with training samples), we provide the hyper-parameter settings used during training below, while the nyyper-paaaetterd used during inference are con- sistent with Appendix E.
steps 10000
batch_size 16
-1r 3e 05
warnup_steps 1000
-laael_smoottieee 0.1
oplimizer Adam
E Details of Fine- Tane-Tnnngg and Inference
gpus 1
steps 20 batch_siza
The following are the hyper--aramend parameters used in full fine-tuning setting SAMSum.
Ir - 05
gpus
warmup_steps
steps 1150
label_smoothing 0.1
-balhhssizz 16
optimizer Adam
Ir 3e 05
sceds 3113 3133333333333311133333333
-warmup_stepe 100
-label_smoothing 0.1
G Data Statistics for Dialogue Datasets
-optimizer Adam
The following the hyper-parameters used in full fine-tuning setting on DIALOGSUM.
Please refer to Table
H Examples of Generated Summaries
gpus
steps 1000
Please refer to Figure 4.
-bbachhhsiaa 16
-Ir 3e - 05
-warmup_stern 100
label_smoothin 0.1
-optimizer Adam
The following are the hyper-parameters used in full fine-tuning setting on TWEETSUMM.
gpus
steps 98
batch_size 16
Ir 3e 05
-warmup_steps 10
Llabe_ssmootii 0.1
-optimizer Adam
All models maintain consistent hyper-parameters
across all datasets during inference.
gpus
Match_size 32 use_cache true
-max Jength 100
min_length 5
beam_size 5
-Lenthhppenaaty
-no wew.eeetttngaammmmmeem size
-early_stopping false
F Zero- and Few-Shot Evaluation Details
For zero-shot evaluation, the optimal summary length limit hyper-parameter max_lengin for

---

Dutaset
#Dialogue #Utterance #Turos/dial. #Tokens/dial. #Tokens/turn #Tokens/summ. Scc//Dom.
Open-Domute Dialogwe
Empathetic Dialogues (Rashkin et al., 2019)
22,878
98,759
4.3
86.5
20.1
34,2
Empathetic
Decode (Nie al., 2021)
22.119
181,580
8.2
157.6
19.2
Multiple
Wizard of Wikipedia (Dinan al., 2019)
19,287
174,624
9.1
213.7
23.5
55.8
Wizard
Wizard Internet (Komeili et al., 2022)
9,025
91.042
10.1
225.0
22.3
55.6
Wizard
Topical: Chat (Gopalakkiishmn al., 2019)
6,524
140,544
21.5
507.7
23.6
61.2
Topical
Persoma Chat (Zhang et al., 2018)
5,648
84,366
14,9
239.9
16.1
$5.0
Persona
Blended Skill Talk (Smith al., 2020)
4,662
62.581
13.4
268.1
20.0
53.8
Multiple
ConyA12 (Dinan et al., 2018)
4,554
67,532
14,8
237.8
16.1
$3.9
Persona
MuTual (Cui
2020)
2.719
14,134
5.2
98.7
19.0
38.1
Daily
CMU DoG (Zhou et al,, 2018)
2,647
$8,406
22.1
422.4
19.1
67.9
Movie
DailDialog
2017)
13,891
Daily
DialogRE YYu et al., 2020)
1,578
20,724
13.1
249.6
19.1
49.2
Chit Chat
FriendsQA (Yang
2019)
16,415
16,9
DREAM (Sun et al., 2019)
345
1,706
4.9
74.3
15.2
32,1
Daily
Castomer Service Dialogue
Woz (Lee et al., 2019)
37,860
431,580
11.4
146.0
12.8
45.2
47
Schema (Rastogi al., 2020)
22,559
453,320
16.1
Taskmaster2 (Byme et al., 2019)
17,083
286,930
16.8
264.7
15.8
$9.8
7
MuliWWZ 2.2 (Zang al., 2020)
13.7
20.5
MSR-F2E (1i et al,, 2018)
8,769
68,406
7.8
150.0
19.2
46.9
3
doc2dial (Feng al., 2020)
4,741
56,221
11.9
260.7
21.9
76.9
Table Data statistics for preprocessed, cleaned and annotated dialogue datasets of Ds # denotes average value. Dial. represents dialogue. Summ. represents ChatGPT--amoote summary. Sce. represents scenario of open-domain dialogue. Dom. represents domains of customer service dialogue.
Dialogue
Freddie: you watching on I've just finished Men.
Kelly: I've just finished The Crown.
Freddie: There should be episades caming
Kelly: OMG! cae't wait! :)
Freddie: Try Outlander or The Tudees the meantime.
I'm watching
Bob: Try Grimm or Sabrina. They're both great!
Dialogue
Kelly: They are horrible and disgusting!
they are funny. They't 'r just scriously. X.
Peytom: I been asking you bring that video gume for
Cameron: Heney. I um not having enough time to come home.
Greg: I'm watching House of Cards though admit the
Peytom: When would you come home?
bering.
will
stay
another guess.
Mike: agrro. Nor the same without Kevin Spacey! You should try The
Peytom: Cant you just deliver that game through the couner?:
Wing if you enjoy political
Cameron: Dost be moin i!
Greg: Netllix original Hodyguard supposed to good from what I've
Peytom: Get the job
then.
heard.
Mary:
ashumed cunfess enjoy stuff Pretty Little
ChatGPT (coro-shur)
Gossip Girl ()
Noney: Don't eshamed enjoy
And my favourite
Peyton asks Cameron to bring video game, but Cameron is out of town and unable home. Peylon suggests delivering game through but Cameron refuses. Cameron wall be out town for mother week and Poyton tells him get the job done and come home ASAP.
Desperate Housewives!!!
Mary: know! I watched it swice!
Alan; I'm catching up with
Nancy: OMG! I used to love Friends!
BaRg-large
ChaiGPT (crroshow)
Cameron will have to stay out of town for another week. He will have to deliver the video game Peyton.
shows Netflix, recommend new ones ouch other, and share their opinions on different ganres They mention shows like Mad The Crown, Outlander, The Tudors, Zombic, Grimm, Sabrina, House of Cards, The West Wing. Bodyguard, Pretty Litle Läans, Gessip Girl. Desperate Housewives, and Friends.
group people discuss
(DXISTUP)
Peyton wunts Cumeron to bring her video game. Cameron will be out lown for another week.
BART-large
Ground Truth
Freddic has just finished watching Vid Men. Kelly his just finishel The Crown. Jim is wutching Zombie. watching House of Cands. Alan is catching up with Friends, Nancy's favourite series is Desperate Housewives.
Poyton expecting Cameron to being video game. probably be out for moother week.
MP4 (DAF-TOP)
Freddis, Kelly, Jim, Bob, Greg, Mary and Alan discussing what they wasching Nedlix.
Ground Truth
Freddie, Kally, Jim, Greg, Bob, Mike, Mary, Alan and Noney are w watching different shows Netfli.
Figure Examples of generated summaries.


Reviews:
[{"review_id": "sSv6GhCABT", "reviewer": "Reviewer_k7jr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful and valuable comments. We will explain your concerns and questions as follows:\n\nQ1: The paper introduces novel strategies for multi-scenario multi-domain dialogue summarization, but it's important to acknowledge potential drawbacks primarily centered around the introduced complexities. The incorporation of a multi-stage pre-training strategy and speaker embeddings adds intricate layers to the model architecture. This increased complexity could lead to challenges in implementation, such as longer development cycles, higher resource requirements, and potential difficulties in reproducing results.\n\nA1: We appreciate your concern regarding the complexities introduced by our multi-stage pre-training strategy and speaker embeddings. Here is our response to this:\n\nFirstly, although the model indeed incorporates additional layers and complexities, these were specifically introduced to address the intricacies associated with multi-scenario multi-domain dialogue summarization. The primary goal of this design is to ensure significant and robust performance in dialogues across different scenarios and domains.\n\nSecondly, regarding longer development cycles: We agree that multi-stage pre-training may require additional time. However, it is important to emphasize that once the pre-training is completed, our model can converge on downstream datasets faster than other models and demonstrates a strong performance advantage. Additionally, we will release all the pre-trained models and code. Therefore, for real-world transfer, only minimal time cost and computational resources are required for further fine-tuning, eliminating the need for additional pre-training.\n\nThirdly, as for the higher resource requirements, we agree that the incorporation of speaker embeddings may lead to a slight increase in inference time complexity. However, compared to dialogue summarization models that incorporate more complex architectures (such as role interactions, dialogue rewriting, extra information, and graph structures), this cost is almost negligible. On the other hand, compared to LLMs with parameter sizes ranging from several billion to hundreds of billions, our model only has 0.4 billion parameters, requiring significantly fewer resources for inference.\n\nLastly, about the potential difficulties in reproducing results: We understand the concern, especially given the complexities involved. To address this issue, we will release all pre-trained models, datasets, code, and training procedures in the open-source community, ensuring that researchers and practitioners can reproduce our results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a significant contribution to the field of dialogue summarization. The proposed MP4 pre-trained model, accompanied by the LCM3DS corpus, addresses the challenge of adapting dialogue summarization across diverse scenarios and domains. The novel multi-stage pre-training strategy and integration of speaker embeddings are notable strengths, leading to impressive performance improvements.", "reasons_to_reject": "\nThe paper introduces novel strategies for multi-scenario multi-domain dialogue summarization, but it's important to acknowledge potential drawbacks primarily centered around the introduced complexities. The incorporation of a multi-stage pre-training strategy and speaker embeddings adds intricate layers to the model architecture. This increased complexity could lead to challenges in implementation, such as longer development cycles, higher resource requirements, and potential difficulties in reproducing results.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a novel approach for multi-scenario multi-domain dialogue summarization. It proposes the MP4 pre-trained model, which employs a multi-stage pre-training strategy to bridge the gap between pre-training and fine-tuning objectives. The model's effectiveness is demonstrated on diverse datasets, showcasing significant performance improvements over existing methods in various settings. Additionally, the paper contributes the LCM3DS corpus, a resource containing multi-scenario multi-domain dialogues and their annotated summaries, enhancing the availability of high-quality training data for this task.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Whclj229JC", "reviewer": "Reviewer_SZZj", "summary": "", "questions": "1. What's the main contribution of this work? The distilled dataset?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful and valuable comments. We will explain your concerns and questions as follows:\n\nQ1: What's the main contribution of this work? The distilled dataset?\n\nA1: The main contribution of this work is multifaceted:\n\n  1. Multi-Stage Pre-trained Model (MP4): This model provides both the academic and industrial communities with a pre-trained model that can be rapidly transferred to downstream datasets across various scenarios and domains. On one hand, compared to other dialogue summarization models, it demonstrates outstanding performance and transfer efficiency and integrates both open-domain dialogue summarization and customer service dialogue summarization. On the other hand, in comparison to LLMs, it has only 0.4 billion parameters, significantly reducing computational resource overhead, and also supports further fine-tuning in an open-source manner.\n\n2. High-Quality Large-Scale Dialogue Summarization Corpus (LCM3DS): This corpus encompasses multi-turn dialogues across a wide range of scenarios and domains. To the best of our knowledge, this is currently the largest, open-sourced, uniformly formatted dialogue collection with the most extensive coverage of scenarios and domains. It can promote the development of various dialogue-related tasks (e.g., dialogue selection, response generation, dialogue comprehension, etc.). Additionally, we have annotated the corresponding summaries using ChatGPT, providing invaluable labels for the field of dialogue summarization. This is reflected not only in pre-training but also in potential future evaluations of dialogue summarization.\n\n3. Demonstrated the feasibility of enhancing small pre-trained models through LLMs: We believe the advantage of LLMs lies in their powerful inference capability suitable for various downstream tasks; for handling specific tasks, small-scale models are already sufficiently robust. Thus, we sought to endow the small-scale pre-trained models with the knowledge of LLMs in specific tasks. Extensive experimental results validated the feasibility of this approach.\n\nQ2: Lack of novelty. Since the domain-adaptive post-training has been proposed before, this main contribution of this work is like weak supervision with ChatGPT or knowledge distillation. For the model level, there is not too much innovation.\n\nA2: We understand the reviewer's concerns regarding novelty, but we would like to clarify our primary motivations and several distinctions:\n\n1. We agree that one of the main contributions is to demonstrate the feasibility of enhancing small pre-trained models through LLMs.\n\n2. One of the main focuses of our study is to bridge the gap between pre-training and fine-tuning objectives in dialogue summarization tasks using a multi-stage pre-training strategy. We didn't dedicate much attention to innovations in model architecture, but rather naturally incorporated speaker embeddings for better dialogue representation.\n\n3. Regarding domain-adaptive post-training, our goal at this stage is to get the model to focus on understanding dialogues across different scenarios and domains. We adjusted the masking ratio for the Token Masking and Token Deletion tasks, increasing it from 15% to 20%, because the information density of dialogues is usually lower than that of pure texts. Additionally, our Utterance Permutation task differs from previous related works, as we only shuffled the order of utterances, hoping that the model can accurately align utterances with roles. Moreover, we designed a new Utterance Masking task, where the model needs to reconstruct dialogues with masked Gap-utterances.\n\nQ3: In Section 2, it mentioned that it added roles for the original dialogue datasets. I would wonder if the operation is also the same for those baselines, especially for the previous works illustrated in the Table 2~4.\n\nA3: Yes, all baseline models represent each turn of the input dialogue with the role followed by the utterance, for example: \"Role: Utterance.\" or \"Role<EOR>Utterance<EOU>\". Our MP4 model adopted the second scheme to concatenate the dialogue.\n\nThank you for your comprehensive comments. We will make the necessary updates in the revised version to further clarify our contributions and address the concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The experimental results look competitive.", "reasons_to_reject": "1. Lack of novelty. Since the domain-adaptive post-training has been proposed before, this main contribution of this work is like weak supervision with ChatGPT or knowledge distillation. For the model level, there is not too much innovation.\n2. In Section 2, it mentioned that it added roles for the original dialogue datasets. I would wonder if the operation is also the same for those baselines, especially for the previous works illustrated in the Table 2~4.\n", "questions_to_authors": "1. What's the main contribution of this work? The distilled dataset?", "paper_topic_and_main_contributions": "This paper focused on the dialogue summarization problem. It first collected 20 dialogue datasets and utilized ChatGPT to annotate the datasets automatically. Then it pre-trained and fine-tuned BART by multi-stage pre-training strategy including domain-aware pre-training, task-oriented pre-training and fine-tuning. Experiments on three public datasets (SAMSum, DIALOGSUM, and TWEETSUMM) showed the effectiveness of the proposed approach.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}, {"review_id": "Z8yarWtOZk", "reviewer": "Reviewer_BDoc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful and valuable comments. We will explain your concerns and questions as follows:\n\nQ1: This work relies on ChatGPT for data annotation, which may introduce biases or errors. Human annotation could be more accurate.\n\nA1: We agree that human-annotated summaries might be more accurate, as they are often considered the gold standard. However, we chose to use ChatGPT for data annotation for several reasons:\n\n1. One of our primary goals is to investigate whether by annotating large-scale multi-scenario multi-domain dialogue summarization pre-training corpus with ChatGPT, we can significantly improve the transfer performance of small-scale pre-trained models on multi-scenario multi-domain downstream datasets.\n\n2. Human annotations, especially on a large scale, are typically resource-intensive and time-consuming. Leveraging LLMs can considerably reduce costs. This not only makes the deployment of enhanced small-scale models in practical projects more feasible but also aids in rapid iterations and optimizations.\n\n3. While human annotation might be seen as more accurate, it is susceptible to inconsistencies and subjectivities across different annotators. LLMs, with their consistent annotation approach, can alleviate such variability. Although LLMs might introduce a small amount of potential biases or errors, these can be further reduced as our pre-trained model undergoes further fine-tuning on human-annotated downstream datasets.\n\nWe deeply understand the reviewer's concerns regarding the choice of data annotation method. In this study, we balanced transfer efficiency against accuracy and believe that ChatGPT annotation is an effective solution to achieve our goals. In the future, we aim to explore the integration of human annotations, combining the efficiency and consistency of LLM annotations with the accuracy of human annotations to further optimize model performance and robustness.\n\nQ2: The pre-training strategies are similar to BART, lacking novelty.\n\nA2: We appreciate your insights regarding the similarities between our domain-aware pre-training and BART. Although our method indeed draws inspiration from BART, we would like to clarify the fundamental reasons for our choices and several distinctions:\n\n1. One of the main focuses of our study is to bridge the gap between pre-training and fine-tuning objectives in dialogue summarization tasks using a multi-stage pre-training strategy. During the domain-aware pre-training (DAP) stage, we aim for the model to focus on understanding dialogues across various scenarios and domains, while avoiding learning knowledge specific to particular tasks. In this way, the model can still adapt to other dialogue-related tasks, such as dialogue sentiment analysis, response generation, and dialogue information extraction. Hence, we have adopted more general pre-training tasks similar to BART.\n\n2. We adjusted the masking ratio for the Token Masking and Token Deletion tasks, increasing it from 15% to 20%, because the information density of dialogues is usually lower than that of pure texts. Additionally, our Utterance Permutation task differs from previous related works, as we only shuffled the order of utterances, hoping that the model can accurately align utterances with roles. Moreover, we designed a new Utterance Masking task, where the model needs to reconstruct dialogues with masked Gap-utterances.\n\nThank you again for your constructive comments. We will further emphasize these differences in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes a large-scale multi-scenario multi-domain dialogue data annotated by ChatGPT, indicating a successful application of LLM to enhance small models.\n2. It provides a valuable datasets for pre-training and evaluating dialogue summarization models.\n3. The paper conducts extensive experiments and analyses to prove the effectiveness of the proposed method. The experimental results show the method achieves SOTA results. The zero-shot and few-shot experiments suggest the capability of multiple scenarios and domains adaptation.", "reasons_to_reject": "1. This work relies on ChatGPT for data annotation, which may introduce biases or errors. Human annotation could be more accurate.\n2. The pre-training strategies are similar to BART, lacking novelty.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper focuses on multi-scenario multi-domain dialogue summarization. It proposes a large-scale dataset, a multi-stage pre-training strategy. The experimental results shows the effectiveness of the proposed data and the pre-training method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a novel approach to multi-scenario multi-domain dialogue summarization, introducing the MP4 pre-trained model with a multi-stage pre-training strategy to enhance pre-training and fine-tuning alignment. Furthermore, the paper contributes the LCM3DS corpus, featuring multi-scenario multi-domain dialogues and annotated summaries, thereby enriching training data quality for this task. The study encompasses the collection of 20 dialogue datasets, automatic annotation using ChatGPT, and multi-stage pre-training involving domain-aware pre-training, task-oriented pre-training, and fine-tuning of BART. Experimental results on public datasets (SAMSUM, DIALOGSUM, and TWEETSUMM) confirm the effectiveness of this proposed approach, emphasizing the importance of both the data and pre-training methodology. However, as the reviewers pointed out, the novelty contributions were insignificant since several previous papers have explored domain-adaptive post-training.
Decision: Accept-Findings