Re$^3$Dial: Retrieve, Reorganize and Rescale Conversations for Long-Turn Open-Domain Dialogue Pre-training
Authors: Jiaxin Wen
Hao Zhou
Jian Guan
Jie Zhou
Minlie Huang
Venue: EMNLP 2023 Main (2023)
Keywords: Dialogue
Large-scale Pre-training
Pre-training data
Multi-turn Dialogue

Abstract: Pre-training on large-scale open-domain dialogue data can substantially improve the performance of dialogue models. However, the pre-trained dialogue model's ability to utilize long-range context is limited due to the scarcity of long-turn dialogue sessions. Most dialogues in existing pre-training corpora contain fewer than three turns of dialogue. To alleviate this issue, we propose the Retrieve, Reorganize and Rescale framework (Re$^3$Dial), which can automatically construct billion-scale long-turn dialogues by reorganizing existing short-turn ones. Given a short-turn session, Re$^3$Dial first employs a session retriever to retrieve coherent consecutive sessions. To this end, we train the retriever to capture semantic and discourse relations within multi-turn dialogues through contrastive training. Next, Re$^3$Dial samples a session from retrieved results following a diversity sampling strategy, which is designed to penalize repetitive or generic sessions. A longer session is then derived by concatenating the original session and the sampled session. By repeating the above process, Re$^3$Dial can yield a coherent long-turn dialogue. Extensive experiments on multiple multi-turn dialogue benchmarks demonstrate that Re$^3$Dial significantly improves the dialogue model's ability to utilize long-range context and thus generate more sensible and informative responses. Finally, we build a toolkit for efficiently rescaling conversations with Re$^3$Dial, which enables us to construct a corpus containing 1B Chinese dialogue sessions with 11.3 turns on average (5X longer than the original corpus). We will release our retriever model, toolkit, and data for public use.

Dial: Retrieve, Reorganize and Rescale Conversations for Long-Turn Open-Domain Dialogue Pre-training
Jiaxin Wen 1.2, Hao Zhou', .Jian Gu Jie Zhou', Minlie Huangl.21
"The CoAI group, Tsinghua University, Beijing, China Department of Computer Science and Technology, Tsinghua University, Beijing, China "Pattern Recognition Center, WeChat AI, Tencent Inc., China wenjx22@mails. tsinghua. edu. cn, aihuangêtsinghuu. edu. cn
Abstract
DelDialaB
Pre-training on large-scale open-domain dia- logue data substantially improve the per- formance of dialogue models. However, the pre-trained dialogue model's ability utilize Long-range context is limited due to the scarcity long-turn dialogue sessions. Most dialogues in existing pre-training corpora contain fewer than three turns of dialogue. To alleviate this issue, we propose the Retrieve, Reorganize and Rescale framework (Re" DRelD,al), which can automatically construct billion-scale long-turn dialogues by reorganizing existing short-ture turn ones. Given short-turn session, Dial first employs a session retriever to retrieve coher- ent consecutive sessions. To this end, train the retriever to capture semantic and discourse relations within multi-turn dialogues through contrastive training. Next, Dial samples session from retrieved results following a di- versity sampling strategy, which is designed penalize repetitive or generic sessions. A longer session is then derived by concatenating the original session and the sampled session. By repeating the above process, Dial can yield a coherent long-ture turn dialogue. Extensive experiments on multiple multi-turn dialogue benchmarks demonstrate that Dial signif- icantly improves the dialogue model's s ability to utilize long-range context and thus generate more sensible and informative responses. Fi- nally, we build a toolkit for efficiently rescaling conversations with Dial, which enables us to construct a corpus containing IB Chinese di- alogue sessions with 1 11.3 turns average (5x longer than the original corpus). Our retriever model, code, and data is publicly available at https://git thub. cwm/thu wathucooii/ee aiaal.
Rirnder
Open- Demnis Dialogue Pre-iraining Corgus
(A)
acdik Ra # 30H fe i et Ro &f 72
Parsors really expect too murc froms thair childives to this soclety, and surh childden sours mentally unbealthy even they achieve succnas futtore.
1u3 sote yrt is mars gase
stuct norking hard from a
Pareott want to childeen to an Tuy Leugue School, but they just spead play
Haueally spenkingg it laad to pregrne practioe what you preach for educating childrens
-ittx xt CD s " wn cr jatt 838 22 Au 919p7
-Whee panennt ger mad, they should ank thems rovo quostiones i. Can what unk your child when you this aget
ugree. You cun't blane others but yourself for your child's
(b)
Figure I: (a) Statistics of several open-domain dialogue pre-training corpora, none of which has more than 3 turss average. Taking EVA can construct new corpus with 1B sessions and 11.3 turns on average, which 5x longer than that of the EVA corpus. (b) An excerpt of the automatically con- structed long-turn turn dialogue by Re*Dial. More detailed examples are presented in Appendix G.
example, Rea
standing goals in Al. Recently, variety of large- scale open-domain pre-trained dialogue models have dramatically promoted this progress (Roller et al., 2020; Zhou et al., 2021; Shuster et al., 2022b). And a critical ingredient to the success these models is the pre-training dialogue corpus. How- ever, while existing dialogue pre-training corpus collects millions to billions of dialogues from pub- lic social media, e.g., Reddit for English (Roller et al., 2020) and Weibo for Chinese (Zhou et al., 2021), long-turn dialogues are highly scarce. More specifically, based on the publicly reported data statistics shown in Figure (a)), most dialogues existing pre-training corpora only have less than three turns. The lack of large-scale long-tun turn di-
1 Introduction
Building intelligent open-domain dialogue systems that can generate coherent and engaging multi-tum dialogues with humans has been one of the long-
Corresponding author

---

alogue data restricts dialogue models from deriv- ing more advanced abilities to utilize long-range context for modeling multi-turn dialogues during pre-training (Xu et al., 2021, 2022b).
responses in multi-turn dialogue. Finally, we de- velop a toolkit for efficiently rescaling conversa- tions with Dial, which enables us to construct a corpus containing 1B Chinese dialogue sessions with 1 11.3 turns on average (5x longer than that of the original EVA corpus). We will make our re- triever model, toolkit, and data public. We believe our work provides new opportunities in long-ture dialogue pre-training the research community.
In this paper, we focus on answering the follow- ing research question:
Can we automatically build a billion- scale long-turn dialogue corpus by reor- ganizing existing short-turn dialogues?
Our contributions can be summarized as follows:
Our basic idea is to construct a long-turn dialogue via recursively retrieving and selecting one consec- utive session from the existing dialogue corpus. De- spite the simplieity of this idea, we still face several challenges to make the constructed corpus effec- tive in enhancing long-turn dialogue pre-training. First, the selected scssion should be coherent with the query session. Otherwise, it will introduce noisy utterances without long-range dependency or break the conversation flow (Liu et al., 2021), which may impact the performance of dialogue models. Second, our in-depth analysis reveals that the retrieved sessions tend be biased to be rel- evant but semantically repetitive with the query or overly generic (e.g.. "A: Haha, it's so cute, B: Haha! IMAO.") due to both the data bias in the dia- logue corpus (Zhou et al., 2021; Lee et al., 2021; Li et al., 2015; Liu et al., 2018) and the model bias of the retriever (Thakur et al., 2021). These biases sig- nificantly lower the diversity and informativeness of the reorganized long-turn dialogues.
We introduce Re which presents novel perspective to alleviate the scarcity of long- turn conversations by automatically building a billion-scale long-turn dialogue corpus via concatenating existing short-turn dialogue.
We propose to train a dense session retriever on massive unlabeled plain dialogue data with contrastive learning to capture the global se- mantic and discourse relations within multi- turn dialogues. We also propose the diver- sity sampling strategy improve the diversity and informativeness of the automatically con- structed corpus.
Experiments three Chinese multi-turn dia- logue benchmarks demonstrate that Dial enhance the model's ability to model long- range context, thereby leading to consistent and significant improvements in different pre- training settings.
We release Rea which contains 1B Chinese dialogue with 1 11.3 turns on average (5x longer than that of the original EVA cor- pus).
To tackle the above challenges, we propose the Retrieve, Reorganize and Rescale framework DRelDial), which employs an Unsupervised Dense Session Retriever (UDSR) to retrieve coherent short-turn dialogues and reorganize them into a long-turn one. We train UDSR through contrastive learning by taking consecutive dialogue segments from the same dialogue positive pairs and those from different dialogues negative pairs. To avoid overly retrieving semantically repetitive or generic sessions, we propose a diversity sampling strategy, effectively improving the diversity and informative- ness of the reorganized long-turn dialogues. Figure ((b) shows an example of the automatically con- structed long-turn dialogue using DeaDi
Related Work
2.1 Large-Scale Open-Domain Dialogue Pre-training
In the past few years, large-scale pre- pre-training has greatly promoted the progress of the NLP commu- nity (Brown et al., 2020). Recently, large-scale pre-training has also become the mainstream ap- proach to building open-domain dialogue models, both in English (Zhang et al., 2019; Roller et al., 2020; Thoppilan et al., 2022) and Chinese (Bao et al., 2020; Zhou et al., 2021; Gu et al., 2022; Wen et al., 2022). Through pre- pre-training massive dia- logue data crawled from public social media. these models exhibit strong conversational ability, signif- icantly outperforming traditional non-pre-trained dialogue models. However, the scarcity of long- turn dialogues in the pre-training corpus hinders
We verify the effectiveness of DDil on three Chinese multi-turn open-domain dialogue bench- marks. Extensive experiments demonstrate that Re3 Dial consistently and significantly enhances the dialogue model's ability to utilize long-range context, leading to more sensible and informative

---

these models from deriving a better ability to uti- lize long-range context for modeling multi-turn dialogues during pre-training. To alleviate this is- sue, we study how to automatically and efficiently build a large-scale long-turn dialogue corpus based on the existing short-turn dialogue corpus.
3.1 Retrieve
Task Definition We define the dialogue session retrieval task as follows. Given a dialogue ses- @Q sion with [Q] utterances Sq = {ug.... uy }. our goal is to choose a |C]-turn dialogue ses- |cl)
sion S. {ud ... ,uee } from a dialogue corpus that should be coherent with Sq- Consequently, S @ Se (ug.... ,uq natural dialogue session of [Q| + |C| turns.
2.2 Retrieval-Augmenter Language Model
[Q] Uer .ue _ can make
Extending neural language models with a retrieval system has been widely studied in various NLP tasks, such as language modeling (Khandelwal et ul., 2019), story generation (Zhang al., 2022), and open-domain QA (Lewis et al., 2020; Izacard and Grave, 2020). The integration of retrieval tech- niques in open-domain dialogue systems also has a long history. Retrieval-based dialogue systems di- rectly return a response via retrieving from a large dialogue corpus (Ji et al., 2014; Zhou et al., 201 8). Moreover, recent works investigate how to gener- ate more accurate responses via retrieving from external knowledge sources (Komeili et al., 2021; Shuster et al., 2022a). While we also leverage a re- trieval system in open-domain dialogue, our work is significantly distinguished from these mainly in that: (1) We aim to enhance the model's ability to utilize long-range context for modeling multi- turn dialogue rather than focusing improving (2) We leverage the retrieval system only for constructing a long- turn dialogue corpus, which is disentangled from the training and inference stages dialogue mod- els. Consequently, our approach does not introduce additional training costs or inference latency.
We observe that previous retrievers either rely solely on local term matching and fail to capture global semantic relations (e.g.. BM25 (Robertson et al., 2009)) or struggle to capture discourse coher- ence within multi-turn dialogues (Liu et al., 2021) (e.g., Contriever (Izacard et al., 2022)). Therefore, these retrievers exhibit unsatisfactory performance in our dialogue session retrieval task. To remedy these problems. we train an Unsupervised Dense Session Retriever (UDSR). By using consecutive dialogue segments from the same dialogue as pos- itive pairs and those from different dialogues as negative pairs, UDSR demonstrates superior capa- bility in capturing global semantic relevance and discourse coherence within multi turn dialogues.
Model Structure Given two dialogue sessions S and S., we encode them using two encoder models, E, and E.. The similarity score is de- fined as the dot product of their representations:
factuality or directly responding.
Contrastive Training We adopt contrastive train- ing to train the dialogue session encoder E,y and E... For each training instance (Sq. S , S J. which contains one query session Sq, one coherent session St and n incoherent sessions { j==..... we optimize the contrastive loss L as follows:
3 Methodology
An overview of Dial is shown in Figure 2. Let
D = {Sa}} be the original dialogue pre-training cor-
pus. For each session S,, Re Diil constructs long- turn dialogues automatically in four steps: (1) Ini- tialize the constructed session Sout with Sü: Sout = S,. (2) Use UDSR to retrieve top-K coherent ses- sions from D: UDSR(S,1) ') = { .a............. 5x) (3) Use diversity sampling to sample consecutive session Se.. (4) Update Sout and obtain a longer session by Sout Sout @ Se.- Let Set be Sy. Go to step 2 until Sout has been updated for L times. L is a hyperparameter to control the number of turns of the constructed dialogue.
+ Σ sim( (5,
C = --loo- esim(S
Positive and Negative Pairs Large-scale posi- tive and negative pairs are crucial for the effec- tiveness of contrastive learning (Izacard et al., 2022). However, considering that there is no available labeled data for this task, we propose to build positive and negative pairs from unla- beled plain dialogues, which are much easier to access. Let {Su,} be an unlabeled dialogue cor- pus, where Sa, = [$1. ... SKK] "
We use the last session S; instead of the full context Sout for retrieval since the representation of Sout needs to be dynamically computed during the iterative retrieval process, leading to significant increase in time cost. More analysis
is a Ky-turn
of the retrieval performance with varying context lengths is illustrated in Appendix E.4.

---

Sampling weight: w/ x
Session
Querry
Open Open-Domain Dialogue Curpun
Large-Srale
06
be beppyyyleeelteeee
Heeelll spoding binkerd
Reteleve
Session
Dialogue- Level seright
Sumplled Connt
Curput- Level Weight
odnesting diibio.
Lesges Sched,, teether
(4) Concatenate
is partly repoaddilly parette- Utalogac Weightt
dntrperntinn so paateephoeene.. Repecitive
Seasion
-6.051
uitles pereue art
Select
-Whea purosteppt mud, dey shoud an Boeadhne quation adwer youwel vig.
<2 m
=2
12 -0.00
4i wimeercllll to too were
1: tapus tareerr
Jusns t'araaa delann respasillly.
68 -11
12 12-100
-0.006
Low Cooprasleve Wetghn
Constructed Long-turn Dialogue
(b)
Figure 2: Overview of Re Dial. (a) Constructing multi-tun turn dialogues via recursively leveraging UDSR to retrieve consecutive sessions from large-scale open-domain dialogue corpus. (b) The proposed diversity sampling strategy assigns each retrieved session a sampling weight, which is a combination of dialogue-Ievel weight and corpus-level weight, aiming to avoid overly repetitive or generic retrieved sessions.
session. We first divide each Sa, into two con- secutive segments Sq. = { ... S... f and Sa, = ( 11 Ski ). where M, is randomly chosen from [2, K; - 2]. We then obtain positive pair (S. Sc,) if = j. Moreover, we consider two kinds of negatives: (1) Easy Negatives: given Sq. we randomly select consecutive session Sty where i + j. (2) Hard Negatives: given Sq. we leverage the top-K sessions retrieved by BM25 as Se,, thereby improving the model's 's ability to differ- entiate those incoherent negatives that have lexical overlaps (Huang et al., 2020).
Dialgoue-Level Diversity Sampling There are widely duplicate substrings in the original dialogue pre-training corpus because: (1) Large-scale open- domain dialogues are mainly collected from pub- lic social media (Roller et al., 2020; Zhou et al., 2021) as follows. Given one post P and multi- ple comments {C,], we derive multiple sessions {(P, C.)3 which share the same prefix P. (2) There are generally duplicate contents in web-crawled datasets. For example, Lee et al. (2021) find that web-crawled datasets contain between 3.04% (on C4) to 13.63% (on RealNews) duplicate substrings. Morcover, such dataset bias in the original dialogue pre-training corpus will be amplified due to the model bias of the retriever since higher lexical overlap generally leads to a higher similarity score, whether in sparse or dense latent space.
3.2 Reorganize
After building the retriever, we can then reorga- nize the existing short-turn corpus into long-turn corpus by recursively retrieving and selecting the consecutive session Sc..
Then, the cross-sample duplicates would become in-context duplicates in the concatenated dialogue, e.g., (P, C2 P, P2C.) We conjecture that such in- context duplication could bias the model towards simply copying context for response generation. To remedy this problem, we introduce the dialogue- level weight S., where set dh, 0 otherwise) if it meets any of the two requirements: (1) Any utterance in Sh. exactly matched with Sout- (2) The longest common substring (LCS) between Sout and Sk o contains more than N words.
In this section, we first provide an in-depth anal- ysis to reveal that the selected session tends to be biased towards relevant but repetitive or plausible but generic. These biases can be attributed to both the dataset bias of the original corpus and the model bias of the retriever. As shown in Section 4.4, these biases lead reduced diversity and informative- ness of the constructed corpus, finally resulting in decreased model performance. To remedy these problems, we introduce the diversity sampling strat- egy at both dialogue-Ieve and corpus-level. For- mally, for each retrieved session sh we derive its sampling weight wh as follows:
Corpus-Level Diversity Sampling A lot of generic meaningless utterances exist in the origi- nal dialogue corpus, e.g.. "A: Haha, it's so cute. B: Haha! LMAO." (Li et al., 2015; Liu al., 2018). Due to their high frequency and compatibility with various dialogue contexts, the retriever is prone to select these plausible but generic dialogues as
dh P.
where df, is a binary dialogue-level weight, and pt: is a numeric corpus-level weight. We then adopt weighted sampling to select Se,-

---

Prestrininng Setting
Protrainul Model
Model Architectrre
Model
we further pre-train an existing pre-trained general language model. (3) Further Pre-training on DM, where we further pre-train an existing pre-trained dialogue model. Table shows the detailed back- bone model information for each setting.
Pte-trining Irom Scratch
non causal decoder
oB
Purther Pee craining on LM Further Pre-trannnng DM
GF12 small ChaGGLM
causel decoder
100M
docoder-deooder
Table I: Backbone model information for each pre- training setting.
Pre-training We extract subset of the EVA pre- training corpus as the original corpus, which con- tains Million dialogue sessions. For Dial, we set L=5, t=p-K=5, and the maximum LCS length N=10. The average number of turns in the original corpus is 2.2, while for the Re Dial-constructed corpus, it significantly increases to 1 1.6. We set the maximum sequence length to 256. For pre-training from scratch, we set the batch size to 512 and the pre-traimin traininiing steps to 10K. For further pre-training the batch size to 256 or 128 and the pre- training steps to 30K. We pre-train the model with the auto-regressive language modeling task. More training details are shown in Appendix B.
consecutive sessions. Consequently, these generic sessions are more frequently sampled than more contentful and specific sessions at the corpus-level during reorganizing, which will decrease the infor- mativeness and diversity of the final corpus, ag- gravating the problem of generic replies generated by dialogue models. To remedy this problem, introduce the corpus-level weight pč, to penalize session for being repeatedly sampled:
Per
+1
where &. is the sampled times of Sk
Benchmarks We conduct evaluations on three widely-adopted Chinese open-domain multi-tum dialogue benchmarks, including KdConv (Zhou et al., 2020). DuLeMon (Xu al., 2022b), and NaturalConv (Wang et al., 2021), each has 16-20 turns on average. Data statistics are shown in Table 9.
3.3 Rescale
We finally build a toolkit for efficiently rescaling dialogue corpus with Dial. We speed up re- trieving with FAISS (Johnson et al., 2019), which achieves 192 searching per second single V100 GPU. Furthermore, we support parallel searching over multi-GPUs, which can achieve 1,536 search- ing per second with 8 V100 GPUs, for example. We also leverage PyArrow2to speed up the process- ing of big data. In practice, we show the efficiency of our toolkit by constructing Re Deil-BB in 4.7.
Metries We adopt the following automatic met- rics for evaluation. PPI wero-shot measures the per- plexity on the test without fine-tuning on the downstream training sets. PPL measures the per- plexity on the test set after iine-tuning. BLEU-N measures the precision of the n-gram overlap be- tween generated and ground-truth responses (Pa- pineni et al., 2002) after fine-tuning. ROUGE-L measures the recall of the n-gram overlap between generated and ground-trutt responses (Lin, 2004) after fine-tuning. Distinct-N measures the percent- age of the unique n n-grams over all the generated n-grams after fine-tuning (Li et al., 2015).
4 Experiment
4.1 Retriever
We train UDSR on a subset of the EVA pre- training corpus (Zhou et al., 2021), which con- tains 102,000049,000011000
examples for the train/validation/eest split. More details of data pro- cessing are provided in Appendix A. We adopt BERT-base (Devlin al., 2018) the encoder backbone. The parameters of Eg and E. are not shared according to our preliminary experiments.
4.3 Main Results
4.3.1
Automatic Evaluation
Table shows the automatic evaluation results. In the zero-shot setting, Real consistently outper- forms the original baseline by a large margin in PPLzero-shot on three benchmarks across different pre-training settings For instance, in the last block. the Re'DDal-traiine model achieves PPL of
4.2 Dialogue Model
Settings We consider three general scenarios where Dial can be utilized for long-turn dialogue pre-training: (1) Pre-training From Scratch, where we pre-train a dialogue model from scratch. (2) Further Pre-training on LM, where
*We also present zero-shot experiment results on English henchmarks in Appendix D.
Chttps://githhin http:///ittuuu.cccomaaaaeae www.apaaacaaarr...ooooooooo

---

Benchmark
Pre-training Data
PPL
BLEU-1
BLEU-2
ROUGE-L
Distinct-2
Pre-tmining From Scnotch
DuLeMon
Original
106,70
61.56
20.74
9.20
17:75
7.87
Rs 'Dial
83.10
$6.09
21.43
9.66
18.75
8.32
KdConv
Original
309.82
42.95
23.83
13.25
21.28
6.601
Realial
14.36
22.95
Original Re"Dial
164.00
22.94
NaturalConv
124.80
57.28
22.14
10.39
23.35
7.98
Further Pre trannnn on LM
Dul eeoo
Original
12.95
10.07
15.10
8.05
19.06
13.78
Re Dial
11.94
9.94
15.34
8.26
19.29
14.00
KdConv
Original
12.13
5.94
22.16
14.71
27.20
9.27
Re"Dial
11.56
15.41
Original Re"Dial
14.11
10.56
16.48
21.89
13.94
NaturalComv
13.26
10.26
17.38
9.04
21.91
14.54
Further Pie training on DM
DuLeMon
Original
48.79
29.77
16.68
7.38
14.80
21 06
Re" Dial
46.25
29.27
17.12
7.55
15.07
22.28
KdConv
Original
60.08
10.87
21.90
13.71
21.38
16.83
Re"Dial
48.58
22.79
14.45
22.09
16.93
NaturalConv
Original
69.92
30.52
8.75
23.45
RePDial
67.78
29.20
18.54
9.10
18.59
24.24
Table 2: Automatic evaluation results. The best performance is highlighted in bold. Note that perplexity is not comparable across different settings since the backbone model uses different vocabulary.
46.25 on DuLeMon, compared to the original base- line's performance of 79. This indicates better ability in multi-turn dialogue modeling. Moreover, beyond benefiting zero-shot performance, Dial can also significantly improve the model's 's perfor- mance after fine-tuning on sizable crowdsourcing high-quality long-turn datasets. Specifically, the Dial--rrradd model achieves better perplexity. BLEU, and ROUGE scores, while showing an im- proved or comparable generation diversity. In sum- mary, these results demonstrate that Re pro- vides well-generale geleraeenrrl data foundation in the of large-scale dialogue pre-training.
40
Original Dkul
25-
20
30
10
Langruum
Shatltar
Lang-urr
Figure 3: Pair-wise human evaluation results of the fur- ther pre-trained dialogue model. We report win rate of each model two test with different context lengths. We use Fleiss' kappa (Fleiss, 1971) to measure inter-annotator agreement (all are moderate agree- ment with 0.4 < 0.6).
4.3.2 Human Evaluation
We conduct a pair-wise human evaluation study the models' performance when provided with di- alogue contexts different lengths. We first ran- domly sample 100 long-turn contexts (consisting of at least six turns) from DuLeMon as the Long- turn test set. We then extract the last utterances from these contexts to form the Short-turn test set. We hence obtain 400 generated responses from the two models. For each pair of responses (one by the Dial--atrad model and the other by the Original-trainee model), three annotators are hired to give preference in sensibleness and informativeness, respectively. Sensibleness mea-
sures whether the response is relevant and consis- tent with the context. Informativeness measures whether the response is informative given the con- text. We adopt majority voting to make final deci- sions among three annotators. As illustrated in Fig- ure 3, the Re'Dial-trainn -trineed model outperforms the baseline in both sensibleness and informativeness This ver-
by large margin on the long-turn ifies that R improves the dialogue model's ability to effectively utilize long-range context to generate more sensible and informative responses.

---

Pre-truining From Scratch
Further Pr-training LM
Further Pre-training DM
Retriever
PPLamsb
BLEU-1
BLEU-2
BLEU-1
BLEU-2
PPL
BLEU-1
BLEU--2
Original
193.51
21.81
10.74
13.06
17.91
10.41
$9.60
18.82
9.98
Random
170.58
10.72
14.92
18.39
10.67
19.05
10.08
BM25
192.94
20.51
10.14
14.69
18.14
10.58
73.43
19.48
10.32
Contriever
154.65
21.97
10.99
13.41
18.62
10.78
61.48
19.34
10.23
Re"iial
135.88
22.74
11.47
12.25
18.70
10.90
54.54
19.48
10.37
Table 3: Comparison of different retrievers. We report the average metric over three benchmarks. Cells are
blue/orange if the constructed long-turn dialogue data increuses/deercases the performance compared the original baseline, respectively.
Aspects
Irrelevance
Local Relevance
Discourse Incoberence
Variants
PPL.
Overlap
Repeat Sampling.
BM25
Contriever
72.10
66.70
61.20
52.50
Re RePDial
49.35
0.17
650.70 1 217.33 33
who dialogue
50.30
0.22
656. 10 64
Dial
97.90
94,90
68.80
who corpus 51.19
0.20
1,609.91 91
Table 4:
Accuracy of discriminating the positive re- trieved session from the incoherent negative session.
Table 5: Effect of the diversity sampling strategy. We report the average PPI. on three benchmarks. Overlap and Repeat Sampling are defined Appendix
4.4 Analysis
locally relevant negative pairs, indicating their perior ability to capture global semantic relevance. (2) Both BM25 and Contriever struggle to capture discourse coherence, showing near-random perfor- mance. (3) UDSR outperforms baselines by a large margin in capturing both global relevance and dis- course coherence, verifying the effectiveness of our training data construction strategy.
Effect of Retriever We compare different ap- proaches to retrieve dialogue sessions and evaluate the final dialogue model performance. We try Ran- dom sampling. a term-based retriever BM25, and a statc-of-the-an dense retriever Contriever. Table 3 presents the results. All baselines bring fewer improvements or even inversely hurt model per- formance, especially zero-shot performance in the further pre-training setting. In contrast, using the retriever in R Dial achieves consistent and signif- icant improvements across different benchmarks and pre-training settings.
Overall, these results indicate that automati- cally building long-turn dialogues to enhance pre- training is non-trivial. Simply improving dialogue turns is insufficient. herent sessions based on both global semantic rel- evance and discourse coherence within multi-turn dialogues rather than relying solely on word over- lap or semantic similarity. Otherwise, it will intro- duce unexpected noise or biases and lead to slightly improved or even decreased model performance.
important to retrieve co-
To gain a deeper understanding of the effective- ness of different retrievers in capturing global se- mantic and discourse relations within multi-turn dialogues, we propose to evaluate the retriever us- ing individual tests in different aspects (Ribeiro et al., 2020). To this end, we first construct positive pairs following the strategy illustrated in Section 3.1 and introduce perturbations create negative pairs. We then compute the retriever's accuracy discriminating between positive and negative pairs, expecting it assigns a higher score to posi- tive pairs. Our evaluation focuses on three aspects: Inrelevacce, Local Relevance, and Discourse Inco- herence. For example, to create a locally relevant negative pair, keep one utterance from the pos- itive session unchanged while replacing the other utterances with randomly sampled session. More details can be found in Appendix E. The results shown in Table 4 reveal that: (1) Dense retrievers demonstrate better performance in discriminating
Effect of Diversity Sampling To further inves- tigate the influence of the proposed diversity sam- pling strategy in Dial, we conduct an ablation study. As shown in Table 5, the dialogue-level and corpus-level weights reduce the bias towards repet- itive and generic sessions and improve the diversity and the informativeness of the constructed corpus expected. Finally, both of them contribute to the pre-trained dialogue model's performance.
Utilizing Long-range Context To manifest the benefits Re Diali we visualize the distribution of PPLzero-shot on samples with varying numbers of dialogue context turns. Specifically, we first

---

Orgiaal 89GS
Contreww
utterances from 1S1,S2, ,S 1). We try two utterance retriever models: (1) Contriever: It is a state-of-the-art dense retriever model. (2) Sentence- BERT (Reimers and Gurevych, 2019): It is an en- coder model fine-tuned for sentence similarity. We set K = 2 in our experiments.
18.0
10.
12.0
17.09
17:03
Summarization-based Context Compression We introduce an additional summarization model to summarize long-term context into short sentences. We try two summarization models: (I) Pegasus- 523M (Zhang et al., 2020): It is widely encoder-decoder model specifically pre-trained and fine-tuned for text summarization. 66B (Zeng et al., 2022): is a widely-adopted instruction-tunell large language model.
Plamaa Conteet Sime
THumber Contoct Time
#Muntber Conteet Time
Figure 4: We report the PP shot varying with the
number of context turns. Re achieves significantly lower perplexity when given longer contexts.
(2) ChatGLM-
select sessions from the original test set that con- tain at least turns. We then truncate their con- texts into different turns and compute the perplex- ity. The results shown in Figure 4 reveal that: (1) In comparison to the Priginal-trainee model, the Re Dial-rrieed model achieves significantly lower perplexity as the length of context increases, No- tably, when evaluating DuLeMon, benchmark specifically designed evaluate the modeling of long-range dialogue history, the perplexity of the Original-trained dialogue model quickly stops de- creasing after giving more than four turns of con- text. This indicates that pre-training on a long-turn scarce corpus restricts the model's utilization of long-range context. And Dial can effectively remedy this problem. (2) Although other retrieval baselines also exhibit sharper decreasing trend in perplexity compared to the Original-traine trained model, they generally yield higher perplexity. This implies that while these baselines enhance the utilization of long-range context, they capture fewer long-range dependencies compared to Re Re' Dial and may even exhibit inferior performance when the local context is more effectively utilized.
We report the average PPIzero-shot over three multi-turn dialogue benchmarks. From the results shown in Table 6, we observe that Dial signif- icantly outperforms all baselines in long-turn dia- logue benchmarks. Moreover, augmenting the dia- logue model with a context summarization model or a retriever shows less improvement or inversely hurts model performance in several cases.
On the one hand, the two-stage framework suf- fers from error propagation due to the introduced summarization model the retriever. For ex- ample, both the summarization model and the retriever may lose important information in the original context. Moreover, the summarization model could also suffer from hallucination prob- lems (Maynez et al., 2020), thereby introducing new noises. In contrast, DDil keeps the origi- nal long-tum turn context unchanged and thus does not noises.
lead information loss introduce On the other hand, we conjecture that augmenting dialogue models with the context summarization model requires further training on summarization- based dialogue datasets (Xu et al., 2022a). In con- trast, & does not require collecting additional training datasets and greatly improves the model performance.
4.5 Comparing with Context Compression Methods
While Dial aims to construct long-turn dia- logue pre-training corpus to enhance the utiliza- tion of long-range context, there is another line of work that focuses on compressing long contexts into short contexts. We hence additionally conduct experiments on retrieval-bacee baseline and a summarization-basel baseline for long-term con- text modeling and compare them with Re
4.6 Case Study
As shown in Table 7, the Original-traines model mainly focuses on local context and tends to gen- crate more generic responses (c.g., "I think the same" in responding the preceding utterance, "they thought it was too risky"). In contrast, the Re Dial-trained dialogue model generates words related to the long-range context (c.g., "fashion de- signer" which has been mentioned nine turns prior),
Retrieval-based Context Compression Given an original context S = {Si, S2, ... Sy), we use SN as the query to retrieve the top-K K most relevant

---

Method
Pre-training From Scratch
Further Pre-training LM
Further Pre-training DM
Original
193.51
13.06
59.60
Retrieval (Sentence-BERT)
169.24 669.66
13.87
Retrieval (Contriever)
15
13.69
64.03
Summarization (Pegasus-523M) Summarization (ChatGLM-666B)
68.16
182.52
13.79
63.48
De'Dial
135.88
12.25
54.54
Table Comparison between Dial and retrieval-based and summarization--aase context compression methods, We report the average PPL over three benchmarks. Cells are bluc/orange if the method increases/decreeses the performance compared to the original baseline, respectively.
Dialogue Context:
enhancing the model's ability utilize long-range context from data-centric perspective. Rea leverages dense retriever trained on massive unla- beled dialogues to improve the coherence of con- catenated sessions. Furthermore, a diversity sam- pling strategy is proposed to penalize repetitive or generic sessions, improving the informativeness and diversity of the constructed corpus. Exten- sive experiments demonstrate that Rea signif- icantly improves the model's performance var- ious multi-turn dialogue benchmarks across dif- ferent pre-training settings due to the better uti- lization of long-range contexts. Finally, we pro- vide a toolkit for efficiently rescaling conversations with Rial and successfully build Re 1B, a large-scale long-turn dialogue corpus that contains B Chinese dialogues with 11.3 turns on average. Our work provides new data foundation for build- ing large-scale pre- pre-trained dialogue models.
A: You're about graduate from the Central Academy of Fine Arts, aren't you?
B: Yes. I'll be 4th year undergraduate.
A: What do you want do after graduating?
B: My dream is to be fashion designer that's definitely
what I'll be doing after graduating.
A: Being a designer is difficult because requires inspiration.
B: know it is difficult. I could get inspiration from Dunhuang
murals or ethnic costumes.
A: So do you want to design Chinese-style clothes?
B: Yes, I think this kind of clothes look more mysterious and
have a sense of history.
A: So are you planning to open studio?
B: I've always had this idca, but I don't 't have enough money.
A: Why don't you find a partner?
B: I tried, but my classmates they didn't 't want to do it, they thought it was too risky.
Original: Well, actually think the same.
Dial: think would good idea find another experienced fashion designer.. which will help you achieve your dream.
Table 7: Generated responses from the model pre- trained on Dial and Original corpus (translated from Chinese to English). We highlight the generated spans that are related to long-range context.
Limitations
Although we have already verified the effective- ness of Rea using UDSR, there are still several directions to further improve the retricval perfor- mance. For instance, while we explore using the BM25 hard negatives in our experiments, there are more advanced negative sampling strategies (Xiong et al., 2020). We will explore these directions and further improve UDSR.
leading to a more sensible and specific response.
4.7 Constructing Dial-BB
To show the efficiency of constructing large-scale long-turn dialogue data with Dial and allow researchers to explore Dial easily, we finally release D an improved corpus based on the original EVA corpus that contains IB sessions with 11.3 turns average (5x longer than that of the original EVA corpus). The whole pipeline costs about five days with 32 V100 32G GPUs.
Besides, despite that R can be adapted any open-domain dialogue corpus in any language, we currently only conduct experiments based on a Chinese open-domain dialogue corpus. It is nec- essary to further collect other language dialogue corpus, such as the English dialogue data from Reddit, and verify the effectiveness of Dial.
5 Conclusion
Moreover, we note that our work just makes the first step to automatically construct a long-turn dia- logue corpus for enhancing long-turn dialogue pre- training. Based on the Re Dial framework, future
This paper presents Dial, a framework that au- tomatically builds cillion-scale long-tun turn dialogues by reorganizing existing short-turn ones, thereby

---

works could further explore: (1) can we flexibly control the conversation flow fit the specific char- acteristics in real long-turn dialogues (e.g., topic- drift) by adjusting the degree of coherence? (2) can we design pre-training tasks to utilize the addi- tional signals in the constructed long-turn dialogue corpus, e.g., the similarity score?
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of Wikipedia: Knowledge-powered conversational agents. In Proceedings of the International Confer- ence on Learning Representations (ICLR).
Joseph Fleiss. 1971. Measuring nominal scale agree-
ment among many raters. Psychological bulletin, 76(5):378.
Ethics Statement
Yuxian Gu, Jiaxin Wen, Hao Sun, Yi Song, Pei Ke, Chujie Zheng. Zheng Zhang, Jianzhu Yao, Xiaoyan Zhu, Jie Tang, et al. 2022. Eva2. 0: Investigating open-domain chinese dialogue systems with large- scale pre-training. arXiv preprini arXiv:2203. 03313..
Our experiments are conducted based on the exist- ing web-crawled dialogue corpus. Despite that the corpus has been preprocessed for safety concerns (e.g., filtering sensitive words) (Zhou al., 2021), Xu et al. (2020) show that the pre-trained dialogue models still have unsafe behaviors, such as gener- ating toxic responses. Therefore, dialogue models should be carefully examined before being made publicly available.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case neural text degeneration. arXiv preprint
Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padmanab- han, Giuseppe Ottaviano, and Linjun Yang. 2020. Embedding- -based retrieval facebook search. Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 2553-2561.
We hire three annotators from a professional data annotation company. We do not ask about any private information in the annotation process. We pay each annotator 0.2$ for comparing each pair of retrieving results. Each comparison costs about 1 minute on average, so the payment is quite reasonable.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense in- formation retrieval with contrastive learning. Trans. Mach. Learn, Res., 2022.
Gautier Izacard and Edouard Grave. 2020. Leverag- ing passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282
Acknowledgements
This work was supported by the National Key Re- search and Development Program of China (No. (021ZD0113304) the National Science Founda- tion for Distinguished Young Scholars (with No. 62125604) and the NSFC projects (Key project with No. 61936010).
Zongcheng Ji, Zhengdong Lu, and Hang Li. 2014. An information retrieval approach to short text conversa- tion. arXiv preprint TmXiv:1408.6986 6988.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Transactions Big Data. 7(3):335-547.
References
Vladimir Karpukhin, Barlas Oğuz. Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.044 04906.
Siqi Bao, Huang He, Fan Wang, Hua Wu, Haifeng Wang, Wenquan Wu, Zhen Guo, Zhibin Liu, and
Xinchao Xu. 2020. Plato-2: 2: Towards building an open-domain chatbot via curriculum learning. arXiv preprint arXiv:2006.167.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. arXiv preprini arXiv: 1.00172.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2021. arXiv
Internet-augmented dialogue generation. preprint arXiv:2107.075mm
Katherine Lee, Daphne Ippolito. Andrew Nystrom. Chiyuan 7hang, Douglas Eck, Chris Callison-Bummi Burch, and Nicholas Carlini. 2021. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499,
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 180:11800004805

---

Patrick Lewis. Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin. Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, Scbastian Riedel, and Douwe Kiela. 2020. -angmeeeld generation for knowledge- intensive nlp tasks.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joinl Conference on Natu- ral Language Processing TEMNLP-IJCNLL pages 3982-3992, Hong Kong. China. Association for Com- putational Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversity-promotingg objec- tive function for neural conversation models. arXiv preprint 1rXiv:1510.030:.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behav- ioral testing of nlp models with checklist. arXiv preprint ar 200.01000.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 81.
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends in Information Re- trieval, 33).3333-88.
Yahui Liu, Wei Bi, Jun Gao, Xiaojiang Liu, Jian Yao. and Shuming Shi. 2018. Towards less generic re- sponses in neural conversation models: A statistical re-weighting method. Proceedings of the 2018 conference empiricul methods in natural language processing, pages 2769-2774.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ouf, Kurt Shuster, Eric M Smith, et al. 2020. Recipes for building an open-domain chatbot. arXiv preprint 2XXi::000.136333
Yongkang Liu, Shi Feng, Daling Wang. Kaisong Song, Feiliang Ren, and Yifei Zhang. 2021. A graph rea- soning network for multi-ture response selection via customized pre-training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13433- 13442.
Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, and Jason We- ston. 2022a. Language models that seek for knowledge: Modular search & generation for di- alogue and prompt completion. arXiv preprint arXiv:2203.I 13224.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan T. McDonald. 2020. On faithfulness and fac- Proceedings
Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung. Moya Chen, Kushal Arora, Joshua Lane, et al. 2022b. Blenderhot 3: deployed conversational agent that continually learns to responsibly engage. arXiv preprint arXiv:2208.031
tuality in abstractive summarization. of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 1906-1919, Association for Com- putational Linguistics.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gre- gory Diamos, Erich Elsen, David Garcia, Boris Gins- burg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. 2017. Mixed precision training. arXiv preprint 17100:17
Eric Michael Smith, Mary Williamson, Kurt Shuster, Jason Weston, and Y-Lan Boureau. 2020. Can you put it all together: Evaluating conversational agents' ability to blend skills. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2021-2030, Online, Association for Computational Linguistics.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 31 311-318.
Nandan Thakur, Nils Reimers, Andreas Rücklé Ab- hishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evalua- tion of information retrieval models. arXiv preprint 2i.::i2:4.0000 04.66663.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20:
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoory Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applica- tions. arXiv preprint arXiv:2201.0825:
International Conference for High Performance Com- puting, Networking, Storage and Analysis, pages 1- 16. IBEE.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, pages 3505-3506.
Xiaoyang Wang, Chen Li, Jianqiao Zhao, and Dong Yu. 2021. Naturalconv: A chinese dialogue dataset towards multi topic-driven conversation.
Jiaxin Wen, Song Yi, and Ke Pei. 2022. Opd: An open- domain chinese pre-trained dialogue model.

---

Lee Xiong, Chenyan Xiong, Ye Li. Kwok-Fung Tang. Jialin Liu, Paul Bennett, Junaid Ahmed, and Arood Overwijk. 2020. Approximate nearest neighbor neg- ative contrastive learning for dense text retrieval. arXiv preprint urXiv:2007.00800
2022, pages 3346-3361. Association for Computa- tional Linguisties.
Hao Zhou, Pei Ke, Zheng Zhang, Yuxian Gu, Yinhe Zheng, Chujie Zheng. Yida Wang, Chen Henry Wu, Hao Sun, Xiancong Yang. et al. 2021. Eva: An open-domain chinese dialogue system with large-scale generative pre-training, arXiv preprint arXiv:2108 MxXiv:218..0144: 01547.
Jing Xu. Da Ju, Margaret Li. Y-Lan Boureau, Ja- Weston, and Emily Dinan. 2020. Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079.
Hao Zhou, Chujie 7heng. Kaili Huang, Minlie Huang, and Xiaoyan Zhu. 2020. Kdconv: A chinese multi-domain dialogue dataset towards multi-turn knowledge-driven conversation. arXiv preprint arXiv:200.041000.
Jing Xu, Arthur Szlam, and Jason Weston. 2021. Be- yond goldfish memory: Long-term open-domain con- versation. arXiv preprint
Jing Xu, Arthur Szlam, and Jason Weston. 2022a. Be- yond goldfish memory: Long-term open-domain con- versation. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 5180-5197. Association for Computational Linguistics.
Xiangyang Zhou, Lu Li, Daxiang Dong, Yi Liu, Ying Chen, Wayne Xin Zhao, Dianhai Yu, and Hua Wu. 2018. Multi-turn response selection for chatbots with deep attention matching network. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 11 112-1122.
Xinchao Xu, Zhibin Gou, Wenquan Wu, Zheng-Yue Yu Niu, Hua Wu, Haifeng Wang, and Shihang Wang. 2022b. Long time no see! open-domain conversa- tion with long-term persona memory. arXiv preprint arXiv:2203.05797.
A Data Information
A.1 Retriever Data
We first derive a subset of the original EVA pre- training corpus that consists of dialogues with more than four turns. We then randomly sample 1,000,000/49,00000
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang. Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b An open bilingual pre-trained model. arXiv preprint 1rXiv:2210.02414.
from this subset for the train/validationn/esst set, respectively. We use the in- batch negative trick in our experiment (Karpukhin et al., 2020). Table shows the statistics of the retriever data.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter J. Liu. 2020. PEGASUS: pre-training with ex- tracted sep-enntenees for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 11328 11339. PMLR.
Split Train Valid
# Num Examples
In- In-Batch # Num Negatives
1,000,000
1023 1024
49,000
1023 1024
Test
1,000
999 0
Saizheng Zhang, Emily Dinan, Jack Urbanck, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.
Table 8: Statistics of retriever data. # Num Examples denotes the number of query sessions. In-Batch denotes whether use the trick of in-batch negatives, which we in the training stage of UDSR. # Num Negatives denotes the number of negatives for each query session, which consists of two parts, i.e., the number of random negatives and the number of BM25 negatives.
Yizhe Zhang. Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019. Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv: 19111990334
A.2 Dialogue Pre-training Data
We randomly sample 5 million dialogue sessions for experiments and billion dialogue sessions for constructing R 1B from the original EVA pre-training corpus.
Zhexin Zhang, Jiaxin Wen, Jian Guan, and Minlie Huang. 2022. Persona- guided planning for control- ling the protagonist's 'ss persona in story generation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15,
A.3 Dialogue Benchmarks
Table 9 shows the data statistics of the three bench- marks. We use the official split of DuLeMon and

---

KdConv. For NaturalConv, considering that the original training set is too large (containing nearly 20,000 sessions), we randomly sample 5,000 from it as the training set. Moreover, as we focus on multi-turn dialogue modeling instead of grounding dialogue in this paper, we only use plain dialogue data for training and evaluation, leaving out any grounding information, such as knowledge, per- sona, or document.
To improve the training efficiency, we adopt the mixed-precision training (Micikevicius et al., 2017), gradient checkpointing. and ZeRO (Rajb- handari et al., 2020) implemented in DeepSpeed (Rasley et al., 2020).
It costs about 72 hours to train UDSR on I mil- lion examples. It costs about 15-48 hours for di- alogue pre-training and 0.5-4 hours for dialogue fine-tuning, depending on different backbone mod- els and downstream benchmarks.
B Training Details
C More Metries
In this section, we provide more training details of our experiment.
Overlap Score For each utterance in multi-turn dialogue, it is computed the length of the LCS between the utterance and its context divided by the length of the utterance. And we use miero- averaging to derive the overlap score of the overall constructed corpus.
For retriever training, we concatenate all utter- ances within a multi-turn dialogue with the <SEP> token. We set the maximum sequence length to 256, batch size to 512. the initial learning rate of Adam W optimizer to 5e-5. The best checkpoint is selected based on the Top-1 recall on the validation set.
Repeated Sampleing Let Cj be the sampled times of the dialogue S; after finishing contructing the corpus. It measures the mean and the standard deviation of the sampled times of the Top- K ses- sions among all the corpus sorted by c. We use K 1000 in our experiments.
For dialogue pre-training, we concatenate all utterances within a multi-turn dialogue with the <SEP> We use the Noam scheduler to adjust the learning rate. We set the maximum sequence length to 256 for decoder-only models and 256 + 64 for encoder-decoder models. For the setting of pre-training from scratch, set the batch size to 512, the initial learning rate of AdamW optimizer 1e-2, and the warmup step to 1K. Due to the limited computational resources and time, we pre- train the 6B non-causal decoder model for 10K steps. For the setting of further pre-training on LM, we set the batch size to 256, the initial learning rate of AdamW optimizer to le-4, and the warmup step to 100. We pre-train GPT2-small*for 30K steps. For the setting of further pre-training DM, we set the batch size 128, the initial learning rate of Adam W optimizer to le-5, the gradient accumulation steps to 2, and the warmup step to 100. We pre-train ChatGLM (Zeng et al., 2022) for 30K steps.
D Experiments on English Benchmarks
Our proposed Rea is a language- framework. To further verify the effectiveness of R in different languages, we conduct experi- ments on an English corpus. Specifically, we col- lect an English dialogue pre-training corpus, which consists of million conversations from Reddit. The average number of turns in the original corpus is 2.3.
We then conduct further pre-training on a GPT- large model. We report PPLzero-shot on three widely-adopted English multi-turn dialogue bench- marks, including Blended Skill Talk (Smith et al., 2020), PersonaChat (Zhang et al., 2018), and Wiz- ard of Wikipedia (Dinan et al., 2019). From the results shown in Table 10, can see that ' achieves significantly lower PPL than the Original baseline. These results demonstrate the effective- ness of Real in English.
For dialogue fine-tuning, keep the same max- imum sequence length with pre-training. We man- ually select the batch size from [32, 64, 128] and the initial learning rate of Adam optimizer from [le-4, 5e-5] based on the perplexity on the valida- tion set. We generate responses using beam search (Holtzman et al., 2019).
E
Analysis of the Retriever
E.1 Constructing Incoherent Examples
Given a human-written K-turn dialogue session S = {u' u², ....... ux }, we first construct the posi- tive pair (S,, St) as follows:
hnttps://huggiiggaac. co/ue// got2-chhinee-ccleecrrussssa "https://hugging" wwtps///ccc///HDD//c/ttggnngnoo

---

Train
Valid
Test
Benchmark
Session
Avg. Turn
Session
Avg. Turn
# Session
Avg. # Turn
Dul_cMon
2,401
16.2
300
16.0
300
16.1
KdConv
3,600
18.5
450
20.7
450
21.6
NaturalConv
5,000
20.1
272
20.1
Table 9: Statistics of three multi-turn open-domain dialogue henchmarks.
Pre-training Data
Blended Skill Talk
PersonaChat
Wizard of Wikipedia
Original
33.72
50.05 39.61
Re Dial
29.17
31.37
48.62
Table 10: PPazersshat in the setting of further pre-training on LM on three English multi-turn dialogue benchmarks. The best performance is highlighted bold.
Retriever
Top-5
Top-20
Speed
BM25
38.04
47.75
22 it/s
S {ut, w. ... w 3) S = {ur u u un}
Contriever
49.60
62.40
192 it/s
UDSR
79.70
89.70
192 it/s
w/ (single)
59.10
70.80
192 it/s
w/ (siiggle)
59.90
74.60
192 it/s
We then introduce the following perturbations to create the negative consecutive session S. in the specific aspect:
Table 1: Automatic evaluation results of different re- trievers. We report the Top-k recall and the inference speed for retrieving the Top-10 sessions from 5M ses- sions.
Irrelevance: create S by randomly sam- pling a session.
Local Irrelevance: we create S. by main- taining one single utterance in S unchanged while replacing the other utterances with randomly sampled session.
SM75
LIGER
90
T00 rccall
90
8
Discourse Incoherence we create S. by us- ing the first three utterances S, i.e. S.
20
20
JK
10K L00K 500K TM
10K 101K 500K JM
40
1K 10c 100C 1W
Tam
Train
Faki
{ul, u², 2.}.
Figure Top-k recall varying with different numbers of training examples.
E.2 Automatic Evaluation
Table 11 shows the automatic evaluation results of the retriever. We can see that UDSR outperforms baselines by large margin. Moreover, also test ablated versions of UDSR in encoding strate- gies, which either encode the last utterance of S, or encode the first utterance of S. for retrieving. The results further demonstrate the importance of capturing global features in this task. We also study the inference speed of the retriever since we aim to build billion-scale pre-training corpus. We can see that with the help of GPU and FAISS, dense trievers can achieve incredible efficiency, process- ing 192 query sessions per second with single V100 GPU. In contrast, BM25 (implemented using ElasticSearch) can only process 22 query sessions per second.
E.3 Sample Efficiency
We investigate how the retriever's performance varies with the number of training examples. As shown in Figure 5, UDSR significantly outperforms BM25, starting from just IK training examples. Moreover, UDSR is more data-hungry as dense retrieval method (Xiong et al., 2020; Karpukhin et al., 2020). While the performance of BM25 is close to convergence with 100K training exam- ples. the performance of UDSR is still markedly improved. Considering that there is massive unla- beled plain dialogue data, the retrieval performance of UDSR might further improved with more training examples.

---

Context Turns
Top-5
Top-20
G Examples of Re'
61.60
77.00
We present two examples of automatically con- structed long-turn dialogues by Ria in Table 13
69.20
83,40
72.20
86.30
Table 12: We report the Top-k recall varying with the number of context turns.
H Instructions for Human Evaluation
We show our instructions for human evaluation Figure 7.
Original
Contest
Figure 6: Attention weights on the context (in log-scale) in the final layer of the dialogue model pre-trained on the Original corpus and Dial averaged on the test set of DuLeMon.
E.4 Effect of Context Length
While we only use the last session as the query for retrieval to reduce the time cost of large-scale retrieval, are interested in how the retrieval per- formance of UDSR varies with different con- text lengths. sample 1000 long-turn (at least 12-turn) dialogues from Weibo data, where the last 3-turn serves as the Sg- We then investigate how the UDSR's performance varies with the number of turns of Sg. As shown in Table 12, UDSR can effectively use longer context and achieve better retrieval performance.
F Attention Weights Distribution
We visualize the attention weights distribution on the context tokens when predicting the next tar- get token in Figure For each token, we select the Top-5 context tokens with the largest attention scores. We then take the average attention score in each interval of five tokens. Compared with the original short-turn pre-training corpus, pre-training Dial achieves better awareness of the older Dial context. The result indicates that pre-training on the automatically constructed long-turn dialogue data with Re learns to better attend to and utilize context for response generation.

---

Automatically Constructed Long-turn Dialogue
Example #1
I really hate to say "okay, its my fault" during an argument.... This attitude of unwillingness to solve the problem only make things words
B: Communication is the best way to maintain good relationship.
..... Even you have different views, don't t immediately dismiss other's views, but first agree with part of it Please take care to have good attitude when communicating...
B: That's true. want communicate properly, and don't t want to fight.....
A: The best way for two people to get along is through both arguments and warmth.... Sometimes your temper is like a sword can hurt someone who cares about you...
B: A good relationship leads to a quick make-up after a fight.....
A: Actually, many times I would first apologize. This is not because I was wrong, but because cherish you- know that have to take care of your feelings-..
B: That's why we say that the one who loves the most is the one who is most humble.
A: I think good relationship is to love each other equally deeply.
B: It's important that girls should not be to humble in relationship, or you will really despise yourself in the end. Loving someone mutual.
Example #2
A: Parents really expect too much from their children this society, and such children are
100% mentally unhealthy even they achieve success in the future.
B: There no need for children to start working so hard from such a young age.
A: Parents want to send their children to an Ivy League School, but they just spend time
on playing phones. It is only pipe dream to expect children to work hard without parents themselves set example.
B: Honestly speaking, it is hard to practice what you preach for educating children.
A: Every time this topic of "tutoring your child's 's homework' comes up. it has lot of resonance. think that the root of the problem is not how well the child learns, but what is the mindset of the parents. When parents want to get angry, they should ask themselves two questions: Can you do what you are asking your child to do when you were this age? 2. Even you could have done it, why should your child have to do it?
B: agree. You can't 't blame others but yourself for your child's IQ.
A: ... Most of what we experienced as children was percussive education We should encour- age children .ore...
B: Yes, recognizing your child's efforts should precede criticizing his shortcomings.
Table 13: Examples of the automatically constructed long-turn dialogue by Re which are translated from Chinese English.

---

Labeling instructions
This study aims to evaluate dialogue generation systems. Specifically, for each dialogue context, the dialogue generation system will generate response.
In the process of evaluation, you will be given on dialogue context and two generated sessions by two difference dialogue generation systems. You need to compare them in terms of sensibleness and informativeness, and give preference (Win, Lose, Tie).
By sensibleness, we mean that the generated response should be semantically relevant and consistent with the context.
By informativeness, mean that the generated response should be informative given the context.
Examples
Sensibleness
Informativeness
Context
A: You're about to graduate from the Central Academy
Response 2 Win
Response
Win
of Fine Arts, aren't you?
B: Yes, I'll be a 4th year undergraduate.
A: What do you want to do after graduating?
B: My dream is to be fashion designer so that's definitely what I'll be doing after graduating. Being a designer difficult because requires inspiration.
B: know difficult. I could get inspiration from Dunhuang murals ethnic costumes.
A: So do you want to design Chinese-styl clothes?
B: Yes, think this kind of clothes look more mysterious and have a sense of history.
A: So are you planning to open a studio?
B: I've always had this idca, but don't have enough money.
A: Why don't i you find partner?
B: tried, but my classmates they didn't want to do it, they thought it was too risky
Generated Response
B: Well, actually I think the same.
Generated Response
B: think it would be good idea to find another experienced fashion designer, which will help you to achieve your dream.
Figure 7: Instructions given to labelers for human evaluations.


Reviews:
[{"review_id": "gxoxnqRYHP", "reviewer": "Reviewer_dkfb", "summary": "", "questions": "Question A. Would you be able to provide the results in terms of Distinct-1 in Table 2?\n\nQuestion B. Why didn't the authors provide the results in terms of PPL, ROUGE-L and Distinct-2 in Table 3?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. The proposed approach was solely evaluated on three Chinese dialogue datasets, which raises concern about cherry-picking. It could be better if the authors would experiment with English dialogue datasets to further demonstrate its effectiveness.\n\nOur framework is language-agnostic. To further verify the effectiveness of Re^3Dial in different languages, we conduct experiments on an English corpus. Specifically, we collect an English dialogue pre-training corpus, which consists of 1 million conversations from Reddit. The average number of turns in the original corpus is 2.3. \n\nWe conduct further pre-training on a GPT-2 large model. We report PPL$_{zero-shot}$ on three widely-adopted English multi-turn dialogue benchmarks, including Blended Skill Talk [1], PersonaChat [2], and Wizard of Wikipedia [3]. From the results shown in the following table, we can see that Re$^3$Dial achieves significantly lower PPL than the Original baseline. These results demonstrate the effectiveness of Re$^3$Dial in English.\n\n| Pre-training Data | Blended Skill Talk | PersonaChat | Wizard of Wikipedia |\n| ----------------- | ------------------ | ----------- | ------------------- |\n| Original          | 33.72              | 39.61       | 50.05               |\n| Re$^3$Dial        | **29.17**              | **31.37**       | **48.62**               |\n\n> 2. As we can see from Table 2, the performance of Re^3Dial is worse than Original in terms of Distinct-2. Moreover, the authors did not provide the results in terms of Distinct-1, raising concern about cheery-picking.\n\nThere is only one case where Re$^3$Dial does not outperform the original baseline (9.24 vs 9.27 in Distinct-2), which is also a comparable performance. We further present the average Distinct-1 metric in the table below.\n\n| Setting | Model | Dist-2 | Dist-1 |\n| ---- | ---- | ---- | ---- |\n| Pre-training From Scratch | Original | 7.17 | 0.90 |\n|                            | Re$^3$Dial | **7.88**  | **1.12** |\n| Further Pre-training on LM | Original   | 12.33     | **2.06**     |\n|                            | Re$^3$Dial | **12.59** | **2.06**     |\n| Further Pre-training on DM | Original   | 20.65     | 2.01     |\n|                            | Re$^3$Dial | **21.15** | **2.07** |\n\n> 3. The experimental results are not satisfactory. The performance improvements of Re$^3$Dial over Original in terms of BLEU-1, BLEU-2, ROUGE-L and Distinct-2 are mostly marginal.\n\nWe would like to further clarify the substantial improvements of Re$^3$Dial.\n\nFirstly, for automatic evaluations (Section 4.3.1), Re$^3$Dial demonstrates substantial improvement in automatic metrics in an open-ended generation task (i.e., open-domain dialogue generation in our experiments). Let us reference the results in DuLeMon [4]. It was observed that increasing model parameters from 110M to 1.6B leads to an improvement of 0.0/0.6 in terms of BLEU-1/2, respectively.  In this light, the performance boost achieved by Re$^3$Dial is substantial. Furthermore, it is essential to underline that Re$^3$Dial outperforms the Original baseline in terms of PPL$_{zero-shot}$ and PPL by a large margin. As pointed out by [5][6][7], perplexity shows a stronger correlation with human judgments than other automatic metrics (e.g., BLEU). \n\nSecondly, considering that automatic metrics cannot fully demonstrate the performance of open-ended dialogue models [5][6][7][8], we further conduct human evaluations (Section 4.3.2). As illustrated in Figure 3, the Re$^3$Dial-trained dialogue model performs significantly better than the Original baseline in sensibleness and coherence in the long-turn test set.\n\n\n> 4.  The authors did not report PPL, ROUGE-L and Distinct-2 in Table 3, which is confusing. \n\nTaking further pre-training on DM for example, we present the detailed evaluation results of the dialogue model trained on the corpus constructed by different retrievers. We report the average metric over three benchmarks. As we can see, Re$^3$Dial achieves the best performance on all automatic metrics. These detailed evaluation results are omitted in the main paper for space considerations.\n\n| Retriever  | PPL$_{zero-shot}$ | PPL       | BLEU-1    | BLEU-2    | ROUGE-L   | Dist-1   | Dist-2    |\n| ---------- | ----------------- | --------- | --------- | --------- | --------- | -------- | --------- |\n| Original   | 59.60             | 22.80     | 18.82     | 9.98      | 18.17     | 2.01     | 20.65     |\n| Random     | 61.83             | 22.93     | 19.03     | 10.08     | 18.24     | 2.05     | 21.03     |\n| BM25       | 73.43             | 22.86     | **19.48**     | 10.32     | 18.28     | 1.99     | 20.74     |\n| Contriever | 61.48             | 22.98     | 19.34     | 10.22     | 18.06     | 1.97     | 20.49     |\n| Re$^3$Dial | **54.54**         | **22.06** | **19.48** | **10.37** | **18.58** | **2.07** | **21.15** |\n\n> 5. The performance of the proposed dense retriever is not promising. Moreover, as we can see in Table 3, improvements made by the proposed Re^3Dial over Random, BM25 and Contriever are somewhat marginal in terms of BLEU-1 and BLEU-2 especially under the setting 'Further Pre-training on LM' and 'Further Pre-training on DM'.\n\nWe would like to further clarify the effectiveness of the dense retriever in Re$^3$Dial.\n\nFirstly, in the zero-shot setting, where the pre-trained dialogue model is directly deployed for applications, other retrievers inversely increase PPL in most cases (Table 3). In contrast, as shown in Table 3 and Figure 4, the Re$^3$Dial-trained model consistently achieves the lowest PPL, especially as the context length grows. These results indicate that Re$^3$Dial significantly outperforms other retrievers in enhancing the models' utilization of long-range contexts. \n\nSecondly, in the fine-tuning setting, despite that fine-tuning on sizable long-turn dialogues weaks the influence of pre-training [9], Re$^3$Dial still consistently outperforms all other baselines (Table 3). Notably, Re$^3$Dial achieves a substantially superior PPL than other retriever baselines (see the table above in Response4). And perplexity has shown a stronger correlation with human judgments than other automatic metrics (e.g., BLEU) [5][6][7]. Moreover, considering that the automatic metrics cannot fully demonstrate the performance of open-ended dialogue models [5][6][7][8], we further conduct a pair-wise human evaluation to study the model's performance trained with different retrievers. We conduct human evaluations on the long-turn test set (Section 4.3.2). The human evaluation results shown in the following table indicate that the dense retriever in Re$^3$Dial leads to significantly better performance.\n\n| Retriever                 | Sensibleness (Win) | Sensibleness (Lose) | Informativeness (Win) | Informativeness (Lose) |\n| ------------------------- | ------------------ | ------------------- | --------------------- | ---------------------- |\n| Re$^3$Dial vs. BM25       | 39                 | 18                  | 42                    | 20                     |\n| Re$^3$Dial vs. Contriever | 36                 | 20                  | 40                    | 21                     |\n\nLastly, we also evaluate the retriever using individual tests in different aspects to gain a deeper understanding of the effectiveness of different retrievers. From the results shown in Table 4, we can see that our proposed dense retriever outperforms baselines by a large margin in capturing global relevance and discourse coherence.\n\n> 6.  More importantly, considering that the focus of this work is to build long-turn dialogue sessions by retrieving relevant short-turn dialogue sessions, it is very critical to train a retriever that really performs well. However, the authors only compared their retriever with two baseline retrievers including BM25 and Contriever, which fails to demonstrate the superiority of the proposed retriever.\n\nWe select BM25 and Contriever as the baselines since they are the previous state-of-the-art non-parametric retriever and dense retriever [10].\n\nWe additionally conduct experiments based on an encoder for sentence similarity (Sentence-BERT) [11]. \nWe report the average PPL$_{zero-shot}$ over three benchmarks. As we can see, Sentence-BERT underperforms Contriever and also significantly underperforms the dense retriever in Re$^3$Dial.\n| Retriever  | Pre-training From Scratch | Further Pre-training on LM | Further Pre-training on DM |\n| ---------- | ------------------------- | -------------------------- | -------------------------- |\n| Original   | 193.51                    | 13.06                      | 59.60                      |\n| Random     | 170.58                    | 14.92                      | 61.83                      |\n| BM25       | 192.94                    | 14.69                      | 73.43                      |\n| Sentence-BERT   | 161.05                    | 13.76                      | 63.39                      |\n| Contriever | 154.65                    | 13.41                      | 61.48                      |\n| Re$^3$Dial | **135.88**                    | **12.25**                      | **54.54**                      |\n\n>7. Typo and missing references.\n\nThanks for pointing these out. We will fix them in our revised version of the paper.\n\n\n[1] Can You Put it All Together: Evaluating Conversational Agents’ Ability to Blend Skills. ACL2020\n\n[2] Personalizing Dialogue Agents: I have a dog, do you have pets too? ACL2018\n\n[3] Wizard of Wikipedia: Knowledge-Powered Conversational agents. ICLR2019\n\n[4] Long Time No See! Open-Domain Conversation with Long-Term Persona Memory. ACL2022 findings.\n\n[5] Towards a Human-like Open-Domain Chatbot. arXiv2020\n\n[6] How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for\ndialogue response generation. EMNLP2016\n\n[7] Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses. ACL2017\n\n[8] RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems. AAAI2018\n\n[9] Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence. ICLR2019\n\n[10] Unsupervised Dense Information Retrieval with Contrastive Learning. TMLR2022\n\n[11] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP2019", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The authors proposed to automatically construct large-scale long-turn dialogue corpus by retrieving, reorganizing and rescaling the existing short-turn dialogue sessions, which barely requires human annotations.\n\n(2) In comparison to BM25 and Contriever, the proposed Unsupervised Dense Session Retriever (UDSR) with contrastive learning effectively discriminates the positive retrieved session from the incoherent negative session.", "reasons_to_reject": "(1) The proposed approach was solely evaluated on three Chinese dialogue datasets, which raises concern about cherry-picking. It could be better if the authors would experiment with English dialogue datasets to further demonstrate its effectiveness.\n\n(2) The experimental results are not satisfactory. As we can see from Table 2, the performance of Re^3Dial is worse than Original in terms of Distinct-2. In addition, the performance improvements of Re^3Dial over Original in terms of BLEU-1, BLEU-2, ROUGE-L and Distinct-2 are mostly marginal. Moreover, the authors did not provide the results in terms of Distinct-1, raising concern about cheery-picking.\n\n(3) The performance of the proposed dense retriever is not promising. First of all ,the authors did not report PPL, ROUGE-L and Distinct-2 in Table 2, which is confusing. Moreover, as we can see in Table 2, improvements made by the proposed Re^3Dial over Random, BM25 and Contriever are somewhat marginal in terms of BLEU-1 and BLEU-2 especially under the setting 'Further Pr-training on LM' and 'Further Pre-training on DM'. More importantly, considering that the focus of this work is to build long-turn dialogue sessions by retrieving relevant short-turn dialogue sessions, it is very critical to train a retriever that really performs well. However, the authors only compared their retriever with two baseline retrievers including BM25 and Contriever, which fails to demonstrate the superiority of the proposed retriever.", "questions_to_authors": "Question A. Would you be able to provide the results in terms of Distinct-1 in Table 2?\n\nQuestion B. Why didn't the authors provide the results in terms of PPL, ROUGE-L and Distinct-2 in Table 3?", "paper_topic_and_main_contributions": "Considering that the capability of pre-trained dialogue models in utilizing long-range contexts is constrained by the scarcity of long-turn dialogue sessions in the pre-trained dialogue corpus, this paper aims to automatically build a large-scale long-turn dialogue corpus by reorganizing short-turn dialogue sessions from existing dialogue corpus.\n\nThe main contributions made by the authors are: 1) they proposed the Retrieve, Reorganize and Rescale framework (i.e. Re^3Dial) to build a large-scale long-turn dialogue corpus by using existing short-turn dialogue sessions; 2) they proposed to train an Unsupervised Dense Session Retriever (UDSR) with contrastive learning to capture the global semantic relevance; 3) they experimented their proposed approach on three multi-turn Chinese dialogue benchmarks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "(1) While all the models under the pre-training setting 'Further Pre-training on DM' are built upon ChatGLM-6B, the authors did not cite the following two papers relevant to ChatGLM-6B. Please consider citing them properly.\n\n@article{zeng2022glm,\n  title={Glm-130b: An open bilingual pre-trained model},\n  author={Zeng, Aohan and Liu, Xiao and Du, Zhengxiao and Wang, Zihan and Lai, Hanyu and Ding, Ming and Yang, Zhuoyi and Xu, Yifan and Zheng, Wendi and Xia, Xiao and others},\n  journal={arXiv preprint arXiv:2210.02414},\n  year={2022}\n}\n\n@inproceedings{du2022glm,\n  title={GLM: General Language Model Pretraining with Autoregressive Blank Infilling},\n  author={Du, Zhengxiao and Qian, Yujie and Liu, Xiao and Ding, Ming and Qiu, Jiezhong and Yang, Zhilin and Tang, Jie},\n  booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},\n  pages={320--335},\n  year={2022}\n}", "Typos_Grammar_Style_and_Presentation_Improvements": "(1) In Table 11 on page 14, there is a typo in the first turn uttered by person A in Example #1. Please replace 'only make things words · · ·' with 'only make things worse · · ·'\n\n(2) In Table 11 on page 14, there is another typo in the last turn uttered by person B in Example #1. Please replace 'girls should not be to humble in a relationship' with 'girls should not be too humble in a relationship,'\n\n(3) In Figure 7 on page 15, there is a typo in the second paragraph in the Labeling Instructions. Please replace 'In the process of evaluation, you will be given on dialogue context' with 'In the process of evaluation, you will be given a dialogue context' or 'In the process of evaluation, you will be given one dialogue context'.\n\n(4) In line 404 on page 6, there is a grammatical error in the sentence 'We compare different approaches to retrieve dialogue session'. Please replace it with 'We compare different approaches to retrieving dialogue session'."}}, {"review_id": "H0jmN3k9uc", "reviewer": "Reviewer_2y1L", "summary": "", "questions": "For the pre-training experiment, how many examples do you use from the Re^3Dial dataset? It's mentioned 5 million examples from EVA corpus are used, is the number the same or larger for Re^3Dial dataset?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. The Re$^3$Dial dataset has 5x utterances. Also, it's unclear if they actually use more conversation from the Re$^3$Dial dataset.\n\nWe use 5 million examples for Re$^3$Dial during pre-training in our experiments, which is the same number as other baselines instead of 5X utterances. Specifically, for each response in the original corpus, Re$^3$Dial constructs a longer context for it. And the loss is only computed on the response tokens.\n\n> 2. The marginal improvements might not be worth the extra efforts. \n\nWe would like to further clarify the substantial improvements of Re$^3$Dial.\n\nFirstly, for automatic evaluations (Section 4.3.1), Re$^3$Dial demonstrates substantial improvement in automatic metrics in an open-ended generation task (i.e., open-domain dialogue generation in our experiments). Let us reference the results in DuLeMon [1]. It was observed that increasing model parameters from 110M to 1.6B leads to an improvement of 0.0/0.6 in terms of BLEU-1/2, respectively. In this light, the performance boost achieved by Re$^3$Dial is substantial. Furthermore, it is essential to underline that Re$^3$Dial outperforms the Original baseline in terms of PPL$_{zero-shot}$ and PPL by a large margin. As pointed out by [2][3][4], perplexity shows a stronger correlation with human judgments than other automatic metrics (e.g., BLEU). \n\nSecondly, considering that automatic metrics cannot fully demonstrate the performance of open-ended dialogue models [2][3][4][5], we further conduct human evaluations (Section 4.3.2). As illustrated in Figure 3, the Re$^3$Dial-trained dialogue model performs significantly better than the Original baseline in sensibleness and coherence in the long-turn test set.\n\n\n[1] Long Time No See! Open-Domain Conversation with Long-Term Persona Memory. ACL2022 findings.\n\n[2] Towards a Human-like Open-Domain Chatbot. arXiv2020\n\n[3] How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. EMNLP2016\n\n[4] Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses. ACL2017\n\n[5] RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems. AAAI2018", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The methodology and experiment are well explained.\n- The idea of retrieve and reorganize to generate long conversations might be useful under some circumstances.", "reasons_to_reject": "- The marginal improvements might not be worth the extra efforts. The Re^3Dial dataset has 5x utterances, but the improvement is marginal. Also, it's unclear if they actually use more conversation from the Re^3Dial dataset.", "questions_to_authors": "For the pre-training experiment, how many examples do you use from the Re^3Dial dataset? It's mentioned 5 million examples from EVA corpus are used, is the number the same or larger for Re^3Dial dataset?", "paper_topic_and_main_contributions": "This paper propose a data augmentation framework to generate long conversations by retrieving multiple relevant short conversations and reorganize them into one long conversation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "n/a", "Typos_Grammar_Style_and_Presentation_Improvements": "n/a"}, "extra_text": {}}, {"review_id": "Tyc4CAKUSB", "reviewer": "Reviewer_YxhS", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. While the methodology proposed in this paper appears to be language agnostic, the experiments are conducted only on Chinese corpora, so it cannot guarantee proper functioning across different languages.\n\nWe conduct further pre-training on a GPT-2 large model. We report PPL$_{zero-shot}$ on three widely-adopted English multi-turn dialogue benchmarks, including Blended Skill Talk [1], PersonaChat [2], and Wizard of Wikipedia [3]. From the results shown in the following table, we can see that Re$^3$Dial achieves significantly lower PPL than the Original baseline. These results demonstrate the effectiveness of Re^3Dial in English.\n| Pre-training Data | Blended Skill Talk | PersonaChat | Wizard of Wikipedia |\n| ----------------- | ------------------ | ----------- | ------------------- |\n| Original          | 33.72              | 39.61       | 50.05               |\n| Re$^3$Dial        | 29.17              | 31.37       | 48.62               |\n\n> 2. The methodologies used within the framework presented in the paper are not significantly novel.\n\nAt its core, our proposed Re$^3$Dial framework provides a novel perspective to alleviate the scarcity of long-turn conversations, i.e., automatically building a billion-scale long-turn dialogue corpus via concatenating existing short-turn dialogue.  \n\nHowever, naive implementations of this novel idea may result in incoherent, low-diversity, or low-informativeness long-turn dialogue corpus. Therefore, our novelty also lies in addressing the following two challenges:\n- **Preserving Dialogue Coherence**: We build a high-performance dialogue session retriever by training on massive unlabeled dialogues, which are readily available in various languages. Additionally, we propose to use individual tests to evaluate the effectiveness of the dialogue session retriever in capturing global semantic and discourse relations within multi-turn dialogues.\n- **Enhancing Dialogue Diversity**: We propose two \"inherent problems\" of the dialogue pre-training corpus, which may decrease diversity during concatenation. And our proposed diversity sampling strategy effectively tackles these concerns.\n\nExtensive experiments demonstrate that Re$^3$Dial provides a well-generalized data foundation in the era of large-scale dialogue pre-training.\n\n\n[1] Can You Put it All Together: Evaluating Conversational Agents’ Ability to Blend Skills. ACL2020\n\n[2] Personalizing Dialogue Agents: I have a dog, do you have pets too? ACL2018\n\n[3] Wizard of Wikipedia: Knowledge-Powered Conversational agents. ICLR2019", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper highlights critical issues in the current open-domain chatbot domain concerning the utilization of corpora and proposes methodologies to address them.\n\n- Furthermore, it defines inherent problems that might arise during the resolution process and provides clear solutions to these issues. The overall process is also well-structured.\n\n- The proposed methodology appears to be generally applicable to corpora based on different languages as well.\n\n- The paper effectively demonstrates the efficacy of the augmented corpus through experiments, and the results are also impressive.", "reasons_to_reject": "- While the methodology proposed in this paper appears to be language agnostic, the experiments are conducted only on Chinese corpora, so it cannot guarantee proper functioning across different languages.\n\n- The methodologies used within the framework presented in the paper are not significantly novel.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses a highly significant issue in the field of open-domain chatbots, which involves extending the current short-turn open-domain dialogue corpus into a long-turn corpus. Moreover, the methodology proposed in this paper, including Re^3Dial, is very effective due to its automated approach. Also, they have a plan to release 1B corpora as open-source data.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WvKXZ5NJMX", "reviewer": "Reviewer_RkmQ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive review!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* a novel and effective technique for dialogue data augmentation is proposed - it addresses the problem of composing longer dialogues of shorter segments while preserving coherence and maximizing diversity of conversations\n* training on such a dataset results in improved dialogue system's performance both on automatic and human metrics\n* the authors contribute the resulting 1B dialogue dataset for public use", "reasons_to_reject": "N/A", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes Re3Dial, a technique to augment dialogue data by creating longer dialogues from the shorter ones. For that, they use a dense retriever to select coherent dialogue segments for combining, and a diversity sampling procedure to increase variability in the dialogue trajectories.\nThe model trained (depending on the setup, it's either a from-scratch model or GPT2-small or ChatGLM) on the resulting augmented 1-billion dialogue dataset outperforms the original model trained on EVA corpus in both automatic metrics on 3 Chinese dialogue datasets - and human metrics on one of them.\n\nThe authors also release the resulting dialogue dataset with 1B dialogues in Chinese with the average length of 11.3 turns.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wf3u70NYoz", "reviewer": "Reviewer_oPUQ", "summary": "", "questions": "This paper introduces a general framework and builds its experimental setup incorporating multiple models. However, it fails to compare any additional baseline methods, such as alternative retrieval enhancements or different approaches to long-term modeling.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. It fails to compare any additional baseline methods, such as alternative retrieval enhancements or different approaches to long-term modeling.\n\nWe additionally conduct experiments on a retrieval-based baseline and a summarization-based baseline for long-term context modeling.\n\nFor the retrieval-based baseline, given an original context $S = {S_1, S_2, ..., S_N}$, we use $S_N$ as the query to retrieve the top-$K$ most relevant utterances from $\\{S_1, ..., S_{N-1}\\}$. We try two utterance retriever models: (1) Contriever: It is a state-of-the-art dense retriever model. (2) Sentence-BERT [1]: It is an encoder model fine-tuned for sentence similarity. We set $K=2$ in our experiments. \n\nFor the summarization-based baseline, we introduce an additional summarization model to summarize long-term context into short sentences. We try two summarization models: (1) Pegasus-523M [2]: It is a widely-adopted encoder-decoder model specifically pre-trained and fine-tuned for text summarization. (2) ChatGLM-66B [3]: It is a widely-adopted instruction-tuned large language model. \n\nWe report the average PPL$_{zero-shot}$ over three multi-turn dialogue benchmarks. From the results shown in the following table, we observe that Re$^3$Dial significantly outperforms all baselines in long-turn dialogue benchmarks. Moreover, augmenting the dialogue model with a context summarization model or a retriever shows less improvement or inversely hurts model performance in several cases. \n\nOn the one hand, this two-stage framework suffers from error propagation due to the introduced summarization model or the retriever. For example, both the summarization model and the retriever may lose important information in the original context. Moreover, the summarization model could also suffer from hallucination problems [4], thereby introducing new noises. In contrast, Re$^3$Dial keeps the original long-turn context unchanged and thus does not lead to information loss or introduce new noises. \n\nOn the other hand, we conjecture that augmenting dialogue models with the context summarization model requires further training on summarization-based dialogue datasets [5]. In contrast, Re$^3$Dial does not require collecting additional training datasets and greatly improves the model performance. \n\n| Method                         | Pre-training From Scratch | Further Pre-training on LM | Further Pre-training on DM |\n| ------------------------------ | ------------------------- | -------------------------- | -------------------------- |\n| Original                       | 193.51                    | 13.06                      | 59.60                      |\n| + Retrieval (Sentence-BERT)    | 169.36                    | 13.87                      | 65.17                      |\n| + Retrieval (Contriever)       | 168.85                    | 13.69                      | 64.03                      |\n| + Summarization (Pegasus-523M) | 172.21                    | 14.20                      | 68.16                      |\n| + Summarization (ChatGLM-66B)  | 182.52                    | 13.79                      | 63.48                      |\n| Re$^3$Dial                     | **135.88**                | **12.25**                  | **54.54**                  |\n\n\n> 2. The overall method lacks substantial innovation.\n\nAt its core, our proposed Re$^3$Dial framework provides a novel perspective to alleviate the scarcity of long-turn conversations, i.e., automatically building a billion-scale long-turn dialogue corpus via concatenating existing short-turn dialogue.\n\nHowever, naive implementations of this novel idea may result in incoherent, low-diversity, or low-informativeness long-turn dialogue corpus. Therefore, our novelty also lies in addressing the following two challenges:\n\n- **Preserving Dialogue Coherence**: We build a high-performance dialogue session retriever by training on massive unlabeled dialogues, which are readily available in various languages. Additionally, we propose to use individual tests to evaluate the effectiveness of the dialogue session retriever in capturing global semantic and discourse relations within multi-turn dialogues.\n\n- **Enhancing Dialogue Diversity**: We propose two \"inherent problems\" of the dialogue pre-training corpus, which may decrease diversity during concatenation. And our proposed diversity sampling strategy effectively tackles these concerns.\n\nExtensive experiments demonstrate that Re$^3$Dial provides a well-generalized data foundation in the era of large-scale dialogue pre-training.\n\n[1] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP2019\n\n[2] Pegasus: pre-training with extracted gap-sentences for abstractive summarization. ICML2020\n\n[3] GLM-130B: An Open Bilingual Pre-trained Model. ICLR2023\n\n[4] On Faithfulness and Factuality in Abstractive Summarization. ACL2020\n\n[5] Beyond goldfish memory long-term open-domain conversation. ACL2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes a Re3Dial conversation framework that automatically constructs billion-scale long-turn dialogues by implementing conversation retrieval, reorganization, and rescaling techniques. The motivation behind this approach is to enhance the dialogue model's ability to utilize long-range context and generate more sensible and informative responses in multi-turn dialogue. It sounds reasonable.", "reasons_to_reject": "This paper introduces a general framework and builds its experimental setup incorporates multiple models. However, it fails to compare any additional baseline methods, such as alternative retrieval enhancements or different approaches to long-term modeling. From my perspective, as an applied article, this is not reasonable. Furthermore, the overall method lacks substantial innovation.", "questions_to_authors": "This paper introduces a general framework and builds its experimental setup incorporating multiple models. However, it fails to compare any additional baseline methods, such as alternative retrieval enhancements or different approaches to long-term modeling.", "paper_topic_and_main_contributions": "The objective of this paper is to introduce the Re3Dial framework, which addresses the issue of pre-trained dialogue models lacking the ability to effectively utilize long-range context. To overcome this limitation, the framework automatically constructs billion-scale long-turn dialogues by implementing conversation retrieval, reorganization, and rescaling techniques. The Re3Dial approach consists of four steps for the automatic construction of long-turn dialogues. Firstly, the constructed session is initialized using the original session. Then, a retrieval system called Unsupervised Dense Session Retriever is employed to retrieve the top-K coherent sessions from the dialogue corpus. Subsequently, diversity sampling is applied to select a consecutive session. Finally, the constructed session is updated and extended by concatenating the sampled session. This process is repeated for a specified number of turns, controlling the length of the constructed dialogue. Experimental results demonstrate that the proposed framework significantly enhances the dialogue model's ability to generate coherent and informative responses in multi-turn dialogues.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the data scarcity problem of long-turn dialogues in pretraining large-scale open-domain dialogue models. The authors introduce a novel and effective "Retrieve, Reorganize, and Rescale" framework that can automatically construct billion-scale long-turn dialogues by reorganizing existing short-turn dialogues. The proposed method has been evaluated on Chinese dialogue datasets, effectively demonstrating its efficacy. The authors have committed to releasing their model, toolkit, and data for public use.

The soundness scores for this paper are as follows: (4, 4, 4, 4, 3). Initially, reviewers expressed concerns that evaluating the method using only a single language might not validate the generalizability of the approach. However, in the rebuttal, the authors presented additional experiments on an English benchmark, which addressed these concerns, leading all reviewers to agree on the robustness of the work.

The excitement scores for this paper are: (3, 3, 4, 4, 4). The majority of reviewers show enthusiasm for this work, primarily because the proposed data augmentation method can be adapted to various languages without requiring manual intervention. This indicates the significant potential and broad applicability of the work.
Decision: Accept-Main