DialoGPS: Dialogue Path Sampling in Continuous Semantic Space for Data Augmentation in Multi-Turn Conversations
Year: 2023

Abstract:
In open-domain dialogue generation tasks, contexts and responses in most datasets are oneto-one mapped, violating an important manyto-many characteristic: a context leads to various responses, and a response answers multiple contexts. Without such patterns, models poorly generalize and prefer responding safely. Many attempts have been made in either multiturn settings from a one-to-many perspective or in a many-to-many perspective but limited to single-turn settings. The major challenge to many-to-many augment multi-turn dialogues is that discretely replacing each turn with semantic similarity breaks fragile context coherence. In this paper, we propose DialoGue Path Sampling (DialoGPS) method in continuous semantic space, the first many-to-many augmentation method for multi-turn dialogues. Specifically, we map a dialogue to our extended Brownian Bridge, a special Gaussian process. We sample latent variables to form coherent dialogue paths in the continuous space. A dialogue path corresponds to a new multi-turn dialogue and is used as augmented training data. We show the effect of DialoGPS with both automatic and human evaluation.

heading: 1 Introduction
text: Open-domain dialogue generation has received significant attention and has made notable advancements (Zhang et al., 2020b; Shuster et al., 2022; OpenAI, 2022). However, it still faces challenges due to the nature of the data. One specific challenge is the many-to-many relationship between contexts and responses in open-domain conversations. A context can lead to various responses, and a response can be relevant to multiple contexts. Unfortunately, most datasets only provide one-to-one
∗Equal contribution. †Corresponding author: Rui Yan (ruiyan@ruc.edu.cn).
A: A man came into the bank with a gun. B: That’s insane! A: Let me tell you what happened.
The original version
The modified version A: I am also afraid… Have a hope. B: Wow! What a great news!! A: Ha ha.. I knew mom. Bye bye.
(a) Discrete replacement causes incoherence.
Latent Variable
A man came into the bank with a gun. That’s insane! Let me tell you what happened.
Mapping Function
Mapped Path Sampled Path
A man robbed the bank.
Are you kidding me?
I saw it in person.
Temporally correlated distributions
(b) Sampled dialogue paths in the continuous semantic space correspond to coherent discrete dialogues.
Figure 1: (a) When replacing each utterance in the original conversation by semantic similarity, the modified dialogue is incoherent. (b) We map dialogues into a continuous semantic space where latent distributions of utterances correlate with each other, and sample dialogue paths for training. Each path corresponds to a discrete multi-turn conversation.
mappings between contexts and responses. This limitation results in models being poorly generalized when they rely on learned one-to-one patterns, making them prone to generating safe yet uninteresting responses (Jiang and de Rijke, 2018; Jiang et al., 2019).
To address this limitation, many attempts (Sai et al., 2020; Qiu et al., 2019; Xie et al., 2022) have been made from a one-to-many perspective which involves constructing multiple responses for a context. Furthermore, some works are proposed from a many-to-many perspective but are limited to singleturn settings. To construct new dialogue sentence pairs, they either replace sentences based on se-
1267
mantic similarity (Zhang et al., 2020a) or sample new sentences from probabilistic models (Li et al., 2019). Next, they adopt BERT (Devlin et al., 2019) or GAN (Goodfellow et al., 2014) discriminators to filter incoherent sentence pairs.
These methods cannot be trivially extended to multi-turn settings. Considering T utterances in a dialogue and K candidates for each utterance, they need to (1) prepare a large sentence set as candidates for replacement or a strong generative model, and (2) check the coherence of the modified conversation at least KT−1 times, which is impractical. Figure 1(a) shows a case in which we replace each utterance in a conversation following Zhang et al. (2020a). The modified conversation is still incoherent across turns. Therefore, to enhance multi-turn dialogue generation from a many-to-many perspective, we resort to a continuous semantic space that satisfies two requirements. First, it describes semantic distributions of utterances, allowing for sampling semantic neighbors of each utterance. Second, latent variables sampled from any two distributions should be temporally correlated, contributing to a new coherent dialogue path in the latent space without requiring post-checks. This path can be utilized as a new training sample to augment the model. Our motivation is illustrated in Figure 1(b).
Driven by this motivation, we propose a novel method for augmenting open-domain dialogues from a many-to-many perspective, called DialoGue Path Sampling (DialoGPS), aiming to enhance generalization and improve the quality of generated responses. Specifically, our approach involves the following steps: (1) We map each utterance in a multi-turn dialogue to a special Gaussian process in a continuous semantic space known as the Brownian Bridge (Revuz and Yor, 2013). (2) For each utterance xi, we sample K latent variables zji , j ∈ [1,K], establishing K different dialogue paths in the bridge. Each path corresponds to a new multi-turn conversation in the discrete space. (3) DialoGPS utilizes an encoder-decoder architecture. To construct augmented data, we mix the latent variable zi with representations of xi in the encoder if xi is part of the context, and in the decoder if it is the response. (4) Finally, we train the model using the augmented data.
To ensure the effectiveness of DialoGPS, we address several key issues. First, traditional Brownian Bridges have deterministic endpoints, which
prevent response sampling and lead our method degenerating into a many-to-one paradigm, further impairing generalization. To overcome this limitation, we derive the formula of endpoint distributions. Second, since augmented data that lacks discrete utterance labels makes the optimization challenging, we propose a self-distillation framework where the model first learns from the ground truth and then distills its knowledge to guide itself in utilizing augmented data.
We evaluate DialoGPS on two multi-turn opendomain datasets. Both automatic and human evaluation show that DialoGPS performs better than strong baselines and even outperforms the model trained on manually denoted multi-reference data, which demonstrates the benefit of the many-tomany augmentation paradigm. Because DialoGPS is plug-and-play, we add it to BART (Lewis et al., 2020) and achieve competitive results with the state-of-the-art model, DialoFlow (Li et al., 2021). Our contributions are as follows: • DialoGPS is the first work to augment multiturn dialogues from a many-to-many perspective. • To ensure the effectiveness of DialoGPS, we have introduced dialogue-specific designs, including endpoint sampling of Brownian Bridges and self-distillation for model optimization. • Experiments conducted on both non-pretrained and pre-trained models show that our DialoGPS method outperforms all baselines.

heading: 2 Related Work: Dialogue Generation Augmentation
text: In general, dialogue generation can be categorized into two groups: task-oriented and open-domain. Open-domain generation is a context-aware process that lasts for turns. The model learns to generate a proper but open response from the preceding utterances (i.e., contexts). Task-oriented dialogues progress for specific purposes and are limited to specific domains, such as obtaining knowledge (Zhao et al., 2020; Tao et al., 2021). However, due to the specific domains in task-oriented dialogues, the many-to-many relationship is not as apparent compared to open-domain dialogues.
In this paper, we focus on open-domain dialogue generation augmentation from an X-to-many perspective. From a one-to-many perspective, Sai et al. (2020) manually denoted multiple responses for a dialogue context. Based on such multi-reference datasets, Qiu et al. (2019) proposed to capture the
common feature in feasible responses and then add the specific feature to obtain the final output, which augments the utility of the data and improves the generalization. Xie et al. (2022) proposed that with only one-to-one data, models can construct pseudotarget data in the decoder and improve the model by bootstrapping. From a many-to-many perspective, existing methods work in single-turn settings. Li et al. (2019) generated multiple context or responses with CVAE (Zhao et al., 2017) and introduced a GAN (Goodfellow et al., 2014) discriminator to filter incoherent sentence pairs. Zhang et al. (2020a) augmented a one-to-one dialogue dataset Dp with an unpaired sentence set Du. They sample sentences from Du and replace the most similar sentences in Dp. They use BERT (Devlin et al., 2019) and knowledge distillation to filter noise in incoherent sentence pairs. Until now, manyto-many augmentation in multi-turn settings are understudied.

heading: 3 Method
text: We first present some preliminaries (§ 3.1). Then, we introduce mapping dialogue texts to the desired latent space (§ 3.2), augmented data construction (§ 3.3), augmented data utilization (§ 3.4), and inference details (§ 3.5). Figure 2 shows the overview of DialoGPS.

heading: 3.1 Preliminary
text: In open-domain dialogue generation, given a multiturn dialogue X = [x0, x1, ..., xT ], the goal is to predict the response xT based on the context X0:T−1. The number of tokens in xt is denoted as |xt|, t ∈ {0, 1, . . . , T}. The i-th token in the xt is denoted as xit. A Brownian Bridge B defined on time range [0, T ] is a special Gaussian process established on deterministic endpoints µ0 and µT . At time t, the latent variable zt follows a Gaussian distribution B(t|µ0, µT ):
zt ∼ B(t|µ0, µT ) = N (µ0+ t T (µT −µ0), t(T − t) T ), (1)

heading: 3.2 Extended Brownian Bridge
text: In DialoGPS, given X , a non-linear function fθ maps each xt to µt, the expectations of the corresponding semantic distribution. Based on µ0 and µT , we can establish a Brownian Bridge, and from which we sample the latent variable zt as the semantic neighbor of xt. Meanwhile, z0, z1, ..., zT compose a coherent dialogue path because in a
Brownian Bridge, the covariance between t1 and t2, with 0 < t1 < t2 < T is t1(T−t2) T , where the constant positive covariance guarantees that B(t1|µ0, µT ) and B(t2|µ0, µT ) are temporally correlated.
However, as defined in Eq. 1, a conventional Brownian Bridge B has deterministic endpoints, which prevents us from sampling for xT , the response, and x0, the first utterance in the context. To avoid degenerating to a many-to-one mode that impairs the generalization, we derive an extended Brownian Bridge β with samplable endpoints. Take the derivation of β(T |µ0, µT ) as example: given a B, both the distance dδ between µT and zT−δ and the summation of dδ and zT−δ follow the Gaussian distribution, we can derive the distribution of zT as follows:
zT−δ ∼ N (T − δ T µT + δ T µ0, δ(T − δ) T )
dδ = µT − zT−δ ∼ N ( δ T µT − δ T µ0, δ(T − δ) T )
   ⇒
zT = dδ + zT−δ ∼ N (µT , 2δ(T − δ) T ).
(2)
Due to the symmetry, z0 follows N (µ0, 2δ(T−δ)T ). Here, δ serves as a hyperparameter. To sum up, we define the extended Brownian Bridge β as:
β(t|µ0, µT ) =    N (µt, 2δ(T − δ) T ), t = 0 or T,
N (µ0 + t T (µT − µ0), t(T − t) T ), otherwise.
(3)
To optimize the mapping function fθ, we follow (Wang et al., 2022) to adopt a contrastive learning framework where positive samples are ordered sentence triplets from the same conversation (xt0 , xt1 , xt2 , t0 < t1 < t2) and negative samples are constructed by randomly replacing the middle point xt1 with other sentences xt′1 from the mini-batch B. The objective is as below:
Lβ = EX  log  1 + ∑ (xt0 ,xt ′ 1 ,xt2 )∈B exp(d(xt0 , xt′1 , xt2 ; fθ))
exp(d(xt0 , xt1 , xt2 ; fθ))
    ,
(4)
where d(xt0 , xt1 , xt2 ; fθ) = − 12σ2t1 ∥fθ(xt1)−(1− t1 t2 )fθ(xt0) − t1t2 fθ(xt2)∥ 2 2 . The essence of Eq. 4 is to optimize the outputs of fθ, i.e., µt0 , µt1 , and µt2 to the linear relationship as defined in Eq. 1. In DialoGPS, a 4-layer MLP serves as fθ. To embed utterance as inputs of fθ, there are many choices such as averaging token embeddings or encoding
by a language model. We leave the embedding details in §5.3.

heading: 3.3 Augmented Data Construction
text: As shown in Figure 2(a), we take Transformer (Vaswani et al., 2017) as the bone architecture. With fθ, an extended Brownian Bridge β is established. We sample latent variables zt ∼ β(t|µ0, µT ) and mix them with representations of corresponding xt. In the encoder, for each utterance xt in the context X0:T−1, we conduct:
e1t , e 2 t , ...e |xt| t = Encoder(xt), êit = W enc x · eit +W encz · zt,
(5)
where eit is the output corresponding to the i-th token in xt from the encoder, i ∈ [1, |xt|]. W encz and W encx are trainable vectors of the same dimension as e and z. Finally, ê is sent to the decoder for cross-attention. We conduct the mixup every decoder layer:
d̂ij = W decj x · dij +W decj z · zT , i ∈ [1, |xT |] , j ∈ [1, N ] , (6)
where N is the number of decoder layers, dij is the self-attention output at position i in layer j. Also, W decj z and W decj x are trainable vectors. d̂j is used as Query, and ê are used as both Key and Value in the cross-attention. For a dialogue text X , we conduct sampling and mixup K times, which is equivalent to providing K extra discrete dialogues X̂k = [ x̂k0, x̂ k 1, ..., x̂ k T ] , k ∈ [1,K] for training. Figure 2(b) shows mixup details.

heading: 3.4 Utilizing Augmented Data by Self-Distillation
text: In general, given X to a dialogue generation model, parameters ϕ of model are optimized by minimizing the negative log-likelihood:
ϕ = argmin ( EX [ − log(Pϕ(xT |X0:T−1])) ]) . (7)
However, as aforementioned, what we obtain are continuous representations of X̂ whereas the corresponding discrete sentences are inaccessible, which makes Eq. 7 intractable. Hence, to utilize the augmented data, we make an assumption that: There is an inaccessible many-to-many dialogue dataset DMtoM . PMtoM describes the conditional distribution of responses given contexts in this dataset. The accessible one-to-one dataset D1to1 is collected by sampling from DMtoM uniformly, and thus P1to1 can be viewed as an approximation of PMtoM .
Based on this assumption, we propose a selfdistillation framework consisting of two steps: (1) It optimizes the model with the original discrete data following Eq. 7. (2) During training, as Pϕ fits P1to1, which is an approximation of PMtoM , the model can use its output given X to teach itself when presented with augmented data, i.e., the representations of X̂:
ϕ = argmin ( DKL [ Pϕ(xT |X0:T−1)||Pϕ(x̂T |X̂0:T−1) ]) ,
(8)
where DKL[·||·] is the KL-divergence (Kullback and Leibler, 1951). In Eq. 8, to remove the gap between utilizing the original discrete data X and the augmented continuous data X̂ in the same architecture, we mix each utterance in X with the
expectations µ0:T . Formally, the overall training objective is to minimize:
L = Lβ︸︷︷︸ Mapping X to β + EX [− log(Pϕ(xT |X0:T−1, µ0:T ))]︸ ︷︷ ︸ Utilizing original discrete data +
1 K
K∑
k
DKL [ Pϕ(xT |X0:T−1, µ0:T )||Pϕ(x̂kT |X̂k0:T−1, zk0:T ) ]
︸ ︷︷ ︸ Utilizing augmented data
(9)

heading: 3.5 Inference
text: The inference goal is to predict xT based on context X0:T−1. First, fθ takes X0:T−1 and outputs corresponding µt for sampling and mixup in the encoder, where t ∈ {0, 1, . . . , T − 1}. Next, the decoder receives the encoder output and an inferred µT to decode the response in an autoregressive manner. To obtain the value of µT , we do not require additional prediction networks. Instead, we can directly derive its value based on the property of Brownian Bridge. Specifically, given the context, we know that for any t:
µt = µ0 + t
T − 1(µT−1 − µ0). (10)
If µT is already known, a Brownian bridge established on µT and µ0 would yield the same µt values. Consequently, we can establish an equality and derive the value of µT as follows:
µt = µ0 + t T (µT − µ0) = µ0 + t T − 1(µT−1 − µ0)
⇒ µT = T T − 1µT−1 − 1
T − 1µ0. (11)
We find that there is hardly a difference in evaluation results when conducting mixup operations with either expectations µ or sampled variables z. To reduce randomness for easier analyses, experiments in below use expectations µ to mixup. Nonetheless, sampling variables gives DialoGPS the ability to generate diverse responses to an arbitrary context and we will discuss it in § 5.4.

heading: 4 Experimental Settings
text: Datasets We conduct multi-turn dialogue generation experiments on two public datasets: DailyDialog (Li et al., 2017) and PersonaChat (Zhang et al., 2018a). DailyDialog contains high-quality multi-turn dialogues collected from daily conversations, and it has many multi-reference versions (Sai et al., 2020; Gupta et al., 2019) denoted by humans, which makes it possible for us to compare
DialoGPS with human annotators. Besides, it is more reliable to evaluate the generalization and performance with multiple references. PersonaChat collects dialogues based on chatters’ profiles. Profiles are not shown to models, so it is more challenging and open to generate proper responses, measuring generalization capacity better.
Baselines and Parameters We compare DialoGPS with (1) Transformer (Vaswani et al., 2017). (2)DD++ (Sai et al., 2020): it is a variant of DailyDialog in which each context has five manually denoted responses. We train a vanilla Transformer on it. (3) TSA (Xie et al., 2022): it is an unsupervised augmentation method in the decoder side. It uses its decoder’s output to construct pseudo-target data which is used to train the model for another round. From a dialogue generation viewpoint, it is a one-to-many method that bootstraps based on one-to-one data. (4) M&D-D (Zhang et al., 2020a): it uses a pre-trained model and BM-25 algorithm to construct new context-response pairs from unpaired sentences. Since it is a single-turn augmentation, given a multi-turn dialogue, we only apply this method to the last two turns. (5) ResBag (Qiu et al., 2019): an augmented VAE-based model. It captures the common feature in the bag of plausible responses and then adds the specific feature to obtain the final output, which utilizes the multiple references better.
Because DialoGPS is a plug-and-play method, we add it to a BARTLarge (Lewis et al., 2020) and compare with DialoFlowLarge (Li et al., 2021). DialoFlow is one of the state-of-the-art pre-trained models in open-domain dialogue generation. It augments the model by modeling the dialogue flow. More details on the implementation and hyperparameters are in Appendix A.1.
Evaluation Metrics We consider three automatic evaluation metrics: BLEU (Papineni et al., 2002), Distinct (DIST) (Li et al., 2016), and BLEURT (Sellam et al., 2020). BLEU measures the word overlap between generated responses and the ground truth. DIST measures the ratio of unique n-grams in the generated responses. Because these two metrics are only sensitive to lexical variation, we evaluate BLEURT, an advanced learned semanticsensitive evaluation metric based on BERT (Devlin et al., 2019). On the evaluation of fine-tuning pre-trained models, we follow (Li et al., 2021) to report METEOR (Lavie and Agarwal, 2007) and
Entropy (Zhang et al., 2018b). For human evaluation, we recruit five evaluators to manually judge 200 samples from each experiment in blind testing, where we set three metrics to comprehensively evaluate the generation quality: whether a response is readable (Read.), coherent (Coh.), and informative (Info.). For each aspect, evaluators can score at ‘bad’, ‘borderline’ and ‘good’.

heading: 5 Results
text: Table 1 shows the automatic evaluation results. On PersonaChat, without access to chatters’ profiles, conversations are so open that there is so much noise in data for models to learn. Therefore, mod-
els prefer safe responses and thus DISTs are relatively low. However, DialoGPS still improves by about 20% in DISTs than the best-performing baseline. Also, BLEU and BLEURT scores imply that DialoGPS matches references more lexically and more semantically. On the multi-reference DailyDialog dataset, DialoGPS gains improvement by a large margin than other strong baselines. Also, most baselines suffer a trade-off between matching the references and diversifying responses. By contrast, DialoGPS performs evenly well on all metrics. DialoGPS also wins 6 out of all 7 metrics compared with the model trained on DD++, the human-written multi-reference training set. Our
results in bold pass the significance test p < 0.01. In Table 2, when adding DialoGPSK=2 to a pretrained BART and fine-tuning on two datasets, it achieves competitive performance as one of the SOTA dialogue generation pre-trained models, DialoFlow. DialoFlow augments the generation with the help of ‘flow’, i.e., the difference of adjacent utterances in continuous space. Their flows are not as flexible as paths sampled from the Brownian Bridge, which is one of the reasons that DialoGPS outperforms DialoFlow in five out of all eight metrics. Table 3 shows human evaluation results. In three metrics, DialoGPS achieves the top rank with solid agreement among evaluators. More evaluation details are in Appendix A.2.

heading: 5.1 Study on Dialogue Paths
text: We conduct an ablation study on the number of sampled dialogue paths K, results are shown in Table 1. On both datasets, with the increase of K, various metrics increase and then reach the bottleneck or slightly decrease. This phenomenon mainly dues to that different from discrete data, sampled paths in continuous space have a information bottleneck, i.e., if K is big enough to cover the most samplable area in the Brownian Bridge, then increasing K further may cause little improvement or even de-
crease due to more noise. We visualize the sampled paths of a conversation with 5 utterances during training in Figure 3. A sample at each time step is denoted as a point and paths are depicted. We can see that the Brownian Bridge area covered by paths is significantly increased when K increases from 1 to 8, but there is a slight difference when K further increases to 16. The visualization confirms automatic evaluation results in Table 1.

heading: 5.2 Component Ablation
text: We study the effect on the performance of the following components in DialoGPS: mixup in the encoder (M.E.), mixup in the decoder (M.D.), and constraints from Eq. 4 that is the optimization of the mapping function (Brown.). The results are reported at the bottom of Table 1. Removing mixup in the decoder (–M.D.) degenerates DialoGPS to a many-to-one mode and thus the performance degrades much, confirming the intuition mentioned in §1. Removing mixup in the encoder(–M.E.) degenerates DialoGPS to a one-to-many pattern which is insufficient compared with the many-tomany pattern, and DIST drops while the BLEU maintains. Nonetheless, the performance is still
competitive with the best one-to-many baseline. Without constraints from Eq. 4 (–Brown.), there is no context-wise correlation among sampled latent variables and the mixup turns to introduce noise. This variant resembles sampling each utterance with a VAE (Bowman et al., 2016; Miao et al., 2016). However, Eq. 11 does not hold anymore so there exist gaps between the inference and the training, and results drop compared to the variant with Eq. 4. Overall, this variant still plays a positive role because adding noise during training is proved to be effective in improving the robustness and generalization of the model (Srivastava et al., 2014; Gao et al., 2021). When there is neither M.D. nor M.E., the method becomes a vanilla transformer.

heading: 5.3 Study on Utterance Representation
text: In §3.3, we defer details on obtaining utterance representations of each turn in a dialogue. We study three variants of encoding an utterance: (1) average embeddings of each token in an utterance (Avg.), (2) average embeddings of each token in an utterance along with position embeddings (Avg. + Pos.), and (3) encode utterances by a GPT-2 (Radford et al., 2019). We conduct this study on the multireference DailyDialog dataset and the results are in Table 4. The simplest method (Avg.) achieves first place. With extra positional information, the performance drops a little, and in this experiment, we observed that the Lβ term in the overall training objective Eq. 9 maintains steadily, but other terms increase a little. An explanation is that features to be mixed with latent variables (e and d) have included positional information and positional information in latent variables introduces redundancy. For (GPT-2), we add a special token ‘<eou>’ at the end of an utterance and view its corresponding output as the utterance representation. (GPT-2) costs much more training time and only beat (Avg.) in one metric. We guess there is an expression capacity gap so we try to (1) train a 4-layer language model to replace the GPT-2 and (2) apply GPT-2 in pre-trained experiments. In both experiments, we do not observe improvement than (Avg.). To sum
up, the simplest (Avg.) achieves the best trade-off between performance and costs so in DialoGPS, we adopt this scheme by default.

heading: 5.4 What Does the Model Learn from Augmented Data?
text: If we mixup with sampled variables instead of expectations during inference, the model obtains the ability to generate diverse responses. Although we do not know what discrete labels augmented data have, to some extent the diverse outputs during inference reflect semantics that augmented data have during training. We provide a case in Table 5. Transformer and ResBag generates incoherent responses, and TSA answers the arrival time but not the way. DD++ reply to the context but does not leads to the follow-up dialogue. M&D-D responds properly but can only provide one answer. We let DialoGPS generate 10 times and report all the outputs along with their respective frequency.
The frequency, the semantics, and lexical features of responses resemble a Gaussian distribution. In this case, ‘you have to go to (get off at) the next stop’ is close to the expectation. As the semantics get farther away, the frequency of other responses are lower. Overall, DialoGPS provides diverse choices to arrive at the barber. This case shows that continuous augmented data do have open dialogue knowledge which is conducive to model generalization.

heading: 6 Conclusion
text: We propose DialoGPS that first augments opendomain and multi-turn dialogue generation from a many-to-many perspective. Specifically, We map dialogues into the continuous semantic space which is modeled by our extended Brownian Bridge and sample dialogue paths to augment training. We propose a self-distillation framework to utilize augmented data despite the inaccessible discrete labels. Empirically, we prove the effect of DialoGPS and study its characteristics. DialoGPS could be a general method that suits seq2seq tasks where the source has multiple sentences and the target is different from the source in semantics, like summarization. However, DialoGPS should be modified according to the unique properties of the task, which is left to study in the future.
Limitations
Similar to other augmentation methods, DialoGPS demands high requirements for computing resources. The training is performed on up to 8 V100 GPUs. On DailyDialog: a vanilla transformer only needs 50 minutes while a non-pretrained DialoGPS takes about 80 minutes when K = 1. Other baselines take about the same amount of time as DialoGPS K = 1. But when DialoGPS achieves its performance peak (K = 16), the training takes 4 hours. Most of time cost comes from sampling which is difficult to be accelerated by GPUs.

heading: Acknowledgement
text: This work was supported by National Natural Science Foundation of China (NSFC Grant No. 62122089), Beijing Outstanding Young Scientist Program NO. BJJWZYJH012019100020098, and Intelligent Social Governance Platform, Major Innovation & Planning Inter-disciplinary Platform for the "Double-First Class" Initiative, Renmin University of China.

heading: A Appendix
text: A.1 Model Implements
In pre-process, we truncate the original long conversations in the dataset with the window size 5. Table 6 shows the dataset statistics.
For non-pretrained experiments, our code is based on fairseq (Ott et al., 2019). We adopt grid search to tune hyper-parameters. On the DailyDialog dataset, the search ranges for learning rate and batch size are {0.00008, 0.00010, 0.00012, 0.00015} and {112, 160}, respectively. On the PersonaChat dataset, the search ranges for learning rate and batch size are {0.00010, 0.00012, 0.00015} and {32, 64}, respectively. We choose the parameter combination with the lowest perplexity in the validation set. Table 7 shows the searched results for each experiment.
Except for batch size and learning rate, the following important settings: the warmup steps are 4000. We use Adam optimizer with β = (0.9, 0.98). Both attention dropout and activation dropout are 0.1. For models trained from scratch, δ on Dailydialog is 12 and 1 3 on PersonaChat. For fine-tuned models, δ is 12 on two datasets. We select the best checkpoint based on the perplexity in the validation set. Early stop patience is 10 epochs. For pre-trained experiments, on both datasets, the batch size is 64 and learning rate is 0.00002. The training is performed on Nvidia V100 GPU. On DailyDialog: our method takes about 80 minutes when K = 1, 4 hours when K = 16, and 8 hours
to finetune a BARTlarge. Because M&D-D does not suit multi-turn settings, we only use it to modify the last two turns with Okapi BM25 algorithm and we finetune BERT on DailyDialog and PersonaChat respectively to measure the fluency between the last two utterances and the fluency between the penultimate sentence and the above as filtration. In our experiments, on two datasets, the paired sentence set Dp is same as the original training set and the unpaired sentence set Du is constructed from all sentences in DD++. On DailyDialog, we use multiple references in DD++ as the response bag of ResBag, and on PersonaChat, we use constructed data from M&D-D as its response bag.
A.2 Evaluation Details
Because some evaluation script links of DialoFlow (Li et al., 2021) are out of date, we can not reproduce NIST (Lin and Och, 2004) scores so we do not report it. This issue was also reported by the community 1. Also, METEOR and Entropy are reproduced. Our reproduced BLEU scores are close to the original paper so we directly quote their results.
Our human evaluators are recruited from Amazon Mturk. In terms of human evaluation, all generated responses are re-capitalized and de-tokenized fairly. The salary for each evaluator is 1 dollar per 10 samples. To give a fair salary, we first evaluate 50 samples by ourselves, calculate the time and effort, and set this amount (samples evaluated by ourselves are just for evaluating the salary, which is not given to evaluators and not reported in the final results).
A.3 QuantiDCE
In addition to the metrics mentioned in the main paper, we further supplement our evaluation with the dialogue-specific metric QuantiDCE (Ye et al., 2021), which measures the coherence between the
1https://github.com/microsoft/DialoGPT/issues/ 72
response and the context. The results show that our proposed DialoGPS outperforms all baseline models.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Following instructions, we add Limitations after Conclusion.

heading: 3 A2. Did you discuss any potential risks of your work?
text: In Limitations.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: The main claims in the paper are stated in the abstract and in the introduction.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We use public datasets and open pre-trained models. These are mentioned in many places in the paper such as Introduction and Experiments.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: We have cited all datasets we use. We have cited open pre-trained models. For example, in Section.1 Introduction and Section.4 Experiments, etc.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? All open code we use are from github where code is licensed under MIT by default.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. In appendix A1, we report the dataset statistics.
C 3 Did you run computational experiments?

heading: In Section 4.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? In terms of parameters, we report model structure, e.g., 4-layer transformer, BART large... which have certain parameters. In appendix A1, we report computational budget and GPU version.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? In 4.1.2 and appendix A1, we discuss experimental setup, including hyperparameter search and best-found hyperparameter values.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We report standard deviation across 5 runs if there’s randomness. We report p-value in t-test and kappa value of human evaluation agreement.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? In 4.1.3, we report evaluation metrics. In 4.1.2, 4.1.3, and 4.5, we report pre-trained models we use.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? In 4.1.3 and 4.2.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? In 4.1.3, we summarized three aspects of evaluation instructions. Also, in appendix A2, before human evaluation, we have de-tokenized and re-capitalized the outputs for a fair and solid evaluation, and thus the instructions are relatively concise.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? We discuss these In appendix A2,
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: However, DialoGPS should be modified according to the unique properties of the task, which is left to study in the future. Limitations
Similar to other augmentation methods, DialoGPS demands high requirements for computing resources. The training is performed on up to 8 V100 GPUs. On DailyDialog: a vanilla transformer only needs 50 minutes while a non-pretrained DialoGPS takes about 80 minutes when K = 1. Other baselines take about the same amount of time as DialoGPS K = 1. But when DialoGPS achieves its performance peak (K = 16), the training takes 4 hours. Most of time cost comes from sampling which is difficult to be accelerated by GPUs