Dynamic Stochastic Decoding Strategy for Open-Domain Dialogue Generation (2024)

Abstract: Stochastic sampling strategies such as topk and top-p have been widely used in dialogue generation task. However, as an opendomain chatting system, there will be two different conversation scenarios, i.e. chit-chat and knowledge-based question answering. In the former situation, responses diversity is essential due to the one-to-many nature in dialogue. The latter, on the other hand, requires less randomness given that stochastic decoding strategy entails the risk of generating incorrect information. As a result, an adaptive and flexible decoding strategy is needed to cope with these two scenarios simultaneously. To this end, we propose the dynamic decoding strategy (DDS), which can adjust the decoding space w.r.t. different contexts. In DDS, both sequence-level and token-level adaptive search can be achieved to adjust the decoding process in a unified framework. Besides, our adaptive algorithm can not only be used during model inference, but it can also be applied during the model training stage to further enhance the performance. Comprehensive experiments indicate that the proposed decoding strategy can consistently improve the performance of pre-trained dialogue models when coupled with four well-used stochastic decoding algorithms.

heading: 1 Introduction
text: Building generative open-domain dialogue system is a significant yet challenging area of deep learning research. It has been widely recognized that the pre-training paradigm, in which large-scale transformer-based models are trained with massive amounts of conversational data, is an effective and promising approach. Some of the more notable works in English include DialoGPT (Zhang et al., 2020b), LaMDA (Thoppilan et al., 2022), Blender (Roller et al., 2021; Shuster et al., 2022), and lately, ChatGPT has attracted great attention
*Corresponding author.
Chit-chat
c: 我好喜欢猫猫(I love cats.) r1: 我好喜欢我的猫猫啊 (I love my cat so much) r2: 我也喜欢猫猫！ (I like cats, too!) r3: 我也喜欢猫猫 (I like cats, too) r4: 我也喜欢猫猫！ (I like cats, too!) r5: 我也好喜欢 (I like it too)
Factual Question Answering
and interest from researchers and the industry. For chinese dialogue models, EVA (Zhou et al., 2021; Gu et al., 2022), PanGu-Bot (Mi et al., 2022) and PLATO (Bao et al., 2020, 2021, 2022) are also excellent options. In recent research, however, it has been demonstrated that decoding strategies play an important role in performance even beyond model architecture (Meister et al., 2022b), whereas standard strategies remain relatively unchanged (Suzgun et al., 2022).
Stochastic decoding algorithms are widely used for dialogue generation task. Users expect varying responses from a chatbot when they input similar queries, or they tend to become bored and lose interest if it only responds with fixed reply. For such a chit-chat scenario, deterministic decoding algorithms, such as greedy search or beam search, are not suitable. Additionally, even when using large pre-trained language models, decoding strategies that aim for high probability output, suffer from in-
11585
credible degeneration issue (Holtzman et al., 2020; Welleck et al., 2020). Consequently, dialogue generation models are inclined to employ stochastic sampling methods such as top-k sampling (Fan et al., 2018) or nucleus sampling (Holtzman et al., 2020), where the probability distribution will be shaped by the temperature T .
Aside from chit-chat, however, there is another scenario for chatbots, namely factual question answering (QA). Unfortunately, since the size of the decoding space required for two different dialog scenarios is different, stochastic sampling methods are not able to handle both simultaneously due to the unified and constant randomness of their decoding processes. As shown in Table 1, with the same temperature, the chit-chat sample has a narrow range of generation, where from r1 to r5 are the same I like cats too-like responses. Whereas, candidates response to the factual question are too diverse, leading to answers are factually incorrect (r1 and r5), with low fluency (r2) or self-contradictory (r1). As a result, the determined sampling randomness will reduce the diversity under chit-chat condition while enlarge it for question answering, which will increase the risk of generating dull responses and wrong answers. In addition, even under the same scenario, different contexts will have varying degrees of decoding flexibility (Csáky et al., 2019). For example, What animals do you like? has larger response space than Do you love cats?. Furthermore, different tokens has different ranges of decoding space within the same utterance (Holtzman et al., 2020).
To resolve the drawbacks of existing stochastic decoding algorithms, we propose a dynamic decoding strategy (DDS) for dialogue generation, which can be combined with mainstream stochastic sampling. The key intuition of dynamic sampling is that the decoding space varies according to the context, therefore the shape of probability distribution should be adjusted adaptively. To achieve this goal, we incorporate an additional diversity predicting head into the dialogue generation model, which is capable of producing the score based on decoding diversity to guide the sampling process adaptively. It only introduces a few parameters and performs decoding at a similar speed to standard dialogue models. The labeled data for training the head is derived from the pre-trained model automatically. Three types of mapping functions are designed, projecting the diversity score to the temperature
for shaping the sampling distribution. In order to control the token generation in a more fine-grained manner, the regression head can be applied to each output token or the whole context, allowing us to control the randomness of decoding at both levels. Apart from inference, adaptive temperature can also be introduced to dialogue training stage to balance the model prediction confidence.
We perform extensive experiments on two union of datasets with two Chinese pre-trained dialogue models. The results show that the DDS can largely improve the performance of four sampling-based decoding algorithms. Human evaluation is also conducted to ensure relevance and fluency of responses while improving diversity.
In summary, our contributions are as follows: • We propose a novel dynamic decoding mecha-
nism for dialogue generation, which can easily be integrated into stochastic decoding strategies and handle different conversational scenarios simultaneously.
• The mechanism can be conducted on both sentence level and token level with three mapping functions, and adaptive temperature training is introduced except for the inference stage.
• Extensive evaluations show that the proposed decoding strategy can largely improve the performance of dialogue models with strong generalization ability when coupled with widely used stochastic decoding strategies.

heading: 2 Background

heading: 2.1 Dialogue Generation
text: In this work, we work with the task of dialogue generation in open-domain, where the input context c = {c1, c2, ...} can be either a chat conversation or a factual question and response r = {r1, r2, ...} is produced accordingly. Dialogue generation models, which are normally pre-trained on massive conversational corpora nowadays, directly models the response probability pθ(r | c), where θ indicates the model parameters. Standard MLE training is used to minimize the negative log-likelihood (NLL) of the training data:
LNLL (Pdata ; θ) = E(c,r)∼Pdata (− logPθ(r | c))
= E(c,r)∼Pdata (− T∑
t=1
logPθ (rt | r<t, c)), (1)
where T is the length of the response r, and the token probability distribution Pθ is typically modeled
as softmax-normalized logits from decoder output zt by:
Pθ (rt | r<t, c) = softmax (zt) (2)
Decoding process is the search for a response token string r∗ according to the given dialogue model θ and context c. Most current generative methods employ one of a few standard decoding strategies, which may be characterized as either deterministic or stochastic in nature.

heading: 2.2 Stochastic Decoding Algorithms
text: Deterministic decoding algorithms like greedy search or beam search, choose the most probable token or path at each step, generating fixed responses through the following form:
r⋆ = argmax r pθ(r | c) (3)
Different from that, stochastic algorithms will generate various responses given the same context by sampling r ∼ pθ(· | c). Based on this, four sampling approaches are briefly presented below.
Temperature Sampling. It is a stochastic sampling method in which the next token is chosen at random based on the new biased probability distribution p
′ θ shaped by the temperature T (Ackley
et al., 1985):
p ′ θ (rt|r<t, c) = exp (pθ (rt|r<t, c) /T )∑ r exp (pθ (r|r<t, c) /T )
(4)
Top-k Sampling Based on temperature sampling, it truncates the probability distribution produced by the model by limiting the sampling space to the tokens with top k highest possibilities before sampling (Fan et al., 2018).
Top-p Sampling. Instead of considering a fixed number of tokens in each decoding step, nucleus (top-p) sampling dynamically selects the smallest set of tokens where the sum of their probabilities is more than the threshold p (Holtzman et al., 2020).
Locally Typical Sampling. It truncates the probability distribution by local informativeness to generate more human-like text (Meister et al., 2022a).

heading: 3 Methodology
text: We propose the dynamic decoding strategy to dynamically compute temperature T ′ w.r.t. different contexts, which replaces T in Equation 4 for all four sampling methods outlined above. The value of this parameter T ′ will vary adaptively according to the size of the decoding space. In this section, we first describe how to build the labeled data about dialogue decoding diversity automatically. After that, we elaborate the regression head trained by it for predicting diversity scores on two levels, which will then be projected to temperature T ′ in accordance with three different mapping strategies. Besides, the dynamic T ′ can also be applied to training stage. The overview of the proposed framework is illustrated in Figure 1.

heading: 3.1 Diversity Score Calculation
text: Labeled data is needed to train the regression head to predict the temperature. D = {(ci, ri)}ni=1 denotes a training set consisting of n dialogues. In order to quantify the range of decoding space available for a given context ci, we seek to determine its diversity score si. To achieve this, instead of expensive human annotations, we construct the labeled data automatically. We are motivated by the strong generation capability of pre-trained dialogue models, which has been trained by a large amount of conversational data from various domains. For each ci ∈ D, the dialogue model generates m candidates {r̂i}m based on it, after which the similarity degree between them will be determined. BERTScore (Zhang et al., 2020a) is a popular learned evaluation metric for doing this. It compares sentences using contextual embeddings from a pre-trained BERT model, computing a similarity score based on the cosine similarity between the sentence embeddings. We trained the Chinese BERT model on wiki2019zh* dataset using the framework from SimCSE (Gao et al., 2021) to calculate the score. The average BERTScore of each {r̂i}m can reflect the diversity of them, deemed as the range of generation space for the context ci. The higher the score, the narrower the range. Consequently, the labeled dataset D′ = {(ci, {r̂i}m, si, )}ni=1 is constructed.

heading: 3.2 Diversity Score Training
text: For training and predicting the diversity score efficiently, we design the regression head based on the dialogue generation model, which maps token representation into a one dimensional vector using two feed-forward networks with non-linearity between them:
score = tanh(WT1 x+ b1)W T 2 + b2 (5)
Then, the predicted score ŝ will be fitted to label si through MSE loss:
LMSE(P ′ data ; θ) = E(c,s)∼P ′data
( (s− ŝ)2 ) (6)
As shown in Figure 1, the regression head can be employed on two levels:
Sentence-level On this condition, the diversity score comes from the head of EOS token (denotes the end of a sentence) of context. Therefore, only ci and si are needed from D′ for training the head.
*https://github.com/brightmart/nlp_chinese_corpus
Token-level For token-level situation, the hidden state of each generated token will provide the diversity score through the regression head. Thus, the head will be trained by each r̂i ∈ {r̂i}m with the same label si.
There are two ways to train the regression head: either individually with other parameters fixed, or jointly with the standard dialogue generation task. In addition, due to some unexpected samples in D′ (please refer to Table 1 and Figure 3), the data filtering process will be conducted before training. Afterwards, the predicted diversity score may be more accurate than the one directly derived from the pre-trained model.

heading: 3.3 Temperature Mapping Strategies
text: After obtaining the diversity score si, we further convert it to guide the dynamic temperature T ′ for Equation 4. As si increases, T ′
should decrease to sharpen the probability distribution of sampling and vice versa. Consequently, three mapping strategies are designed: • Linear Mapping
T (s) = hs+ t0, (7)
where k is the slope. • Exponential Mapping
T (s) = hs + t0, (8)
where h < 1 is the radix to adjust the sharpness of mapping function. • Inverse Sigmoid Mapping
T (s) = h
h+ e s h
+ t0, (9)
where e is the mathematical constant, and h ≤ 1 is a hyperparameter to adjust the sharpness. All t0 is
the offset to make T (s) equals 1 when s reaches the mean value. A visual representation of different mapping strategies is provided in Figure 2. In this way, a dynamic temperature T ′ can be constructed to guide the decoding process adaptively.

heading: 3.4 Dynamic Temperature in Training
text: In addition, same as the inference stage, the temperature T ′ can shape the probability distribution pθ of decoder output z during training process by:
piθ = exp(zi/T ′ )∑
j exp (zj/T ′) , (10)
Thus, the dynamic temperature training can be conducted to balance the model prediction confidence
of chit-chat and factual question answering scenarios respectively. Considering the one-to-many labels, the former is suitable for low confidence training, whereas the latter requires a higher degree of confidence due to the certainty of the knowledge.

heading: 4 Experiments

heading: 4.1 Dataset
text: For training, we use two datasets with different data size to verify the effectiveness of the proposed decoding strategy in two conversation scenarios, each of which contains a chit-chat and a QA dataset. The first is the union (US) of Diamante (Lu et al., 2022), a human-written chit-chat dialogue dataset, and PersonQA, a question answering data about persons.
Both of them are small but with high-quality. The second dataset (UL) has much larger size, consisting of LCCC-base (Wang et al., 2020), and LQA, which includes longer explanations in responses. We calculate the diversity score of each dataset, and then mix the data within the same union. Figure 3 depicts the similarity scores of LCCC and LQA, showing that QA scenario scores are holistically larger than those of chit-chat. The overall trend is in line with expectations, while there are some noise samples with much higher scores in LCCC and lower ones in LQA. Table 1 shows the cases from those parts and it is what we need to solve through our method. Therefore, we filter these extreme data by dropping samples whose score is lower than 0.6 in QA dataset and higher than 0.7 in chit-chat dataset. Table 4 provides the statistics of both unions for training the regression head. Please see Appendix B for more details about QA dataset. For test, all the four sub-sets are evaluated separately. In this work, we mainly focus on Chinese datasets, but we also conduct additional test in Section 4.5 to verify the multilingual availability.

heading: 4.2 Training Settings
text: We take two Chinese pre-trained models: PanGuBot (Mi et al., 2022) containing 350M parameters and EVA2.0 (Gu et al., 2022) with 300M parameters as the underlying generation models to demonstrate that our method is applicable to a wide range of architectures. The regression head is trained for 3 epochs and only takes 0.27% and 0.20% parameters for PanGu-Bot and EVA2.0 respectively. DDS is introduced to four widely used stochastic decoding strategies at sentence level with inverse sigmoid mapping. We set k = 3, p = 0.9, τ = 0.9 for topk, top-p, typical sampling respectively, and T = 1
for all of them including Temperature sampling as common settings. In main experiments, we adopt sentence-level DDS, given that its lower costs than token-level one. The responses are generated 5 times per test.

heading: 4.3 Automatic Evaluation
text: For automatic evaluation, we divide metrics into two groups because chit-chat and QA datasets require different evaluation aspects. For factual QA datasets, the most important thing is to verify the knowledge accuracy w.r.t. the ground truth, thus we adopt the following metrics: BLEU-{1,2,3,4} (Chen and Cherry, 2014), Rouge-{1,2,L} (Lin, 2004) and F1. While for chatting datasets, considering there will be multiple responses for one context, the metrics above are not suitable. Therefore, we utilize these three metrics to evaluate the diversity: Distint-{1,2,3} (Li et al., 2016), Ent-{1,2,3} (word entropy) (Csáky et al., 2019) and BERTScore (calculating the similarity score between five generated responses given the same context).
Table 2 shows the results from PanGu-Bot. As can be seen, the proposed dynamic decoding strategy (DDS) improves the performance of all four well-known stochastic decoding algorithms on four datasets, confirming its general applicability and superiority. Specifically, for LQA and PersonQA, all metrics obtains the best scores, indicating that DDS can generate more accurate answers for QA scenario. Under the same settings, the higher Distinct and Ent scores of Diamante and LCCC verify the diversity in chit-chat scenario. Appendix A shows some generated cases. Table 3 summarizes the result from EVA2.0 in a zero-shot setting, which illustrates similar trends. This observation demonstrates that the proposed DDS can be applied to different model architectures and learning manners.

heading: 4.4 Human Evaluation
text: For chit-chat dataset, although label-related metrics are not suitable, it is also necessary to evaluate its relevance (Rel.) and fluency (Flu.) besides the diversity. So we conduct human evaluation as a supplement to automatic experiment. Rel. reflects how likely the generated response is relevant to its context. Flu. reflects how likely the generated response comes from human. We collect 100 samples for each decoding setting from Diamante and employ three annotators to judge whether the response is in compliance with above standards. Table 5 summarizes the human evaluation results. We can see that the proposed approach has similar results compared with baselines, which indicates that dynamic decoding method maintains the relevance and fluency of responses while improving its diversity. We use Fleiss’s kappa (Fleiss, 1971) to measure the inter-annotator agreement.

heading: 4.5 Multilingual Availability

heading: CQ BLEU-4 F1 ROUGE-2 ROUGE-L
text: Although the proposed method was tested on Chinese corpora, it could work for other languages as well. To demonstrate this, we select English datasets as additional study, ComplexQuestions (Bao et al., 2016) for QA and DailyDialog (Li et al., 2017) for chit-chat. The superior results from Table 6 with top-p sampling support the multilingual availability of DDS. The linguistic phenomena in English differ greatly from those in Chinese, making this experiment a good test of the applicability of the proposed method to non-Chinese languages.

heading: 4.6 Token Level DDS
text: Dynamic decoding at the token level is more finegrained than that at the sentence level. The Figure 4
depicts that the diversity score (the higher, the narrower decoding space) shows a rising trend over the generation step, which is consistent with the heuristic motivation of Lee et al. (2022) that generating the latter part of a sentence require less decoding randomness. Table 7 shows the results at both two levels. The scores of token level on both two datasets are higher than base, verifying the effectiveness of it. Different from Diamante, PersonQA does not perform better at the token level than it does at the sentence level. This may be because the higher randomness of former part within the utterance than sentence level, thus it needs further design for mapping strategy. Figure 4 has shown the effectiveness of predicting diversity score at token level, and we leave the study of exploiting the potential of it as future work.

heading: 4.7 Study of mapping strategies
text: In this section, we study the effectiveness of different mapping strategies. As shown in Table 8, all three types of mapping functions can largely improve the performance on both two scenarios. We simply set h for them as 5, 0.01 and 0.02 respectively and actually the hyperparameters do not need to be specially adjusted. For example, the slope of linear mapping can influence the performance, but
as shown in Table 9, all five different values can outperform the fixed temperature sampling.

heading: 4.8 Domain Adaptation
text: We conduct experiments with out-of-domain test data on EVA2.0 for further generalization evaluation. For chit-chat scenario, we choose CDConv (Zheng et al., 2022), a high-quality dataset for detecting contradiction problem. We only select the first turn of each conversations, where the query is basically the question in chit-chat scenario. For QA scenario, we employ BaikeQA, a QA dataset from Chinese Wiki. The results from Table 10 show that DDS can still outperform the basic decoding strategy, which indicates the generalization ability.

heading: 4.9 Dynamic Training
text: To evaluate the effectiveness of dynamic training (DT), we train the LM head and regression head jointly. The results of Table 11 show that dynamic training is effective in improving performance. The dynamic training and decoding can be performed simultaneously, and the higher performance of DT+DDS indicates that the performance can be further enhanced.

heading: 5 Conclusion
text: In this paper, we discuss the drawbacks of commonly used standard decoding methods for opendomain dialogue generation task. To overcome
them, we present a novel dynamic decoding strategy, DDS, to handle different conversational scenarios concurrently. It can adaptively adjust the decoding space according to different contexts at both sequence and token levels with three mapping functions. Moreover, we further boost the performance by introducing the dynamic temperature to training stage. Extensive experiments demonstrate the superiority and generalization of proposed decoding method.
Limitations
The following are our limitations:
• The contribution for our work may go beyond dialogue generation task. Nowadays, more and more tasks are combined in one model, especially the large language model like ChatGPT. Given that different tasks have different optimal hyper-parameter for decoding temperature, it is badly needed to adjust the temperature adaptively to handle all tasks simultaneously. But we haven’t expended proposed strategy to LLMs.
• Since there is no suitable public Chinese QA conversational dataset available, the QA datasets we utilize are collected through our internal efforts and haven’t been released publicly now. It may be difficult to reproduce our results in this manner.
• Considering reranking approach is very popular and effective for text generation, we
haven’t evaluated the performance of combining it with our proposed method.

heading: Acknowledgements
text: This work is supported by the Beijing Natural Science Foundation, China (Nos. 4222037, L181010).

heading: A Case Study
text: Table 12 and 13 shows some cases generated 5 times by the proposed method and baselines on both two scenarios. From where we can see that the diversity of responses under chit-chat is enhanced and the correctness in QA scenario is also improved.

heading: B Details about QA Datasets
text: Both PersonQA and LQA are our internal data, and each sample is a question-answer pair. Questions are collected from Internet, and answers are manually annotated. Both questions and answers will be classified. PersonQA contains QA pairs
about persons and LQA includes samples that have longer explanations. All the data is anonymized and nonsensitive.
C Insights for Mapping Strategies
We design three mapping strategies in order to cover all major types of mapping trends. Specifically, As shown in Figure 2, Linear mapping simply projects the diversity score to temperature linearly. Exponential mapping has flat slope when diversity score is near the mean value while sharp slope at either end. Conversely, Inverse Sigmoid mapping shows a different trend. According to Section 4.7, all three types of strategies can work on both two scenarios.

heading: Future Work
text: Figure 4 has shown the effectiveness of predicting diversity score at token level, and we leave the study of exploiting the potential of it as future work