Venue: NAACL 2024

# LSTDial: Enhancing Dialogue Generation via Long- and Short-Term Measurement Feedback

Guanghui Ye\({}^{1}\), Huan Zhao\({}^{1*}\), Zixing Zhang\({}^{1}\), Xupeng Zha\({}^{1}\), Zhihua Jiang\({}^{2}\)

\({}^{1}\)College of Computer Science and Electronic Engineering, Hunan University, China

\({}^{2}\)Department of Computer Science, Jinan University, China {yghui,hzhao}@hnu.edu.cn

Corresponding authors.

###### Abstract

Generating high-quality responses is a key challenge for any open domain dialogue systems. However, even though there exist a variety of quality dimensions especially designed for dialogue evaluation (e.g., coherence and diversity scores), current dialogue systems rarely utilize them to guide the response generation during training. To alleviate this issue, we propose LSTDial (**L**ong- and **S**hort-**T**erm **D****ialogue), a novel two-stage framework which generates and utilizes conversation evaluation as explicit feedback during training. Specifically, we fine-tune pre-trained dialogue systems through using turn-level quality feedback in the first stage and further train ever-improving dialogue agents through using dialogue-level quality feedback in the second stage. By using our approach on dialogue systems, capable of enabling dialogue generation with both short-term capabilities (generating more fluent, relevant and varied responses at the turn-level) and long-term capabilities (generating more coherent, engaging and informative responses at the dialogue-level). We implement LSTDial on four strong baseline models and experiment with two open-domain dialogue datasets. Experimental results show that LSTDial achieves significant improvement, enabling to generate better dialogue responses in terms of both human and automatic evaluation.

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 5857–5871 June 16-21, 2024 ©2024 Association for Computational Linguistics

## 1 Introduction

Generating high-quality responses is a key challenge for any open domain dialogue systems [19, 30, 3]. Current state-of-the-art systems do this by training agents with supervised learning on large amounts of labeled text data. However, these labeled data are usually reference responses or pre-defined task goals instead of multi-faceted quality dimensions designed for dialogue evaluation (e.g., relevance, coherence, and engagement, etc.), which leads to a great gap between dialogue generation and evaluation. Response generation systems are typically trained by optimizing the average likelihood of the training data (Maximum Likelihood Estimation, MLE), without a clear signal on the progression of the ongoing conversation [33].

A possible solution to the above issue is to introduce quality annotations of the ongoing conversation during training. However, selecting relevant and meaningful quality dimensions and generating corresponding measurement annotations are crucial. In the dialogue evaluation task, an evaluation

Figure 1: A motivation example of comparing MLE and LSTDial (ours). (a) MLE does not aim at any dialogue quality. Conversely, LSTDial achieves (b\({}_{1}\)) the short-term objective via training with short-term quality feedback and (b\({}_{2}\)) the long-term objective via training with long-term quality feedback.



metric can be evaluated at the turn and dialogue level Yeh et al. (2021); Mehri and Eskenazi (2020). In a turn-level evaluation setting, the goal is, given prior dialogue history (frequently denoted as context) \(c\) and a response \(r\), the metric will assign a score \(s\) of \(r\) conditioned on \(c\) while assuming a measured quality dimension \(q\). Examples of turn-level quality dimensions include fluency, relevance, diversity, appropriateness, and safety, etc Jiang et al. (2022). Conversely, in a dialogue-level evaluation setting, the goal is to evaluate the performance throughout the full dialogue. Examples of dialogue-level quality dimensions include coherence, engagement, topic depth, informativeness, and consistence, etc Zhang et al. (2022, 2023). Because human evaluation is both expensive and time-consuming, automatic evaluation metric models that strongly correlate with human judgments have been proposed. Intuitively, if the mentioned-above quality annotations can be automatically generated and recast into explicit feedback, it is promising to study ever-improving dialogue agents that learn through interaction.

Feedback-based reinforcement learning (RL) has gained increasing attention in dialogue generation Yu et al. (2023); Cai et al. (2023); Zhu et al. (2023). Because human-generated feedback is expensive to obtain, researchers have devised learned feedback generators while assuming one can train downstream models to utilize generated feedback Zhou et al. (2023). Motivated by this, we design feedback functions which depict multi-sided qualities of conversation and utilize generated feedback to guide response generation in the RL-based setting. Considering the balance between efficiency and performance, we fine-tune pre-trained dialogue systems through using turn-level quality feedback (called _short-term feedback_ in our method) while train ever-improving dialogue agents via continual learning through using dialogue-level quality feedback (called _long-term feedback_ in our method). A motivation example is shown in Figure 1.

To this end, we propose the LSTDial (**L**ong-**and **S**hort-**T**erm **D**ialogue) framework. There are two stages involved in the learning process. In the first stage (STDial), we first extract single-turn dialogues from the training set of the experimental dataset and then calculate their turn-level evaluation scores. After this, we fine-tune dialogue systems via a multi-objective loss function including MLE and short-term feedback. In the second stage (LTDial), we further train the first-stage dialogue system but in the RL-based mode. Specifically, we first generate a response \(r\) via the training system conditioned on current history \(c\). Then, we calculate dialogue-level evaluation scores of the conversation <\(c\), \(r\)> as current reward or feedback of \(r\). Particularly, to estimate future reward of \(r\), we generate a future response \(f\) via user simulator Peng et al. (2022); Hu et al. (2023) while assuming the conversation is ongoing. Next, we calculate turn-level evaluation scores of <\(r\), \(f\)> as future reward or feedback of \(r\). Finally, these two rewards are integrated and adapted in policy updating for RL. A novel feature of our methodology is generating and utilizing feedback that accesses relevant and meaningful conversation evaluation for better response generation.

The contributions of this paper are three folds:

* We propose LSTDial, a novel framework which utilizes turn- and dialogue-level conversation evaluation as explicit feedback during training for the first time, alleviating the gap between dialogue generation and evaluation.
* Our approach enables both short-term capabilities (generating more fluent, relevant and varied responses at the turn-level) and long-term capabilities (generating more coherent, engaging and informative responses at the dialogue -level) for dialogue generation.
* We conduct comprehensive experiments on two popular open-domain dialogue datasets and the results show that LSTDial achieves significant improvement in terms of both human and automatic evaluation.

## 2 Related Works

### Dialogue Generation

In recent years, significant progress has been made in the field of open-domain response generation. GPT-2 Radford et al. (2019) is a Transformer-based model which improves performance in a log-linear fashion across tasks. Then, DialoGPT Zhang et al. (2020) is trained on 147M conversation-like exchanges extracted from Reddit comment chains, enabling to generate relevant, content-rich and context-consistent responses. In contrast with DialoGPT, GODEL Peng et al. (2022) leverages a new phase of grounded pretraining that require information external to the 

current conversation to produce good responses. The most related work to ours is PLATO-2 Bao et al. (2021) trained via curriculum learning. There are two stages involved in the learning process. However, the main difference between previous dialogue systems including PLATO-2 and ours is that we elaborately design the quality-driven training method while the others mainly optimize the average likelihood of the training data. For more advances on dialogue systems, we refer to Ni et al. (2023); Deng et al. (2023) for interested readers.

### Dialogue Evaluation

Automatic dialogue evaluation metrics can be divided into two categories: rule-based and model-based Yeh et al. (2021). Rule-based metrics employ heuristic rules to evaluate responses while model-based metrics train deep networks or language models on specific dialogue data. Recently, combined model-based metrics have sprung up. USL-H Phy et al. (2020) composites three quality groups (understandability, sensibleness, likability) in a linear hierarchy. \(IM^{2}\)Jiang et al. (2022) trains sub-metrics with each targeting a specific dimension and integrates them into meaningful categorical metrics. Some recent studies Mendonca et al. (2023); Duan et al. (2023) propose ensemble models that take advantage of the strengths of current evaluation models with prompting Large Language Models, achieving state of the art results on the newest DSTC dialogue evaluation challenge. For more details of dialogue evaluation and DSTC challenges, we refer the readers to Zhang et al. (2021); Rodriguez-Cantelar et al. (2023).

## 3 Methodology

The approach proposed in this paper belongs to the augmentation task of dialogue generation, and the specific objectives and motivations have been elaborated in Figure 1. Figure 2 shows the architecture of our method LSTDial. First, we crafted two sets of conversation evaluators to generate long- and short-term feedback (described in SS3.1 and SS3.2, respectively). Then, the two-stage training of LSTDial will be presented in SS3.3 and SS3.4.

### Short-term Feedback

We integrated three turn-level dialogue evaluation models together to construct Turn-level Evaluator (i.e., TEval). TEval eventually generates a Short-term Feedback (i.e., ST-Feed) and participates in the training of short-term stage. In the figure 2(c), the Fluency, Relevance and Diversity evaluation models included in TEval are presented respectively. **Fluency**: We used VUP Valid Utterance Prediction) proposed by USL-H Phy et al. (2020) for Fluency. We run VUP with the original settings. **Relevance**: Similar to the Fluency metric, we proposed this metric to enhance the relevance prediction by using negative sampling. Specifically, the objective is to discern whether a given context-response pair is relevance or not. RoBERTa-base Liu et al. (2019) is adopted as the text encoder. **Diversity**: We used a new automatic evaluation metric, Sem-Ent Han et al. (2022), which can measure the semantic diversity based on the semantic distribution of generated responses.

\[Feed_{ST}=\alpha_{1}s^{\text{(Fit)}}+\alpha_{2}s^{\text{(Rel)}}+\alpha_{3}s^{ \text{(Div)}} \tag{1}\]

where we set \(\alpha_{1}\) = \(\alpha_{2}\) = \(\alpha_{3}\) = \(\frac{1}{3}\) , which meaning that all three turn-level scores contribute the same to the final short-term feedback.

Notably, the design details of the evaluation model are in Appendix B. A description of all the qualities used to build the feedback is in the Appendix A. We follow DSTC 11 track 41 to select quality in feedback.

Footnote 1: [https://chateeval.org/dstcl1](https://chateeval.org/dstcl1)

Footnote 2: [https://huggingface.co/roberta-large-mnli](https://huggingface.co/roberta-large-mnli)

### Long-term Feedback

In the figure 2(d), we also carefully constructed the Dialogue-level Evaluator (i.e., DEval) to generate the Long-term Feedback (i.e., LT-Feed). DEval is capable of evaluating the Engagement, Coherence, and Informativeness content of the entire conversation. **Engagement**: We trained a discriminatory model using the RED Reddit-based Engagement Dataset Xu et al. (2022). The RED dataset is derived from Reddit and is carefully curated using a unique distant-supervision framework. In this framework, emotional, attentional, behavioral, and reply engagement are combined to form a single score called ENDEX. **Coherence**: We considered the use of the GRADE Huang et al. (2020) as our model for coherence evaluation. We got \(k\) - 1 adjacency pairs for a dialogue containing \(k\) utterances and hence \(k\) - 1 coherence scores. The coherence score at the dialogue-level is calculated by averaging the \(k\) - 1 scores. **Informativeness**: We adopted a pretrained natural language inference (NLI) model2 to calculate the topic depth of a 

conversation to represent the informativeness of a response.

\[Feed_{LT}=\beta_{1}s^{(\text{Eng})}+\beta_{2}s^{(\mathrm{Coh})}+\beta_{3}s^{ (\mathrm{Inf})} \tag{2}\]

Finally, we averaged the three dialogue-level scores, setting \(\beta_{1}\) = \(\beta_{2}\) = \(\beta_{3}\) = \(\frac{1}{3}\) to get the LT-Feed. All the qualities used to construct the feedback are described in the Appendix A. Details of the design of the evaluation model are given in Appendix B.

### Short-term Training

This section will introduce the short-term training of dialogue system, which we refer to as STDial. Figure 2(a) describes in detail the training process. In this strategy, we leverage the TEval introduced in SS3.1 as a form of short-term feedback to enhance the short-term generation performance of the dialogue system. We introduce an additional feedback ( eq. 4) term alongside the generative loss (eq. 3) within the overall loss equation (eq. 5), as demonstrated in Equation 3-5:

\[L_{MLE}=\sum_{t=1}^{len}p\left(y_{t}~{}|~{}C;y_{<t}\right)\log\left(q\left( \hat{y}_{t}~{}|~{}C;y_{<t}\right)\right) \tag{3}\]

\[F_{ST=}\normalsize Eval\left(C,q\left(.~{}|~{}C;y_{<t}\right)\normalsize\right. \tag{4}\]

\[L_{ST}=L_{MLE}-\lambda\cdot F_{ST} \tag{5}\]

where \(C\) is the context and \(q\in R^{|V|\cdot len}\) in the Equation 3 is the softmax output produced by STDial (i.e., the predicted response matrix \(\hat{R}\)). The term \(\hat{y}_{t}\) indicates the decoder's response at the \(t^{th}\) word it generates. The \(L_{MLE}\) of Equation 3 is the cross-entropy loss from STDial. In the Equation 4, the function _Eval_ refers to TEval, which will generate the short-term feedback \(F_{ST}\). The TEval module processes the input as a one-hot representation in a standalone assessment scenario. This means that the input consisting of _len_ tokens first passes through an embedding lookup layer. This embedding lookup layer converts the input into a matrix of shape \(\mathbb{R}^{D\times len}\) to rest of the network. \(D\) represents the dimension of the word embeddings. To enable the differentiability in the loss, we avoid using _argmax_ to get the decoded token. Instead, we access the output of the softmax layer, which represents the likelihood distribution of output lengths across the entire vocabulary (\(\mathbb{R}^{|V|\times len}\)). In order to obtain the same input matrix \(\mathbb{R}^{D\times len}\) for TEval network, we use this distribution to perform a weighted embedding lookup throughout the entire vocabulary. The input form of the transformed predicted responses \(\hat{R}\) is:

\[\mathbb{R}^{D\times len}=\mathbb{R}^{D\times|V|}\times\mathbb{R}^{|V|\times len} \tag{6}\]

Then the context \(C\) and the predicted response \(\hat{R}\) will get the same matrix form \(\mathbb{R}^{D\times len}\), which is concatenated and fed into TEval to get the final feedback. We weighted the feedback by the hyperparameter \(\lambda\) in Equation 5. We set \(\lambda\) to 15 by

Figure 2: An architectural illustration of the introduced LSTDial method. LSTDial consists of (a) Short-term Training and (b) Long-term Training. In (a) we use (c) Short-term Feedback incorporated into regular MLE training to improve the short-term capabilities (relevance and diversity, etc.) of the dialogue. In (b) we build on the (e) RL Iteration algorithm and use (d) Long-term Feedback as a reward to improve the long-term capabilities of the dialogue (coherence and engagement, etc.).

 

grid search for optimising the final loss on the development set. The STDial is trained to minimize cross-entropy loss while maximizing ST-Feed. The TEval dialogue evaluation model is trained on the original corpus, following which the parameters are frozen. The updated loss is back-propagated to update STDial.

### Long-term Training

Figure 2(b) describes in detail the training process and components of LTDial, which is mainly based on Reinforcement Learning (RL) iterations as well as the construction of effective feedback. In this section, RL iteration will be introduced first, followed by reward composition, which is involved in long-term training.

#### 3.4.1 RL Iteration

**History Generator**: In the context of the \(t\)-th utterance, the dialogue state is derived from the dialogue history, denoted as \(H_{t}\) = <\(C_{1}\), \(R_{1}\), ..., \(C_{t}\)>. We define \(k\) as the iteration variable. When \(k\) = 1, we take the first utterance of the dialogue sample as the initialized dialogue history: \(H_{1}\) = \(C_{1}\). The purpose of History Generator is to update \(H_{t}\) to \(H_{t+1}\). When moving to the next iteration \(k\) = \(t+1\), the new history \(H_{k}\) is assigned as <\(C_{1}\), \(R_{1}\), ..., \(C_{t}\), \(R_{t}\), \(C_{t+1}\)> such that the newly-generated \(R_{t}\) and \(C_{t+1}\) can be appended to the previous history \(H_{t}\).

**LTDial**: LTDial takes the dialogue history \(H_{t}\) as input and generates the action \(R_{t}\). We employed the stochastic policy gradient algorithm to optimize LTPial. In our method, the policy takes the form of the dialogue generation network (i.e., \(P(R_{t}|H_{t};\theta)\)), and is defined by the parameters \(\theta\) of LTDial. The objective to maximize is the expected future reward \(\mathbb{E}[r_{k}]\):

\[L_{LT}(\theta)=\mathbb{E}[r_{k}] \tag{7}\]

According to the theorem of policy gradient, The gradient of the expected reward can be calculated given the parameter \(\theta\) of LTDial as:

\[\nabla_{\theta}L_{LT}(\theta)=E\left[\nabla\log P\left(R_{t}\mid H_{t};\theta \right)\cdot r_{k}\right] \tag{8}\]

where \(r_{k}\), which involves long-term feedback, will be described in detail in the next section. We recommend that readers who are interested in the implementation of the policy gradient algorithm refer to Francois-Lavet et al. (2018).

**Combiner**: Combiner acts as a connector to add the newly-generated utterance (e.g., action \(R_{t}\)) to the end of the previous history.

**User Simulator**: We introduced a User Simulator Hu et al. (2023) to imitate human interaction with the dialogue system to obtain better training data for the online RL. Our method simulates a conversation between the User Simulator and our LTDial, and let them take turns talking with each other. We used the base version of GODEL3 Peng et al. (2022) as our User Simulator. The whole interaction is shown in Figure 2(e). Through extensive experiments, we chose iteration variable \(k\) = 5 to ensure moderate long-term training difficulty and better performance.

Footnote 3: [https://github.com/microsoft/GODEL](https://github.com/microsoft/GODEL)

#### 3.4.2 Reward Composition

In our approach, the reward composition of \(t\)-th RL iteration consists of the current reward \(r^{c}_{t}\) and the future reward \(r^{f}_{t}\). For the dialogue history \(H_{t}\) and response \(R_{t}\), the current reward computes the score of <\(H_{t}\), \(R_{t}\)>, which measures the importance of \(R_{t}\) to the overall dialogue history \(H_{t}\). Conversely, the future reward computes the score of <\(R_{t}\), \(F_{t}\)> where \(F_{t}\) is the content of the conversation that will probably be talked about in the following (i.e., Follow-up utterances), which measure the importance of \(R_{t}\) to the future utterances \(F_{t}\).

**Current Reward**: We used long-term feedback as current reward \(r^{c}_{t}\):

\[r^{c}_{t}=Feed_{LT}(H_{t},R_{t}) \tag{9}\]

Figure 3: The calculation procedure for Future Reward.

 



**Future Reward**: The entire process of calculating future reward \(r_{t}^{f}\) is shown in Figure 3. We used the User Simulator as generator \(G\) and the top-k sampling to generate future dialogues, while using TEval as a score \(S\) to pick \(k\)-best future utterances. By using \(G\) and \(S\) twice, we finally get the set of future utterances \(F_{t}\) = (\(f_{1},f_{2},\ldots,f_{k}\)) about \(R_{t}\) and the importance \(W_{t}\) = (\(w_{1},w_{2},\ldots,w_{k}\)) about \(F_{t}\). Then, we spliced \(R_{t}\) with all \(f_{i}\) to get \(k\) <\(R_{t}\), \(f_{i}\)> pairs. Finally it is scored using TEval and weighted using \(w_{i}\):

\[r_{t}^{f}=\sum_{i=1}^{k}w_{i}Feed_{ST}(R_{t},f_{i}) \tag{10}\]

The final reward \(r_{final}\) consists of \(r_{t}^{c}\) and \(r_{t}^{f}\):

\[r_{final}=\delta_{1}r_{t}^{c}+\delta_{2}r_{t}^{f} \tag{11}\]

where \(\delta_{1}\) and \(\delta_{2}\) are learnable weights for training.

## 4 Experiment Setup

### Datasets

Following the works Cai et al. (2020); Sun et al. (2023) on augmented dialog generation, we evaluated our method over two English open domain dialogue datasets: DailyDialog Li et al. (2017) and Opensubtitles Lison and Tiedemann (2016). For the purpose of our study, we conducted pre-processing on these two datasets. We extracted \(T-1\) single-turn dialogues [(\(u_{1}\), \(u_{2}\)), (\(u_{2}\), \(u_{3}\)), ..., (\(u_{T-1},u_{T}\))] from a multi-turn dialogue (\(u_{1},u_{2},\ldots,u_{T}\)), where \(u\) represents an utterance. We gathered all dialogue pairs, reduced the repeat pairs, and divided them into training, validation and test sets. We splited the DailyDialog dataset to 54,894/6,000/5,700, and OpenSubtitles to 64,000/8,000/8,000.

### Baselines

We compared our method on following classic generation baselines: (1) **Transformer** Vaswani et al. (2017): an encoder-decoder architecture that relies solely on attention mechanisms. (2) **GPT-2**Radford et al. (2019): a large-scale pre-trained language model, which is fine-tuned using the full training dataset. (3) **PLATO** Bao et al. (2021): a pretrained dialogue generation model based on UniLM Dong et al. (2019). We utilized the v1 version with the 132M parameters. (4) **BART** Lewis et al. (2020): a pretrained sequence-to-sequence transformer model. We used the base version with the 110M parameters.

### Implementation Details

Our implementation is based on the open-source toolkit Transformers Wolf et al. (2020). All the experiments are conducted on 4 nvidia 3090 24GB GPUs. We adopted AdamW optimizer Loshchilov and Hutter (2019) with a learning rate of 3e-5 and a batch size of 16 for training. The best performing model on the validation set was selected for testing. The choice of all the hyperparameters depends on how they perform on the development set. We used top-k sampling Fan et al. (2018) during dialogue generation with the temperature as 0.7. The discount factor \(\gamma\) to 0.95 during the reward accumulation phase of RL. During the iteration of reinforcement learning, we can choose different Iteration variable \(k\) to balance the computation cost and the performance. We test \(k\in\{3,5,7\}\) on both datasets and find \(k\) = 5 can get moderate long-term training difficulty and better performance.

### Main Results

#### 4.4.1 Automatic Evaluation

We instantiated our method on a set of mainstream dialogue generation models including Transformer, GPT-2, PLATO and BART. The automatic evaluation results are shown in Table 1. For each pair such as Plato+LST (i.e., using the long- and short-term training method in this paper) vs. Plato (i.e., using the original MLE method), as can be seen, our model consistently surpasses vanilla baselines across all automated metrics on both datasets, underscoring the effectiveness and versatility of our methodology. It can be observed that the results of GPT2 and Tranformer improve more significantly compared to models with larger number of parameters such as BART and PLATO. This is mainly due to the fact that large-scale models such as BART and PLATO already had a better dialogue generation capability.

#### 4.4.2 Human Evaluation

Table 2 reports the result of human evaluation. We employed three annotators to assess the quality of the responses generated on the DailyDialog dataset. Totally, 100 randomly sampled responses generated by each model are rated by each annotator on six different dimensions. Ratings ranged from 1 to 3, indicating poor, normal and good (Assessment questions are consistent with the quality description in Appendix A). We conducted two groups of experiments. The first group

(PLATO+LST vs. PLATO) proves the effectiveness of the training method proposed in this paper, and the PLATO model using our method significantly outperforms the origin PLATO in terms of all the evaluation aspects. In the second group (STDial vs. LTDial), STDial has better performance in turn-level quality, while LTbial has better performance in dialogue-level quality. This suggests that long-term and short-term training methods (LSTDial) can accomplish the long-term (Engagement/Coherence/Informativeness) and short-term(Fluency/Diversity/Relevance) objectives of dialog generation, respectively.

#### 4.4.3 LLMs Evaluation

Since rule-based evaluation metrics are not effective in evaluating the performance of dialogue generation, we turn our attention to large language models. Specifically, we used GPT-4 [19] to conduct a dialogue evaluation of LSTDial and its comparison models on the same 100 well-collected samples as in the previous section. This process begins with an instruction that directs the Large Language Models (LLMs) to provide both reasons and ratings for a given dialogue. To enhance the comprehension of LLMs, we also supply evaluation criteria standards. See Appendix C for complete instruction. In Figure 4 the results of the GPT-4 Evaluation are presented. In the left figure, it is demonstrated that LSTDial improves the results across the board in six areas, while STOdial and LTPial are only effective in the short- or long-term qualities. The right figure, on the other hand, demonstrates the significant enhancement of using LSTDial on different models.

### Ablation Study

We list the results of the ablation study in Table 3, aiming to investigate the influence of different modules in our proposed method. It can be seen that if we removed STDial or LTDial, the performance of all metrics drops. This suggests that both the short-term and long-term training modules proposed in this paper are important for enhanced dialogue generation. We also observed that after removing LTDial, the Distinct-1/2/3 are reduced by 41.32%

\begin{table}
\begin{tabular}{l|c|c c c|c c c c|c c} \hline  & Models & Dist-1 \textasciitif{} & Dist-2 & Dist-3 & BLEU-1 \textasciitif{} & BLEU-2 & BLEU-3 & BLEU-4 & Len. & PPL ↓ \\ \hline (a) & Transformer & 1.19 & 6.21 & 15.13 & 13.29 & 2.28 & 1.13 & 0.76 & 6.74 & 25.25 \\  & Transformer (\(\blacktriangle\)) & **4.55** & **16.64** & **32.14** & **33.76** & **27.91** & **26.55** & **18.78** & **12.24** & **18.07** \\ \cline{2-11}  & GPT-2 & 2.16 & 7.44 & 16.15 & 15.27 & 2.84 & 1.66 & 0.78 & 7.01 & 15.91 \\  & GPT-2 (\(\blacktriangle\)) & **4.96** & **19.04** & **35.77** & **17.63** & **25.15** & **19.73** & **11.45** & **10.74** & **14.39** \\ \cline{2-11}  & PLATO & 6.20 & 28.41 & 49.93 & 31.81 & 24.85 & 20.42 & 16.12 & 8.31 & 12.73 \\  & PLATO (\(\blacktriangle\)) & **6.29** & **33.76** & **65.28** & **36.18** & **29.92** & **24.97** & **19.62** & **11.22** & **7.65** \\ \cline{2-11}  & BART & 7.64 & 24.42 & 40.99 & 9.97 & 8.01 & 6.69 & 5.33 & 6.35 & 11.59 \\  & BART (\(\blacktriangle\)) & **7.95** & **34.16** & **64.39** & **18.08** & **13.96** & **12.47** & **8.17** & **10.87** & **6.48** \\ \hline \hline (b) & Transformer & 1.57 & 3.28 & 6.39 & 8.76 & 2.35 & 1.21 & 0.87 & 7.41 & 27.83 \\  & Transformer (\(\blacktriangle\)) & **3.81** & **9.45** & **23.67** & **38.13** & **30.77** & **26.38** & **22.69** & **11.02** & **25.76** \\ \cline{2-11}  & GPT-2 & 3.12 & 4.32 & 7.29 & 10.97 & 3.30 & 2.15 & 1.15 & 8.47 & 26.47 \\  & GPT-2 (\(\blacktriangle\)) & **5.07** & **13.66** & **30.17** & **23.96** & **23.42** & **18.05** & **13.43** & **9.85** & **25.16** \\ \cline{2-11}  & PLATO & 6.06 & 25.55 & 47.82 & 35.81 & 28.85 & 24.14 & 19.26 & 10.83 & 24.25 \\  & PLATO (\(\blacklozenge\)) & **7.45** & **25.87** & **63.65** & **45.76** & **35.98** & **31.63** & **26.07** & **13.91** & **5.51** \\ \cline{2-11}  & BART & 7.17 & 23.92 & 39.30 & 10.24 & 8.37 & 7.06 & 5.65 & 7.51 & 21.84 \\  & BART (\(\blacktriangle\)) & **9.87** & **36.75** & **68.43** & **16.32** & **15.33** & **15.81** & **9.04** & **10.12** & **4.43** \\ \hline \end{tabular}
\end{table}
Table 1: Automatic evaluation results (%) on (a) DailyDiago and (b) OpenSubtitles. The symbol “\(\blacklozenge\)” indicates that the model is trained using our proposed method. Bold numbers mean that the enhancement to the best baseline is statistically signifcant (a two-tailed paired t-test with p-value ¡0.01).

Figure 4: GPT4-Evaluation. Variant of LSTMial (left); Original model using the LSTDial method (right).

 

/ 37.62% / 51.03%, respectively. However, the BLEU-1/2/3/4, only dropped an average of 9.39%. This indicates that LTDial, has a greater impact on diversity metric. An more obvious phenomenon is the dramatic reduction of BLEU-1/2/3/4 by 71.92% / 92.69% / 95.18% / 96.49% respectively after removing STDial. This is due to the fact that STDial incorporates MLE training, which is capable of generating dialogue responses with high BLEU values. When only the RL-based LTbial is left, the BLEU values naturally drops significantly since ground-truth is not involved in the training. Furthermore, most of the metrics decreased after removing the corresponding feedback in both STDial and LTPial, which are in line with our expectations.

### Discussion

We aim to address the following research questions (RQs) about LSTDial: (1) Does the two-type feedback work effectively enough? (2) How does the order of the two-stage training influence the method? (3) Can our method outperform other RL-based methods? (4) Can it be effectively implemented on LLMs? (5) What insights can be drawn from the case study?

**RQ1: Feedback Validity**

The previous section 4.4.2 presents the 100 samples scored by human. To verify the validity of the two types of feedback, we calculated the fine-grained correlation of short-term feedback as well as long-term feedback with human evaluation on there samples, respectively. From Table 4, it can be observed that the two types of feedback has significant correlation with human evaluation4. Considering the substantial individual differences in evaluating open-domain conversations, we observed that our evaluators (contained TEval and DEval) can be used to provide effective feedback for a human-chatbot conversation.

Footnote 4: Pearson correlation of 0.2-0.5 can be proved to be moderately to highly significant with human ratings in dialogue evaluation tasks Yeh et al. (2021).

#### RQ2: Impact of Training Order

We empirically find that it is easier for dialogue system to converge when training at the long-term stage after it is already trained at the short-term stage. Thus, we only report the results of the designed order (short-term first and then long-term) in the paper. Here, we compare with the other order (long-term first and then short-term) and report the results at different RL steps, as shown in Table 5, using the GPT-4 evaluation guidelines given in Appendix C. The experiments employed the PLATO model trained by our LSTDial. The results show that the improvement in the long-term first is less efficient compared to the short-term first.

#### RQ3: Comparison with RL-based methods

We conducted experiments to compare the effectiveness of LSTMial with other RL-based ap

\begin{table}
\begin{tabular}{l|c|c} \hline Metric & Pearson Corr & Spearman Corr \\ \hline Flu. & 0.24 & 0.25 \\ Rel. & 0.48 & 0.45 \\ Div. & 0.35 & 0.33 \\ \hline \hline Eng. & 0.28 & 0.27 \\ Coh. & 0.41 & 0.43 \\ Inf. & 0.34 & 0.34 \\ \hline \end{tabular}
\end{table}
Table 4: Fine-grained correlation of short-term feedback (top) as well as long-term feedback (down) with human evaluation, respectively. All values are statistically significant to p ¡ 0.05, unless in italic.

\begin{table}
\begin{tabular}{l|c c c|c c c c|c c} \hline  & Dist-1 \(\uparrow\) & Dist-2 & Dist-3 & BLEU-1 \(\uparrow\) & BLEU-2 & BLEU-3 & BLEU-4 & Len. & PPL \(\downarrow\) \\ \hline LSTDial & **4.55** & **16.64** & **32.14** & **33.76** & **27.91** & **26.55** & **18.78** & **12.24** & **18.07** \\ w/o STOI Dial & 3.75 & 14.28 & 30.07 & 9.48 & 2.04 & 1.28 & 0.66 & 10.48 & 19.31 \\ w/o LTDIial & 2.67 & 10.38 & 15.74 & 31.91 & 24.89 & 23.73 & 16.77 & 8.97 & 20.42 \\ \hline \hline STDial & **2.67** & **10.38** & **15.74** & **31.91** & **24.89** & **23.73** & **16.77** & **8.97** & **20.42** \\ w/o ST-Feed & 1.19 & 6.21 & 15.13 & 34.33 & 25.43 & 24.59 & 16.76 & 6.74 & 25.25 \\ \hline \hline LTDial & **3.75** & **14.28** & **30.07** & **9.48** & **2.04** & **1.28** & **0.66** & **10.48** & **19.31** \\ w/o LT-Feed & 2.11 & 9.38 & 20.56 & 9.32 & 0.98 & 1.21 & 0.43 & 9.83 & 18.70 \\ w/o Future Reward & 3.04 & 12.76 & 25.73 & 8.45 & 1.47 & 1.09 & 0.65 & 10.21 & 19.25 \\ \hline \end{tabular}
\end{table}
Table 3: The ablation test of our model (%) on DailyDialog, which is implemented on the basic Transformer. Numbers in bold mean that the improvements to the ablation models are statistically signifcant.

\begin{table}
\begin{tabular}{l|c c c c c} \hline \hline Steps & \multicolumn{2}{c}{Short-term First} & \multicolumn{2}{c}{Long-term First} \\  & Eng. & Coh. & Inf. & Eng. & Coh. & Inf. \\ \hline
20000 & 2.68 & 2.94 & 2.48 & 1.83 & 2.07 & 1.80 \\
40000 & 3.27 & 3.34 & 3.08 & 2.11 & 2.26 & 1.90 \\
60000 & 3.62 & 3.81 & 3.56 & 2.14 & 2.20 & 1.92 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Results of dialogie-level evaluation of two order settings at different steps of long-term training.

 

proaches applied to dialogue generation tasks. Table 6(a) shows the results of LSTDial as well as two open-source RL-based methods VHRL [14] and HRLG [20] on DialyDialog. The above experimental results show that our LSTDial outperforms both of VHRL and HRLG in terms of all metrics except for BLEU-3 (where LSTDial is slightly inferior to VHRL). In particular, for Dist-3, BLEU-4, and PPL, LSTDial exhibits a significant advantage over other methods, showing that the model enhanced with our method can generate responses that are more diverse, fluent, and similar responses to the reference. In addition, the reasons why the RL algorithm should be used in the long-term stage rather than short-term are given in Appendix D.

#### RQ4: Implementation with LLMs

To investigate whether LSTDial is effective on LLMs, we implemented our approach on a small-sized LLM called INstructGPT (1.3B) 5 [7] and showed its comparative results on the DailyDialog dataset in Table 6(b). The above experimental results show that (1) INstructGPT outperforms PLATO and even PLATO(\(\blacktriangle\)) in terms of all metrics, showing the powerful ability of LLMs for our task; (2) INstructGPT(\(\blacktriangle\)) slightly outperforms INstructGPT in terms of all metrics except for BLEU-2, showing that our method is still effective for the implementation on LLMs.

Footnote 5: We reproduced INstructGPT based on: [https://github.com/LanXiu0523/RLHF_instructGPT](https://github.com/LanXiu0523/RLHF_instructGPT)

### RQ5: Case Study

We randomly sampled cases of dialogue responses from DailyDialog and then compared the MLE to our approach (implemented on the PLATO model). From Table 7(a), it can be seen that the model using LSTDial can generate a more diverse and interesting response. We have further observed responses generated by our model across different stages in Table 7(b). It can be observed that the baseline's response (i.e., the original PLATO without our trainings method) is simple and generic, while the STDial response is more informative and relevant. In particular, LSTDial tends to ask questions and show emotion to further deepen the dialogue. This response demonstrates long-term capabilities at the dialogue-level. See Appendix E for more case studies along with detailed explanations.

## 5 Conclusion

In this study, we introduced LSTDial, a novel framework that utilizes turn- and dialogue-level conversation evaluation as explicit feedback during training. By bridging the gap between dialogue generation and evaluation, LSTDial enables dialogue systems to exhibit both short-term capabilities at the turn-level and long-term capabilities at the dialogue-level. Experimental results on two popular open-domain dialogue datasets demonstrate the significant improvement achieved by LSTDial in terms of both human and automatic evaluation metrics. These findings highlight the importance of incorporating conversation evaluation into dialogue system training for generating high-quality responses. Future research will explore more feedback mechanisms to better utilize feedback to enhance dialogue generation.

\begin{table}
\begin{tabular}{l|c|c c c|c c c c|c c} \hline  & Models & Dist-1 \(\uparrow\) & Dist-2 & Dist-3 & BLEU-1 \(\uparrow\) & BLEU-2 & BLEU-3 & BLEU-4 & Len. & PPL \(\downarrow\) \\ \hline (a) & VHRL [14] & 4.10 & 13.73 & 25.07 & 32.45 & 25.72 & **26.68** & 17.73 & 9.41 & 21.22 \\  & HLRLG [14] & 4.21 & 12.65 & 20.33 & 31.43 & 26.89 & 24.95 & 15.47 & 10.73 & 19.86 \\  & LSTDial (ours) & **4.55** & **16.64** & **32.14** & **33.76** & **27.91** & 26.55 & **18.78** & **12.24** & **18.07** \\ \hline \hline (b) & INstructGPT & 13.21 & 40.76 & 69.75 & 38.15 & **30.68** & 25.71 & 19.95 & 14.33 & 6.41 \\  & INstructGPT (\(\blacktriangle\)) & **15.10** & **41.13** & **69.76** & **39.14** & 29.92 & **26.40** & **20.48** & **15.02** & **6.38** \\ \hline \end{tabular}
\end{table}
Table 6: Automatic evaluation results (%) on DailyDialog. (a) Comparison of our approach (instantiated on the naive Transformer) with two RL-based dialogue generation methods. (b) We implemented our approach on INstructGPT (denoted by “\(\blacklozenge\)”) to explore the applicability of LSTDial for Large Language Models (LLMs).

\begin{table}
\begin{tabular}{p{28.5pt} p{341.4pt}} \hline - So you have all the news on broadcast? \\ \hline (a) & **Ground-truth:** & Yes, it’s so helpful. \\  & **MILE:** & Yep, got it. \\  & **LSTDial**: & Yes, I do. It keeps me informed! \\ \hline - You need to know the type of shoes you want . \\ \hline (b) & **Baseline**: & I’m not sure. \\  & **STDial**: & Guess I’ll decide on the shoes once \\  & I’m at the store. \\  & **LSTM**: & Knowing the type of shoes I want \\  & is key, but have you ever had a pair of shoes that \\  & pleasantly surprised you? \\ \hline \end{tabular}
\end{table}
Table 7: (a) Response generated by MLE and our method (the top case). (b) Case of response generation at different stages of the model (the bottom case).

 



### Limitation

Firstly, due to limited resources, this paper did not attempt to use larger evaluation models to generate feedback, as well as LLMs with a larger number of parameters to apply our approach. Secondly, in the SS3.4.1 RL Iteration phase, we defaulted to taking the first utterance of the dialogue sample as the initialized dialogue history, without attempting to use a greater number of utterances. This may affect the difficulty and effectiveness of dialogue interaction, as a greater number of initial utterances would lead to richer background knowledge for the conversation. Finally, a key factor contributing to the successful performance of LSTDial is the sequence of short-term and long-term training, meaning that LSTDial needs to be trained strictly in order.

## Ethics Statement

There are no ethical issues involved in this study. All datasets used in this paper are publicly available. Due to the limitations of rule-based metrics, we conducted a human evaluation of the response generation quality. We recruited three part-time postgraduate students to conduct the human evaluation with clearly defined evaluation rules. They were paid 5 CNY per sample for their work during the evaluation period.

## Acknowledgements

This work was supported by National Natural Science Foundation of China (NSFC Grant No. 62076092).

## References

* A. M. Abadi, R. H. Larochelle, J. L. Andersen, S. B. Kessler, D. E. C. Smith, P. G. Baker, and T. W. Lee (2018)A review of machine learning techniques for language understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-10. Cited by: SS1.
* N. Ahmed, F. Chen, Y. Song, C. Zhang, X. Zhao, and D. Yin (2020)Data manipulation: towards effective instance learning for neural dialogue generation via learning to augment and reweight. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 6334-6343. External Links: Link, Document Cited by: SS1.
* 12th National CCF Conference, NLPCC 2023, Foshan, China, October 12-15, 2023, Proceedings, Part III, pp. 376-387. Cited by: SS1.
* Y. Deng, W. Lei, W. Lam, and T. Chua (2023)A survey on proactive dialogue systems: problems, methods, and prospects. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR, China, pp. 6583-6591. Cited by: SS1.
* L. Dong, N. Yang, W. Wang, F. Wei, X. Liu, Y. Wang, J. Gao, M. Zhou, and H. Hon (2019)Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 13042-13054. Cited by: SS1.
* H. Duan, J. Wei, C. Wang, H. Liu, Y. Fang, S. Zhang, D. Lin, and K. Chen (2023)Botchat: evaluating lms' capabilities of having multi-turn dialogues. CoRRabs/2310.13650. Cited by: SS1.
* A. Fan, M. Lewis, and Y. Dauphin (2018)Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia, pp. 889-898. Cited by: SS1.
* A. Ferron, A. Shore, E. Mitra, and A. Agrawal (2023)MEEP: is this engaging? prompting large language models for dialogue evaluation in multilingual settings. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 2078-2100. Cited by: SS1.
* V. Francois-Lavet, P. Henderson, R. Islam, M. G. Bellemare, and J. Pineau (2018)An introduction to deep reinforcement learning. CoRRabs/1811.12560.



Sarik Ghazarian, Ralph M. Weischedel, Aram Galstyan, and Nanyun Peng. 2020. Predictive engagement: An efficient metric for automatic evaluation of open-domain dialogue systems. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, pp. 7789-7796. Cited by: SS1.
* S. Han, B. Kim, and B. Chang (2022)Measuring and improving semantic diversity of dialogue generation. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 934-950. External Links: Link Cited by: SS1.
* Z. Hu, Y. Feng, A. T. Luu, B. Hooi, and A. Lipani (2023)Unlocking the potential of user feedback: leveraging large language model as user simulators to enhance dialogue system. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM 2023, Birmingham, United Kingdom, October 21-25, 2023, pp. 3953-3957. External Links: Link Cited by: SS1.
* L. Huang, Z. Ye, J. Qin, L. Lin, and X. Liang (2020)GRADE: automatic graph-enhanced coherence metric for evaluating open-domain dialogue systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 9230-9240. External Links: Link Cited by: SS1.
* Z. Jiang, G. Ye, D. Rao, Di Wang, and X. Miao (2022)Im2: an interpretable and multi-category integrated metric framework for automatic dialogue evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, H. Walls, C. Cortes, R. Fergus, P. Dollar, and R. Garnett (Eds.), pp. 11091-11103. External Links: Link, Document Cited by: SS1.
* K. Kann, A. Ebrahimi, J. J. Koh, S. Dudy, and A. Ronconce (2022)Open-domain dialogue generation: what we can do, cannot do, and should do next. In Proceedings of the 4th Workshop on NLP for Conversational AI, ConvAI@ACL 2022, Dublin, Ireland, May 27, 2022, pp. 148-165. External Links: Link, Document Cited by: SS1.
* M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer (2020)BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 7871-7880. External Links: Link, Document Cited by: SS1.
* J. Li, M. Galley, C. Brockett, J. Gao, and B. Dolan (2016)A diversity-promoting objective function for neural conversation models. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pp. 110-119. External Links: Link, Document Cited by: SS1.
* December 1, 2017
- Volume 1: Long Papers, A. F. Cohn, Y. Bengio, and J. Salam (Eds.), pp. 986-995. External Links: Link, Document Cited by: SS1.
* P. Lison and J. Tiedemann (2016)Opensubtitles2016: extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation LREC 2016, Portoroz, Slovenia, May 23-28, 2016, European Language Resources Association (ELRA), Cited by: SS1.
* Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019)Roberta: a robustly optimized BERT pretraining approach. CoRRabs/1907.11692. External Links: Link, 1907.11692 Cited by: SS1.
* I. Loshchilov and F. Hutter (2019)Decoupled weight decay regularization. In Proceedings of the 7th International Conference on Learning Representations, ICLR, Cited by: SS1.
* Sh. Mehri and M. Eskenazi (2020)Unsupervised evaluation of interactive dialog with dialoppt. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGdial 2020, 1st virtual meeting, July 1-3, 2020, pp. 225-235. External Links: Link, Document Cited by: SS1.
* J. Mendonca, P. Pereira, J. Paulo Carvalho, A. Lavie, and I. Trancoso (2023)Simple LLM prompting is state-of-the-art for robust and multilingual dialogue evaluation. CoRRabs/2308.16797. External Links: Link, 2308.16797 Cited by: SS1.
* M. Mesgar, S. Bucker, and I. Gurevych (2020)Dialogue coherence assessment without explicit dialogue act labels. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 1439-1450. External Links: Link, 2002.0202 Cited by: SS1.
* J. Ni, T. Young, V. Pandelea, F. Xue, and E. Cambria (2023)Recent advances in deep learning based dialogue systems: a systematic survey. Artif. Intell. Rev.56 (4), pp. 3055-3155. External Links: Link, ISSN 1573-649X, Link Cited by: SS1.
* L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askel, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe (2019) 

2022. Training language models to follow instructions with human feedback. In _Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022_.
* Park et al. (2023) ChaeHun Park, Seungil Chad Lee, Daniel Rim, and Jaegul Choo. 2023. Density: Open-domain dialogue evaluation metric using density estimation. In _Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 14222-14236.
* Peng et al. (2022) Baolin Peng, Michel Galley, Pengcheng He, Chris Brockett, Lars Liden, Elnaz Nouri, Zhou Yu, Bill Dolan, and Jianfeng Gao. 2022. MODEL: large-scale pre-training for goal-directed dialog. _CoRR_, abs/2206.11309.
* Phy et al. (2020) Vitou Phy, Yang Zhao, and Akiko Aizawa. 2020. Deconstruct to reconstruct a configurable evaluation metric for open-domain dialogue systems. In _Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020_, pages 4164-4178. International Committee on Computational Linguistics.
* Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. _OpenAI Blog_.
* Rodriguez-Cantelar et al. (2023) Mario Rodriguez-Cantelar, Chen Zhang, Chengguang Tang, Ke Shi, Sarik Ghazarian, Joao Sedoc, Luis Fernando D'Haro, and Alexander Rudnicky. 2023. Overview of robust and multilingual automatic evaluation metrics for open-domain dialogue systems at DSTC 11 track 4. _CoRR_, abs/2306.12794.
* Rohmatillah and Chien (2023) Mahdin Rohmatillah and Jen-Tzung Chien. 2023. Hierarchical reinforcement learning with guidance for multi-domain dialogue policy. _IEEE ACM Trans. Audio Speech Lang. Process._, 31:748-761.
* 23, 2021_, pages 300-325. Association for Computational Linguistics.
* Saleh et al. (2020) Abdelrhman Saleh, Nutasha Jaques, Asma Ghandehartioun, Judy Hanwen Shen, and Rosalind W. Picard. 2020. Hierarchical reinforcement learning for open-domain dialog. In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020_, pages 8741-8748. AAAI Press.
* Sato et al. (2020) Shiki Sato, Reina Akama, Hiroki Ouchi, and Jun Suzuki and. 2020. Evaluating dialogue generation systems via response selection. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020_, pages 593-599. Association for Computational Linguistics.
* Stasaki and Hearst (2022) Katherine Stasaki and Marti A. Hearst. 2022. Semantic diversity in dialogue with natural language inference. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022_, pages 85-98. Association for Computational Linguistics.
* Sun et al. (2023) Bin Sun, Yitong Li, Fei Mi, Weichao Wang, Yiwei Li, and Kan Li. 2023. Towards diverse, relevant and coherent open-domain dialogue generation via hybrid latent variables. In _Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023_, pages 13600-13608. AAAI Press.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 5998-6008.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remii Louf, Morgan Funtowicz, Joe Davison, Sam Shlieifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Pu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Xu et al. (2022) Guangxuan Xu, Ruibo Liu, Fabrice Harel-Canada, Nischaal Reddy Chandra, and Nanyun Peng. 2022. Endex: Evaluation of dialogue engagingness at scale. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 4884-4893. Association for Computational Linguistics.
* Yeh et al. (2021) Yi-Ting Yeh, Maxine Eskenazi, and Shikib Mehri. 2021. A comprehensive assessment of dialog evaluation metrics. _CoRR_, abs/2106.03706.
* Yu et al. (2023) Xiao Yu, Qingyang Wu, Kun Qian, and Zhou Yu. 2023. KRLS: improving end-to-end response generation in task oriented dialog with reinforced keywords learning. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_, pages 12338-12358.
* Zhang et al. (2023) Chen Zhang, Luis F. D'Haro, Chengguang Tang, Ke Shi, Guohua Tang, and Haizhou Li. 2023. xdial-eval: A 

multilingual open-domain dialogue evaluation benchmark. In _Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023_, pages 5579-5601.
* Zhang et al. (2022) Chen Zhang, Luis Fernando D'Haro, Qiquan Zhang, Thomas Friedrichs, and Haizhou Li. 2022. Fined-eval: Fine-grained automatic dialogue-level evaluation. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Eminutes, December 7-11, 2022_, pages 3336-3355. Association for Computational Linguistics.
* Zhang et al. (2021) Chen Zhang, Joao Sedoc, Luis Fernando D'Haro, Rafael E. Banchs, and Alexander Rudnicky. 2021. Automatic evaluation and moderation of open-domain dialogue systems. _CoRR_, abs/2111.02110.
* Zhang et al. (2020) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT : Large-scale generative pre-training for conversational response generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, ACL 2020, Online, July 5-10, 2020_, pages 270-278.
* Zhou et al. (2023) Yujia Zhou, Zhicheng Dou, and Ji-Rong Wen. 2023. Enhancing generative retrieval with reinforcement learning from relevance feedback. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_, pages 12481-12490.
* 26th European Conference on Artificial Intelligence, September 30
- October 4, 2023, Krakow, Poland
- Including 12th Conference on Prestigious Applications of Intelligent Systems (PAIS 2023)_, pages 3148-3155.

## Appendix A Quality Description

The dialogue metrics used in this paper include the following two categories.

**Turn-level**:

_Fluency_: Responses are free of grammatical and semantic errors.

_Relevance_: Responses are on-topic with the immediate dialogue history.

_Diversity_: The response is informative, with long sentences including multiple entities and conceptual or emotional words.

**Dialogue-level**:

_Engagement_: Throughout the dialogue, the system displays a likeable personality.

_Coherence_: Throughout the dialogue, the system can maintain a good conversation flow.

_Informativeness_: Throughout the dialogue, the system provides unique and non-generic information.

## Appendix B Feedback Details

### BT-Feed Details

**Fluency**: We used VUP (Valid Utterance Prediction) proposed by USL-H Phy et al. (2020) for Fluency. The authors trained a model based on BERT-base to capture the Fluency of an utterance by classifying whether it is valid. For doing this, they applied many rules to get a negative sample, e.g., word reorder, word drop, and word repeat. We ran VUP via following the original setting. **Release**: Similar to the Fluency metric, We proposed this metric to enhance the relevance prediction by using negative sampling. Specifically, the objective is to discern whether a given context-response pair is relevance or not. To create positive samples (\(c_{i}\), \(r^{+}_{i}\)) for this binary classification task, we leveraged two consecutive utterances (\(u_{i}\), \(u_{i+1}\)) extracted from existing human-human dialogue corpora Li et al. (2017), where \(u_{i}\) serves as the context and \(u_{i+1}\) represents the corresponding response. Similar to the work of Sato et al. Sato et al. (2020), we selected the utterances in the dataset that are the most similar to the positive example as the negative samples (\(c_{i}\), \(r^{-}_{i}\)). RoBERTa-base Liu et al. (2019) is adopted as the text encoder. **Diversity**: According to previous studies Stasaski and Hearst (2022), these lexical-level evaluation metrics such as Distinct-n (Dist-\(n\)) Li et al. (2016) often fail to capture semantic diversity. So we used a recent automatic evaluation metric, Sem-Ent Han et al. (2022), which can measure the semantic diversity based on the semantic distribution of generated responses. Sem-Ent correlates with human judgments on response diversity more than other automatic diversity metrics.

### BT-Feed Details

**Engagement**: Engagement is widely acknowledged as a crucial evaluation criterion for assessing the quality of dialogue system Ghazarian et al. (2020). As a result, we trained a discriminatory model using the RED Reddit-based Engagement Dataset Xu et al. (2022). The RED dataset is derived from Reddit and is carefully curated using a unique distant-supervision framework.

 

this framework, emotional, attentional, behavioral, and reply engagement are combined to form a single score called ENDEX. Subsequently, a hyper-parameter threshold is utilized to group posts into positive and negative samples based on this score.

* 1 adjacency pairs for a dialogue containing \(k\) utterances and hence \(k\)
- 1 coherence scores. The coherence score at the dialogue-level is calculated by averaging the \(k\)
- 1 scores. **Informativeness**: During the human-human interaction, when the interlocutors deeply dive into a topic, they tend to convey a large amount of information (Zhang et al., 2022). We adopt a pre-trained natural language inference (NLI) model6 to calculate the topic depth of a conversation to represent the informativeness of a response. More specifically, given a dialogue of \(k\) utterances, a pre-trained NLI model is used to provided entailment score to each utterance pair in the dialogue. In total, there are \(\frac{k(k-1)}{2}\) entailment scores per dialogue. The dialogue-level entailment score is the average of all utterance-pair entailment scores in the dialogue.

Footnote 6: [https://huggingface.co/roberta-large-mnli](https://huggingface.co/roberta-large-mnli)

## Appendix C GPT-4 Evaluation Guideline

Figure 5 and Figure 6 elaborate the turn-level and dialogue-level GPT-4 evaluation guidelines, respectively. Steps to conduct the evaluation are:

**Step1**. Clarify task requirements by reading the instruction.

**Step2**. Read the dialogue, and the system response carefully.

**Step3**. Give some brief analysis from the aspects mentioned before.

**Step4**. Read the Definitions and Criteria provided above to help you with your in-depth analysis.

**Step5**. On a scale of 1-5, evaluate the above three aspects of the dialogue and produce the required output.

 

Why not Use RL at Short-term Stage

The RL-based training is more complicated and time-consuming than the multi-task learning. At the short-term stage, we leverage the multi-task learning to quickly train a high-quality dialogue system via turn-level feedback. After that, we move to the long-term stage and continuously train the resulting system via the complicated RL-based training to enhance replies considering the quality of the whole dialogue. However, if we utilize the RL-based training in the first stage, such design will make the structure of LSTDial very complicated, probably bringing negative influence to the model's reliability and generalization in practical scenarios.

## Appendix E Detailed Case Analysis

Our model has the following three incremental variants as it is trained:

(a) **Baseline**: the original dialogue system without our trainings.

(b) **STDial**: baseline + short-term training.

(c) **LSTDial**: STDial + long-term training.

We analyze the change of generated responses across these models of different stages with the following two random samples obtained from user inputs to the above models.

#### 10.1.1 Analysis1

(See table 8):

(1) The baseline's response is simple and generic, expressing a positive attitude towards the user's intention, but it lacks specific information or elements to further engagement in the conversation. (i.e., Lacking of short- and long-term capabilities)

(2) The STDial response is more informative and relevant due to mentioning the benefits of running for staying energized and enjoying the outdoors. However, it still lacks a proactive conversation guidance. (i.e., Having short-term capabilities but lacking of long-term capabilities)

(3) The LSTDial response goes a further step, not only affirming the benefits of running but also adding a subjective evaluation and asking a question to further engage in a deeper-in conversation. This kind of response not only facilitates a deeper conversation but also shows an interest in the user's preferences and plans, thereby enhancing user engagement. (i.e., Having both short-term and long-term capabilities)

#### 10.1.2 Analysis2

(1) The baseline model provides a poor response, displaying uncertainty about the weather.

(2) The STDial model offers a specific and informative response about the weather, showing the improved diversity and relevance. Compared to the baseline's response, the STDial response exhibits the short-term capabilities (i.e., fluency, relevance, and diversity).

(3) The LSTDial model enriches the response by adding a subjective evaluation of the weather and engaging the user in the conversation, increasing participation.

Compared to STDial, we observe that LSTDial tends to ask questions and express opinions actively, which is very friendly for users to participate in multiple rounds of conversations. Therefore, after the two-stage training, LSTDial can enhance long-term capabilities (i.e., coherence, engagement, and informativeness) beyond short-term capabilities.

\begin{table}
\begin{tabular}{l l} \hline \hline User: Hí, what’s the weather like today? \\ \hline
**Baseline**: I’m not sure. \\
**STDial**: Today’s weather is sunny with a temperature of 25 degrees Celsius. \\
**LSTM**: Today’s weather is sunny with a temperature of 25 degrees Celsius. It’s a great day to go outside and enjoy the sunshine. What are your plans for the day? \\ \hline \hline \end{tabular}
\end{table}
Table 9: Case of response generation at different stages of our model (case 2).