SLIDE: A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluation (2024)

Abstract: The long-standing one-to-many problem of gold standard responses in open-domain dialogue systems presents challenges for automatic evaluation metrics. Though prior works have demonstrated some success by applying powerful Large Language Models (LLMs), existing approaches still struggle with the oneto-many problem, and exhibit subpar performance in domain-specific scenarios. We assume the commonsense reasoning biases within LLMs may hinder their performance in domainspecific evaluations. To address both issues, we propose a novel framework SLIDE (Small and Large Integrated for Dialogue Evaluation), that leverages both a small, specialised model (SLM), and LLMs for the evaluation of open domain dialogues. Our approach introduces several techniques: (1) Contrastive learning to differentiate between robust and non-robust response embeddings; (2) A novel metric for semantic sensitivity that combines embedding cosine distances with similarity learned through neural networks, and (3) A strategy for incorporating the evaluation results from both the SLM and LLMs. Our empirical results demonstrate that our approach achieves state-of-the-art performance in both the classification and evaluation tasks, and additionally the SLIDE evaluator exhibits better correlation with human judgements. Our code is available at https:// github.com/hegehongcha/SLIDE-ACL2024.

heading: 1 Introduction
text: Open-domain dialogue generation is an important research topic in the field of Natural Language Processing (NLP) (Zeng et al., 2021; Wang et al., 2021; Xiao et al., 2023; Tang et al., 2023b,a; Yang et al., 2024a). Evaluating such dialogues, however, is challenging due to the
∗ Equal contribution † Corresponding authors
prevalent one-to-many issue where one conversational context may have multiple reasonable, yet semantically different responses. Additionally, adversarial negative responses, which have word overlap with given contexts, also raise further challenges in adequately evaluating open-domain dialogue (Sai et al., 2020).
Existing evaluation methods centered on word-overlap, such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), METEOR (Banerjee and Lavie, 2005)) and semantic-embedding based approaches (e.g., BERTScore (Zhang et al., 2020) and BARTScore (Yuan et al., 2021)), evaluate responses via calculating the similarity between the generated response and contexts or gold references. Therefore, they struggle to adequately address the one-to-many problem as a context may permit many different responses. Recently, there are some novel approaches proposed to leverage the commonsense reasoning capabilities learned in Large Language Models (LLMs) for a wide range of NLP tasks, such as dialogue generation, evaluation and sentiment analysis (Loakman et al., 2023; Liu et al., 2023; Yang et al., 2024b; Chiang and yi Lee, 2023; Yang et al., 2024c). However, there are still some limitations of LLM-based evaluation. For example, Lin and Chen (2023) propose LLM-EVAL to evaluate open-domain dialogue systems with a range of LLMs and identify numerous issues including the significant reliance on prompt phrasing, which may artificially limit the effectiveness of the metrics and the evaluation quality changed with prompt formulation, which makes scoring unreliable and unrobust. Additionally, Wang et al. (2023b) state that without additional support, LLMs alone do not constitute equitable
15421
evaluators. In addition, LLMs face significant challenges in generating accurate and dependable assessments when evaluating both positive and adversarial negative responses. Although LLMs can classify and evaluate adversarial negative responses with a degree of proficiency, their performance significantly diminishes when dealing with multiple positive responses, which is the one-to-many nature of open-domain dialogue, a reasonable response that matches well to context could have significantly different semantics to its reference. As for small task-specific model (SLM), which is cost-effective alternatives, reveal a propensity towards positive responses. Nonetheless, the deployment of SLM is challenging, as they require extensive training data. This requirement often leads to issues related to data scarcity and a lack of sufficient examples covering a wide range of topics within open-domain dialogue, presenting obstacles to their effective implementation and scalability.
Therefore, we combine a SLM with LLM to solve the one-to-many issue, and we also resolve the data scarcity problem through data augmentation via LLMs. Initially, we train a SLM using contrastive learning to minimise the cosine distance between the positive responses and the context embeddings, and enlarge the distance between the embeddings of adversarial negative responses and the context embeddings. After fine-tuning with contrastive learning, these compact models can discern between positive and adversarial negative responses adequately. Subsequently, we divide the representations of the responses into two categories: robust and non-robust vectors. Through contrastive learning, we only retain robust vectors for subsequent steps while excluding non-robust ones. Following this, we proceed to calculate the contextual-response distance, and obtain a probability value from the classification model. Finally, an integrated score is composed by combining these two values as the final evaluation score of the SLM model. Ultimately, the evaluation score produced by the SLM and the score from the LLM is further integrated as the final evalu-
ation score, SLIDE, used for evaluation. A series of experiments are conducted to analyse the effectiveness of our proposed framework, including a classification task and an evaluation task. For the classification task, we use the SLM to classify the positive and adversarial negative responses from the DailyDialog++ dataset. Experimental results demonstrate that our SLM achieves state-of-the-art (SOTA) performance, even when compared to the results of some LLMs (e.g., GPT-3.5). We additionally note that the SLM achieves higher accuracy in positive samples, whilst the LLM performs better for negative example responses. For the evaluation task, SLIDE also shows a good correlation with human scores.
Our contributions can be summarised as the following:
• We propose a new evaluator for opendomain evaluation, namely SLIDE (Small and Large Integrated for Dialogue Evaluation). To the best of our knowledge, this is the first attempt to combine these models in the open-domain dialogue evaluation task.
• To better measure the semantic differences between dialogues, we introduce a novel evaluation score, which is integrated from the derivative of embedding cosine distances, and the similarity probability created by neural networks. The score has been demonstrated to be very effective in classifying positive and negative responses, leading to a SOTA result on the open-domain dialogue datasets.
• We augment existing dialogue evaluation datasets with multiple positive and adversarial negative responses. This enhanced dataset can be used in fine-tuning the open-domain SLM model.
• We conduct a range of experiments to demonstrate the proposed SLIDE effectively addresses the shortcomings of singular SLM-based or LLM-based opendomain dialogue evaluation models regarding the one-to-many reference problem.

heading: 2 Related Work

heading: 2.1 Dialogue evaluation metrics
text: Traditional n-gram-based evaluation metrics like BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and METEOR (Banerjee and Lavie, 2005) assess the overlaps of words between candidate responses and a reference standard. On the other hand, metrics based on embeddings, such as Extrema (Forgues et al., 2014) and BERTScore (Zhang et al., 2020), transform the responses and references into high-dimensional representations to evaluate semantic similarities.
Recent years have witness an increasing interest in developing the trainable metrics. RUBER (Tao et al., 2018) evaluates the similarity between the generated response, its context, and the actual reference. DEB (Sai et al., 2020), a BERT-based model pre-trained on extensive Reddit conversation datasets, was introduced to improve evaluation effectiveness. The Mask-and-fill approach (Gupta et al., 2021), utilising Speaker Aware (SA)-BERT (Gu et al., 2020), aims to better understand dialogues. MDD-Eval (Zhang et al., 2021) was developed for evaluating dialogue systems across various domains by using a teacher model for annotating dialogues from different domains, thus supporting the training of a domain-agnostic evaluator. This method, however, relies on human labels and extra training data, which our proposed approach does not require. Lastly, CMN (Zhao et al., 2023) was designed to effectively address the one-to-many nature of open-domain dialogue evaluation in the latent space, yet it does not account for adversarial negative examples.

heading: 2.2 LLM-based Evaluators
text: The application of LLM for evaluation purposes has seen significant interest due to its impressive capabilities across various tasks. GPTScore was introduced by Fu et al. (2023) as a method for multi-faceted, tailor-made, and training-independent evaluation. Preliminary investigations by Wang et al. (2023a) into the efficacy of evaluators based on LLM have been conducted. Kocmi and Federmann (2023) have
utilised GPT models for assessing machine translation quality. Furthermore, Liu et al. (2023) developed G-EVAL, leveraging GPT-4 to evaluate across multiple tasks such as dialogue response generation, text summarization, data-to-text generation, and machine translation. Despite these advancements, the use of LLM-based metrics to assess adversarial negative responses within open-domain dialogue scenarios remains unexplored.

heading: 3 Methodology

heading: 3.1 Model Architecture
text: As shown in Figure 1, our approach can be divided into two phases: the training process and the evaluation process. During the training phase, our focus is on training the SLM to classify positive and adversarial negative responses. In the evaluation phase, we integrate the SLM and LLMs to evaluate open-domain dialogue responses.

heading: 3.2 Training Process
text: we first use contrastive learning to train the SLM to discern positive responses from adversarial negative responses.
We employ Sentence-Transformer (Reimers and Gurevych, 2019) to encode the context and responses. Specifically, we encourage the embeddings of positive responses to be closer to context embeddings, whilst negative adversarial responses move further away. Given a triplet of < xci ,x p i ,x a i >, where 1 ≤ i ≤ n, xci represents context, x p i represents positive responses, and xai represents adversarial negative responses, the goal of the training process is to minimise the distance between xpi and x c i while maximising the separation between xai and xci . The loss function is defined as follows:
htypei = Encoder(x type i )
Lout = max(||hci − hpi || − ||hci − hai ||+margin, 0) (1)
in which margin is a hyperparameter and type = {c, p, a}. hi is the hidden state of the xi, whilst ||hci − hpi || − ||hci − hai || refers to the cosine distance.
To enhance the precision of classification, we undertake a disentanglement process. The response embeddings are disentangled into two different sub-representations: robust embeddings and non-robust embeddings. The robust embedding can be interpreted as a salient feature for classification, whilst the non-robust embedding constitutes noise that could act as an interfering element, potentially leading to wrong predictions.
We denote the robust and non-robust embeddings of positive responses hpi as {hpri ,hpni }, respectively. On the other hand, {hari ,hani } represents the robust and non-robust embeddings of adversarial negative responses hai . It is imperative that the robust and non-robust embeddings within both positive responses and adversarial negative responses maintain clear distinctions. Similarly, the separation between the robust embeddings of different types of responses should be maintained. Therefore, we define our training loss function as follows:
Lins_same_pos = z1 ∗ d21 + (1− z1) ∗max(margin− d1, 0)2 Lins_same_neg = z2 ∗ d22 + (1− z2) ∗max(margin− d2, 0)2 Lout_robust = z3 ∗ d23 + (1− z3) ∗max(margin− d3, 0)2 (2)
where dj, 1 ≤ j ≤ 3 represents the cosine similarity and zj, 1 ≤ j ≤ 3 indicates whether a pair of vectors match; with zj = 0 denoting no match and consequently a greater distance between the vectors. Conversely, zj = 1 signifies a match leading to a closer distance. To encourage divergence between the robust and non-robust vectors within both positive or negative classes, we set z1 = z2 = 0. Similarly, for the robust vectors across positive and negative classes to diverge, we assign z3 = 0.
d1 = ||hpri − hpni || d2 = ||hari − hani || d3 = ||hpri − hari || (3)
In addition, we develop a classification network to classify each factor. The process is defined as follows:
h = concat(hc,hres)
pi = Softmax(Linear(h))
Lcls = 2∑
i=0
yi ∗ pi (4)
where hres encompasses {hpri ,hpni ,hari ,hani }, while yi = {0, 1, 2} is the label. Specifically, yi = 1 corresponds to h pr i , yi = 0 relates to hari ; for all other cases, yi = 2.
In summary, the total loss function is
L = Lout + Lins_same_pos + Lins_same_neg + Lout_robust + Lcls (5)

heading: 3.3 Evaluation Process
text: After training the SLM, we proceed to the evaluation phase. We first follow G-EVAL (Liu et al., 2023) and prompt LLM to evaluate the given context and response. The prompt could be found in A.1. Therefore we obtain a score referred to as ScoreLLM. Subsequently, SLM is employed to encode both the context and the response. This step involves segregating the response encoding into two distinct embeddings—robust and non-robust. The mathematical representation of this process is given by
hc = Encoder(xc)
hr = Encoder(xr)
hr,robust,hr,non = sep(hr) (6)
Then we calculate the cosine distance between context and response, as well as the probability for the given response.
d = cosine_similarity(hc,hr,robust) sd = (d− dmin)/(dmax − dmin) sp = p(y = 1)
= Softmax(Linear(hc,hr,robust)) (7)
where d is the cosine similarity between a context and a response. The normalised distance is denoted by sd, while sp represents the prediction probability of the classifier. Experimental evidence from training suggests that positive responses tend to be closer in embedding space to their corresponding contexts than negative ones, implying that dpos < dneg. Furthermore, it has been observed that for negative responses, sp is typically lower than for positive ones. Consequently, it can be deduced that for positive responses, sd − sp will be smaller than for negative ones. Based on these insights, we define a new probabilistic score for our SLM as follows:
ScoreSLM = 1− sd + sp (8)
Finally we design an integration strategy to integrate ScoreSLM and ScoreLLM. Our empirical findings suggest that while the SLM demonstrates greater accuracy in identifying positive responses, the LLMs shows a greater performance on classifying negative ones. We define the final score as follows:
Score = { ScoreSLM,ScoreSLM ≥ 0.5, ScoreLLM,ScoreLLM < 0.5, (ScoreSLM + ScoreLLM)/2, otherwise
(9)

heading: 4 Experimental Setup

heading: 4.1 Dataset
text: We conduct dialogue evaluation on three opendomain dialogue datasets: DailyDialog++ (Sai et al., 2020), TopicalChat (Gopalakrishnan et al., 2019), and Personachat (Zhang et al., 2018). The DailyDialog++ dataset contains 9,259 contexts in the training set, 1,028 in the validation set, and 1,142 in the test set. Each context is accompanied by five positive responses, five random negative responses, and five adversarial negative responses. However, the PersonaChat and TopicalChat datasets lack multiple positive and adversarial negative responses. To address this issue, we employ GPT-4 to generate both positive and adversarial negative responses and positive responses. The prompt for generating responses could be found in A.2. As a results, the generated training and test sets for both PersonaChat and TopicalChat datasets comprises 2,000 contexts. Each context within these datasets is further enriched with a set of responses, consisting of five positive responses and five adversarial negative responses.
Validation of the generated responses is conducted through a structured three-step process. Initially, a classification prompt is designed for the LLMs to determine the validity of the positive responses generated by GPT-4. Invalid responses are discarded, and we prompt the GPT-4 for subsequent generation attempts. Secondly, we randomly select 1,200 generated responses from the test set for assessment by human annotators. A significant majority of these responses are appropriate to their
contexts, exceeded 98%. The final phase involves a comparative analysis with the humanannotated benchmark dataset DailyDialog++. A SLM trained on training set of the DailyDialog++ dataset with 88% accuracy is utilised to classify the generated responses without incorporating distance metrics. The classification accuracy is 83% on the PersonaChat test set and 85% on the TopicalChat test set.

heading: 4.2 Baselines
text: We select a range of widely used baseline metrics. The word-ovelap and embeddingbased metrics includs BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), METEOR (Banerjee and Lavie, 2005), Embedding-Average (Wieting et al., 2016), Vector-Extrema (Forgues et al., 2014), BERTScore (Zhang et al., 2020), Unieval (Zhong et al., 2022), and BARTScore (Yuan et al., 2021). For the LLM-based metrics, we select recently proposed baselines, including G-EVAL (Liu et al., 2023) and another LLM-based metric proposed by Chiang and yi Lee (2023) which we denote LLM-Chiang, as well as Gemini.

heading: 4.3 Evaluation Set
text: As for the evaluation set, we select 600 contextresponse pairs from each of three open-domain dialogue datasets—namely DailyDialog++, PersonaChat, and TopicalChat—resulting in a total of 1,800 samples. Then the evaluation set was evaluated by three human annotators, all of whom possess proficiency in English, to ensure an accurate assessment. They are instructed to thoroughly read each context-response pair and then rate them according to four distinct criteria including naturalness, coherence, engagingness, groundedness, employing a 1-5 Likert scale for their assessments.
The four criteria are consistent with Zhong et al. (2022) and are defined as follows: (1) Naturalness: The degree to which a response is naturally written; (2) Coherence: The extent to which the content of the output is wellstructured, logical, and meaningful; (3) Engagingness: The degree to which the response is engaging; and (4) Groundedness: The extent to which a response is grounded in facts
present in the context. After obained four scores for each criteria, an aggregate score is calculated by averaging across all criteria to provide a comprehensive measure of each response’s quality. Additionally, the Inner-Annotator Agreement (IAA) between three evaluators are evaluated through Cohen’s Kappa (Cohen, 1960), which indicate a moderately strong agreement level (0.4-0.6) among annotators, with a value of 0.53, validating the reliability of the human evaluation.
We employ Gemini, GPT-3.5-turbo-1106 and GPT-4-1106 for all experiments. For SLM, DistilBERT (Sanh et al., 2019) is utilised.

heading: 4.4 Experimental Results

heading: 4.4.1 Dialogue Classification Task
text: We conduct dialogue classification on the on DailyDialog++ dataset as it contains the human-annotated labels for each contextresponse pair. We calculate the classification accuracy using SLM and LLMs (i.e., GPT-3.5 and GPT-4). The classification prompt for the LLMs could be found in A.3. We also use SLM that trained on the identical dataset for the ablation study.
As Table 1 shows, SLM employs both probability and distance measures, achieves superior performance with an accuracy of 91.05%, which exceeds the 88.91% accuracy achieved by GPT-4. Experimental results show that the SLM particularly excels in the classification of positive responses, achieving the highest accuracy of 91.83%. Conversely, GPT-4 exhibits the strongest performance in classifying adversarial negative responses, achieving
97.40% accuracy. The performance of Gemini surpasses that of GPT-3.5, yet remains inferior to that of GPT-4. These results indicate that while LLMs, such as GPT-4, possess robust capabilities in recognising adversarial negative responses, they still struggle with the one-to-many problem, especially when dealing with semantically diverse positive responses. On the other hand, SLM with the integration of distance and probability metrics shows exceptional performance in recognising opendomain positive responses. Based on these insights, we propose a hybrid approach that combines the SLM, a task-specific model, with LLMs for comprehensive dialogue evaluation. This approach is designed to leverage the distinct strengths of both types of models to enhance the overall performance in open-domain dialogevaluations.

heading: 4.4.2 Dialogue Evaluation
text: As shown in Table 2, we investigate the correlation between automatic evaluation metrics and human judgments across three datasets. The word-overlap metrics, including BLEU, ROUGE, and METEOR, exhibit a weak positive correlation with human scores (<0.4), among which ROUGE-2 showing the least correlation (<0.2). Similarly, embedding-based metrics, such as Embedding and BARTScore, demonstrate a weak correlation with human evaluations, ranging from 0.2 to 0.4. Notably, BERTScore shows the strongest correlation to human judgments among both word-overlap and embedding based traditional metrics. On the other hand, Unieval achieves the suboptimal results (<0.3) on the DailyDialog++ and PersonaChat datasets .
As for LLM-Based metrics, there is a notable improvement in correlation with human scores compared with traditional metrics, most show correlations greater than 0.6. Specifically, Ours (LLMs-only(GPT-4) exhibits superior performance compared with G-EVAL and LLM-Chiang and achieves the highest correlation in both the PersonaChat (0.737 for Pearson and 0.729 for Spearman) and TopicalChat datasets (0.745 for Pearson and 0.736 for Spearman). Moreover, the variance in per-
formance between G-EVAL and LLM-Chiang indicates the significant impact of model selection (i.e., GPT-4 or GPT-3.5) and prompt design on LLM-based metric effectiveness .
As for the SLM, Our SLM-Dis and Prob approach that incorporate both distance and probability measures, revealing a strong correlation with human evaluations (>0.6). As for the SLIDE method, it combines an SLM with LLMs. SLIDE (GPT-4 ) achieves the strongest correlation with human ratings on the DailyDialog++ dataset (0.773 for Pearson and 0.704 for Spearman). Furthermore, our LLM metrics, characterised by simpler prompt designs compared to those used in GEVAL and LLM-Chiang, do not individually surpass these benchmarks. However, the amalgamation of LLM and SLM techniques results in a significant enhancement in correlation strength, surpassing both G-EVAL and LLMChiang, particularly when employing GPT-3.5. The same trend is also observed when utilising Gemini. These two distinct models could demonstrate the effectiveness of integrating LLM and SLM techniques.
The datasets generated by GPT-4, namely PersonaChat test set and Topical-Chat test set, exhibit the strongest correlations with the GPT-4-based metrics, including our proposed method. It indicates the inherent advantage of using model-specific metrics for evaluating datasets generated by the same model, thereby leading to a more accurate and relevant assessment of these two datasets.
To validate the effectiveness of distance metric used in SLM and SLIDE, we conduct several ablation studies. It can be observed that Ours (SLM-Dis-Prob) gives better performance than the model variant with the Dis (SLM-Dis-only) or Prob (SLM-Prob-only) component only. This suggests that SLM with distance metric is more effective in evaluating open-domain dialogues.

heading: 4.5 Example Visualisation
text: To demonstrate the effectiveness of disentanglement, we selected a subset of examples from the DailyDialog++ dataset and employed T-SNE for visualisation, as depicted in Fig-
ure 2. Each example includes a context, five positive responses, and five adversarial negative responses. The vector representations prior to disentanglement are illustrated in the left plot, whereas those post-disentanglement are shown on the right. Post-disentanglement visualisations reveal a more distinct clustering of responses in relation to their contexts. By utilising these refined representations to train a classifier, we observed an improvement in classification accuracy on the test set of DailyDialog++ dataset, from 87% to 88.19%. Additional examples are available in Appendix A.5.

heading: 4.6 Case Study
text: We conduct qualitative analyses through case studies, which are shown in Table 3. Each case shows the conversational context as well as the corresponding gold-standard reference and thegenerated responses. We compare our evaluation metric with nine different baselines. To simplify the comparison, we normalise all scores to a range of 1-5 Likert scale. Note
that the normalisation is only applied to the case study, and is not implemented in our main experiments. In the first case, the response candidate is positive, whereas the response is an adversarial negative response in the second case. Our SLIDE metric correlate well with the human ratings. For example, in the first scenario, only our models rate a positive score (>3), which is consistent with the human score. More examples could be found in Appendix A.4.

heading: 5 Conclusion
text: In this paper, we introduce a novel automatic evaluation metric (SLIDE) that integrates SLM with LLMs for open-domain dialogue evaluation. Our approach involves initially training a SLM through iterative contrastive learning stages, followed by leveraging the combine strengths of SLM and LLMs to create a superior evaluation metric. This metric exhibits enhanced correlation with human judgments in comparison to those derived exclusively
from LLMs. Furthermore, experimental results reveal that LLMs-based metric struggles with classifying and evaluating open-domain dialogues, which is the one-to-many nature. Therefore, we design a novel method for merging scores from SLM and LLM to enhance dialogue evaluation. Experimental results show that our SLIDE model outperforms a wide range of baseline methods in terms of both Pearson and Spearman correlations with human judgements on three open-domain dia-
logue datasets, and deals well with the one-tomany issue in open-domain dialogue evaluation.
Ethics Statement
In this paper, we introduce SLIDE, a novel automatic evaluation metric that integrates SLM and LLM for assessing open-domain dialogue systems. The advantage of SLIDE lies in its utilisation of both SLM and LLM, enhancing the evaluation of open-domain dialogues. However, a potential downside is that SLIDE might award high scores to responses that are inappropriate or offensive under certain conditions. Therefore, it is crucial to carefully review the content of training datasets prior to training SLIDE to mitigate this issue.
Limitations
Although our proposed method performs well in evaluating the open-domain dialogue systems, it also has some limitations. Primarily. Although our model first combines SLMs and LLMs for automatic evaluation metrics, it is a simple combination according to its characteristics. In the future, we will conduct a deeper combination of LLM and SLM to make a better evaluation on the open-domain dialogue system. In addition, because the PersonaChat dataset and TopicalChat dataset are augmented by LLM, not the real human-annotated dataset, our model does not perform the best on these
two datasets. We need to further employ the difference between LLM-generated datasets and human-annotated dataset, and further analyse the effectiveness of our proposed model.

heading: Acknowledgments
text: This study was partially supported by the National Science Foundation (IIS 2045848 and 2319450). Part of the work used Bridges-2 at Pittsburgh Supercomputing Center through bridges2 (Brown et al., 2021) from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by NSF grants #2138259, #2138286, #2138307, #2137603, and #2138296.

heading: A Appendices
text: A.1 Prompt for evaluating the engagingness of the response
Engagingness Please rate the dialogue response. The goal of this task is to rate dialogue response. Note: Please take the time to fully read and understand the dialogue response. We will reject submissions from workers that are clearly spamming the task. How engaging is the text of the dialogue response? (on a scale of 0-1, with 0 being the lowest)
Example: Conversation History: Is there something wrong? I enjoy having your daughter in my class. I’m glad to hear it.
Dialogue Response: I enjoy listening jazz music in my free time.
Evaluation Form (scores ONLY): Engagingness: 1
Input: Conversation History:
Response:
Evaluation Form (scores ONLY): Engagingness:
A.2 Prompt for generating response You are a conversational dialogue generator. Given a conversation context, , which includes 2 speakers[annotated as FS(FirstSpeaker) and SS(SecondSpeaker)], and a response. Your task is to generate five diverse positive response and five adversarial negative response respectively.
Positive Response
Positive response is valid for the conversation context.
Adversarial Negative Response Adversarial negative responses have a significant word overlap with the conversation context but are still irrelevant response, which may not have any relation to the context. You need to choose some words (do not include stopwords such as "I", "you", "are", etc.) from the conversation context and use them directly or indirectly while writing the adversarial negative responses. Indirect usage here refers to using words closely related to the context words. For example,using synonyms,antonyms, homonyms, subwords, or other words that are known to frequently co-occur with the words in the context (e.g., the words "flexibility" and "injuries" co-occur with "acrobatics").
The following are five examples of a conversation context and response, and the corresponding prediction.
Example Context: FS:Is there something wrong? SSI enjoy having your daughter in my class. FS:I’m glad to hear it. Positive response: She is so brilliant. Her behavior is good in the class. I would love to hear that she knows every rules and regulation. I was shocked to know that she is your daughter. She answers all my questions.
Adversarial Negative Responses: I enjoy listening jazz music in my free time. I need pin drop silence in the class. If I hear someone talking they will be sent out of the class. I am glad you enjoyed the magic show organised by our team. I think there was something wrong with the CCTV camera installed in the class. This is the wrong method to solve the problem. Please be attentive in the class.
A.3 Prompt for classifying response
You are a classifier. Given a conversation context, which includes 2 speakers[annotated as FS(FirstSpeaker) and SS(SecondSpeaker)], and a response. Your task is to classify this response whether is positive or negative.
Positive Response Positive response is valid for the conversation context.
Adversarial Negative Response Adversarial negative responses have a significant word overlap with the conversation context but are still irrelevant response, which may not have any relation to the context. You need to choose some words (do not include stopwords such as "I", "you", "are", etc.) from the conversation context and use them directly or indirectly while writing the adversarial negative responses. Indirect usage here refers to using words closely related to the context words. For example,using synonyms,antonyms, homonyms, subwords, or other words that are known to frequently co-occur with the words in the context (e.g., the words "flexibility" and "injuries" co-occur with "acrobatics").
Your output format is only the “Positive” or “Negative”.
Example The following are five examples of a conversation context and response, and the corresponding prediction.
Example 1:
Context: FS: Is there something wrong? SS: I enjoy having your daughter in my class. FS: I’m glad to hear it.
Response: She is so brilliant. Prediction: Positive
Example 2: Context: FS: Is there something wrong? SS: I enjoy having your daughter in my class. FS: I’m glad to hear it.
Response: I enjoy having your daughter in my class. Prediction: Positive
Example 3: Context: FS: Is there something wrong? SS: I enjoy having your daughter in my class. FS: I’m glad to hear it.
Response: I’m glad to hear it. Prediction: Positive
Example 4: Context: FS: We have to pick up Conrad before the party. SS: Alright, no problem. FS: We’re supposed to meet him at Cal’s Bar at 10
Response: I pushed the problem aside; at present it was insoluble." Prediction: Negative
Example 5: Context: FS: Is there something wrong? SS: I enjoy having your daughter in my class. FS: I’m glad to hear it.
Response: I think there was something wrong with the CCTV camera installed in the class. Prediction: Negative
A.4 Case Study We display some other samples from DailyDialog++ dataset. From Table 4, We can see that our SLIDE model have a close score with
human score.
A.5 Example Visualisation We display some other visualization example in this section. From Figure 3, We can see that the positive responses become closer to context after disentanglement.
15435

heading: Future Work
text: In the future, we will conduct a deeper combination of LLM and SLM to make a better evaluation on the open-domain dialogue system. In addition, because the PersonaChat dataset and TopicalChat dataset are augmented by LLM, not the real human-annotated dataset, our model does not perform the best on these
two datasets. We need to further employ the difference between LLM-generated datasets and human-annotated dataset, and further analyse the effectiveness of our proposed model