MEEP: Is this Engaging? Prompting Large Language Models for Dialogue Evaluation in Multilingual Settings
Authors: Amila Ferron
Amber Shore
Ekata Mitra
Ameeta Agrawal
Venue: EMNLP 2023 Findings (2023)
Keywords: automatic evaluation of dialogue
dialogue evaluation
multilingual
metrics
engagingness
prompting
LLM
large language model
multilinguality
TL;DR: We find that using selected novel prompt constructs with LLMs, including our comprehensive definition of engagingness, outperforms state-of-the-art methods on evaluation of engagingness in dialogue across multiple languages.

Abstract: As dialogue systems become more popular, evaluation of their response quality gains importance. Engagingness highly correlates with overall quality and creates a sense of connection that gives human participants a more fulfilling experience. Although qualities like coherence and fluency are readily measured with well-worn automatic metrics, evaluating engagingness often relies on human assessment, which is a costly and time-consuming process. Existing automatic engagingness metrics evaluate the response without the conversation history, are designed for one dataset, or have limited correlation with human annotations. Furthermore, they have been tested exclusively on English conversations. Given that dialogue systems are increasingly available in languages beyond English, multilingual evaluation capabilities are essential. We propose that large language models (LLMs) may be used for evaluation of engagingness in dialogue through prompting, and ask how prompt constructs and translated prompts compare in a multilingual setting. We provide a prompt-design taxonomy for engagingness and find that using selected prompt elements with LLMs, including our comprehensive definition of engagingness, outperforms state-of-the-art methods on evaluation of engagingness in dialogue across multiple languages.

MEEP: Is this Engaging? Prompting Large Language Models for Dialogue Evaluation in Multilingual Settings
Amila Ferron and Amber Shore and Ekata Mitra and Ameeta Agrawal
Department of Computer Science, Portland State University {aferron, ashore, ekata, ameeta)@pdx. edu
Abstract
Dialogue:
Prompt
"Hil", "Wow, hello..."
As dialogue systems become more popular, evaluation of their response quality gains im- portance. Engagingness highly correlates with overall quality and creates sense of connec- tion that gives human participants filling experience. Although qualities like co- herence and fluency are readily measured with well-worn automatic metrics, evaluating gagingness often relies on human assessment, which is costly and time-conee consuming process. Existing automatic engagingness metrics evalu- ate the response without the conversation his- tory, designed for one dataset, have lim- ited correlation with human annotations. Fur- thermore, they have been tested exclusively on English conversations. Given that dialogue sys- tems are increasingly available in languages beyond English, multilingual evaluation capa- bilities essential. We propose that large language models (LLMs) may be used for eval- uation of engagingness in dialogue through prompting, and ask how prompt constructs and translated prompts compare in a multilingual setting. We provide prompt-design taxon- omy for engagingness and find that using se- lected prompt elements with LLMs, including comprehensive definition of engagingness. outperforms state-of-the-art methods on evalu- ation of engagingness in dialogue across multi- ple languages.
Prompt Design Elements
Response:
"It was an interesting trip..."
Dialogue
Dialogue:
Response
ful-
"(Hola!". "Guau, hola...".
Engagingness Definition
Response:
"Fue un viaje interesante...
Dialogue: "E.
'RE!
1699.
EN: 90
Response:
LLM
ZH: 90
Figure I: Overview of proposed methodology. MEEP, on datasets in English, Spanish, and simplified Chi- nese. Our prompt is distinguished by its definition of engagingness, drawn from previous research, linguistic analysis, and the study of inclusivity. Extensive exper- iments informed the use of additional prompt design elements. Scores shown here are in a range from (not engaging) 100 (very engaging).
Cohen et al., 2022). Development of models that produce more engaging responses be supported by automatic metrics complement human anno- tation.
Despite the ground-breaking work of existing metrics for engagingness, they evaluate the sponse without the conversation history (Xu al., 2022), or are designed for specific dataset (Liu et al., 2023), and are not highly correlated with human annotations (Ghazarian et al., 2020). Al- though multi-dimennsinal automatic metrics are desirable, they have limited success with complex qualities like engagement (Mehri and Eskenazi, 2020a; Deng al., 2021: Zhang et al., 2022a). En thusiasm for multi-dimensional evaluation is bal- anced by calls to develop metries that measure spe- cific dialogue qualities in order to complement ex- isting metrics (Gehrmann et al., 2022; Mchri et al., 2022; Colombo et al., 2022).
1 Introduction
Dialogue systems are becoming more popular, but their quality is usually evaluated in terms of metrics such fluency, coherence, or sensibility. Recent advancements large language model-based di- alogue systems enable high levels of proficiency, thus shifting the emphasis from fluency evalua- tion to more nuanced aspects such as engagingness, which has emerged as an important quality of dia- logue systems (Yu et al., 2016; Zhou et al., 2022; Code available at https:/ / thub. con/PortNLP/MEES

---

Furthermore, dialogue systems are increasingly available in languages beyond English so is im- portant to be able test systems in these languages (Rodríguez-Canter Cantelar et al., 2023). With the rise of virtual assistants, the importance of engaging dia- logue (Richardson et al., 2023) and multilingualism has increased significantly, they need to actively assist users, facilitate knowledge exploration, and foster effective communication, making engaging- ness vital parameter evaluating dialogues. Our research therefore develops an automatic metric for engagingness that can evaluate dialogue responses in multiple languages.
An extensive evaluation across ten datasets and three languages (English, Spanish, and Chinese).
Related Work
Automatic Dialogue Metries From the develop- ment of PARADISE, the earliest automatic met- ric to evaluate dialogue (Walker et al., 1997), reference-based automatic metrics such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and BERTSCORE (Zhang al., 2020) have been used extensively for evaluation of dialogue but have weak correlation with human judgments (Liu al., 2016; Novikova al.. 2017), and do not suit the multiplicity of valid responses possible in any con- versation (Zhao et al., 2017; 7; Mehri et al., 2022).
We establish a comprehensive definition of en- gagingness in conversation and construct an ex- tensive range of prompts designed with our dimen- sions of engagingness and prompt engineering tech- niques (see Figure 1). These prompts are employed across several LLMs, which show impressive ca- pabilities in many tasks and have had limited ex- ploration into their use for automatic evaluation of dialogue.
Automatic Metrics for Engagingness Dialogue Ghazarian et al. (2020) use engagingness as a mea- sure of dialogue system performance in the first attempt to learn engagingness scores at the utter- ance level from annotated data. This BERT-based metric achieves modest correlations with human notated scores, and leaves room for future improve- ment. recent effort at creating more robust automatic metric for engagingness is ENDEX (Xu et al., 2022). This method uses several features a Reddit- based dataset to learn engagingness scores and evaluate the responses in a turn-level dataset without considering the dialogue context. Some multi-dimensional dialogue metrics include engag- ingness one of the dimensions measured (Mehri and Eskenazi, 2020a: Zhang al., 2021), however, the engagingness scores for these metrics are not highly correlated with human evaluation. Other multidimensional dialogue metrics use results from human-directed questions about interestingness to evaluate engagingness for an imprecise approxi- mation (Deng et al., 2021; Zhong et al., 2022; Liu et al., 2023).
To our knowledge, this is the first work to ex- tensively test prompt design for the dedicated eval- uation of engagingness in dialogue in multilin- gual setting. We test the prompts first on English- only datasets, and select the best-performing set of prompting methods to test against four strong bascline metrics on Spanish-language and sim- plified Chinese- lannusee datasets. We also test translated versions of these prompts. We find that our proposed framework MEEP: Metric for Engagingness Evaluation using Prompting can be effective for evaluation of enganginess in dialogue in multiple languages with the broader implication that LLMs can be used for evaluation of complex qualities of dialogue in multiple languages through prompting alone.
Our contributions are as follows:
A thorough definition of engagingness ing five dialogue engagement dimensions in- formed by previous engagingness research and a linguistics- based approach. It frames the goals of engagingness in conversation in terms of human-aligned dimensions indepen- dent of existing datasets and provides a more nuanced target for the development of engag- ing dialogue systems.
Multilingual Automatic Dialogue Metries Multilingual dialogue models are commonly evalu- ated using similarity measures such as SacreBLEU, SCORE, BLEU, and perplexity (Agarwal et al., 2021; Zhang et al., 2022b). Development of non-similarity-base automatic metrics for a multilingual context remains an open question for research. The recent DSTC11 shared task proposal for Task 4 acknowledges this lack as motivation for the task (Rodriguez-Canted et al., 2023).
A novel method for measuring engagingness in dialogue using a set of prompt engineering approaches that improve LLM performance.
Large Language Models for NLG Evaluation

---

Gao et al. (2023) find that for text summarization evaluation, ChatGPT provides scores in the style of human evaluation, and that adding dimension definitions leads to slight improvements in corre- lation with human annotations. GEMBA (Kocmi and Federmann, 2023) achieves saate-of-thh-rr per- formance for machine translation evaluation using GPT-3.5 and larger. Most attempts to study the effectiveness of prompting in the context of auto- matic evaluation use modified versions of annota- tor instructions as prompts (Koemi and Federmann, 2023; Wang et al., 2023; Luo et al., 2023; Fu et al., 2023; Gao et al., 2023). Prompt techniques im- prove through the use of clarifying examples (Yuan et al., 2021) or assigning LLMs roles (Shen et al., 2023). Svikhnushina and Pu (2023) use new di- alogues generated by LLM interacting with chatbot that are then scored with the same LLM.
Interesting
Specific
Relevant
Correct
Semantically appropriate
Understandable
Spearman
Poarson
Overall
0.4
0.8
Figure 2: Correlation of several dialogue dimensions with engagingness as annotated in the FED dataset.
To ground our definition, we start with the dictio- nary. Dictionary definitions identify three consis- tent aspects of the quality of being engaging: atten- tion, interest, and participation AMerriam-Webster 2023; Wiktionary, 2023; Dictionary, 2023). A re- ply is engaging when it gets the user's attention, interests the user, and incites participation from the user. These three aspects form the core of what we expect an engagingness metric to measure. We propose the following five subdimensions of engagingness:
Large Language Models for Evaluation of Dia- logue G-EvAL (Liu et al., 2023), evaluates sum- marization and dialogue response generation, us- ing prompts that include definitions of the task and quality to be evaluated, Chain of Thought reason- ing, and scoring function. GPTSCORE (Fu et al.. 2023) provides an adaptable metric using shot instruction with prompt template that includes the task specification and a definition of the aspect to be evaluated. Each has only been tested on one (English) dataset.
Engaging responses are diverse, not repetitive and generic. When a model produces a standard or generic re- sponse, it can constrain the responses the user can reasonably produce (Stasaski and Hearst, 2023), creating predictable dialogue which limit engagingness.
Response Diversity:
3 Engagingness in Dialogue
Interactional Quality: Engaging responses encourage response from the user. A defini- tionally necessary component of engagement is participation, so signs of elicited participa- tion in the user, such as 1) the presence of any response, and 2) the presence of a high-quality response, demonstrate this aspect.
Engagingness is central to our conception of con- versation, as demonstrated by the prevalence of the word "conversation" as a collocate² of the verb "engage" (COCA). What then is engagingness? In prior research, several metrics conflate engaging- ness with interestingness (Deng et al., 2021; Zhong al., 2022: Liu et al., 2023). Others provide no definition and the training of these metrics rely in- stead on the implicit judgements from the training set annotators (Logacheva et al., 2018). We posit that there is no standard definition of engagingness in NLP because of the subjectivity of human judg- ment. Humans, like models, cannot be relied upon for self reporting (Gao et al., 2021: Rosenman et al., 2011). We put forward a definition of engagingness that is useful for our goal: create an evaluation tool that will improve dialogue model performance.
Interestingness: We pose interestingness a necessary component of engagingness. Re- search has commonly used interestingness as a proxy for engagingness (Deng et al., 2021; Zhong et al., 2022; DSTC1 2023; Liu et al., 2023), and we see high correlation between scores for engagingness and interestingness in the FED dataset (Mehri and Eskenazi, 2020a) (see Figure 4). However, that there is one-to-one correspondence be- tween interestingness and engagingness. Inter- estingness captures factual appeal (Mehri and Eskenazi, 2020b), while definitions of engag- ingness emphasize active participation. Both
do not believe
AA collocate is a word that appears in conjunction (or in the same context) as the other word with greater than random frequency.

---

Prompt Code
Description
Naive
simple baseline prompt; directly asking for score
Naive+R
Naive prompt with role assignment
HD
prompt derived from instructions given human annotators of engagingness
HD+R
the human-directed prompt
role assignment
MEEP
an intro like the HD prompt, and short phrases for each of our six subdimensions of engagingness the MEEP prompt with the "such as" phrase
MEEP+SA
MEEP+R
MEEP prompt with role assignment
MEER+SA-R
the MEEP prompt
"such as" phrase and role assignment
MEEP+SA-DIAN
MEEP+SA prompt edited used dialogue-level
MEEP+SA+R prompt edited to be used the dialogue-level
Table I: Overview of prompt styles. Some prompts are also translated into other languages.
contribute to meaningful conversations from different perspectives.
naive prompt approach is valid as its own path of inquiry, and similar prompt is used in Wang et al. (2023).
Engaging responses are specific the conversational context. This is another aspect of avoiding generic user re- sponses. See et al. (2019) show that control- ling for specificity improves the engagingness of their dialogue model.
Contextual Specificity:
Human-directed Prompts (HD) Since LLMs are developed to approximate natural language and natural linguistic interaction, we theorize that an evaluation prompt styled after the same instructions that human annotators were given should lead to increased performance over the Naive prompt.
Engaging responses create a sense of belonging, not one of othering (Powell and Menendian, 2022; Alexander and Andersson, 2022). Even an interesting conversation can quickly turn dull when we feel misunderstood, unheard, and looked down on. Conversations where rapport is established, on the other hand, are often enjoyable even in the absence of con- tent. This aspect of engagingness is particu- larly underexplored in the research.
Othering:
Prompts With Our Proposed Dimensions of En- gagingness (MEEP) We incorporate each of our five subdimensions of engagingness into a word or phrase in order to form the definition of engaging- ness within the prompt. This is demonstrated in Figure 3.
Annotated Prompt
Subdimensions
Score the following response... the following quelities: variety of response according the rontext, likelihoed of mncouraging the other partic pant to respond, likellhood encouraging a quality resporse the other partic pant, interestingness, specificity and Ikelingge of creating a senze belonging for the other part cipant.
Response Diversity Interactional Quality Interestingness
1. Contextuel Specificity
Ethaning
4 MEEP: Metric for Engagingness Evaluation using Prompting
We design several categories of prompts. A straight- forward naive prompt acts as the basis against which we compare our more developed prompt sets. We adapt dialogue annotator directions into prompts for the human-directed prompt set and our own proposed sub-dimensions of engagingness form the third prompt set. For each prompt set, we create version casting the LLM role. Table provides an overview of the prompt types, and the full text of each prompt is available in Appendix A. We experiment with prompts in English as well as Spanish and Chinese.
Figure 3: The MEEP prompt annotated to show the five engagingness subdimensions.
"Such As" and Role Assignment (SA, R) Yuan et al. (2021) find that even when controlling for the presence of example responses, the phrase "such as" increases performance in the domain of ma- chine translation evaluation. When added to our prompts, it is accompanied by an example response. Role assignment is theorized to increase the qual- ity of the responses generated by dialogue models (S "Svikhnushina and Pu, 2023), and we explore its utility in dialogue response evaluation. We incor- porate role assignment using methods specific the model used. When possible, we make the as- signment directly in the API call, using the "sys- tem" option. If this method not supported by
Naive Prompt (Naive) We use the Naive prompt baseline comparator from which develop more performant prompts. Koemi and Federmann (2023) report best performance from the least con- strained prompting method, implying that this

---

the model, the role assignment is prefixed to the prompt text.
5.2 Benchmark Datasets
We use four existing datasets, all from the 11th Dialogue System Technology Challenge, Track 44 (DSTC11 2023). Challenge organizers ma- chine translated two English turn-level datasets into Spanish and Chinese, and two Chinese dialogue- level datasets into English produce total of datasets spanning three languages. Annota- tions from the original datasets are used test the datasets in the additional languages.
Translation (P) We translate a selection of our prompts from English into Spanish and Chinese, to match the language of the dialogues evaluated. Spanish text was manually translated, and Chinese translations are with GPT-4 and checked by a hu- man translator for quality.
5 Experiments
FED The Fine- Fine-grained Evaluation of Dialogue (FED) dataset (Mehri and Eskenazi, 2020a) pro- vides annotations of English-language dialogues first collected by Adiwardana et al. (2020) by way of multi cilliidimeesional evaluations per dialogue turn, including engagingness as one of the evaluated di- mensions.
We first test our prompt combinations using two LLMs on the English-language FED dataset. Five examples from these tests are randomly selected, on which we perform qualitative analysis. We then select the best performing prompts to test over six multilingual turn-level datasets, two in each of English, Spanish, and Chinese. For these tests, we evaluate the prompts in English as well as the versions translated into the language of the dataset. The scores reported by each L.LM/prompt style pair over six datasets are correlated with the human- annotated scores to evaluate the proposed methods and four strong baseline methods. A final set of ex- periments evaluates the performance of the highest- performing prompts on dialogue-level datasets in English and Chinese and correlates with human- annotated scores. These are compared with the strongest baseline method from turn-level testing.
The Persona-See (SEE) dataset (See et al.. 2019) expands the PersonaChat (Zhang et al., 2018) 8) task to provide human annotations at the turn-level, evaluating multiple dimensions conversation per dialogue. They consider enjoyment to be an equiv- alent measure of engagement, relying on users' active involvement and positive perception. We regard it as reliable proxy for engagingness. Orig- inally containing over 3, 3,000 annotated dialogues, this is much larger than the other datascts, seen in Table 2. For efficiency and consistency of scale, randomly select subset of 300 dialogues for comparative analysis. These same 300 dialogues are used each of English, Spanish, and Chinese.
SEE
5.1 Large Language Models
For our base model, we select text-davinci--eee (GPT-3.5) (Ouyang et al., 2022),
KDCONV The Knowledge- driven Conversation (KDCONV) dataset consists of Chinese-language dialogues between two humans who are given knowledge graphs from which to draw their re- sponses (Zhou et al., 2020).
(ChatGPT), and gpt-3. 1tt-3.5-turbb-0300 (ChatGPT0613) from
the GPT-series of LLMSs (Radford et al., 2022). We set the temperature to O, top_p to 1. and n to I. Presence penalty and frequency penalty are set to O, logit bias to null, and we request the best of 1. We also use LLaMA2 (Touvron et al., 2023) in the 7B size with the chat_completion function with max_seep max_seq_len set to 1024, max_gen_lest set to None, temperature set to 0, max_maaaaaaa sax_batch_ss size set to 1, and top_p set to 1. We do not set a max length for generation because the responses are long and the location of scores within the returned text is unpredictable.
The large-scale cleaned Chinese conver- sation dataset (LCCC) contains conversations from Chinese social media posts of users heuristically identified to be human (Wang al., 2020). These dialogues combined with those from publicly available databases and cleaned.
LCCC
Dialogues for KDCONV and LCCC were manu- ally annotated at the dialogue-Ievel for engaging- ness and machine translated into English for the DSTC11 Track Challenge.
We note that ChatGPT is available at the time of writing as gpt-3.5-turbo 5-turbo in the free version, and gpt-4 in the paid version. We take the liberty of using ChatGPT as the short- hand for 2ot-3.5-turdte of testing.
Translation Quality Analysis We briefly look at dataset translation quality to assess the validity of
the free version at the time
hhttps://chatevalloor//sscc

---

Dataset
samples
Language
Annotation level
COMET-20
COMET-21
CosSiml
CosSim2
FED-EN
375
English
FED-ES
375
Spanish
turn
0.458
0.117
0.846
0.914
FED-ZH
375
Chinese
0.323
0.084
0.789
0.883
SEE-EN
300
English
SEE-ES
300
Spanish
turn
0.510
0.128
0.873
0,921
SEE-7H
300
Chinese
turn
0.376
0.102
0.800
0.875
KDCONV-EN
392
English
dialogue
0.242
0,053
0.788
0.857
KDCONV-ZE ZH
392
Chinese
dialogue
LCCC-EN LCCC-ZH
22
English
dialogue
-0.132
-0.40
0.690
0.776
22
Chinese
dialogue
Table Dataset specifications, with average machine translation quality of datasets that translated. The highest scores for each measure are in bold. Second highest scores are underlined.
the annotations as an accurate measure of engaging- ness for translated datasets. This informs analysis our test results in Section 6.2. Translations into Spanish and Chinese were obtained by the DSTC11 organizers with the MS Azure service and Tencent Machine Translation service", respectively.
et al., 2022). It accomplishes this by formulating single evaluation dimensions as boolean question- answering (QA) tasks. In evaluating the engaging- ness of a response, UNIEVAL takes three parame- ters: the conversational context, an additional con- text (such as an intriguing fact), and the response itself. To facilitate testing with the FED dataset which does not include additional fact, emply string used the additional context. We UNIEVAL multilingual context where it had not been previously evaluated.
The datasets include four measures of translation quality for every utterance: two quality estimator metries from COMET versions from 2020 (Rei al., 2020) and 2021 (Rei et al., 2021), and two cosine similarity measures from embeddings gen- crated from the original utterance and translation. Two methods of embedding generation were used. Details are available in Appendix B. We average the provided translation quality scores across all ut- terances for each translated dataset and list them in Table 2. The Spanish-language datasets score con- sistently higher than the Chinese-language datasets,
GPTSCORE evaluates dialogue response gener- ation at the turn-level across eight dimensions, including engagingness (Fu et al., 2023). Their in-
prompt for evaluation of engagingness cluded in Appendix A. Our approximation of their methodology is adapted from their code for text summarization evaluation. A prompt that in- cludes task definition, identification of the qual- ity be evaluated (engagingness), and the dia- logue, followed by "Answer: Yes.", is passed the LLM with temperature=0, max_ max_ tokenskkn=0, logprobs=0, echo=True, and n=0. The log prob- abilities returned for the token "Yes" become the score for engagingness.
5.3 Baselines
The results of our proposed method are compared against four recent baselines.
ENDEX evaluates the engagingness of dialogues based on human reactions (Xu al., 2022). This approach uses fine-tuned RoBERTa-large models on datasets evaluated at the turn level, measuring dimensions of engagement including attentional, behavioral, emotional, and task-based aspects of replies. Each response is given binary score for engagingness. Due the inherent constraints of ENDEX, we encountered difficulties in adapting languages other than English.
G-EVAL an LLM-based evaluator comprising a prompt, Chain-of-Thouuht reasoning, and a scor- ing function (Liu et al., 2023). It achieves strong correlations with human evaluations using GPT-3.5 and GPT-4 models in NLG tasks, including the assessment of engagingness in dialogues. We their GPT-3.5 model, which shows higher correla- tions, and modify the prompt for testing datasets without additional context, as seen in Appendix A.
UNIEVAL employs T5-Large as the underlying model to assess any dialogue dimension, includ- ing those that are not seen during training (Zhong
'ht www.cooo.t.com//enu/////oo////// ccmm/en-us/prduuctss soersttiiee-erriccssttrrsssss Phttps://wwwr tencentcloud com/ com/ products/tmt

---

Prompt
GPT3.5
ChatGPT
ChatGPT0613
LLaMA2-7B
Average
S
P
P
S
P
S
S
P
Naive
0,445
0.521
0.434
0.463
0,416
0.435
0.356
0.272
0.413
0.423
Naive+R
0.416
0.466
0.464
0.469
0.471
0.480
0.356
0.322
0.427
0.434
HD
0,509
0.511
0.462
0.472
0.511
0.535
-0.007
0.046*
0.369
0.391
HD+R
0.492
0,500
0.497
0,508
0.518
0.526
0.112
0.029*
0.405
0.391
MEEP
0.475
0,508
0.548
0.553
0.511
0,504
0.333
0,111
0.467
0,419
MEEP+R
0.494
0.507
0.540
0,543
0,514
0.516
0,402
-0.057*
0.487
0.377
MEEP+SA
0.532
0,520
0.558
0.573
0.537
0,549
0.410
0.091
0.509
0,433
MEEP+SA-R
0.526
0.517
0.568
0,566
0.541
0.548
0.376
0.113
0,503
0.436
Table 3: Results for prompts: Naive, Sluan--Directed (HD), and MEEP, with ablation results for prompt elements such as (SA) and system role (R). Spearman (S) and Pearson (P) results are listed for each experiment. Results marked with are not statistically significant. The highest result for each column is in bold, with the second highest result underlined. The FED-EN dataset was used for testing.
6 Results and Analysis
almost entirely performs worse with the prepended system role, indicating that the system role prompt not beneficial when used with GPT-3.5 in this set- ting. This combined with mixed performance when used on LLaMA2-7B suggests that it may add a level of complexity to which the simpler models cannot rise.
6.1 Effect of Prompt Styles
Effect of MEEP Average correlations of system output and human annotations for each prompt shown in Table 3. Prompts including MEEP gen- erally give higher correlation than experiments not including MEEP in the prompts. The method with lowest correlation is the human-directed prompt, with average correlations of 0.420 (S) and 0.429 (P). The HEEP-based prompts have average corre- lations of 0.492 (S) and 0.481 (P). This indicates that including our fine-grained subdimensions of engagingness in prompting for evaluation of engag- ingness is an important component of an effective evaluation method.
Comparing Models Considering that ChatGPT is an interactive LLM specifically trained on con- versational data, we are curious to see whether it brings additional gains to our task. In Table 3, we see that ChatGPT has an average of 0.023 (S) and 0.012 (P) higher correlations than GPT-3.5 across all Inglish-languagg turn-level experiments. ChatGPT0613 has similar though slightly lower ag- gregate performance to that of ChatGPT. LLaMA has 0.151 (S) and 0.302 (P) lower average perfor- mance than GPT3.5, considering statistically signif- icant results only. GPT-3.5 has better performance than ChatGPT and ChatGPT0613 for the simplest prompts, when used without system role assign- ment, otherwise ChatGPT is the model that is most aligned with human annotations.
Effect of 'such as' As seen in Table 3, prompts with our engagingness dimensions (MEEP) and the use of 'such as' have the highest and second highest correlations, in 5 of 8 measures. Prompts including 'such as' have average correlations of 0.506 (S) and 0.492 (P). Their performance dominates both with and without defining the system role.
6.2 Performance in Multilingual Setting
Effect of system role In Table 3, we that the use of system role has mixed impact. Performance for ChatGPT and ChatGPT0613 improves with sys- tem role most cases, while performance varies for LLaMA2-7B and generally falls for GPT-3.5. Defining the system role gives the most consistent gains in performance when used with ChatGPT on prompts that do not include MEEP. The inconsis- tent improvements with the use of system role may be because it offers a contextual shift that is not as closely aligned as other methods, like MEEP. This more approximate alignment at times moves the model to a less-alls aess-ed position than it would be if using a better aligned method alone. GPT-3.5
6.2.1 Turn-level Evaluation
Table 4 reports Spearman coefficients from ex- periments on multilingual turn-level datasets with prompts in English, Spanish, and Chinese (Pear- son coefficients are available in Appendix C). Our highest-performing English prompts have an average of 0.064 higher correlation than the higghet--peffrmiin bascline metric (G-EvAL GPTSCORE) for that dataset. Average increases of the highest-correlated English prompt over the highest-correlated baseline are 0.060, 0.082, and 0.050 for the English, Spanish, and Chinese- language datasets, respectively.

---

Model
Prompt
FED
SEE
EN
ES
Z.H
EN
ES
7H
ENDEX
0.290
0.164
UNIEVAL
0.190
0.258
0.076*
0.015*
0.073*
0.038
GPTSCORE
GPTSCORE
0.176
0.146
0.230
0.087*
0.153
0.140
G-EVAL
G-EVAL
0.488
0.448
0.402
0.194
0.131
0.062
GPT-3.5
MEEP+SA
0.532
0,481
0.451
0.236
0.223
0.189
ChatGPT
MEEP+SA
0.558
0.516
0.431
0.169
0.138
0.133
ChatGPT
MEEP+SA+R
0.568
0.542
0.435
0.160
0,150
0.140
ChaGGPT-0613
MEEP+SA+R
0.550
0.471
0.400
0.214
0.200
0.175
GPT-3.5
SEA
0.438
0.520
0.128
0.085*
ChatGPT
MEEP+SA7
0.500
0.408
0.161
0.149
ChatGPT
MEEP+SA+
0,525
0.444
0.123
0.168
ChatGPT-0613
MEEP+SA+R?
0.542
0.374
0.273
0.227
Table 4: Correlation results using Spearman coefficient for multilingual turn-level experiments. All results are statistically significant except those labeled with * "4" denotes a translated version of the prompt into the language the dataset. Highest results are bold.
Model
Prompt
KDCONV
LCCC
EN
ZH
EN
ZH
G-EvAL
G EVAL-DIAL
0,327
0.189
0.149
0,248
GPT-3.5
MEEP+SA-DIAL
0.286*
0.223
0.282
0.287
ChatGPT
MEEP+SA-DIAN
0.178
0.392
0.073*
0.191
ChatGPT
SIEP+SAAR-DIAL
0.185
0.428
-0.004*
0.193
ChatGPT-0613
MEEP+SA+R SA-L-DIAL
0.186
0.362
0,032*
0.117
Table 5: Correlation results using Spearman coefficient for multilingual dialogue-level experiments. Results that are not statistically sigmificant are marked with * Highest scores for each dataset are in bold.
Results indicate that correlation using our method is highest for English-language dialogues, followed by Spanish-language dialogues, with Chinese-language dialogues having the lowest cor- relations. We would expect see comparable per- formance because they are all high-resource lan- guages. The unexpected results may be related to translation quality of the datasets, which is lower for the Chinese datasets as seen in Table 2. A comparison of results for the Spanish and Chinese datasets in Figure 4 shows that translating prompt into the language of the dataset does not consis- tently improve correlation, but our best scores for the Spanish and Chinese datasets are nevertheless seen with translated prompts. The ChatGPT06 13 model provides several of these highest correla- tions, indicating that improved system role capabil- ities with this model may have included multilin- gual system role training.
Spearman Coufiiclent
0.5
ChaRGPT MEEP-SA
ChaxPPT MEEP-BA+R
0.4
ChattPP (-000
0.3
-GPT33 T[MERP-SA
ChargPT TIMEEP+SA)
ChusoPT
TIMEEP-SAHR)
0.1
CheeGPT0613
T(MEEP-SA+R)
0.0
FED-ES
FED-ZH
SEE-ES
SEE-ZH
Datasets
Figure 4: Correlation of each prompt on the Spanish and Chinese datasets.
Qualitative Analysis An example from qualita- tive analysis is provided in Table 6, with more ex- amples in Appendix D. The randomly- amples show mixed consistency with general trends observed with testing. We note that MEEP prompts improve performance over aggregate results from HD and Naive prompts, although exceptions can be found. The effect of the system role is difficult to discern at this scale. ChatGPT performs better or similarly to GPT-3.5 in four out of five examples, holding with the pattern exhibited in the data.
ex-
6.2.2 Dialogue-level Evaluation
Results for dialogue-level evaluation are in Table 5. Achieving statistically significant results is less consistent on the dialogue-level than the turn- level datasets, especially for the translated English- language versions. Despite this, we can see hetter performance with our prompts than with the base- line.
We see a pattern of GPT-3.5 more often giving scores that are higher than the ground truth anno- tations. When we average the scores for the top- performing models, GPT-3.5 and ChatGPT, across all the model/prompt combinations used in Table 3, we find that GPT-3.5 produces scores appreciably higher than ChatGPT, with averages of 0.8138 and

---

Dialogue: "Hi!"
naive prompt. We infer that an increase with the more powerful model, the context must be appropriately set, as with MEEP+SA. and that simpler prompts are more appropriate for smaller LLMs.
"Hii! How are you?"
am good"
"You're welcome! How long until you re done with train- ing models?"
"We keep on trying to improve them so guess it Illl be a while"
The results for our MEEP prompts used on multilingual datasets show improvement over state-of-the-art baselines in the evaluation of engagingness in dialogue across Chinese. Spanish, and English. This is also consistent across turn-level and dialogue-Ieved datasets.
Response: "That's & understandable. Good luck, hope goes smoothly!"
Normalized average human annotations: 0.6
Prompt
GPT-3.5
8
ChatGPT
Naive
0,9
0.3
0.85
0.25
Naive
1.0
0.4
0.85
0.25
HD
1.0
0.4
0.5
0.1
HI++R
1.0
0)4
0.5
0.1
In the multilingual setting, for turn-level dia- logues, automatic evaluation is most strongly correlated with human evaluation when the prompt is in the language of the dialogues.
MEEP
0.9
0.3
0.8
0.2
MEEP+R
0.9
0.3
0.8
0.2
MEEP+SA 0.8
0.2
0.8
0.2
MEEP+SA+R
0.9
0.3
0.8
0.2
Table 6: Comparison of results for one example dia-
Conclusion and Future Work
logue from the FED dataset for qualitative analysis. The distance from the ground truth (d) for each score is listed the right of the score.
We propose a novel method for estimating engag- ingness in dialogue using LLMs as automatic eval- uators. Our method - MEEP outperforms state- af- baselines, and when used in prompts taining a 'such as' phrase with examples, leads to better correlation with human annotated scores. This performance demonstrated in multilingual context, using prompts and dialogues in English, Spanish, and Chinese and for both turn-level and dialogue-Ievel evaluation. Our findings indicate that there is promise in the evaluation of other com- plex dialogue qualities such model's ss ability to provide emotional support through similar use of prompting with LLMs. We see opportuni- ties for future work in the development and use of non-translated datasets in languages other than English, with annotations for a well-defined mea- sure of engagingness. In the future, we would like to continue to explore lighter models like LLaMA (Touvron et al., 2023) ORCA (Mukherjee et al., 2023) with our method for more energy-efficins approach (Appendix E).
0.7307, respectively. This indicates that GPT-3.5 has a positive bias in evaluation when compared to ChatGPT, which may reveal underlying limitations in the model's 's capability.
7 Discussion
Insights from our experiments showcase the power our proposed LLM-based evaluator as follows:
Our MEEP dimensions of engagingness im- prove alignment with human annotations and are an effective component of an LLM--base metric, improving over state-of-the-art met- rics in evaluation of engagingness.
Clear improvement seen with the use of such as with clarifying examples in this con- text and we conclude that examples written in this form will improve dialogue evaluation performance with these LLMs.
Defining the system role offers improvement when used with ChatGPT ChatGPT0613 and prompts that naive. It does not im- prove performance when used with LLaMA2- 7B, or GPT-3.5.
Ethical Considerations
The use of LLM-based metrics in evaluating lan- guage models raises concerns regarding potential bias (Gao and Wan, 2022) and self-reinforcementt (Liu et al., 2023). Language models like GPTs are trained on large datasets, which may contain biases and inaccuracies that can impact evaluation tasks. This is particularly important in the context of self-Al feedbuck (Fernandes et al., 2023), where LLM- metrics may prefer LLM -eg--geee
ChatGPT performs better than GPT-3.5 with more complex prompts. When prompts are in their simplest form, GPT-3.5 has higher correlation with human annotations. LLaMA gives highest correlations when used with the

---

texts, potentially reinforcing biases. More specifi- cally, the dialogues and annotations are produced by crowd workers with unknown backgrounds. Ide- ally, they would represent a wide range of ethnic, racial, and economic backgrounds, with varied di- alects. We are not aware of the composition of the workers selected. Since their annotations become our ground-truth, there is possibility that we base evaluation of engagingness that of a pop- ulation that does not fully represent world-wide diversity. Differences in dialect, tone, or wordiness can be interpreted uniquely depending on cultural or individual preferences. By potentially limiting our definition of engagingness to what is seen in these datasets, we admit the possibility of training dialogue systems that are engaging to limited pop- ulation, limiting the accessibility of the tools that use these systems.
smallest version due limited resources.
The lack of transparency in Al models presents a challenge as it hampers a comprehensive under- standing of the assessment process. These limi- tations highlight the importance of further explo- ration, diverse datasets, and increased transparency to strengthen the validity and applicability of our research findings on engagingness in conversations.
Acknowledgements
We thank the anonymous reviewers and the PortNLP group for their insightful comments and suggestions. We express gratitude to Yufei Tao and Russell Scheinberg for assistance with translation, and to Ackta Shah, PhD for suggesting othering and belonging as possible impact on engaging- ness. This research was supported by the National Science Foundation under Grant No. CRII:RI- 2246174.
While more engaging dialogue systems have promising applications in domains like virtual as- sistants and medical support, the use of evaluation metrics beyond evaluation can lead to unintended consequences. Additionally, ethical issues arise due to the lack of transparency of Al models, as well as privacy and data security concerns when crucial to be
References
Daniel Adiwardana. Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang. Apoory Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoe V. Le. 2020. Towards a human- like open- domain chatbot. CoRR, abs/2001.0990 109977.
handling sensitive information. aware of these considerations and prioritize the eth- ical and responsible use of LLMs and evaluation metrics.
Vibhav Agarwal, Pooja Rao, and Dinesh Babu Jayagopi. 2021. Towards code-mixed Hinglish dialogue gener- ation. In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI, pages 271-280, Online. Association for Computational Lin- guistics.
Limitations
Our research on engagingness in conversations is limited to existing datascts available for this spe- cific quality. We see promise in the creation of dataset with intentionally diverse perspectives engagingness for the development of evalua- tion metries that can represent the plurality of user backgrounds.
Jessica Joelle Alexander and Camilla Semlov Ander- sson. 2022. Fællesskab and belonging. In Míriam Tuan-Torres González, Yvette Tetteh, and EJ Top- pin, editors, Puper Series: On Belonging in Europe. Othering Belonging Institute.
COCA. Corpus of Contemporary American English. https:/ www.:/engllss..... english- wwwshhooorr..ooo org//oca// Ac- cessed: June 5, 2023.
Our evaluation of prompt styles is not exhaustive, as the possibilities are vast. We limit our prompt styles to those found to be useful in existing re- search, with strong theoretical basis support them. We leave for further exploration the evalua- tion of wider range of prompt styles.
Deborah Cohen, Moonkyung Ryu, Yinlam Chow, Orgad Keller, Ido Greenberg. Avinatan Hassidim, Michael Fink. Yossi Matias, Idan Szpektor, Craig Boutilier, and Gal Elidan. 2022. Dynamic planning in open- ended dialogue using reinforcement learning.
In Spanish- and Chinese-language dialogue eval- uation, our findings are limited to the evaluation of translations from English. For a more robust multilingual evaluation, we would use datasets cre- ated in each language and evaluated by speakers of those languages.
Pierre Colombo, Maxime Peyrard, Nathan Noiry, Rohert West, and Pablo Piantanida. 2022. The glass ceiling automatic evaluation in natural language generation.
Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. 2021. Compression, transduction. and creation: A unified framework for evaluating
Our experiments with LLaMA use only the

---

natural language generation. In Proceedings the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7580-7605, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human align- ment.
Varvara Logacheva, Mikhail Burtsev, Valentin Malykh, Vadim Polulyakh, and Alcksandr Seliverstov. 2018. comaa-too-haaton
Oxford English Dictionary. 2023. Engaging.
Convai dataset of topic-oriented dialogues. In The NIPS '17 Competition: Building Intelligent Systems, pages 47-57, Cham. Springer International Publishing.
DSTC1 2023. Dstell: Dialogue system technol- ogy challenge 11. https///chhaealloggddsttttt org/dstc11. [Accessed 06-Jun-2023)
Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chatgpt factual inconsistency evaluator for text summarization.
Patrick Fernandes, Aman Madaan, Emmy Liu, António Farinhas, Pedro Henrique Martins, Amanda Bertsch, José G. C. de Souza, Shuyan Zhou, Tongshuang Wu, Graham Neuhig, and André F. T. Martins. 2023. Bridging the gap: A survey on integrating (human) feedback for natural language generation.
Shikib Mehri, Jinho Choi, Luis Fernando D'Haro, Jan Deriu, Maxine Eskenazi, Milica Gasic, Kallirroi Georgila, Dilek Tur,, Zekang Li, Verena Rieser, Samira Shaikh, David Traum, Yi-Ting Yeh, Zhou Yu, Yizhe Zhang, and Chen Zhang. 2022. Re- port from the nsf future directions workshop on auto- matic evaluation of dialog: Research directions and challenges.
Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire.
Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Ship- ing Yang, and Xiaojun Wan. 2023. Human-like like sum- marization evaluation with chatgpt.
Shikib Mehri and Maxine Eskenazi. 2020a. Unsuper- vised evaluation of interactive dialog with DialoGPT. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 225-235, 1st virtual meeting, Association for Computational Linguistics.
Mingqi Gao and Xiaojun Wan. 2022. Social biases in automatic evaluation metrics for nlg.
Nan Gao, Mohammad Saicdur Rahaman, Wei Shao, and
Flora D. Salim. 2021. Investigating the reliability of self-report data in the wild: The quest for ground truth.
Shikib Mehri and Maxine Eskenazi. 2020b. USR: An unsupervised and reference free evaluation metric for dialog generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681 18777 Online. Association for Computational Linguistics.
Sebastian Gehrmann, Elizabeth Clark, and Thibault Sel-
lam. 2022. Repairing the cracked foundation: A sur- vey of obstacles in evaluation practices for generated text.
Merriam- Webster. 2023. Engaging.
Sarik Ghazarian, Ralph Weischedel, Aram Galstyan, and Nanyun Peng. 2020. Predictive engagement: An efficient metric for automatic evaluation of open- domain dialogue systems. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7789- 7796.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawa- har, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. Orca: Progressive learning from complex explanation traces of gpt-4.
Jekaterina Novikova, Ondřej Dušek, Amanda Cer- cas Curry, and Verena Rieser. 2017. Why need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natu- ral Language Processing, pages 2241-2252, Copen- hagen, Denmark. Association for Computational Lin- guistics.
Tom Koemi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality.
Chin-Yew Lin. 2004. ROUGE: package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Chia Shia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Nose- worthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metries for dialogue response generation. In Proceedings of the 2016 Conference Empirical Methods in Natu- ral Language Processing, pages 2122-2132, Austin, Texas. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: A method for automatic evalu- ation of machine translation. In Proceedings of the

---

40th Annual Meeting on Association for Computa- tional Linguistics, ACL '02, page 311-318, USA. Association for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
John A. Powell and Stephen Menendian. 2022. On belonging: An introduction to othering belonging europe. In Mfriam quan-Torres González, Yvette Tetteh, and EJ Toppin, editors, Paper Series: On Belonging in Europe. Othering Belonging Institute.
Marilyn A. Walker, Diane J. Litman, Candace A. Kamm, and Alicia Abella. 1997. PARADISE: A frame- work for evaluating spoken dialogue agents. In 35th Anmual Meeting of the Association for Com- putational Linguistics and 8th Conference of the Eu- ropean Chapter of the Association for Computational Linguistics, pages 27 1-280, Madrid, Spain. Associa- tion for Computational Linguisties.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine McLeavey, and Nya Sutskever. 2022, Robust speech recognition via large-scale weak su- pervision.
Ricardo Rei, Ana C. Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, André F. T. Martins, and Alon Lavie. 2021. Are references really needed? unbabel-IST 2021 submission for the metrics shared task. In Pro- ceedings of the Sixth Conference Machine Trans- lation, pages 1030-1040, Online. Association for Computational Linguistics.
Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu and Jie Zhou. 2023. Is chatgpt a good nlg evaluator? preliminary study.
Yida Wang. Pei Ke, Yinhe Zheng, Kaili Huang, Yong Jiang, Xiaoyan Zhu, and Minlie Huang. 2020. A large-scale chinese short-text conversation dataset. In Natural Language Processing and Chinese Com- puting: 9th CCF International Conference, NLPCC 2020, Zhengzhou, China, October 14-18, 2020, Pro- ceedings, Part I, page 91-103, Berlin, Heidelberg. Springer- Verlag.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Unbabel's participation in the WMT20 metrics shared task. In Proceedings of the Fifth Con- ference on Machine Translation, pages 911 110- On- line. Association for Computational Linguistics.
participaoo
Christopher Richardson, Sudipta Kar, Anjishnu Ku- mar, Anand Ramachandran, Omar Zia Khan, Zeynab Raeesy, and Abhinav Sethy. 2023. Learning re- trieve engaging follow-up queries. In EACI. 2023.
Wiktionary. 2023. Engagiig.
Guangxuan Xu, Ruibo Liu, Fabrice Harel-Canada, Nis- chal Reddy Chandra, and Nanyun Peng. 2022. En- Dex: Evaluation of dialogue engagingness scale, In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4884-4893 Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics
Mario Radríguez-Cantella, Chen Zhang, Chengguang Tang, Ke Shi, Sarik Ghazarian, João Sedoc, Luis Fer- nando D'Haro, and Alexander Rudnicky. 2023, Overview of robust and multilingual automatic eval- uation metrics for open-domain dialogue systems at dste 11 track 4.
Robert Rosenman, Vidhura Tennekoon, and Laura G. Hill. 2011. Measuring bias in self-reported data. In- ternational Journal Behavioural and Healthhare Research, 224):300-322.
Zhou Yu, Leah Nicolich-Henkin, Alan W Black, and Alexander Rudnicky, 2016. A Wizard-of-Oa study on non-task-oriented dialog systems that reacts to user engagement. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 55-63, Los Angeles. Associa- tion for Computational Linguistics.
Abigail See. Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? how controllable attributes affect human judgments. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Humun Language Technologies, Volume (Long and Short Papers), pages 1702-1723, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text genera- tion. In Advances in Neural Information Processing Syxtems, volume 34, pages 27263-27277. Curran As- sociates, Inc.
Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. 2023. In chatgpt we trust? measuring and characterizing the reliability of chatgpt.
Chen Zhang, Yiming Chen, Luis Fernando D' Haroo. Yan Zhang, Thomas Friedrichs, Grandee Lee, and Haizhou Li. 2021. DynaEval: Unifying turn and di- alogue level evaluation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Confer- ence on Natural Language Processing (Volume I: Long Papers), pages 5676-5689 Online, Association for Computational Linguistics.
Katherine Stasaski and Marti A. Hearst. 2023. Pragmat- ically appropriate diversity for dialogue evaluation.
Ekaterina Svikhmushina and Pearl Pu. 2023. Approx- imating human evaluation of social chatbots with prompting.

---

Chen Zhang, Luis Fernando D'Haro, Qiquan Zhang, Thomas Friedrichs, and Haizhou Li. 2022a. FineD- eval: Fine-grained automatic dialogue-level evalu- ation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3355, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Qingyu Zhang, Xiaoyu Shen, Ernie Chang, Jidong Ge. and Pengke Chen. 2022b. Mdia: A benchmark for multilingual dialogue generation in 46 languages.
Saizheng Zhang, Emily Dinan, Jack Urbanck, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of Association for Computational Lin- guistics (Volume I: Long Pupers), pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse- leveel level diversity for neural dialog models using conditional variational autoencoders. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics Voolume
1: Long Popers), pages 164-664, Vancouver, Canada. Association for Computational Linguisties.
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards unified multi- dimensional evaluator for text generation. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 2023- 2038, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Hao Zhou, Chujie Zheng. Kaili Huang, Minlie Huang. and Xiaoyan Zhu. 2020. KdConv: A Chinese multi-domain dialogue dataset towards multi-turn knowledge-drieg conversation. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, pages 7098-7108, Online. As- sociation for Computational Linguistics.
Pei Zhou, Hyundong Cho, Pegah Jandaghi, Dong-Ho Lee, Bill Yuchen Lin, Jay Pujara, and Xiang Ren. 2022. Reflect, not reflex: Inference-based common ground improves dialogue response quality. In Pro- ceedings of the 2022 Conference Empirical Meth- ods in Natural Language Processing, pages 10450- 10468, Abu Dhabi, United Arab Emirates. Associa- tion for Computational Linguistics.

---

A
Full Text of Prompts
Prompt
Description
Code
Naive
Score the following response given the corresponding dialogue context continuous scale from 0 to 100, where score zero means 'disengaging' and score of 100 means "very engaging'. Assume the response immediately follows the dialogue context. Dialogue context: <dialogue>
Response: <dialogue>
Score:
HD
You will be given conversation between two individuals. You will then be given several potential responses for the next turn in the conversation. Is the response engaging?
A response is considered engaging if it can engage the user. This might be inquisitive question an interesting response that can be followed-up on.
A score means No: the response is boring and does little to engage the user. there.
Oh wow! That's cool!
A score of 2 means Somewhat: the response is not particularly engaging but still leaves room for follow-up.
My favourite colour is blue.
Nope. I'm not very good with cooking.
A score of means Yes: the response is actively engaging the user and trying to move forward conversation.
have feeling that if you can dodge a wrench you can dodge a ball.
What kind of shows do you like?
Dialogue context: <dialogue>
Response:
<dialogue>
Score:
Table 7: Full text for English-language prompts Naive and HD used in our experiments.

---

MEEP
Score the following response given the corresponding dialogue context on continuous scale from 0 to 100, where score of zero means 'disengaging' and score of 100 means "very engaging' Assume the response immediately follows the dialogue context. Consider that engagingness of response is defined by the following qualities: variety of response ac- cording to the context, likelihood of encouraging the other participant to respond, likelihood encouraging a quality response from the other participant, interestingness, specificity, and likelihood of creating a sense of belonging for the other participant.
Dialogue context: <dialogue>
Response: <response>
Score:
MEEP
Score the following response given the corresponding dialogue context continuous scale from 0 to 100, where score zero means "disengaging" and score of 100 means 'very engaging'. Assume the response immediately follows the dialogue context. Consider that engagingness according to the context (such as responding to 'Hi how are you?" with feel magnificent, because just successfully defended my PhD! How are you?" instead of 'Good, how are you?"), likelihood of encouraging the other participant to respond (such as 'I love legos! I like using them to make funny things. Do you like legos?" instead of 'I like legos.), )), likelihood of encouraging quality response from the other participant, interestingness, specificity, and likelihood of creating sense of belonging for the other participant. Dialogue context: <dialogue>
SA
response is defined by the following qualities: variety of response
Response:
<response>
Score:
Table 8: Full text for Anglish-languaar prompts MEEP and MEEP+SA used in our experiments.

---

Prompt Code
Description
ES(MEEP
Evalúí la siguiente respuesta dada el contexto de diálogo correspondiente en una escala continua del 0 al 100, donde una puntuación de cero significa "desinteresante" y una puntuación de 100 signífica "muy interesante". Supongamos que respuesta sigue inmediatamente después del contexto de diálogo. Considera que la cualidad de una respuesta interesante se define por las siguientes características: variedad de respuesta de acuerdo al contexto (por ejemplo, responder a "Hola, ξcómo estás?" con "Me siento magnífico porque acabo de defender exitosamente mi tesis doctoral. YY tú?" en lugar de "Bien, dy tú?"), probabilidad de incentivar al otro participante responder (por ejemplo, MiMe encantan los legos! Me gusta usarlos para hacer cosas divertidas. ?T gustan los legos?" en lugar de "Me gustan los legos."). probabilidad de incentivar una respuesta de calidad del otro participante, interés, especificidad y probabilidad crear un sentido pertenencia para el otro participante.
+SA)
Contexto de diálogo: <dialogue>
Respuesta: <response>
Puntuación:
ZH(MEEP
THE & B HETTENEEDDIIIONEEEEEEEEEEEEEE
+SA)
T 1003) DO
#A (MO f M " BY
E ??
fR' FF? ", NNEA ") " R (M)
fl:: " TAAAR
- 7)
# <dialogue>
F <response>
+R
You are an expert evaluator of dialogue.
ES(+R)
Eres experto en evaluación de diálogos.
ZH(+R)
Table 9; Full text for MEEP+SA prompt translated into Spanish and Chinese; system role prompt in English, Spanish, and Chinese

---

Prompt
Description
Code
MEEP
In this task, you will be shown part of dialogue. Score the dialogue on continuous scale from 0 to 100, where a score zero means 'disengaging' and score of 100 means "very engaging". Consider that engagingness of each dialogue is defined by the following qualities: variety responses according to the context (such as responding to 'Hi how are you?" with feel magnificent, because just successfully defended my PhD! How are you? instead of 'Good, how are you?"), likelihood of encouraging the other participant to respond (such as 'I love legos! like using them make funny things. Do you like legos?" instead of 'I like legos.") ") likelihood of encouraging quality responses from the other participant, interestingness, specificity, and likelihood of creating sense of belonging for the other participant. Consider the overall engagingness of the conversation. Dialogue: <dialogue> Score: <score>
+SA
-DIAL
G-EVAL -DIAL
You will be given conversation between two individuals. Your task is to rate the conversation on one metric. Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it needed.
Evaluation Criteria:
Engagingness (1-3) the conversation dull/interesting?
A score (dull) means that the conversation is generic and dull.
A score of 2 (somewhat interesting) means the conversation is somewhat interesting and could engage you in the conversation (e.g.. an opinion, thought)
A score of 3 (interesting) means the conversation is very interesting or presents an interesting fact
Evaluation Steps:
Read the conversation carefully.
2. Rate the conversation on scale of 1-3 for engagingness, according to the criteria above.
Provide a brief explanation for your rating, referring to specific aspects of the conver-
sation.
Example:
Conversation: <dialogue>
Evaluation Form (scores ONLY):
-Engagingness:
Table 10: Full text for dialogue- MEEP+SA-DIAL and G-EVAL-DIAL prompts. G-Eval prompts edited slightly for applicability to dialogue-level evaluation.

---

Prompt
Description
Code
GPTSCORE
Answer the question based the conversation between human and Al.
Question: Are the responses of Al engaging? (a) Yes. (b) No.
Dialogue context: <conversation>
Response: <response>
Answer: Yes.
G-EVAL
You will be given conversation between two individuals. You will then be given one potential response for the next turn in the conversation.
Your task
the responses on one metric.
Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer
needed.
Evaluation Criteria:
Engagingness (1-3) Is the response dull/interesting?
-A score of 1 (dull) means that the response is generic and dull.
-A score of 2 (somewhat interesting) means the response is somewhat interesting and could engage you in the conversation (e.g., opinion, thought)
A score of 3 (interesting) means the response is very interesting presents interest-
ing fact
Evaluation Steps:
Read the conversation, the corresponding fact and the response carefully.
Rate the response on a scale of 1-3 for engagingness, according to the criteria above.
3. Provide a brief explanation for your rating, referring to specific aspects of the response and the conversation.
Example:
Conversation History: <dialogue>
Response: <response>
Evaluation Form (scores ONLY):
Engaginggees::
Table 11: Full for GPTSCORE and G-EVAL prompts.

---

B Translation Quality Metrics
COMET-20 and COMET-21 are quality es- timator metrics from the COMET models -mm20-coomet-ee--aaaeee (Rei et al., 2020) and wmt221-comt--ge-mmm (Rei et al., 2021). CosSiml and CosSim2 measure cosine similarity of the orig- inal utterance and the translation after generat- ing embeddings for both with the SentenceTrans- former library 7. They use the multilingual mod- els distiluse-base-me hagluuee-bae-mmtttiiiniaaa-aaa and ingga re- spectively.
/htpss//wwwwsbet.nee//

---

C Results for Multilingual Datasets Pearson Coefficients
Model
Prompt
FED-EN
FED-ES
FEZ-ZH
SEE-EN
SEE-ES
SEE-ZH
ENDEX
0.260
0.159
UNTEVAL
0.155
0.161
0.079
0.008
0.063*
0.024
GPTSCORE
GPTSCORE
0.174
0.102
0.222
0.076
0.158
0.115
G-EVAL
G-EvAL
0.452
0.427
0.371
0.254
0.219
0.171
GPT-3.5
MEEP+SA
0.520
0.495
0.467
0.265
0.254
0.202
ChatGPT
MEEP4SA
0.573
0.562
0.449
0.205
0.172
0.168
ChatGPT
MEEP+SA+R
0.566
0.573
0,439
0.213
0.197
0.179
ChatGPT-0613
MEEP+SA+R
0.556
0.490
0,406
0.277
0.250
0.231
GPT-3.5
MEEP+SAf
0.020
0.493
-0.092*
0.092
ChatGPT
MEEP+SAf
0.498
0,414
0.185
0.174
ChatGPT
MEEP+SA+R+
0.512
0.440
0. 0.166
0.168
ChatGPT-0613
MEEP+SA+R1
0.519
0.362
0.283
0.263
Table Pearson correlation results on multilingual datasets. 't' denotes human-translated version of the prompt into the language of the dataset. All results statistically significant except those labeled with 9, Best results are highlighted in bold. Second best results are underlined.

---

D Select Examples from English-language FED dataset
Dialogue: "Hi!"
"Hey there! How's going?"
"Hi, how are you?"
"I'm doing well. thank you. How are you?"
"I'm fine. What's up?"
"Not much. How about you?"
"nothing new. Anything exciting happening recently?"
"Not really. You?"
"I watched a movic recently which was pretty good"
"That's good. What movie?"
"the irishman, have you seen it?"
"I have not. Should 1?"
"I think so"
"Alright, I'll put it on my watch list."
"How about you? Have you watched anything interesting recently?"
"I've been waiting to see the new Star Trek movie."
"what is it about?"
"The adventures of Kirk, Spock, and McCoy."
"when was it released?"
"It just came out today." "are you sure, I think it's been out for while"
"I'm positive. I know because I just watched it."
"well, the fact you just watched doesn't mean it was released today"
"That's what I said, I did watch it today."
"ok, I'll put it
watchlist"
"I.et me know what you think of it when you watch it."
"sure, I need to go, nice talking you"
Response: "Good talking to you too."
Normalized average of human annotations: 0.4
Prompt
GPT-3.5
ChatGPT
Naive
0.4
0.8
0.4
Naive
0.8
0.4
0.8
0.4
0,4
0.5
HD+R
0.1
0.1
MEEP
0,2
0.1
MEEP4R
0.7
0.2
0.1
MEEP+SA
0.2
0.5
0.1
MEEP4SA+R
0.7
0.2
Table 13: Comparison of results for one example dialogue from the FED dataset for qualitative amalysis. The distance from the ground truth (8) for each score listed to the right of the score.
Dialogue: "Hi!"
"Have you seen any gond movies lately?"
"Last weekend saw 'The Parasite", Ever heard of it?"
Response: "No, Why did you pick that movie?"
Normalized average human annotations: 0.9
Prompt
GPT-3.5
ChatGPT
Naive
0.8
0.1
0.6
0.3
Naive R
0.8
0.1
0.6
0.3
HD
1,0
0.1
0.5
0.4
HD+R
0.1
0.5
0.4
MEEP
0.8
0.1
0.6
0.3
MEEP4R
0.8
0.1
0.6
0.3
MEEP+SA
0.8
0.1
0.6
0.3
MEEP+SA+R
0.8
0.1
0.6
0.3
Table 14: Comparison results for one example dialogue from the FED dataset for qualitative analysis. The distance from the ground truth (8) for each score listed to the right of the score.

---

Dialogue: "Hi!"
"Heyy!"
"How's your day going?"
"Having fun, hut have bit of headache How ahout you?"
"I'm good! Just getting ready watch basketball game. Hope your headache goes away!"
"Haha, thanks! Never get them,
super weird Who's playing?"
"Hmm, try taking some REDACTED TERM! I'm going see the Warriors haven't i Honestly probably wouldn't 't
Grizzlies. Ever play?" ahle identify which sport they played"
"Haha, all the animal names confuse as well! Who came up with these naming conventions? I'd name my team something more original: The New York Chairs!, sounds beller to me :) actually the REDACTED TERM opening remarks"
"Oh cool! What is 'NeurIPS'?"
"Neural Information Processing Systems"
"Oh wow! Sounds fancy! What is about?" "It's s the higgest MI. conference in world"
"Oh, ML?"
Response: "Yes, machine learning"
Normalized average human annotations:
Prompt
GPT-335
CharGPT
6
Naive
0,8
0.0
0.8
0.0
Naive R
0,8
0.0
0.9
0.1
HD
0,0
0.8
1.0
0.2
HD+R
0.0
0.8
1.0
0.2
MEEP
0.4
0.4
0.8
0.0
MEEP+R
0.7
0.1
0.7
0.1
MEEP+SA
0.4
0.4
0.7
0.1
MEEP+SA+R
0.4
0.4
0.7
0.1
Table 15: Comparison results for one example dialogue from the FED dataset for qualitative analysis. The distance from the ground truth (8) for each score is listed to the right of the score.
Dialogue: "Hi!"
"Hi, how's 's it going?"
"All good! It's cold"
Response: "It's pretty here too."
Normalized average human annotations: 0.7
Prompt
GPT-3.5
5
ChatGPT
6
Naive Naive + R
0.8
0.1
0.7
0.0
0.8
0.1
0.8
0.1
HD
0,5
0.2
0.5
0.2
HD+R MEEP
0,5
0,2
0.5
0.2
0.8
0.1
0.7
0.0
MEEP+R
0.8
0.1
0.7
0.0
MEEP+SA
0.7
0.0
0.5
0.2
MEEP+SA+R
0.7
0.0
0.5
0.2
Table 16: Comparison of results for example dialogue from the FED dataset for qualitative analysis. The distance from the ground truth (8) for each score is listed to the right of the score.

---

E
Carbon Emissions
Researchers are actively considering environmental implications and making efforts to address and re- duce the effects associated with the deployment of large-scale NLP models. CodeCarbon.io io is dedi- cated emission tracker library designed to quantify carbon emissions accurately. NLP techniques al- ways vary accuracy and generalizability depend- ing upon hardware variations. We addressed this by accounting for our hardware specification and recorded reliable emissions estimations and pro- grammatic energy usage readings from CodeCar- bon. The total energy consumed (E) is determined using the following formula:
h) 1.103 codecarbon kWh
With our CO2 emission results, we converted our emissions the time submission to human- understandable emission parameters like "miles driven by an average gasoline-powered passen- ger vehicle" using EPA Greenhouse Gas Equiva- lencies Calculator and found that our total emission for our core research phase was about 0.16 miles "driven by an average gasoline-powered passen- ger vehicle". This does not include energy usage by OpenAl to service our API calls. It is also note- worthy that our GPU energy consumption is 0.854 kWh which is comparable to 0.0001 "barrels of oil consumed'. whereas full masked language model training cost 1200 times higher. We also analyze that energy usage and efficiency are es- sentially a function of running time, assuming the same hardware.
Carbon Emission
Parameter
Data Recorded
Duration
18 Hr
Emissions
1.44E-01
Rmissions Rate
2.8E-03
GPU Power
83E+02
GPU Energy
8.54E-01
Energy Consumed
1.04E+00
Table 17: Carbon Emission data for this project. The recorded data inclusive of failed and successful test cases during this project's s core phase.


Reviews:
[{"review_id": "zS718zET06", "reviewer": "Reviewer_ppYc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### TLDR:\nDiscussed each feedback from the reviewer with facts. Provided importance of the Engagingness in dialogue evaluation and its sub-dimensions, and reported performance of new results. \n\n&nbsp;\n### Our Response:\n\nThank you for your detailed feedback.\n\n+ “Authors use Microsoft Azure & Tencent services for translation on existing English datasets”: \n    +  **Our response :** \n    We’d like to clarify that we did not generate these translations. These translations were provided by the original authors of the datasets. We used the existing publicly available datasets without any modifications. \n&nbsp;\n\n&nbsp;\n+ “MEEP achieves a better Spearman and Pearson coefficient of about 0.05 than baselines”: \n    +  **Our response :** \n    The average improvement in the submitted version is 0.072 above baselines and our updated results yield improvement of 0.090 above baselines (referred in Table 4). We have calculated the difference between the best of our method (highlighted score) and the best of baseline results, and averaged these over the FED and SEE datasets.\n\n&nbsp;\n+ “Limited scope to engagingness”: \n    +  **Our response :** \n    Recent research is focused on making “engaging conversation” with dialogue agents with deeper, open-ended interactions, and sustainable communication to give users a human-like experience. Engagingness is more complex than other dimensions such as fluency or understandability. Existing multi-dimensional metrics have had limited success measuring engagingness and a focused metric such as ours can be a way to develop this capability. Both UniEval and GPTScore, which we use as baseline metrics, are multidimensional metrics (including engagingness as one of the dimensions) that consistently showed lower performance   when compared to our proposed method.\n\n&nbsp;\n+  “The evaluation is limited to ChatGPT & GPT-3.5, which may limit the generalisation of this method. Evaluation on open-source models like Pythia, Llama would beneficial to make the claims more general”: \n      +  **Our response :** \n    We agree that evaluation of open-source models is beneficial, however, unfortunately, at the time of our research implementation and experiments, access to large-scale open-source models that we wanted to explore (such as Llama) was rather limited. We are currently evaluating newer open-source models and will be able to integrate these results into the revised version.\n\n&nbsp;\n+ “Sub-dimensions of engagingness is subjective, would need more explanation.”: \n     +  **Our response :**    \n We can certainly clarify this in the paper. It is worth noting that definitions of engagingness are often subjective, a factor that's commonly overlooked in research. To address this, we performed statistical analysis, as illustrated in Figure 2, focusing on the most prevalent sub-dimensions of dialogue evaluation metrics alongside engagingness. Notably, we opted for \"interestingness\" as an additional sub-dimension due to its strong correlation with engagingness, a choice substantiated by inputs from linguistics, dictionary definitions, and relevant semantic sources cited in our paper. The difficulty of defining engagingness as a quality is in part why it is such a valuable metric, as other less nuanced qualities are easier to train for and evaluate and have accordingly made up a significant portion of prior progress. \n\n&nbsp;\n+ “Translated datasets and code used for this evaluation technique has not been made public.”\n    +  **Our response :**   Both datasets (FED and SEE) and their translations are publicly available (DSTC11, 2023). Our code, including identification of the subset of examples used for testing from the Persona-See datasets, will be made available upon acceptance of our paper.\n \n&nbsp;\n___\nDSTC11. Dstc11: Dialogue system technology challenge 11. https://chateval.org/dstc11. [Accessed 06-Jun-2023].", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "# Strengths\n- MEEP is essentially prompting technique, which achieves better Spearman and Pearson coefficient of about 0.05 than baselines. It can be interpreted as statistically significant correlation. \n-  The method can also be used in conjunction to other prompting techniques like Such As and Role Assignment (SA, R) achieving better performance.\n- Various baselines like GPTSCORE, G-EVAL  have used for comparison are meaningful and widely used by community.", "reasons_to_reject": "# Weakness\n- **Limited scope** of paper to specific 'Engagingness' evaluation.\n- The evaluation is limited to **ChatGPT & GPT-3.5**, which may limit the generalisation of this method. Evaluation on open-source models like **Pythia, Llama** would beneficial to make the claims more general.\n- **Sub-dimensions** of engagingness is subjective, would need more explanation.\n- Translated datasets and code used for this evaluation technique has not been **made public**.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Firstly, Authors describe engagingness as important factor as it highly correlates to sense of humane behaviour during conversation. \nThey propose these sub-dimensions of engagingness:\n- Response Diversity\n- Interactional Quality\n- Sentiment\n- Interestingness\n- Othering\n- Contextual Specificity\n\nAuthors believe, engagingness is really important metric for conversational AI. Their Auto LLM evaluation would be significant for the community.\n\nMEEP(Metric for Engagingness Evaluation using Prompting) is prompting technique to measure engagingness in dialogues across multiple languages. \n\nTo test their effectiveness of technique, They run it in multilingual settings. For getting multilingual datasets, Authors use Microsoft Azure & Tencent services for translation on existing English datasets. Authors use COMET and Cosine similarity to measure translation quality.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LFTWMtQ6OY", "reviewer": "Reviewer_amui", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### TLDR: \n\nAddressed more theoretical aspects of MEEP, explained results, improved translational quality and re-ran the experiment, also included dialogue level results.\n\n&nbsp;\n### Our Responses:\n\nThank you for your helpful feedback.\n\n+ “There might be not enough theoretical discussion and in-depth analyses which help readers understand the prompt design. More motivations and insights are needed.”\n     +  **Our response :** \n    Thank you for this comment. We have cited our sources for the inspiration for each prompt style in section 4. The black box nature of these LLMs limits us in our ability to perform an in-depth analysis of why each prompt style works. Prompt engineering remains an active area of research, and we hope that our findings will pave the way for more robust, reliable, and flexible methods of evaluation for conversations not only in English but also other languages.\n\n&nbsp;\n+ “Considering that this work is all about evaluation, there might be a lack of experiments currently. It might be beneficial to conduct more evaluation experiments categorized by language types and dialog content”: \n    + **Our response :** \n        While more evaluation is certainly always beneficial, what is currently possible is somewhat limited due to the number of suitable datasets that are available with annotations of engagingness in dialogue which are necessary for this study. Our research focuses on automatic evaluation of engagingness in dialogue which has received little attention. Our extensive evaluation considers 3 recent LLMs, 4 state-of-the-art baseline models, 12 distinct prompts, 6 benchmark datasets, and 3 languages. \n\n&nbsp;\n+ “Although the style design is clean, the prompts are not well-organized (Table 6, 7). All sentences squeeze together. ”: \n    + **Our response :** \n    This can be easily fixed for the camera-ready version. Regarding the sentences squeezing together, the prompts are written in the paper exactly as they are input into the models. This is in paragraph form in some cases. We welcome more discussion about this.\n\n&nbsp;\n+ “The Chinese translation of the proposed prompt (Table 7) is bad.”: \n    + **Our response :** \n    \n    Thank you for this note. We have obtained a new translation from another expert and rerun relevant models to obtain updated results using the new translated prompt. In general, the results are slightly better using the new translation.\n    + The new translation is:  \n\n         根据相应的对话背景，对以下回应在0到100的连续刻度上进行评分，其中0分表示“不吸引人”，100分表示“非常吸引人”。假设回应紧随对话背景之后。请考虑回应的吸引力是由以下特质定义的：根据背景的回应多样性（例如对“你好，你怎么样？”的回应是“我感觉很棒，因为我刚成功地为我的博士学位进行了答辩！你怎么样？”而不是“好的，你怎么样？”）、鼓励其他参与者回应的可能性（例如：“我喜欢乐高！我喜欢用它们制作有趣的东西。你喜欢乐高吗？”而不是“我喜欢乐高。”）、鼓励其他参与者提供高质量回应的可能性、有趣性、具体性和为其他参与者创造归属感的可能性.  \n        对话背景：<dialogue>  \n        回应：<response>  \n        评分： \n  \n    + The updated results are in the last four rows of Table 4, in the columns FED-ZH, and SEE-ZH:\n        + We see that the results for FED-ZH are 0.072 higher and results for SEE-ZH are 0.020 higher on average (omitting comparing statistically insignificant results) than those obtained with the previous translated prompt. For both datasets we see a new highest correlation. This indicates that the translated prompts provide better correlation than English-only prompts.\n\n            | Model | Prompt | FED-EN | FED-ES | FED-ZH |SEE-EN | SEE-ES | SEE-ZH|\n            | ------ | ------ |------ | ------ |------ | ------ |------ | ------ |\n            | ENDEX | -- |   0.290  | -- |   -- | 0.164 |  -- | -- |\n            | UNIEVAL |-- | 0.190 | 0.258| 0.076* | 0.015* | 0.073* | 0.038*\n            | GPTSCORE| GPTSCORE|0.176 | 0.146 | 0.230 | 0.087* | 0.153 | 0.140\n            | G-EVAL |  G-EVAL |0.488 | 0.448 | 0.402| 0.194 | 0.131 | 0.062*\n            | ------  ------ |------ | ------ |------ | ------ |------ | ------ |\n            | GPT3.5 | MEEP+SA | 0.532 | 0.481 | 0.451 |**0.236** | 0.223 |0.189\n            | ChatGPT |MEEP+SA | 0.558 | 0.516| 0.431 | 0.169 | 0.138| 0.133\n            | ChatGPT | MEEP+SA+R  |**0.568**|**0.542**| 0.435| 0.160 | 0.150 |0.140\n            | ChatGPT-0613 | MEEP+SA+R | 0.550 |0.471| 0.400| 0.214| 0.200 | 0.175\n            | ------  ------ |------ | ------ |------ | ------ |------ | ------ |\n            | GPT3.5 |MEEP+SA †|--|0.438  |**0.520**| --| 0.128 | 0.085* \n            | ChatGPT | MEEP+SA †| --| 0.500 | 0.408| -- | 0.161 | 0.149\n            | ChatGPT |  MEEP+SA+R †| --|0.525 | 0.444 | -- |0.123 | 0.168\n            | ChatGPT-0613 |  MEEP+SA+R †| --|**0.542**| 0.374| -- |**0.273**|**0.227**  \nCaption: Correlation results using Spearman coefficient for multilingual experiments. All results are statistically\nsignificant except those labeled with *. ‘†’ denotes a human-translated version of the prompt into the language of\nthe dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes a thorough definition of engagingness with six dimensions based on previous studies, which gives the evaluation approach certain linguistic interpretability.\n\nThe ablation study disentangles the effects of different parts in prompt design (MEEP, HD, SR). The overall prompt style is clean.\n\nThe SCC and PCC results are impressive.This paper proposes a thorough definition of engagingness with six dimensions based on previous studies, which gives the evaluation approach certain linguistic interpretability.\n\nThe ablation study disentangles the effects of different parts in prompt design (MEEP, HD, SR). The overall prompt style is clean.\n\nThe SCC and PCC results are impressive.", "reasons_to_reject": "There might be not enough theoretical discussion and in-depth analyses which help readers understand the prompt design. More motivations and insights are needed.\n\nThe engineering part might still need refinement.\n\n* Considering that this work is all about evaluation, there might be a lack of experiments currently. It might be beneficial to conduct more evaluation experiments categorized by language types and dialog content.\n* Although the style design is clean, the prompts are not well-organized (Table 6, 7). All sentences squeeze together.\n* The Chinese translation of the proposed prompt (Table 7) is bad.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper offers a prompt framework for automatic dialogue engagingness evaluation. By identifying six dialogue engagement dimensions in prompt designing, the proposed method makes use of LLMs' multilingual dialogue understanding ability, and obtains higher correlation coefficients than previous approaches.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "m3rBsDH8BO", "reviewer": "Reviewer_F5G1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### TLDR: \n\nIncluded more experiments in dialogue-level, reported all prompt styles used, explained novelty of our work and reported new results.\n\n&nbsp;\n### Our Responses:\n\nThank you for your insightful comments.\n\n+ “This work is limited to turn-level evaluation of engagingness, and there is no more valuable dialogue-level evaluation.”: \n    + **Our response :**  \nThere is limited availability of datasets with annotations for engagingness at the dialogue-level for languages other than English. Nonetheless, the flexibility of our proposed method allows easy generalization to dialogue-level datasets. We have run new experiments and report the results below. The datasets have human annotations of engagingness for Chinese-language dialogues  (DSTC11, 2023). Each dataset has also been translated into English by the dataset authors. KDCONV-EN and KDCONV-ZH have 22 human-annotated dialogues. LCCC-EN and LCCC-ZH each have 295 human-annotated dialogues. The datasets are publicly available.  Our proposed method outperforms the baseline model (G-Eval) on 3 out of 4 datasets. We can easily integrate these results in the revised version.\n\n| Model | Prompt | KDCONV-EN | KDCONV-ZH | LCCC-EN | LCCC-ZH|\n| ------ | ------ |------ | ------ |------ | ------ |\n| G-Eval |--|**0.327***|0.189  |0.149| 0.248|\n| GPT3.5 |MEEP+SA|0.286*|0.223*|**0.282**|**0.287**\n| ChatGPT |MEEP+SA |0.178*|0.392*|0.073*|0.191\n| ChatGPT |MEEP+SA+R|0.185*|**0.428**|-0.004*|0.193\n| ChatGPT-0613 |MEEP+SA+R|0.186*|0.362*|0.032*|0.117 \n\nCaption: Correlation results using Spearman coefficient for dialogue-level experiments. * denotes results that are not statistically significant.\n\n&nbsp;\n+ “This paper is about multilingual evaluation, but in fact, only English translation to Spanish or Chinese evaluation is conducted, so the results and conclusions are not necessarily reliable”: \n\n    +   **Our response :** To our knowledge, datasets with annotations for engagingness are only available in these languages with translations. Use of translated datasets is standard in this setting and has been done in contemporary studies (Mendonca et al., 2023; Platek et al., 2023; DSTC11, 2023).\n\n\n&nbsp;\n+ “The styles of prompt are relatively single, and in fact, it can be attempted to evaluate more styles.”: \n\n    + **Our response :** \n        We have evaluated many styles and included the 12 that are most effective. The evaluated styles presented in the paper include  \n        + a Naive prompt,  \n        + a Naive prompt with system role assignment,   \n        + a Human-Directed prompt,  \n        + a Human-Directed prompt with system role assignment, \n        + a prompt including our MEEP dimensions, \n        + a prompt including our MEEP dimensions and system role assignment, \n        + a prompt including our MEEP dimensions and some examples using ‘such as’,\n        + a prompt including our MEEP dimensions and some examples using ‘such as’, all translated into Chinese for use on Chinese dialogues,\n        + a prompt including our MEEP dimensions and some examples using ‘such as’, all translated into Spanish for use on Spanish dialogues,\n        + a prompt including our MEEP dimensions and some examples using ‘such as’ with system role assignment,\n        + a prompt including our MEEP dimensions and some examples using ‘such as’ with system role assignment translated into Chinese for use on Chinese dialogues, and\n        + a prompt including our MEEP dimensions and some examples using ‘such as’ with system role assignment translated into Spanish for use on Spanish dialogues.\n        \n    The selection of this set of prompts was careful to include a range of styles commonly used from the simple and straight-forward Naive prompt, to the often-used adaptation of Human-Directed prompts (Gao et al. 2023; Liu et al. 2023), system role assignment, and the previously successful use of ‘such as’ as seen in other work (Yuan et al. 2021 (BARTScore)). It is important to evaluate these untested methods to provide evidence to confirm or deny the validity of their use. We contribute MEEP – our careful multidimensional definition of engagingness – as a successful new style for evaluation of engagingness in dialogue. We additionally evaluate the effectiveness of prompting in the language of the dialogues and find that it improves the correlation of automatic evaluation with human judgment.\n\n\n\n&nbsp;\n+ “Table 4 and Table 9 lack some results, but the authors did not provide convincing reasons.”\n     + **Our response :** \nThank you for this comment. The reasons are:\n    \t+ A) Rows 1-2, Column **Prompt**: \n    \tThere are no input prompts for EnDex and UniEval because they either have their own prompts or use a different method for evaluation.\n\t    + B) Row 1, Columns **FED-ES**, **FED-ZH**, **SEE-ES**, **SEE-ZH**: \n\t    EnDex isn’t able to process dialogues in other languages, as noted in Section 5.3.\n    \t+ C) Rows 9-12, Columns **FED-EN**, **SEE-EN**: \n    \tThe prompts in these rows are translated into the language of the dialogue/response pairs (as noted in the table caption). For example, a Chinese-language prompt is used in the evaluation of Chinese-language dialogue/response pairs. English dialogue/response pairs don’t have corresponding translated prompts because the prompts are already in English. Results for this combination are shown in rows 5-8, columns FED-EN, and SEE-EN. Listing them in the lower section would be redundant.\n\n&nbsp;\n+ “The novelty of the method for measuring engagingness in dialogue is relatively limited, and is basically prompt engineering.” \n     + **Our response :** \nOur research contributes novel methods of evaluation (an open problem in NLG) in a challenging problem (engagingness in dialogue) while also evaluating the emergent abilities of LLMs (Wei et al., 2022). Given a well-crafted prompt, we investigate whether these LLMs can be used to generate automatic evaluations of dialogue response engagingness.\nWe address a unique challenge in the evaluation of engagingness – a particularly complex quality of dialogue – and present a solution that draws from interdisciplinary perspectives. Our carefully developed dimensions of engagingness produce results that surpass state of the art metrics as well as ‘basic/simpler’ prompts. Our research further develops limited prior work on comparing prompt-based methods of evaluation to other baselines.\n\n&nbsp;\n___\nDSTC11. Dstc11: Dialogue system technology challenge 11. https://chateval.org/dstc11. [Accessed 06-Jun-2023].\n\nMingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023. Human-like summarization evaluation with chatgpt.\n\nYang Liu, Dan Iter, Yichong Xu, Shuohang Wang Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using GPT-4 with better human alignment.\n\nJohn Mendonça, Patrícia Pereira, Helena Moniz, Joao Paulo Carvalho, Alon Lavie,  Isabel M. Trancoso. Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluation (Track 4). Dialogue System Technology Challenge 11. 2023 (accepted).\n\nOndřej Plátek, Ondrej Dusek, Patricia Schmidtova,  Vojtech Hudecek, Matheusz Lango. Three Ways of Using Large Language Models to Evaluate Chat. Dialogue System Technology Challege 11. 2023 (accepted).\n\nJason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, William Fedus. Emergent Abilities of Large Language Models. Transactions on Machine Learning Research. 2022\n\nWeizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. In Advances in Neural Information Processing Systems, volume 34, pages 27263–27277. Curran Associates, Inc.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes large language models with prompt to evaluate the engagingness of dialogue in multilingual setting.  Some discussions may have insight for future research.", "reasons_to_reject": "1. This work is limited to turn-level evaluation of engagingness, and there is no more valuable dialogue-level evaluation.\n2. This paper is about multilingual evaluation, but in fact, only English translation to Spanish or Chinese evaluation is conducted, so the results and conclusions are not necessarily reliable.  In addition, the styles of prompt are relatively single, and in fact, it can be attempted to evaluate more styles.\n3. Table 4 and Table 9 lack some results, but the authors did not provide convincing reasons.\n4. The novelty of the method for measuring engagingness in dialogue is relatively limited, and is basically prompt engineering.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes large language models with prompt to evaluate the engagingness of dialogue in multilingual setting.\n", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the paper presents promising results in terms of evaluation of engagingness, addressing concerns related to theoretical depth, prompt refinement, diverse experiments with additional models, translation quality in Chinese,  and novelty would strengthen its overall contribution to the field.
Decision: Accept-Findings