xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark
Authors: Chen Zhang
Luis Fernando D'Haro
chengguang tang
Ke Shi
Guohua Tang
Haizhou Li
Venue: EMNLP 2023 Findings (2023)
Keywords: Automatic Dialogue Evaluation
Multilingual Dialogue
Large Language Models

Abstract: Recent advancements in reference-free learned metrics for open-domain dialogue evaluation have been driven by the progress in pre-trained language models and the availability of dialogue data with high-quality human annotations. However, current studies predominantly concentrate on English dialogues, and the generalization of these metrics to other languages has not been fully examined. This is largely due to the absence of a multilingual dialogue evaluation benchmark. To address the issue, we introduce xDial-Eval, built on top of open-source English dialogue evaluation datasets. xDial-Eval includes 12 turn-level and 6 dialogue-level English datasets, comprising 14930 annotated turns and 8691 annotated dialogues respectively. The English dialogue data are extended to nine other languages with commercial machine translation systems. On xDial-Eval, we conduct comprehensive analyses of previous BERT-based metrics and the recently-emerged large language models. Lastly, we establish strong self-supervised and multilingual baselines. In terms of average Pearson correlations over all datasets and languages, the best baseline outperforms OpenAI's ChatGPT by absolute improvements of 6.5% and 4.6% at the turn and dialogue levels respectively, albeit with much fewer parameters. The data and code are publicly available at https://github.com/e0397123/xDial-Eval.

xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark
Chen Zhang"
Luis Fernando D'Haro"
Chengguang Tang'
Ke Shi" Guohua Tang"
Haizhou Li
National University of Singapore, Singapore
Universidad Politécnica de Madrid, Spain "Tencent AI Lab, China
Shenzhen Research Institute of Big Data, School of Data Science, The Chinese University of Hong Kong, Shenzhen, China chen_zhang@u. nus. edu
Abstract
There are two distinct paradigms of automatic evaluation, reference- based, and reference-free. Reference- metrics, such as BLEU (Papineni et al., 2002) and Embedding Average (Mitchell and Lapata, 2008) are widely adopted in the research community due to their easy implementation and general applicability to various types of dialogues. Yet, they can be misleading due to the poor correla- tion with human judgment (Liu et al., 2016; Mehri et al., 2022a). On the other hand, reference-free metrics bypass the reliance on references and di- rectly estimate the quality of a single response (turn level) or a multi-turn dialogue (dialogue level).
Recent advancements in reference-free learned metrics for open-domain dialogue evaluation have been driven by the progress in pre-trained language models and the availability of dia- logue data with high-quality human annota- tions. However, current studies predominantly concentrate on English dialogues, and the gen- eralization of these metrics to other languages has not been fully examined. This is largely due to the absence of a multilingual dialogue evalu- ation benchmark. To address the issue, we in- troduce xDial-Eval built on top of open-source English dialogue evaluation datasets. xDial- Eval includes 12 turn-level and 6 dialogue- level English datasets, comprising 14930 an- notated turns and 8691 annotated dialogues respectively. The English dialogue data extended to nine other languages with commer- cial machine translation systems. On xDial- Eval, we conduct comprehensive analyses of previous BERT-base metrics and the recently- emerged large language models. Lastly, establish strong self-supervised and multilin- gual baselines. In terms of average Pearson correlations over all datasets and languages, the best baseline outperforms OpenAl's Chat- GPT by absolute improvements of 6.5% and 4.6% at the turn and dialogue levels respec- tively, albeit with much fewer parameters. The data and code are publicly available at https: //gi thub. com/ htt///33991/////// /3/xaaallx-ilem -Dval-Eval.
Currently, there's growing interest in creating model-based, reference-free metrics. One line of work focuses on learning discriminative metric with self-supervised learning model is trained to distinguish high-quality responses/dialogues from low-quality responses/cialogues based on weak supervision signals that are automatically con- structed from human-human dialogue data (Yeh et al., 2021). These metrics benefit from the BERT- based language models (Devlin et al., 2019; Liu et al., 2019) and the availability of high qugh-tqulity dialogue corpora (Li et al., 2017; Zhang et al., 2018). With the recent advancement of large lan- guage models (LLMs) (Brown et al., 2020; Tou- vron et al., 2023a; Ouyang and et al., 2022: Ope- nAI, 2023), an emerging line of work is to treat the LLMs as annotators, which judge the quality of re- sponses/dialogues through prompting (Gupta et al., 2022; Huynh et al., 2023; Fu et al., 2023). Differ- ent from the BERT-based metrics, such metrics are generative in nature.
1 Introduction
Open-domain dialogue evaluation is a long-lasting challenge to dialogue system research (Mehri et al., 2022a). Currently, human evaluation is the most reliable way holistically judge the quality of the dialogue. Due to the high costs and low repro- ducibility of human evaluation, automatic metrics are proposed to complement it.
A common attribute of both metric categories is that they are not trained on dialogue evaluation data with human quality annotations, yet they ex- hibit significant potential in simulating how hu- mans perform evaluations at the turn or dialogue level. Despite the significant progress in the field, current research predominantly focuses on the En-
'll methods examined in the paper are not trained on data with human quality annotations. XDial-Fval is purely used for testing purposes.

---

glish language, other languages receive insufficient attention. To expand the scope of automatic dia- logue evaluation research beyond English and thor- oughly investigate the language generalization po- tential of model-based reference-free metrics, we propose a large-scale multilingual open-domain di- alogue evaluation benchmark, named xDial-Eval Eval. To construct XDial-Eval, we curate 12 turn-leved and 6 dialogue-level open-source English evalua- tion datasets. In total, the turn-level and dialogue- level datasets comprise 14930 -amant-d turns and 8691 human-annotated multi-turn dia- logues respectively. Then, we translate the English data into nine different languages with commercial machine translation models.
reference-free metries for automatic dialogue eval- uation (Ych et al., 2021).
BERT-Based Discriminative Metries - A series of works (Sai et al., 2020; Huang et al., 2020; Sinha et al., 2020; Lan al., 2020; Mehri and Eskenazi, 2020b; Phy et al., 2020; Pang al., 2020; Zhang et al., 2021c) targets turn-level evaluation and lever- ages self-supervised learning. They rely on nega- tive sampling strategies, such as random utterance replacement and word order shufiling, to generate synthetic data for training discriminative models. Another group of metrics is learned to holistically judge the quality of multi-turn dialogues (Mesgar et al., 2020; Zhang et al., 2021a; Ghazarian et al., 2022b; Zhang et al., 2022a) with similar idea that a model is trained to distinguish original human- human dialogues from negative samples generated by strategies, such as utterance order shuffling.
In addition, we comprehensively assess the per- formance of the two metric categories on xDial- Eval. Pertaining to the discriminative category, pre- vious state-of-the-art (SoTA) BERT-based methods are analyzed, while for the generative category. we evaluate the multilingual dialogue evaluation capa- bility of recent LLMs, especially the instruction- tuned variants, such ChatGPT", Alpaca (Taori al., 2023), and Vicuna (Chiang et al., 2023). In our analysis, we systematically explore the effects of training data, training strategies, and pretrained models on multilingual dialogue evaluation.
All the metrics rely on either BERT (Devlin al., 2019) RoBERTa (Liu et al., 2019) for dialogue context understanding and modeling the utterance- level interactions. Although they exhibit good cor- relations with human evaluation on English bench- mark datasets, their transferability across different languages has not been explored. In this study, we select representative discriminative metrics and conduct correlation analyses the multilingual XDial-Eval benchmark.
Lastly, we introduce strong multilingual self- supervised baselines on XDial-Eval. Specifically, we fine-tune the LLMs on synthetic instruction data built from human -huann dialogues. The fine- tuned models are found to exhibit much stronger multilingual dialogue evaluation capability than the original LLMs. Motivated by the complemen- tary nature of generative and discriminative metrics, we perform metric ensemble, which yields strong correlations with human evaluation and language generalization capability on xDial-Eval, even out- performing the powerful ChatGPT, which has been recently proven to be a superior reference-free text evaluator (Chen et al., 2023a; Liu et al., 2023).
A.M-d Generative Metrics The evolution of large language models (LLM) has drastically changed the NLP landscape (Brown et al., 2020; Chowdhery et al., 2022; Muennighoff et al., 2023; Touvron et al., 2023a). Especially, the line of works on instruction-tuning of LLMs (Ouyang and et al., 2022; Bai et al., 2022; Chung et al., 2022; Ope- nAI, 2023; Taori et al., 2023; Chiang et al., 2023; Chen et al., 2023b) has led to assistants that exhibit remarkable ability in under- standing and following users' instructions.
purpose Al
In the context of automatic dialogue evalua- tion, these LLMS can serve as unified evaluators of both the response and the multi-turn dialogue quality through prompting with task-specific tem- plates. For example, Gupta et al. (2022) intro- duces InstructDial, which consists of 48 diverse dia- logue tasks in a unified text-to-text format. Models tuned on the InstructDial dataset demonstrate good zero-shot performance in the dialogue evaluation task. Huynh et al. (2023) conduct a comprehensive analysis of the dialogue evaluation capability of LLMs with varying model types, sizes, choices of
2 Related Work
A long-lasting goal of automatic dialogue eval- uation to fully approximate human evalua- tion, which is quantified by strong correlations (0.8+) with ground-truth human quality annota- tions (Mehri et al., 2022a). Recent research is dedicated to developing powerful model-based

---

training data, etc. They observe that the smaller language models fine-tuned on instructions and dialogue-specific data can outperform very large language models. We move beyond both works by comprehensively exploring the LLMs' ability in evaluating multilingual dialogue.
3
The xDial-Eval Benchmark
3.1 Benchmark Details
The xDial-Eval benchmark is built top of open- source English evaluation datasets. The statisties of the datasets are outlined Table 1. The English datasets comprise 14930 annotated turns and 8691 annotated multi-turn dialogues. We utilize the Mi- crosoft Azure translation service" to translate all the datasets into nine diverse languages: Chinese (ZH), Spanish (ES), German (DE), French (FR), Japanese (JA), Korean (KO), Hindi (HT), Arabic (AR), and Russian (RU). In total, translation ef- fort comprised 183,495 unique utterances and cost approximately 400 USD. We keep the original dia- logue quality annotations from the English datasets for the newly translated dialogue data. Examples of the translated data are included in Appendix B.
More recently, several works (Chen et al., 2023a; Liu et al., 2023; Lin and Chen, 2023) study the dialogue evaluation capability of closed-source instruction-follooiin LLMs via prompting, such OpenAl's ChatGPT and Anthropic Claude (Bai et al., 2022). These closed-source LLMs exhibit strong zero-shot correlations with human evalua- tion. Yet, a major limitation is that it is not always feasible adapt such models custom data. Our work serves as the first study exploring the dialogue evaluation capability both the closed-source and the recent open-source instruction-tune LLMs on multilingual dialogue evaluation data. Addition- ally, we move beyond prompt engineering by per- forming task-specific finetuning of the open-source LLMs boost the dialogue evaluation ability and language generalization of such models.
Tum--Lavel Datzects
Alrotance AT Wnthlrreancc
#CtsHhpp Words
ADims
Persono USR (202103)
300
9.3
95.47 12.0
Phreora-Phan (2005)
900
5.1
48./ 11.5
24./11..
43229
DailDilog GRADE (2000)
100
30
25.0/ 10.8
Daily Diilllgllgg-aao (2000)
960
47
47.51 11.0
MialyDialg--Ce Gupta (2019)
500
4.9
49.9 10.9
120056)
360
22.4
Empeehaiic-RRAA (2000)
300
30
Multilingual Open-Domain Dialogue Exist- ing studies on multilingual open-domain dialogue mainly target dialogue response selection and dia- logue generation (Lin et al., 2021) with little em- phasis on automatic dialogue evaluation. For ex- ample. Sato et al. (2018) constructs multilingual dataset, which contains the Ubuntu IRC logs in 12 different languages for response selection. Zhang al. (2021d) proposes MRS, multilingual reply suggestion dataset with ten languages. More re- lated our work, Rodríguez- et (2023) recently held the shared task on "Robust and Mul- tilingual Automatic Evaluation Metrics for Open- Domain Dialogue Systems" at DSTC11². They release a series of multilingual evaluation datasets and a multilingual deep AM-FM (Zhang et al., 2021b) baseline. Yet, their released data is only limited 3 languages. Most of the data target turn- level evaluation with limited amount of dialogue- level annotated data. On the contrary, our proposed benchmark, Evaal-Eval, contains large-scale anno- tated data at both turn and dialogue levels, spanning 10 different languages.
FED-Tixx (20200) ConTeeF-Tom (3022)
375
1044
87.31 13.3
1066
33
21 67| 10:00
Dialogne-Lvd Dutacets
Mustance
PUtt//lssaace
WWodds/Utt
ADima
iead (2002)
1,920
6.0
12.4
Paraon So:
3,316
12.0
Relibble Eval
2,955
21.2
ConTtel (10226)
119
17.9
FED-Nia (2000s)
125
12.7
92
11
Itumnn Eval (2023)
286
12.0
11.6
Table I: Statistics of Finglish evaluation datasets in XDial-Eval. Ctx, Hyp, and Dim refer to context, hy- pothesis, and dimension respectively. Dimension means the annotated response /dialogue quality aspect.
3.2 Translation Quality Assurance
We verify the translated data quality with both au- tomatic and human evaluation. Due to the high costs of running a full evaluation, we randomly sample 100 utterances from each translated dataset, summing up to 1700 translated utterances per non- English language and total of 15300 translation pairs 3 Both the automatic and human evaluation re- sults suggest a high quality of the translated xDial- Eval data.
Automatic Measures Due to the absence of tar- get language references, we apply direct qual-
'https:// com/ coin-gbbazuree stw.tieeeeeeviiiettraslssse. 'ConTureeETTum & ConTurEi- Dira--iaal share the same data.
hhttps://dstcl/ .dst..dccoccccc.cc.......... www.enitootttaaaaaeeeee

---

i. ity estimation with three different models: Ope- nAI's GPT-4 model (OpenAI, 2023), Unbabel's wm122-comettiwii-d (Rei et al., 2022) and wmt23- cometkiwi-da-xl (Rei et al., 2023) models. In ad- dition to quality estimation, we perform additional back-translation of the translated content to En- glish using Azure MT and then conduct reference- based evaluation comparing the English source and back-translated' utterances.
language pairs, totaling 3150 instances. Quality check on a random subset of the human evaluation data conducted to ensure their reliability. Ad- ditionally, for the 350 English-to- to-Chinese transla- tions, three authors of the paper manually evaluated the translation. The inter-annotator agreement is 0.578, indicating medium agreement. The average human evaluation scores for EN-ZH, EN-ES, EN- DE, EN-FR, EN-JA, EN-KO, EN-HI, EN-AR, and EN-RU language pairs are 4.75, 4.68, 4.59, 4.53, 4.39, 4.32, 4. 4.17, 4.41, and 4.71 respectively. The human evaluation results agree with the automatic results, showcasing that the translation quality of XDial-Eval is good.
For quality estimation with GPT-4, we prompt the model to evaluate the adequacy of each trans- lation pair on a scale of to 5, with I de- notes poor adequacy and 5 denotes perfect ad- equacy. The instruction template is outlined in Appendix A. For the reference-based evalu- ation, we adopt sacreBLEU" (Papineni et al.. 2002), BERTSccoTe' (Zhang al., 2020), and BLEURR* (Sellam et al., 2020) to assess the 15300 English source and Pack-translation pairs. The au- tomatic evaluation results are summarized in Ta- ble 2. We can observe that the scores of the auto- matic metrics are generally high.
Model-Based Reference-Free Metries
Mathematical Formulation Let Ddd denote a di- alogue evaluation dataset in xDial- with index i and language l. D either comprises NN number of multi-turn dialogues (dialogue-level) J num- ber of context-response pairs (turn-level), Each data instance can be either a multi-ture dialogue d; a context-response pair (c) s.
1.) E D where E {I...., Each d' or r is evalu- ated by human judges using predefined Likert scale to measure specific aspects of dialogue or response quality. Given the multifaceted nature of quality and our particular interest in the language generalization of the automatic metrics, we are concentrating our analysis on "coherence" at the dialogue level and "context relevance" the turn level, which the most studied dimensions in the dialogue evaluation literature (Yeh et al., 2021). We will explore other dimensions in future works.
Lang
BERTTSoore
BLEUKE
GPI4
ComelKivt22
Cometklwizi
BN-211
17.85
0.691
0.371
4.58
0.817
0.739
id
EN-ES
$6.58
0.767
0.820
4.6
0.845
0.764
EN-DE
50.30
0.754
0.817
4.63
0.344
0,744
EN
01708
0.11
4.59
0.812
0,009
0.84
EN.X0
40.23
0.707
0.782
4.26
0.46
0,767
FX-H1
49.35
0.726
0.797
4.40
0.91
0
EN-AR
48.54
0.737
0.794
4.39
0.35
0,736
0.741
4.40
0.825
0.731
0.798
Table 2: Automatic Evaluation Results. Score ranges of BLEU, BERTScore, BLEURT, GPT-4, wmt22- cometkiwi-da (CometKiwi22), and wmi23-cometkiwi- da-xl (CometKiwi23) are [0, 100), [0, IJ, [0. 1]. 5], [0, 1]. and [0, 1] respectively.
We denote the mean quality score assigned by human annotators to a specific data instance as hj. i.1 The metric model, M, assigns metric score sj to dff or ryy i.l The performance of M on Ddd is as- sessed by computing the Pearson or Spearman cor- relations, px id between s = (s.,,..., 3d H (h].....,h .. The higher the p² the bet- ter the performance of M D M with strong evaluation capability achieves high [o] ΣiEn p where Ω is either the collection of turn- turn-level datasets or the collection of dialogue-level datasets in XDial-Eval. The language generalization of M can be assessed by [T] Σiez ( T Σien p²1) where L is the of languages covered by xDial-Eval.
Human Evaluation We also conduct the human evaluation to cross- cross- validate the translation quality. Specifically, we collaborate with a service provider recruit native speakers, who are proficient in both English and their native language, such as Chinese, Spanish, etc. The bilingual speakers are instructed to evaluate the quality of translations from English to their mother tongue. Each speaker assesses 350 translation pairs, rating them on a 1-5 scale, where I indicates poor translation and 5 indicates excellent translation. The human evaluation covered nine
and
'ht hhttps://huggiiggaaae co/ co/spaces/
http://hhh...cmmmoooeeeeeerrrr/a hlab//masterr/checpponttsst
BERT-Based Discrminative Metrics In this

---

#*# Instruction:
### Instruction:
Given the input dialogue between a human (HHJ) and a chatbot ([C)) below, predict whether the dialogue is coherent?
Given following dialogue context and re- sponse, predict whether the response is relevant to the context.
### Input:
### Input:
[H] My dog puked all over the carpet few
Dialogue Context:
weeks back.
hello, tell something about yourself'
[C] Oh no! Did you clean it up?
am 56 years old and i am housekeeper. do have any hobbies interests ?
[H]
[C] Did you clean it up?
[H] yes did
Dialogue Response:
[C] That's great! I'm glad it worked out for you!
like to read and write poetry.
Response:
### Response:
No
Yes
(a) Dialogue-Level Evaluation
(b) Turn-Level Evaluation
Table 3: Alpaca-7B Instruction Template. The bold text is the expected output and the rest are the input to the LLM.
study, we select two SoTA BERT-based metrics in the open-domain dialogue evaluation literature: PoE (Zhang et al., 2023) and FineD-Eval (Zhang et al.. 2022a). Both are self-supervised metrics designed for turn-level and dialogue-level eval- uation respectively. Their detailed descriptions are outlined in Appendix C.I. The original PoE and FineD-Eval are based RoBERTa-Large (354M) and RoBERTa-Base (125M) (Liu et al.. 2019) respectively. adapt them for the multi- lingual dialogue evaluation task, we reimplement them with XLM-R-Large (550M) and XLM-R- base (270M) (Conneau et al., 2020) respectively.
cant negative impact on model performance.
When scoring dy with FineD-Eval or r' with PoE, the flattened token sequence of di il (cj .T tively. The scalar normalized score sj in the range [0, 1J is output from the models accordingly.
1/ is input into FineD-Eval or PoE respec-
i.
ALseBased Generative Metries We select di- verse set of popular LLMs with different backbones and pretraining data. Due to the fast development pace of LLMs, the following list not exhaustive: LLaMA-7B (Touvron et al., 2023a), LLaMA-2- 7B (Touvron et al., 2023b), Baichuan-2 (Yang et al., 2023), Alpaca-7B (Taori et al., 2023), Vicuna- VI. (Chiang et al., 2023), BLOOM-7B (Scao et al., 2023), Phoenix-7B (Chen et al., 2023b), Palcon-7B (Almazrouei et al., 2023), and OpenAl's ChatGPT (gpt=3.5-turbo-0000)) Due computa- tion limitations, the larger LLMs variants are not explored in this study. The detailed descriptions of each model are included in the Appendix C.2.
To finetune the models, need multilingual training data. The original synthetic data for train- ing PoE comprises approximately 2M English context-response pairs while that of Fiae contains roughly 90K multi-turn dialogues. Both training datasets contain a balanced number of pos- itive and negative data instances We employ the machine translation model (Costa-jussà et al., 2022) convert the En- glish data into the other 9 languages. For easy ref- erence, we denote the multilingual training datasets as xPoE-Tum Turn and FFined-Dial' We didn't 't use the high-quality Azure MT service for training data translation because the costs are exorbitant and we can also check whether training on multilingual data with lower translation quality causes a signifi-
open-source NLLB-200-33
i.l To score d or r with the LLMs, first need to convert the input instance the corresponding model- and specifcc instruction-bant based prompt templates. Take Alpaca as an example, table 3 displays the specific prompts for both dialogue- level and turn-level evaluation tasks". Follow- ing Gupta et al. (2022), we frame the evaluation task binary classification problem. Given input prompt, we specifically focus on the proba- bilities related to the label words "Yes" and "No" as generated by the language model. Then, we
"For PoE, positive and negative refer to relevant and irrel- evant responses while for FineD-Eval, they refer coherent and incoherent dialogues.
10 "Given that xPoE-Turn and xFined-Dian sizes are 10 times their original English datascts, we sample subset equal to the original English size for model training.
AAppendix D includes more example instruction templates for other models and languages.

---

P(Yes))(P(Ys)) + P(No)) and P(Yes) serves 1! d sj of d r As we do not have access to the output probabilities of the closed-source Chat- GPT, we prompt ChatGPT to explicitly provide normalize the probability of "Yes" as P(Yes)
thetic multilingual dialogue data to determine if this strategy improves their language generalization. In particular, we examine two groups of LLMs. The first is vanilla LLMs without instruction tuning. in- cluding LLaMA-7B, BLOOM-7B, LLaMA-2-7B, and Baichuan-22 The second group includes the instruction-tuned variants: Alpaca-7B and Phoenix- 7B. By comparing these two groups after finetun- ing the multilingual synthetic data, we study whether two-stage instruction finetuning is useful, i.e., an LLM first finetumed on penrrol- instruction data and then further adapted to custom data. Additionally, we also want to find out which 7B open-source LLM possesses the strongest mul- tilingual generalization.
numerical score to d' or r s.d
on the scale of to 5. Proposed Metrics A drawback of LLMs is their lack of specialized focus. While these models are constructed to function as versatile Al assistants ca- pable of managing a variety of NLP tasks, their per- formance may not measure up to domain-specific experts when dealing with specialized tasks or par- ticular domains. Hence, we propose to further adapt the open-source LLMs to custom data de- signed for automatic dialogue evaluation.
Furthermore, we explore the ensemble of the finetuned LLMs and the BERT-based diserimina- tive metrics and examine whether their difference in training objectives helps complement cach other in their dialogue evaluation abilities. Since the finetuned LLMs and the BERT-based metrics pro- duce scores ranging from 0 to 1 (described in $4), achieve the ensemble by simply calculating the arithmetic mean of their respective output scores.
An additional note that current LLM finetun- ing often uses annotted data. However, due to challenges in scaling up high-quality, human- annotated training data collection for open-domain dialogue evaluation, and the proven success of SoTA BERT-based metrics using automatically- constructed synthetic dialogue data, propose investigate whether LLMs can also benefit from finetuning with synthetic data.
Specifically, we reuse the xPoE-Tunm and xFined- Dial datasets described in the previous section to perform iiatruction-tuning of the LLMs. To speed up the experiments, we sample subset of 100k (10k per language) from each dataset. transform the data into in- struction format using model-spe -seecffc prompt tem- plates, similar to those Table 3. Then, the LLMs are finetuned on the 200K multilingual instruction data using the Low-Rank Adaptation (LoRA) tech- nique (Hu et al., 2022). The dialoguc/response scoring process of the finetuned LLMs is the same binary classification formulation discussed above.
6 Results & Analysis
Table 4 and present the key experiment results 13
6.1 Zero-Shot Performance of LLMs
Subsequently,
Vanilla vs Instruction-Tuned LLMs Table 5's "LLMMs-Zerosoo" category shows that vanilla LLMs exhibit low correlations with human evalua- tions across all languages. In contrast, instruction- tuned LLMs outperform their vanilla counterparts, with better average Pearson correlations at both turn and dialogue levels. For example, Alpaca- 7B achieves average Pearson improvements of 0.218 and 0.336 over LLaMA-7B at turn and di- alogue levels respectively. Similar improvements are seen when comparing Vicuna-77 7B to LLaMA- 7B and Phoenix-7B to BLOOM-7B. These sig- nificant boosts in performance are credited to the closer alignment of instruction-tuned LLMs with humans' task-solving abilities.
5 Experiment Setup
We explore the effects on multilingual dialogue evaluation with both zero-shot prompting and fine- tuming the LLMs on different types of data For LLM finetuning, we first investigate the cross- lingual generalization of the LLMs when only fine- tuned on the English synthetic data. Models with different pretrained backbones are chosen for inves- tigation: PoE, Fined-Eval, Alpaca-7B, Phoenix-7B, LLaMA-2-7B, and Baichuan-2-15 2-BB.
Impact of Backbone Models Baichuan-2-77 and Falcon-77 7B perform much better than other vanilla
Full experiment results on each dataset can be found at https://docs.got google. www./ppeeadseeeeeeeeeeeeee Lit. edi edsppssasnnniing and Additional supporting analyses are included in Appedix E
Secondly, we finetune the LLMs using the syn-
12 The computation details are outlined in Appendix F.

---

Turn-Level
Models
EN
ZH
ES
DE
FR
KO
AR
RU
AVG
PoE
0.487
0.444
0.455
0.459
0.459
0.436
0.425
0.359
0.426
0.440
0.439
Alpaca-7B
0.499
0.347
0.410
0.422
0,421
0.262
0.221
0.163
0.199
0.352
0.330
Phocnix-7B
0.439
0.392
0.377
0.327
0.417
0.282
0.201
0.372
0.398
0.270
0.348
LLaMA-2-7B
0.509
0.390
0.458
0.414
0.445
0.313
0,311
0.242
0.246
0.381
0,371
Baichuan-2
0.556
0.513
0.469
0.468
0.472
0.385
0,349
0.256
0.294
0.416
0.418
Dialogue-Level
Finel)
0.376
0.349
0.354
0.351
0.356
0.363
0.347
0.320
0.310
0.344
0.347
Alpaca-7B
0.408
0.309
0.355
0.345
0.364
0.219
0.194
0.199
0.200
0.314
0,291
Phoenix-7B
0.334
0.342
0,248
0.238
0.297
0.246
0.179
0.277
0.310
0.202
0.267
LLaMA-2-7B
0.372
0.298
0.348
0.354
0.342
0.245
0.241
0.201
0.207
0.313
0.292
Baichuan-2-7B
0.359
0.328
0.289
0,319
0.322
0.268
0.269
0.228
0.256
0.289
0.293
Table 4: Language- wise average turn-level (over 12 datasets) and dialogue-leved (over datasets) Pearson correlations of models finetuned English data only. The corresponding Spearman results be found in Table 16.
LLMs terms of correlation at both turn and dia- logue levels, possibly due to its pretraining data's similarity with XDial-Eval benchmark data. Un- like LLaMA, pretrained on general web text like CommonCrawl and Wikipedia, or BLOOM, pre- trained 498 HuggingFace NLP datasets, Falcon- 7B uses blend of filtered web data and curated high-quality corpora like social media conversa- tions (Penedo et al., 2023). Baichuan-2-44 2--7B is pretrained on large-scale and diverse multilingual data, totaling 2.6 trillion tokens. Alpaca-7B and Vicuna-7B generally perform better in Latin and Cyrillic languages. However, Phoenix-7B exhibits consistent performance across languages. For instance, Alpaca-7B achieves 0.25 average Pearson correlations in English, Spanish, German, French, and Russian, but < 0.2 in other languages at the turn level. Phoenix-7B shows less perfor- mance variation. Similar trends are observed at the dialogue level. Differences in language generaliza- tion are largely due to their backbone models and instruction-tuninn data. BLOOM and Baichuan multilingual LLMs while LLaMA and Falcon mainly pretrained on English text. Additionally, Phoenix-7B is finetuned with multilingual instruc- tion data while the other instruction- -uriiionntteedddd models are mainly finetuned with English instruction data. In $6.3, we explore whether further finetuning the LLMs on our multilingual dialogue data leads to language generalization improvement.
eters and pretraining data. It also benefits from higher quality instruction data for finctuning than models like Alpaca-7B 7B and Phoenix-7B, and fur- ther gains from reinforcement learning from buman feedback (RLHF), enhancing its alignment with hu- man problem-solving skills.
6.2 Effects of Training on English Data Only
Table 4 shows that all models perform optimally on English evaluation datasets at both turn and di- alogue levels, as expected. We can observe that PoE and FineD-Eval perform consistently well across languages. Interestingly, their performance matches their respective multilingual finetuned vari- ants (Table 5), implying that XLM-R is strong cross-lingual encoder.
Finetuning the LLMs the English synthetic data not only brings improvements in English but also in other languages on turn-level datasets. The extent improvements of different models can differ significantly across various languages. For example, the Al Alpaca-77 (n model, as shown in Table 4 and in the more substantial performance improvement in Latin languages (>0. (=),14), compared to Hindi and Arabic (<0.04). On the other hand, Phoenix-7B has a more consistent performance boost across different languages.
section of Table 5,
At the dialogue level, the average correlation score of LLaMA-2-7B after finetuning on the En- glish synthetic data improves by over twice as much compared to when using zero-shot prompting (0.121 0.292). For Baichuan-2-7B. the absolute improvement is around 5% (0.241 0.293). How- ever, the improvement brought by finetuning is less prominent for the instruction-tume LLMs. For ex- ample, a slight improvement from 0.255 to 0.267
ChatGPT Performance Without finetuning,
ChatGPT consistently excels over other LLMs in all languages, demonstrating its outstanding mul- tilingual evaluation ability. This aligns with prior studies (Chen et al., 2023a). The superior perfor- mance is attributed to its instruction- cnatucttion-tunngg on a stronger foundation model, terms of both param-

---

Turn-Level
Category
Models
EN
ZH
ES
DE
FR
JA
KO
H1
AR
RU
AVG
BERT Based
PoEs
0.464
0.437
0.441
0.454
0.455
0.424
0.417
0.361
0.422
0.436
0.431
LLaMA-7B
0.038
0.025
0.094
0.028
0.037
0.071
0.015
-0,020
0.016
0.072
0.038
LLaMA-2-7B
0.065
0.084
0.029
0.101
0.108
0,066
0,073
0.010
0.064
BLOOM-7B
0.044
0.134
0.100
0.019
0,084
0.017
0.005
0.048
0,099
0.062
0.061
LLMs-Zerohhot
Falcon-7B
0.143
0.127
0.155
0.088
0,151
0.093
0.011
0.068
0.109
0.077
0.102
Baichuan-2-7B
0.175
0.134
0.118
0.133
0.117
0.102
0.139
0.092
0.119
0.129
0.126
Alpaca-78
0.337
0.197
0,269
0.269
0.156
0.131
0.131
0.250
0.218
Vicuna-7B
0.211
0,165
0,226
0.186
0,217
0.160
0.119
0.119
0.144
0.197
0.175
Phoenix-7B
0,298
0.249
0.281
0. 190
0.265
0. 166
0.112
0.214
0.224
0.174
0.217
CharGPT
0.471
0.433
0.467
0.462
0.459
0.415
0.365
0.346
0.398
0.423
0.424
LLaMA-7Bf
0.363
0.267
0.245
0.274
0,271
0.232
0.223
0.216
0.277
0.258
LLaMA-2-78f
0.565
0.484
0,510
0.506
0,523
0,436
0.416
0.355
0.378
0.478
0.465
LLMs-FT (ours)
BLOOM-7B1
0.273
0.197
0,320
0.199
0,300
0.197
0.013
0.214
0.175
0.123
0.201
Falcon-7B+
0,415
0.450
0.465
0.440
0,468
0.295
0.180
0.149
0.196
0,283
0.334
Baichumn-2-7Bf
0.541
0.505
0.515
0.501
0.453
0,444
0.388
0.412
0,480
0.475
Alpaca-7Bt
0,548
0.405
0.491
0.483
0,489
0.327
0.318
0.307
0.309
0,444
0.412
Pho
0.481
0.435
0.461
0.366
0.465
0.323
0,264
0.410
0.435
0.334
0.397
LLaMA-7B + PoEt
0.476
0.443
0.448
0.462
0.466
0.431
0.423
0,371
0.425
0,442
0.439
LLaMA-2-7B-
0.558
0.498
0.518
0.520
0.528
0,470
0.455
0,406
0.444
0.494
0.489
Fiseeblle (ours)
BLOOM-7B PoEt
0.485
0.444
0,461
0.460
0,474
0.425
0.418
0.376
0.431
0,440
0.441
Falcon-78 + PoEt
0.494
0.479
0.485
0.488
0.499
0.419
0.400
0.355
0.411
0.437
0.447
Raichuan-2-78. PoEt
0.544
0.500
0.508
0.504
0,514
0.464
0.455
0.416
0.447
0.484
0.484
Alpacn-7B+ PoFf
0.543
0.461
0.503
0.504
0.511
0.420
0,412
0.387
0.413
0.476
0.463
Phoemix:7B + PoFf
0.503
0.463
0.479
0.451
0.487
0.410
0,388
0.420
0.455
0.426
0.448
Dialogue-Leve
BERT-Based
FineDt
0.386
0.354
0.362
0.362
0.372
0.346
0.341
0.343
0.339
0.376
0.358
ILLMA 7B
0.190
0.190
0.226
0.196
0.151
0.141
0,120
0,027
0.035
0,151
0.143
LLaMA 2:7B
0.036
0.091
0.125
0.165
0,027
0.128
0.127
0.121
BLOOM 7B
0.071
0.212
0,063
0.063
0,122
0.104
0,058
0.097
0,122
0.078
0.099
LLMs Zeroshot
Falcon-7B
0,286
0.240
0,248
0.268
0,153
0.113
0.107
0.134
0.168
0.219
0.194
B-
0,296
0.316
0,270
0.258
0,274
0.211
0.198
0,156
0.201
0.235
0.241
Alpaca-78
0,441
0.321
0.386
0.404
0.301
0,268
0.208
0.270
0.356
0.336
Vicuma-
0.347
0.234
0,243
0.260
0,242
0.209
0.220
0.132
0.148
0,231
0.226
Phoenix: 7B
0,312
0.292
0,264
0.261
0.291
0.254
0,163
0.253
0.253
0,206
0.255
ChaiGPT
0.419
0.407
0.395
0.378
0.310
0.324
0.385
0,363
0.376
LBt
0.237
0.201
0.208
0,240
0.173
0.169
0.151
0,207
0.195
LLMss (ours)
LLaMA-2-7B
0.444
0.401
0.405
0.407
0,410
0.363
0.359
0.319
0.343
0.404
0.386
BLOOM-7B1
0,289
0.235
0,269
0.249
0,253
0.175
0.132
0,288
0.274
0.136
0.230
Falcom-BB*
0,376
0.366
0,314
0.334
0.320
0.231
0.146
0.142
0.197
0.174
0.260
Baichumn-277Bt
0,344
0.329
0.309
0.315
0.275
0.323
0.278
0.325
0.304
Alpaca-7Bt
0.420
0,362
0,383
0.394
0.379
0.309
0.263
0.255
0.278
0.351
0.339
Phocnix-7Bt
0.339
0.324
0,328
0.293
0.321
0,275
0.229
0.321
0.316
0.259
0,300
LLaMA-7B + Di
0.405
0.364
0.371
0.368
0.379
0.353
0.349
0,349
0.346
0,384
0.367
LLaMA-2-7B- FineD
0.477
0,436
0.399
0.394
0,380
0.438
Ensemble (ours)
BLOOM-7B- FineDt
0.405
0,373
0,384
0.374
0,387
0.348
0.341
0.374
0.370
0.373
0.373
Falcoe-7B + FineDr
0.445
0.413
0.397
0.403
0.407
0.356
0.345
0,341
0.346
0.377
0.383
Baichwan-2-7B- FineDt
0.402
0.379
0.366
0.371
0.374
0.339
0.369
0.333
0.369
0,364
0.367
Alpaca-7B- FineDt
0.461
0.407
0.425
0.434
0.427
0.369
0.347
0.342
0.357
0.410
0.398
Phoenix-7B + FineDt
0.403
0.373
0.377
0.356
0.379
0.340
0.317
0.368
0.363
0.338
0.361
Table 5: Language-wiss average turnllevel (over 12 datasets) and dialogue-level (over 6 datasets) Pearson correlations of different models. The Spearman results can found in Table 17. "LL.Ms-Zerosho"" means models applied directly without finetuning, whereas "L.LMs-FT" represents finetuned models. The best score for each language is highlighted in bold and models finetuned on synthetic dialogue data are accompanied with a t.
is observed for Phocnix-7B. The performance of Alpaca-7B even drops from 0.336 0.291. A possible explanation is that the instruction-tur LLMs already possess certain knowledge necessary for multi-turn coherence evaluation. Finctuning on the synthetic data doesn't 't bring much additional knowledge.
6.3 Effects of Training on Multilingual Data
BERT-Based Metrics- We can observe from Ta- ble 5 that both PoE and FineD-Eval are strong metrics for multilingual dialogue evaluation. PoE achieves an average Pearson score of 0.431 at the turn level, outperforming ChatGPT and all the finetuned LLMs, except for LLaMA-2-7B and Baichuan-2 2-7B. FineD-Eval achieves 0.358 at the

---

dialogue level. Its performance is only slightly worse than ChatGPT zero-shot prompting and the finetuned LLaMA-2-7B. Second, the performance both metrics is quite consistent across different languages. Hence, we can conclude that BERT- based discriminative metrics that work well on En- glish dialogues can also generalize to other lan- guages with strong multilingual encoders and mul- tilingual training data.
languages. Meanwhile, Phoenix-7B and Baichuan- 2-7B, with a multilingual pretrained backbone, dis- play more consistent performance in different lan- guages.
Metric Ensemble- - The ensemble of the LLMs and the BERT-based metrics yields strong multilingual dialogue evaluators at both turn and dialogue levels. The best combinations, LLaMA-2-7B PoE and LLaMA-2-7B + FineD-Eval outperform ChatGPT by 6.5% and 4.6% in terms of the average Pearson correlations the turn and dialogue levels respec- tively, albeit without RLHP and the size of their trainable parameters is less than 7.5B. Finally, we can observe that ensemble generally leads to bet- ter multilingual dialogue evaluation capability than individual BERT-based or -Lseed metrics.
Finetuned LLMs- Comparing "LLMs-FT" mod- els with their "LLMs- "ZrMssot counterparts shows that finetuning the LLMs with multilingual syn- thetic dialogue data significantly improves both di- alogue evaluation and language generalization. The observation confirms our claim in $1. For instance, LLaMA-7B zero-shot prompting, with an average Pearson correlation of 0.038, performs poorly in all languages. Finetuning LLaMA-7B on the mul- tilingual synthetic data boosts the performance to 0.258 with nearly 0.2 absolute improvement in most languages. At the dialogue level, LLaMA- 7B's average Pearson correlation increases from 0.143 to 0.195. Similar observations can be made w.r.t. other "LLMs-FT" models. Notably, fine- tuning LLaMA-2-7B leads to the most significant improvement both turn and dialogue levels, from 0.064 to 0.465 and 0.121 to 0.386 respectively.
7 Conclusion
This paper introduces EDial-Eval, a multilingual dialogue evaluation benchmark featuring 14930 an- notated turns and 8691 dialogues in 10 languages. Both automatic and human evaluation validate the high quality of -vaal-Eval. Additionally, we exam- ine the performance of BERT-based metrics and emerging LLMs on this benchmark. Key take- aways include: (1) SoTA BERT-based metrics, backed by strong cross-lingual encoders and mul- tilingual training data, can effectively handle mul- tilingual dialogue evaluation. (2) Recent general- purpose instruction-followe as unified multilingual dialogue evaluators. Future work could refine optimization techniques and use high-quality, task-specific data for fine-tuning the LLMs. (3) Ensembling BERT-based metries and LLMs outperforms ChatGPT in correlation with human judgment and achieves good language gen- eralization (4) Despite the ensemble approach's 's good correlations, multilingual automatic dialogue evaluation remains unsolved. XDial-Eval could serve as valuable benchmark for tracking future research progress in this field.
When comparing Alpaca-7B to LLaMA-7B or Phoenix-7B to BLOOM-7B (in the "LLMs-FT" category), we observe that the Alpaca-7B out- performs LLaMA-7B (or Phoenix-7B outperforms BLOOM-7B) by significant margins in all lan- guages at both turn and dialogue levels. The ob- servation suggests that a two-stage finetuning pro- cess, i.e., finetuning on general-purpose instruction data followed by finetuning on custom data, helps achieve stronger LLM- dialogue evaluators.
LLMs show promise
Additionally, we can observe that the improve- ments of instruction-based LLMs (Alpaca-7BB and Phoenix-7Bi 7B) at the dialogue level are less signifi- cant than at the turn level. Such finding is similar to what we have observed in $6.2. Future work may explore how to introduce high-quality data that car- ries richer information and benefits the coherence evaluation of multi-turn dialogues.
Limitations
Our investigations involving open-source LLMs are restricted to their 7B variants. Future research should explore whether improving the scale of these LLMs, such as using their respective 13B, 40B, or 65B variants, enhances their capability for zero-shot dialogue evaluation, and whether larger LLMs can better generalize to various languages after fine-tuning with multilingual dialogue data.
Lastly, while finetuning with multilingual data boosts LLMs performance in all languages, varia- tions still occur depending on their pretrained back- bone model. For instance, Alpaca-7B, I.L.MA- 7B, and Falcon-7B, which use an English-focused pretrained backbone, perform optimally in Latin

---

Additionally, open-domain dialogue evaluation is multi-faceted in nature, i.c., there are many eval- uation aspects to consider at both turn and dialogue levels. Our research concentrates primarily on the context relevance of responses and the coherence of multi-turn dialogues respectively. Future studies might consider investigating how to prompt LLMs to assess various aspects, as well as exploring strate- gies to fine-tune these LLMs for optimization of multi-diimnnioonl evaluation.
projects: ASTOUND (101071191 HORIZON- 1.--222.PPHHIDDDERAALLEEEESS funded by the European Commission, BE- WORD (PID202I-120610BB-CC MCIN/AEI/1O.. 10339//000001339/011110 3039/501 10001 1033 and, as ap- propriate, by "ERDF A way of making Europe", by the "European Union", and the Research Grants for Young Investigators from Universidad Politécnica de Madrid (GENUSS:POYO--OVEEES,) TAXTYC-32-K61X37) funded by Comunidad de Madrid. Finally, we also would like to thank Tencent AI Lab for providing support to this work.
funded by
Furthermore, multilingual automatic dialogue evaluation, particularly at the dialogue level, re- mains an unsolved challenge. Despite the solid performance of BERT-based metrics and LLMs en- sembles on x xDaal-Eval, there's considerable room for improvement. There two major limitations: (I) Subjectivity in dialogue evaluation Deter- mining "good" or "successful" dialogue can be highly subjective, relying on factors such as the conversation's s goals, participants, and social and cultural context. Sometimes, even human judges find it challenging (Smith et al., 2022). Possible so- lutions could involve generating expert-annotated dialogue data of high quality creating custom metrics designed for varying evaluation scenarios. (2) Long context modeling Some dialogues in XDial-Eval can be quite lengthy, exceeding the maximum token limits of open-source LLMs (> 2048 tokens). We currently address this by trunca- tion, which, unfortunately, results in information loss. Future research could focus on improving the modeling of long dialogues.
References
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Al- shamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, et al. 2023. Falcon-4OB: an open large language model with state-of-the-art perfor- mance.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, al. 2022. Constitutional AI: Harmlessness from AT feed- back.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Yi Chen, Rui Wang. Haiyun Jiang, Shuming Shi, and Ruifeng Xu. 2023a. Exploring the use of large lan- guage models for reference-fie free text quality evalua- tion: A preliminary empirical study. arXiv preprint arXiv: Arxiv-2044.00253
Ethics Statement
Zhihong Chen, Feng Jiang. Junying Chen, Tiannan Wang, Fei Yu, Guiming Chen, Hongbo Zhang. Juhao Liang, Chen Zhang, Zhiyi Zhang, Jianquan Li, Xi- ang Wan, Benyou Wang. and Haizhou Li. 2023b. Phoenix: Democratizing ChaiGPT across languages. arXiv preprint arXiv: Arxiv-2304. 2304.00553.
The XDial-Eval benchmark originates from pub- licly accessible English datasets. To support and propel future exploration in the domain of auto- matic dialogue evaluation, and in harmony with the initiatives of other researchers, we commit to making the data and all our models open-source for future research endeavors.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, et al. 2023. Vieuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality.
Acknowledgement
This work is supported by Human Robot Col- laborative AI under its AME Programmatic Funding Scheme (Project No. A18A2b0046), the National Natural Science Foundation of China (Grant No. 62271432), and the Internal Project Fund from Shenzhen Re- search Institute of Big Data under Grant No. T00120220002, This work is also a result of the
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, et al. 2022. PaLM: Scaling language modeling with path- ways.
Hyung Won Chung. Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghami, Siddhartha Brahma, al. language models. arXiv preprint arXiv:2210. 1416.
2022. Scaling

---

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Parameter-effiient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Marta R. Costa-jusså, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, et al. 2022. No language left behind: Scaling human- centered machine translation.
Lishan Huang, Zheng Ye, Jinghui Qin, Liang Lin, and Xiaodan Liang. 2020. GRADE: Automatic graph- enhanced coherence metric for evaluating open- domain dialogue systems. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 9230 -2400. Online. Association for Computational Linguistics.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jessica Huynh, Cathy Jiao, Prakhar Gupta, Shikib Mehri, Payal Bajaj. Vishrav Chaudhary, and Max- ine Eskenazi. 2023, Understanding the effectiveness of very large language models on dialog evaluation. In The 13th International Workshop on Spoken Dia- logue Systems Technology.
Jinlan Fu, See-Kiong Ng. Zhengbao Jiang, and Pengfei Liu. 2023, GPTScore: Evaluate as you desire.
Tianbo Ji, Yvette Graham, Gareth Jones, Chenyang Lyu, and Qun Liu. 2022. Achieving reliable human assess- ment of open-domain dialogue systems. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 16-477777, Dublin, Ireland. Association for Computational Linguistics.
Sarik Ghazarian, Behnam Hedayatnia, Alexandros Pa- pangelis, Yang Liu, and Dilck Hakkani- Tur. 2022a. What is wrong with you?: Leveraging user senti- ment for automatic dialog evaluation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 4194-4204, Dublin, Ireland. Association for Computational Linguistics.
Tian Lan, Xian-Ling Mao, Wei Wei, Xiaoyan Gao, and Heyan Huang. 2020. PONE: A novel automatic eval- uation metric for open-domain generative dialogue systems. ACM Trans. Inf. Syst., 39(1).
Sarik Ghazarian, Nuan Wen, Aram Galstyan, and Nanyun Peng. 2022b. DEAM: Dialogue coherence evaluation using AMR-based, semantic manipula- tions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume I: Long Papers). pages 771-785, Dublin, Ire- land. Association for Computational Linguistics.
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, et al. 2022. The bigscience ROOTS corpus: A 1.6TB composite multilingual dataset. In Thirty-sixth Conference on Neural Information Pro- cessing Systems Datasets and Benchmarks Track.
Prakhar Gupta, Cathy Jiao, Yi-Ting Ych, Shikib Mchri, Maxine Eskenazi, and Jeffrey Bigham. 2022. In- structDial: Improving zero and few-shot general- ization in dialogue through instruction tuning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 505- 525, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Fighth International Joint Conference on Nat- ural Language Processing ( (Volume 1: Long Papers). pages 986-995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, and Jeffrey Bigham. 2019. Investigating evaluation of open-domain dialogue systems with human generated multiple references. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 379-391, Stock- holm, Sweden. Association for Computational Lin- guistics.
Yen-Ting Lin and Yun-Nung Chen. 2023. LLM-Eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language mod- els.
Zhaojjang Lin, Zihan Liu, Genta Indra Winata, Samuel Cahyawijaya, Andrea Madotto, Yejin Bang, Etsuko Ishii, and Pascale Fung. 2021. XPersona: Evaluating multilingual personalized chatbot. In Proceedings
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.

---

the 3rd Workshop on Natural Language Process- ing for Conversational AI, pages 102-112, Online. Association for Computational Linguistics.
Niklas Muennighoff. Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, et al. 2023. Crosslingual generalization through multitask finetuning.
Chia- Siia-WeiTe Liu, Ryan Lowe, Iulian Serban, Mike Nose- worthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2122-2132, Austin, Texas. Association for Computational Linguisties.
OpenAl. 2023. GPT-4 technical report.
Long Ouyang and et al. 2022. Training language mod- els to follow instructions with human feedback. In Advances in neural information processing systems.
Bo Pang, Erik Nijkamp, Wenjuan Han, Linqi Zhou, Yix- ian Liu, and Kewei Tu. 2020. Towards holistic and automatic evaluation of open-domant dialogue gener- ation. In Proceedings of the 58th Annual Meeting of the A Asociiaion for Computational Linguistics, pages 3619-3629, Online. Association for Computational Linguistics.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG evaluation using GPT-4 with better human alignment. arXiv preprint arXiv: 9xxi-2333.11663
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv: 11692.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation, In Proceedings of the 40th Annual Meeting of the Association for Compu- national Linguistics, pages 31 1-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguisties.
Shikib Mehri, Jinho Choi, Luis Fernando D' Haro, Jan Deriu, Maxine Eskenazi, Milica Gasic, Kallirroi Georgila, Dilek Hakkani-Tur, Zekang Li, Verena Rieser, Samira Shaikh, David Traum, Yi-Ting Ych, Zhou Yu, Yizhe Zhang, and Chen Zhang. 2022a. Re- port from the NSF future directions workshop on automatic evaluation of dialog: Research directions and challenges.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier. Ebtesam Almazrouei. and Julien Launay. 2023. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116.
Shikib Mehri and Maxine Eskenazi. 2020a. Unsuper- vised evaluation of interactive dialog with DialoGPT. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 225-235, 1st virtual meeting. Association for Computational Linguistics.
Vitou Phy, Yang Zhao, and Akiko Aizawa. 2020. Decon- struct to reconstruct a configurable evaluation metric for open- doen-domiin dialogue systems. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4164-4178, Barcelona, Spain (On- line). International Committee on Computational Lin- guistics.
Shikib Mehri and Maxine Eskenazi. 2020b. USR: An unsupervised and reference free evaluation metric for dialog generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681-707, Online. Association for Computational Linguistics.
Ricardo Rei, Nuno M. Guerreiro, José Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, José G. C. de Souza, and André F. T. Martins. 2023. Scaling up COMETKIWE: Unbabel-IST 2023 submission for the quality estimation shared task. arXiv preprint arXiv: 2309.11925.
Shikib Mehri, Yulan Feng, Carla Gordon. Seyed Hos- sein Alavi, David Traum, and Maxine Eskenazi. 2022b. Interactive evaluation of dialog track at DSTC9, In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5731- 5738, Marseille, France. European Language Re- sources Association.
Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha. Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F, T. Martins. 2022. CometKiwi: IST-unbabel 2022 sub- mission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Com- putational Linguistics.
Mohsen Mesgar, Sebastian Büikkr,, and Tryna Gurevych. 2020. Dialogue coherence assessment without ex- plicit dialogue act labels. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4439-1450, Online. Association for Computational Linguistics.
Jeff Mitchell and Mirella I. Lapata. 2008 Vector-based models of semantic composition. In Proceedings of ACL-08: HLT. pages 236-244, Columbus, Ohio. Association for Computational Linguistics.
Mario Rodríguez- Chen Zhang, Chengguang Tang, Ke Shi, Sarik Ghazarian, João Sedoc, Luis Fer- nando Haro, and Alexander Rudnicky. 2023. Ro- bust and multilingual automatic evaluation metrics

---

for open-domain dialogue systems. In DSTCII: The Eleventh Dialog System Technology Challenge, 24th Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), Prague, Czechia
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An sstructioo-following LLaMa model. https:// // gi thub. wwt.:occoocaaaaaee http:/tttsuabbb/aaaa http:/sttffodddappaaaa
Ananya B. Sai, Akash Kumar Mohankumar, Siddhartha
Arora, and Mitesh M. Khapra. 2020. Improving di- alog evaluation with a multi-reference adversarial dataset and large scale pretraining. Trunsactions of the Association for Computational Linguistics, 8:810- 827.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez. Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. LLaMA: Open and efficient foundation language models. ARXIV.
Motoki Sato, Hiroki Ouchi, and Yuta Tsuboi. 2018. Addressee and response selection for multilingual conversation. In Proceedings of the 27th Inter- national Conference on Computational Linguisties. pages 3631 1343644, Santa Fe, New Mexico, USA. As- sociation for Computational Linguistics.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, et al. 2023b, Llama 2: Open foundation and fine-tuned chat mod- cls.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, et al. 2023, Bloom: A I 76b--parmmeter open-access multilingual language model.
Yizhong Wang. Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self generated instructions.
Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? how controllable attributes affect human judgments. Proceedings of the 2019 Conference of the North American Chapter of the Association for Compula- tional Linguistics: Human Language Technologies, Volume (Long and Short Papers). pages 1702-1723, Minneapolis, Minnesota. Association for Computa- tional Linguisties.
Aiyuan Yang, Bin Xiao, Bingning Wang. Borong Zhang, Ce Bian, Chao Yin, et al. 2023. Baichuan 2: Open large-scale language models.
Yi-Ting Yeh, Maxine Eskenazi, and Shikib Mehri. 2021. A comprehensive assessment of dialog evaluation metrics. In The First Workshop on Evaluations and A Assessments of Neural Conversation Systems, pages 15-33, Online, Association for Computational Lin- guistics.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text genera- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892, Online, Association for Computational Linguistics.
Chen Zhang, Yiming Chen, Luis Fernando D d'Haro, Yan Zhang, Thomas Friedrichs, Grandee Lee, and Haizhou Li. 2021a. DynaFval:: Unifying turn and dialogue level evaluation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages for Computational Linguistics.
Koustuv Sinha, Prasanna Parthasarathi, Jasmine Wang. Ryan Lowe, William L. Hamilton, and Joelle Pineau. 2020. Learning an unreferenced metric for online dialogue evaluation. In Proceedings the 58th An- nual Meeting of the Association for Computational Linguistics, pages 2430-2441, Online. Association for Computational L Linguistics.
Online. Association
Chen Zhang, Luis Fernando D' D'Haro, Rafael F. Banchs, Thomas Friedrichs, and Haizhou Li. 2021b. Deep AM-FM: Toolkit for Automatic Dialogue Evaluation, pages 53-69. Springer Singapore, Singapore.
Erie Smith, Orion Hsu, Rebecca Qian, Stephen Roller, Y-Lan Boureau, and Jason Weston. 2022. Human evaluation of conversations is an open problem: com- paring the sensitivity of various methods for eval- uating dialogue agents. In Proceedings of the 4th Workshop on NLP for Conversational Al, pages 77 97, Dublin, Ireland. Association for Computational Linguistics.
Chen Zhang, Luis Fernando H'roo Qiquan Zhang. Thomas Friedrichs, and Haizhou Li, 2022a. FineD- eval: Fine grained automatic dialogue- evalu- ation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3336-3355, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Ekaterina Svikhnushina, Anastasiaa Filippova, and Pearl Pu. 2022. iEval: Interactive evaluation framework for open- open-domain empathetic chatbots. In Proceedings of the 23rd Annual Meeting of the Special Interest Group Discourse and Dialogue, pages 419-431, Edinburgh, UK. Association for Computational Lin- guistics.
Chen Zhang, Luis Fernando Haro, Qiquan Zhang. Thomas Friedrichs, and Haizhou Li. 2023. PoΣ: A panel of experts for generalized automatic dia- logue assessment. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:1234-1250.

---

Chen Zhang, Grandee Lee, Luis Fermando D'Han Haroo, and Haizhou Li. 2021c. D-Score: Holistic dialogue evalu- ation without reference. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 29:2502- 2516.
PROMPT EXAMPLE
Given the following English source sentence, evaluate the adequacy of the French translation on scale of to S. Ad- equacy is defined "to what extent the translated sentence preserves the semantic meaning of the source sentence"
Chen Zhang. João Sedoe, Luis Fernando D' Haro, Rafael Banchs, and Alexander Rudnicky. 2022b. Automatic evaluation and moderation of open-domain dialogue systems. In DSTCIO Workshop in AAAI 2022.
English source sentence:
was able attend the lectures last week. Can you help me understand some concepts?
Mozhi Zhang. Wei Wang. Budhaditya Deb, Guoqing Zheng. Milad Shokouhi, and Ahmed Hassan Awadal- lah. 2021d. A dataset and baselines for multilingual reply suggestion. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the I 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1207-1220, Online, Association for Computational Linguistics.
French translation:
n'ai pas pu assister aux conférences de semaine dernière. Pouvez-vous m' aider comprendre certains con- cepts??
Provide your output the format:
adequacy: x where x is the corresponding numerical rating.
Table 6:
An example prompt template of machine trans- lation evaluation with GPT-4
Saizheng Zhang, Emily Dinan, Jack Urbanck, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: I have dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 2204-221 2213, Melbourne, Australia. Association for Computational Linguistics.
2019) and a scoring head. is trained in multi- task manner with large-scale multi-domain dia- logue data. PoE scores response by averaging the scores of all domain-specific sub-metries.
Tianyi Zhang. Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with bert. In International Conference on Learning Representations.
FineD-Eval targets multi-dimensional dialogue- level evaluation. It consists of three sub-metrics focusing on coherence, likability, and informative- ness evaluation respectively. The sub-metrics are trained to rank the good-quality dialogue ahead of its poor-quality counterpart. FineD-Eval scores dialogue by computing the arithmetic mean of indi- vidual sub-metric scores assigned the dialogue.
Tianyu Zhao, Divesh Lala, and Tatsuya Kawahara. 2020. Designing precise and robust dialogue response eval- uators. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisticx, pages 26-33, Online. Association for Computational Lin- guistics.
C.2 LLM-Based Generative Metries
A MT Evaluation Prompt Template
LLaMA-Series LLaMA is a pre-trained causal LLM from Meta. It is currently the most popu- lar open-source substitute for the closed-source GPT-3 (Brown et al., 2020). Its smallest vari- ant, LLaMA-7B, trained on trillion tokens (mainly in English). LLaMA-2 is an optimized version of LLaMA with a better mixture pre- training data and training strategies. LLaMA-2-7B is pretrained on 2T tokens, which is double that of LLaMA-7B. Alpaca is finetuned from LLaMA-7B on 52K instruction-follooinng demonstrations. The data are generated from OpenAl's 's text-davinci-003 model with self-instruct (Wang al., 2023). Vi- cuna finetuned on 70K user-shared conversations collected from ShareGPT. Both models adopt the Low-Ram Adaptation (LoRA) (Hu et al., 2022) ap- proach for finetuning. Both Alpaca and Vicuma
Table 6 is an example prompt we use for translation quality evaluation of xDial-Eval data with GPT-4.
B Examples of xDial-Eval Data
Table 7 and Table 8 show example data instances from the DailyDialog-Zhar Zhao (Zhao et al., 2020) turn-level and the lEval (Svikhnushina et al., 2022) dialogue-level datasets respectively.
Model Descriptions
C.1 BERT-Based Discriminative Metrics
PoE- consists of a pre-trained transformer encoder and a collection of domain specificc sub-metrics. Each sub-metric contains an adapter (Houlsby et al.,

---

English
Chinese
Spanish
Context:
well, does look
4 33 ti
bucno, jodmo so ve!
it's a perfect fit.
es apuste perfecto.
pay for now
déjame pagarlo ahorn.
Response:
cash, unditt card, or debit card??
10 1 1A
efectivo. tarjeta de crédito tarjeta de débito!
German
French
Japanese
Context:
Nun, sieht aus?
Eh bien, ça ressemble?
ist eine perfekte Passform.
est un ajustement parfait.
Lass mich jeizt daftr hezahlen.
Laissez-moi payer cela maintenant.
4
Resgonse:
Bargeld, Kreditkarte oder Dehitkane?
Argent comptant, carte crèdit ou cane débir?
Korean
Russiun
Context:
2.4. of A 1 414
Hy, kas anO naatow/pp??
a 4
RoGBo/ibe aare Setureriiee so ano veftracc
Respunse:
marriIaa xpizurzooft 1an zeGeromolt saprod??
Table 7: Example data instance from turn-level dataset Daily Zhao (Zhao et al., 2020) in 8 different languages. Hindi and Arabic data are not included due to latex compatibility issues. The relevance rating of the response 5 out 5.
English
Chinese
Human:
really glad finished my service for military What kind of service it?
# #[ TELARS
Chatbot:
X 2 ?
Human:
modical service
Chatbot:
Oh no, what was for? For
f
team
#
Chatbot:
What it for?
Fil FAA??
Spanish
Japanese
Human:
Fstahu may comtento de haber terminuddo mi servicio pura el ejércitn.
Te "
Chatbot:
(Quć tipo servicio era?
Human:
Servicio Médico
Chatbot:
Oh no. apara qué fuxe?
h 19. 212
Human:
Para el equipo médico de combate
&
Chatbot:
Para qué sirvió?
German
Korean
Human:
Ich war wirklich froh, dass ich meinen Militänfienst heendel hatle Um für einen Service handelte sich?
a #472131/35/3777666
Chatbot:
Human:
Sanitätsdiensi
Chatbot:
nein, wozu war das gut?
2. 114, o
Human:
Für dus medizinische Geleechtsteumm
E 1 10?
Chatbot:
Wozu diente es?
French
Russian
Human:
J' J'étais vraiment content d' avvvir terminé mon service militaire
3аR0нTR0 сиулкбу aриии
Chatbot:
De quel type de service s' agiissittil??
Yoo sno Gaza 3a yeayra?
Human:
Service médical
мсапынссоо обслужмaаe
Chatbot:
Oh non, à quoi servait-il?
O ner, Jumr nero sro fizna нyжkо??
Human:
Pour I' équipe médicale de combat
Aas 6ocaott нолионнско RоManиии
quoi servait-il?
Abn vero aro 6aat nyseo!?
Table 8: Example data instance from dialogue-Ievel dataset IEval (Svikhnushina et al., 2022) different languages. Hindi and Arabic data are not included due to latex compatibility issues. The overall rating of the dialogue is out of
are developed to mimic the general instruction- following capability of ChatGPT.
on mixed multilingual instruction data and conversation data. The instruction data of Phoenix is collected with self-instruction (Wang et al., 2023) while the conversation data is mainly ChatGPT-distilled. As the collected data are mainly in English, they are then translated into other languages to support the post-training process of
BLOOMZ 14
BLOOM-Series: BLOOM is large-scale mul- tilingual causal LLM released in the BigSeience workshop. It is pretrained on the ROOTS cor- pus (Laurençon et al., 2022) in 46 natural languages and 13 programming languages. Phoenix is an open-source multilingual ChatGPT-like model that can handle 40+ languages. is finetuned from
A variant of BLOOM finetuned on diverse multilingual data, which cover wide arryy of NL.P tasks, such as question answering, topic classification, and program synthesis.

---

Human:
Human:
[H] My dog puked over carpet few
Dialogue Context:
weeks back.
hello, tell me something about yourself am 56 years old and am housekeeper. you have any hobbies interests
[C] Oh no! Did clean up?
[H] yes
[C] Did you clean it up?
[H] did
Dialogue Response: like read and write poetry.
[C] That's great! glad it worked for you!
Above is a dialogue between a human ([H) and chatbot ([CJ). the dialogue coherent?
Given the dialogue context and response above, is the response relevant the context?
Assistant:
Assistant: Yes
No
(a) Dialogne-Leve Evaluation
Tum-LLvel Evaluation
Table 9: Vicuna-7B Instruction Template.
Human:

Human:
>
[H] My dog puked over the carpet few
Dialogue Context:
weeks back.
hello, tell me something about yourself
[C] Oh no! Did clean up?
am 56 years old and am housekeeper.
[H] yes
you have any hobbies or interests
[C] Did you clean it up?
[H] did
Dialogue Response: to read and write poetry..
[C] That's great! glad worked out for you!
Above is a dialogue between a human ([H)) and chatbot ([CJ). the dialogue coherent?
Given the dialogue context and response above, is the response relevant the context?
Assisttttam
<ssssssistattt<>>
No
Yes
(a) Dialogue-Level Evaluation
Turn-Level Evaluation
Table 10: Phoenix-77 7B Instruction Template.
My dog puked all the carpet few
Dialogue Context:
weeks back.
hello, tell me something about yourself'
[C] Oh no! Did you clean it up?
am 56 years old and am housekeeper. do you have any hobbies interests
yes
[C] Did you clean it up?
[H] yes did
Dialogue Response:
[C] That's great! I'm glad it worked for you!
like to read and write poetry
Question: given the input dialogue between human ([H)) and chatbot ([C)). whether the dialogue is coherent?
Question: given the context and response, pre- dict whether the response is relevant to the con- text?
Answer: No
Answer: Ves
(e) Dialogue-Level Evalustion
Turn-Level Evaluation
Table Falcon-7B Instruction Template.
Phoenix.
2023) enhanced with curated corpora, such as so- cial media conversations, books, and technical pa- pers. The training data is mainly in English and French.
Falcon- Falcon (Almazrouci al., 2023) a large- scale pretrained causal language model released by The Technology Innovation Institute. It is trained on 1,500B tokens of RefinedWeb (Penedo et al.,
BaiChuan-2 BaiChuan-2 (Yang et al., 2023) en-

---


 
 


 
 

are helpful. respectful and honest assistant. Always answer as helpfully us possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, danger- ous, or illegal content. Please ensure that your responses socially unbiased and positive in nature.
You helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include uny harmful, unethical, racist, sexist, toxie, danger- ous, illegal content. Please ensure that your responses are socially unbiased and positive in nature.
If question does not make sense. is not factually coherent, explain why instead of swering something not correct. If you don't know the answer to question, please don't share false information. 

If a question does not make any sense, or is not factually coherent, explain why instead of an- swering something not correct. If you don't know the answer question, please don't share false information. <</SY>>>
Given the following dialogue between human (IHI a chatbot (IC)), whether dialogue is coherent
Given the context and response, predict whether the response is relevant the context
Dialogue Context:
[H] My dog puked over the carpet few weeks back.
hello, tell me something about yourself" am years old and am housekeeper. do you have any hobbies or interests
[C] Oh no! Did you clean up?
[H] yes
[C] Did you clean it up?
Dialogue Response:
[H] yes did
like to read and write poetry.
[C] That's great! glad it worked for you!




Yes
No
(b) Tunn-Level Evaluation
(e) Dialogue-Level Evalustion
Table 12: LLaMA-2 Instruction Template.
<reserved Given the input dialogue be- tween a human ([H)) and chathot ([C)) below, whether the dialogue is coherent.
<reserved 106> Given the context and response below, predict whether the response relevant to the context.
[H] My dog puked over carpet few weeks back.
Dialogue Context:
hello, tell me something about yourself am 56 years old and am housekeeper you have any hobbies interests
[C] Oh no! Did you clean up?
[H] yes
[C] Did you clean it up?
[H] did
Dialogue Response:
[C] That's great! glad worked for you!
like read and write poetry
<reserved 107>
<reserved 107> Yes
No
(a) Dialogue-Lvvel Evalustion
Turn-Level Evaluation
Table 13: Baichuan-2 Instruction Template.
compasses a series of large-scale multilingual lan- guage models, consisting of models with 7 bil- lion and 13 billion parameters, and is trained from scratch massive dataset comprising 2.6 trillion tokens. The pretraining data is collected from va- riety of sources, such as general internet webpages, books, research papers, codebases, and more.
instruction-folloving conversational Al developed by OpenAI. The training of ChatGPT follows the instructGPT development pipeline (Ouyang and et al., 2022): (1) supervised finetuning from strong pre-trained large language model on high- quality
instruction data; (2) Align the finetuned language model with humans' inten- tions and goals leveraging reinforcement from hu-
ChatGPT is closed-source general-purpose purpose

---

man feedback. ChatGPT and its successor, GPT- 4 (OpenAI, 2023) are currently the most powerful Al assistants that are highly capable of solving var- ious NLP tasks. Many open-source LLMs, such as those described in the LLaMA-series and BLOOM- Series, are trained to mimic their abilities.
Figure 1 and 2 show the inter-lingual Pearson cor- relations of FineD-Eval / PoE, ChatGPT, LLaMA- 7B, Alpaca-7B, BLOOM-7B, and Phoenix-7B on FED-Dial and FED-Turn respectively.
We observe that FineD-Evvll and PoE both display very consistent inter-lingual correlation pat- terns. The correlations among language pairs are -0.9 for FineD-Eval and > 0.7 for PoE. The ob- servation showcases that both metrics can provide consistent evaluation of translation- based multi- lingual dialogue data. Additionally, we can observe that ChatGPT displays stronger inter-linuull cor- relations on FED-Dial than FED-Turn. The observation partially explains why ChatGPT has a more consistent performance across different lan- guages at the dialogue level than at the turn level (refer to Table 5 and Table 17).
D More Examples on Instruction Template
This section presents the instruction templates of Vicuna-7B (Table 9), Phoenix-7B (Table 10), Falcon-7B (Table 11), LLaMA-2-7B (Table 12), Baichuan-2-7B (Table 13), and ChatGPT (Ta- ble 14). We try to closely follow the instruction templates of the open-source LLMs used during their supervised finetuning stage.
Furthermore, Phoenix-7B and Alpaca-7B exhibit stronger inter-lingual correlations than their respec- tive backbone models, LLaMA-7B and BLOOM- 7B. The observation supports our conclusion in $6.3 that a two-stage finetuning process (adapt instruction-tuned models on custom multilingual data) yields dialogue evaluators that are more ro- bust across different languages.
E Additional Analyses
Table 16 and Table 17 the corresponding Spear- man correlations of Table 4 and Table 5 respec- tively. Similar observations the ones in $6 can be made with Table 16 and Table 17.
Natural Non-English Dialogues In addition the translation-based xDial-Eval benchmark, we analyze the performance of finetuned models on organic non-English dialogue data using five Chinese dialogue evaluation datasets released by Rodríguez-Cantelar et al. (2023). Table 15 presents the detailed results. We can observe that Baichuan-2-7B is the best among all models at the turn level while LLaMA-2-7B performs the best at the dialogue level. Similar to the observations in $6.3, the ensemble of BERT-based metrics and LLMs leads to even stronger correlations with hu- man evaluation. The results demonstrate that our proposed metrics are not just proficient in manag- ing translation-based multilingual dialogue data, but they also deliver strong performance when ap- plied to natural non-English data.
Lastly, we observe that the inter-lingual corre- lations of Alpaca-7B are notably stronger within Latin-language pairs, compared to other lan- guage combinations. In contrast, Phoenix-7B dis- plays a more evenly distributed correlation pattern among various language pairs. This observation reinforces the findings presented in Section 6.3, suggesting that Alpaca-7B's performance excels in Latin languages, likely due to its English-centric pretrained backbone and initial fine-tuning on En- glish instruction data. On the other hand, Phoenix, equipped with a multilingual pretrained backbone and an initial fine-tuning on multilingual instruc- tion data, exhibits more uniform performance across all languages.
Correlations Among Languages - this sec- tion, we analyze the interdependence of judgments given to dialogue data in differentt languages by different models. Specifically, we choose the FED-Turn (Mehri and Liskenazi, 2020a) and FED- Dial (Mehri and Eskenazi, 2020a) datasets for anal- ysis. The purpose is to check whether the models can provide consistent judgments to multilingual dialogue data with the same semantic meanings.
Correlations Among Metries We delve deeper into examining the extent of agreement among the evaluations provided by various model-based met- rics. To accomplish this, we calculate the corre- lations between different pairs of metrics, focus- ing specifically on their performance turn-level datasets: FED-Turn & Daily-Dialog-Zhao (Zhao et al., 2020), and dialogue-level datasets: FED-Dial & IEval-Dial (Svikhnushina et al., 2022) in En- glish. This approach allows us to discern whether these metrics complement each other. Figure 3
15

---

Given the dialogue between human ([H)) and a chatbot ([CJ) below, evaluate the coherence of the dialogue on a scale of 1-5:
Given the dialogue context and response helow, evaltuate the contextual relevance response on a scale of 1-5:
[H] My dog puked
the carpet few [C] Oh no! Did you clean up?
Dialogue Context:
weeks back.
hello, tell me something about yourself' um 56 years old and i am housekeeper you have any hobbies interests
[H] yes
[C] Did you clean it up?
[H] yes did
Dialogue Response:
[C] That's great! I'm glad it worked for you!
like read and write poetry.
Provide the numerical rating in the following
Provide the numerical rating in the following
format:
format:
Rating: X;
Rating: X;
(a) Divl Evaluation
(b) Turn-Level Evaluation
Table 14: ChatGPT Instruction Template.
Turn-Level
Models
ECM-Turn
LCCC-Turn
HC-Turn
Average
PoE
00.570
0.464/ 0.435
0.436 003351
0.477 / 00.452
Falcon-7B
0,581 /0.590
0.436/ 0.421
0.534 0.486
0,517 / /0.499
Alpaca-77 7B
0.476 00.800
0.409 0.390
0.482 00.44
0.456 / 10.439
Phoenix-7BB
0.494 / 70.562
0.475 0.465
0,464 0.398
0.478 / 00447
LLaMA-2-7B
0.527 70.551
0.436/ 0.415
0,515 0.471
0.493 / 0.479
Baichuun-2-BB
0,522 / 0.614
0.449 0.438
0,551 0,523
0,508 / 0.525
PoE Falcon-7B
0.592 / 00618
0.496
0.460
0.527 00452
0.538 / 00.510
Pol Alpaca-7B
0.555 / 0.573
0.481 0.446
0.503 0.441
0.513 / 00.47
Pol Phoenix-7B
0,558 0.598
0.512/ 0.478
0,497 0.391
0.522 0..499
PoE LLaMA-2-7B
0.576/ 0.607
0.494/ 0.457
0.528
0.462
0.532 0.508
PoE BBichhann---1
0.567 0.6188
0.506 0.468
0.563 0 488
0.545 0.525
Dialogue-Level
Models
LCCC-Dial
HC-Dial
Average
FincD
0.386 00.379
0.563 0.578
0.475 / 0.479
Falcon-7B
0.329 0.338
0.678 0.698
0.504 0.518
Alpaca-7B
0.135 /0.118
0.687 0.690
0.411 /0.404
Phoenix-7B
0.245 /0.234
0.692 0.696
0.465
LLaMA-2-7B
00.263
0.735 0.741
0,500 / 0.502
Baichuan-2-15
0.252 0.225
0.675 0.706
0.464
FineD + Falcon-7B
0.456 / 00456
0.712 0.727
0.584 / 0,592
FineD Alpaca-7B
0,338 / 00.35
0.717 / 0.717
0.528 / 00,521
FineD Phoenix-7B
0.353 / 0..349
0.722 / 0.722
0.538 / 0,536
Finel) + L.JaMA-2-7B
0.403 / 00.40
0.751 / 0.758
0.577 / 0.580
FineD Baichuan-2-7B
0.345 0.342
0.709 0.719
0.527 / 0.531
Table 15: Pearson/ / Spearman correlations of different models (after finetuning on the multilingual dialogue data) on natural Chinese dialogue evaluation datasets.
presents the inter-metric Pearson correlations. We observe that the judgments provided by differ- ent metrics are not consistent and the correlation patterns differ on different datasets. Furthermore, while the correlations between PoE and Alpaca-7B. and between PoE and Phoenix-7B, are reasonably good, they aren't excessively strong. This suggests that these pairs of metrics complement each other without being overly similar. This insight hints at
the potent performance of an ensemble compris- ing PoE and Alpaca-7B, PoE and Phoenix- particularly on turn-level datasets. Similar observa- tions can be made w.r.t. FineD-Eval and Alpaca-7B and FineD-Eval and Phoenix-7B. These insights provide a rationale for the conclusions we drew in the "Metric Ensemble" subsection of & 6.3, where we found that an ensemble of BERT- BERT-based met- rics and LLMs creates highly effective multilingual

---

Turn-Level
Models
EN
Z.H
ES
DE
FR
JA
KO
HI
AR
RU
AVG
PoE
0,465
0.431
0.438
0,442
0.444
0.419
0,411
0.349
0.409
0.426
0.423
Alpaca-7B
0.482
0.336
0.405
0.419
0.418
0.248
0.029
0.155
0.186
0.346
0.302
Phoenix-7B
0,456
0.409
0.399
0.325
0.434
0.288
0.202
0.386
0.409
0,269
0.358
LLaMA-2-7B
0.523
0.401
0.456
0.415
0.450
0.324
0.324
0.242
0.248
0,388
0.377
Baichuan-2-7B
0.561
0.556
0.476
0.473
0.481
0.382
0.349
0.254
0.307
0.417
0.426
Dialogue-Level
Finel)
0.376
0.346
0.356
0.345
0.351
0.362
0.345
0.301
0.299
0.342
0.342
Alpaca-7B
0,400
0.306
0.351
0.338
0.361
0.205
0.208
0.189
0.172
0.309
0.284
Phoenix-7B
0.354
0.346
0.267
0.242
0.317
0.245
0.175
0.272
0.328
0.206
0.275
LLaMA-2-7B
0.375
0.288
0.334
0.347
0.322
0.246
0.244
0.199
0.209
0.304
0.287
Baichuan-2-70
0.371
0.338
0.270
0.311
0.317
0.271
0.278
0.224
0.249
0.277
0.291
Table 16: Language-wise average turnleevll (over 12 datasets) and dialogue-level (over 6 datascts) Spearman correlations of models finetuned English data only.
FineD D
ChatGPT
1.0
100
1.00
0:91
0.95
0:92
0.86
0:90
0.89
0.92
5
1.00
0:59
0.72
0.72
0.71
0.63
0.56
0.54
0.59
0.68
0.91
1.00
0.92
0.89
0.89
0.89
0.86
0.30
1.00
0.63
0.68
0.69
0.56
0.62
0.63
0.8
0.8
0
0.92
0.90
0.91
0.92
3.00
0.66
g
0.93
0.89
0:93
0:33
0.68
0.68
1.00
0.60
0.70
0.6
0.6
0:92
0.89
0.94
0.93
1.00
0.67
0.09
0.90
0.08
0.93
0.71
0.69
0.72
0.72
1.00
0.70
0.54
0.62
0.87
0.71
0.86
0.87
0.88
0.87
0.87
1.00
0.87
0.88
0.87
0.88
0.63
0.67
0.69
0.67
0.70
1.00
0.58
0.58
0.66
0.65
9
0.88
0:89
0.90
0.88
0.89
8.87
1.00
0.99
0:86
0.90
0.56
0.55
0.59
0.52
0.54
0.58
1.00
0.45
0.33
0.57
0.90
0:89
0.91
90
3.90
88
0.86
1.00
0.88
0.
0.54
0.56
0.54
0.53
0.62
0.58
0.45
1.00
0.67
0.59
0.2
0.2
0.89
0.86
0.91
0.89
1:00
0.90
0.70
1.00
0.64
2
2
0.66
0.0
0.0
en
zh
ar
LLaMA-7B
Alpaca-7B
1.0
10
8
1.00
0.46
0.66
0.32
0.49
0.67
1.00
0.55
4
0.46
1.00
0.42
0.53
1.00
0.75
0.75
0.8
0.8
1
0.66
0.35
1.00
0.73
0.65
0.42
0.52
0.43
0.55
0.82
0.91
0.75
1.00
0.89
0.90
0.69
0.67
0.59
0.56
0:63
g
0.75
0.50
0.71
1.00
0.78
0.55
0.67
0.51
0.64
0.71
8
0.86
0.79
0.69
1.00
0.91
0.70
0.67
0.53
0.58
0.63
0.6
0.73
0.51
0.65
0. 78
1.00
0.46
0.61
0.54
0.60
0.76
0.90
0.76
0:90
0.91
1:00
0.72
0.67
0.60
0.56
0.63
0.56
0.54
0.42
0.55
0.46
1.00
0.64
0.22
0.43
0.48
0.71
0.73
0.69
0.70
0.72
1.00
0.59
0.63
0.59
0.66
0.4
0.4
5
0.58
0.34
0.52
0.51
0.55
0.88
0.67
0.57
0.39
0.72
0.59
0.53
0.2
0.2
8
0.49
0.42
0.55
0.62
1.00
0.57
0.56
0.56
1.00
0.52
0.67
0.55
0.45
0.52
0:83
0.57
en
zh
85
de
ko
hi
ar
1
8n
zh
es
de
la
ko
hi
ar
2M
BLOOM-7B
Phoenix-78 7B
1.00
0.57
0.09
0.76
0.07
0.23
0.15
0.76
0.78
0.55
57
3.00
0.89
0.03
0.56
0.60
0.62
0.84
0.79
0.76
0.57
1:00
0.63
0.61
0.57
0.39
0.17
0.61
0.62
0.42
0.82
1.00
0:84
0. 0.78
0.85
0.66
0.67
0.84
0.76
0.78
0.8
0.8
4
0.89
0.63
1.00
0.77
0.90
0.37
0:20
0.77
0.83
0.55
0.89
84
1:00
0:93
0.90
0.65
0.70
0.82
0.81
0.80
0.76
0.61
0.77
.0
0.81
08
0.26
0.80
0.71
0.57
0.83
78
0:83
00
0.85
0.65
0.67
0.87
0.76
0.81
0.0
0.6
0.87
0.57
0.90
0.79
0.80
0.85
0:90
0.83
0.84
0.30
0.66
0.65
0.66
0.60
0.65
0.4
0.4
9
0.15
0.37
0.20
1:00
0.22
0.26
0.17
0.70
0.62
0.69
2
0.61
0:77
0.40
1.00
0.59
0.84
0.87
0
0.63
1:00
0.75
0.76
8
0.78
0.62
0.83
0.71
0.80
0.37
0.26
0.80
1.00
0.63
0.79
0.75
0.61
0.76
0.83
0.60
0.82
0.75
1.00
0.76
0.58
0.47
0.65
0.57
0.65
0.36
0.17
0.59
0.63
1.00
0.76
0.78
0.80
0.81
0.84
0.65
0.69
0.76
0.76
1:00
en
zh
85
de
fr
ko
hl
ar
11
en
zh
85
de
fr
ja
ko
hl
ar
ru
9.0
Figure| 1: Inter-lingual Pearson correlations of different models on FED-Dial dataset.
dialogue evaluators that outperform ChatGPT.
F
Reprodueibility
Computation Details All experiments were car- ried out using single 40GB A GPU card. We

---

Turn-Level
Category
Models
EN
7
F$
DE
FR
JJA
KO
HI
AR
RU
AVG
BERT Based
PoEt
0.474
0.440
0.452
0.455
0.459
0.422
0.423
0.367
0.424
0.438
0.435
LLaMA-7B
0.038
0.034
0.083
0.028
0.038
0.058
0.014
-0.009
0.015
0.059
0.036
LLMMA-2-78
0.068
0.081
0.083
0.041
0.041
0.101
0.108
0.070
0,073
0.022
0.069
BL BLO0M-78
0.046
0.144
0.101
0.010
0.080
0.023
0.002
0.043
0.088
0.072
0.061
LLMs-Zeroshot
Falcon-7B
0.144
0.121
0.154
0.094
0.141
0.093
0.011
0.069
0.108
0.076
0.101
Beichwan-2-77
0.178
0.156
0.132
0.107
0.153
0.097
0.134
Alpaca-7B
0,340
0.202
0.266
0,281
0.278
0.157
0.136
0.129
0.161
0.244
0.219
Vicuna-7B
0,205
0.162
0.225
0.178
0,209
0.166
0.114
0.113
0,144
0.202
0.172
Phoenix-7B
0,300
0.283
0.277
0.188
0,284
0.188
0.136
0.229
0,270
0.184
0.234
CharGPT
0.479
0,435
0.473
0,474
0,463
0.414
0.364
0.347
0.397
0,430
0.428
LLa
0,363
0.268
0.238
0,261
0.261
0.226
0.222
0.216
0,208
0.266
0.253
LLaMA-2-7B1
0.553
0.470
0,501
0,500
0,516
0.424
0.409
0.350
0.375
0.468
0.457
LLMs-FT (ours)
BLOOM-7B?
0,315
0.233
0.359
0,229
0,321
0.219
-0.039
0.232
0.184
0.138
0.219
Falcon-7Bf
0.423
0.439
0,466
0,449
0.479
0.282
0.166
0.146
0,185
0.275
0.331
Baichuan 2-BB
0,569
0.514
0.523
0,509
0.517
0.461
0.446
0.388
0.415
0.478
0.482
Alpaca 7B
0.561
0.397
0.493
0.483
0.486
0.328
0.310
0.300
0.304
0.437
0.410
Phoenix
0.428
0.469
0.311
0.252
01,42
0.327
0.396
MaMM--7 PoE+
0.482
0.437
0.444
0.446
0.462
0.426
0.427
0.368
0.420
0.434
0.439
11aMA-2-7- PoEt
0.552
0.488
0,510
0.512
0.520
0.457
0.445
0.390
0.452
0.483
0.479
Ensemble (ours)
BLOOM-7B + PoFf
0.498
0.450
0.472
0.456
0.477
0.418
0.423
0.379
0.436
0.446
0.445
Falcon 7B PoEt
0.467
0.477
0.478
0.494
0.385
0.367
0.328
0.379
0.408
Baichuan-2-7B- + PoEt
0.553
0.503
0.513
0.506
0.514
0,461
0.454
0.401
0,440
0.480
0,482
Alpaca-7B + PoB
0,555
0.451
0.510
0.507
0.513
0.407
0.396
0.369
0.394
0.396
0.450
Phoemix-BB- PoEt
0,506
0.453
0.481
0.433
0.485
0.385
0.365
0.413
0.448
0.406
0.438
Dialogue-Level
BERT-Based
FinDDt
0.379
0.347
0.359
0.355
0.364
0.343
0.338
0.331
0.331
0.371
0.352
LLaMA-7B 7B
0,174
0.172
0.205
0,181
0.130
0.155
0.128
0.012
0,022
0.131
0,131
LLaMA-2-7B
0.037
0.190
0.156
0,104
0.166
0.123
0.182
0.030
0.146
0.125
0.126
BLOOM-7B
0.086
0.208
0.074
0.076
0.145
0.125
0.072
0.098
0.133
0.097
0.111
Falcon-7B
0,272
0.236
0.233
0,254
0.117
0.083
0.133
0.145
0.169
0.221
0.186
LLMss Zerosbot
Alpaca-7B 7B
0.436
0.324
0.387
0.397
0.389
0.294
0.271
0.211
0,273
0.345
0.333
Baichuan 2athuan---78 2-7B
0.299
0.293
0.261
0,258
0,251
0.205
0.208
0. 0.128
0.193
0.214
0.231
Vicuna
0.314
0.229
0,250
0.227
0.210
0.225
0.145
0.160
0.220
Phoenix 7B
0,278
0.288
0.228
0,234
0.251
0.254
0.158
0.237
0,239
0.191
0.236
CharGPT
0,406
0,362
0.394
0,388
0.394
0.368
0,302
0.313
0,378
0.355
0.366
MaMA-7B#
0.228
0. 0.194
0.203
0.226
0.257
0.181
0,160
0.154
0.179
0.215
0.200
11MMA-2-78f
0.194
0.203
0.257
0.181
0.160
0.154
0.179
0.215
0.200
LLMs-FT (ours)
BLOOM-7B7
0,315
0.263
0.268
0,246
0.271
0.159
0.134
0.283
0.279
0.129
0.235
Falcon-7Bt
0,353
0,349
0,300
0,314
0.312
0.205
0.123
0,140
0.184
0.145
0.243
Baichuan-2-BBt
0.365
0,337
0.330
0.327
0,335
0.298
0,342
0,279
0.325
0.327
0.327
Alpaca-7B*
0,393
0.343
0.360
0.370
0.354
0.292
0.249
0.246
0.261
0.328
0.320
Phocnix-7B#
0,346
0.341
0,338
0,303
0.325
0.274
0.231
0.330
0,321
0.259
0,307
LLaMA-7B + FineDt
0.395
0.358
0,371
0,361
0.373
0.353
0,346
0,343
0.341
0.381
0.362
LLaMA-2-7B- + FineD?
0.457
0.418
0.431
0.427
0.427
0.387
0,383
0,370
0.375
0.429
0.410
Ensemble (ours)
BLOOM-7B FineD*
0.398
0.364
0.379
0.346
0.338
0.363
0.368
0.366
Falcon-7B 7B + FineD>
0.430
0.403
0.393
0.393
0.397
0.344
0.337
0.326
0.337
0.365
0.372
Baichuan 2-7B + FineDt
0.407
0.381
0.372
0.375
0.380
0.349
0.379
0.330
0.364
0.376
0.371
Alpaca FineDt
0.390
0.411
0.417
0.409
0.357
0.338
0.330
0.397
Phoenix 7B FineDI
0.402
0.377
0.386
0.359
0.382
0.341
0.320
0.373
0.363
0.338
0.364
Table 17: Language-wise average turn-level (over 12 datasets) and dialogue-level (over 6 datascts) Spearman correlations of different models. "LLMMs-Zerosoo"" means models applied directly without finetuning, whereas "LLMs-FF" represents finetuned models. The best score for each language is highlighted in bold and models finetuned on synthetic dialogue data are accompanied with a f.
use the alpaca-lora library 16 for LLM fine-tuning with low-rank adaptation. During finetuning, we utilize batch size of 128 with 4 gradient accumula- tion steps. The learning rate, cutoff' length, dropout rate, and number of epochs are set to 0.0003, 1024, 0.05, and 3 respectively. For the LoRA parameters, the rank (r-) and alpha are set to 8 and 16 corre-
spondingly. The target module for low-rank adapta- tion includes the query and key projection matrices. The total number of trainable parameters approx- imately 4M, which accounts for around 5.5% to 6% of the LLMs' full parameter size. The finetuning of each LLM with LoRA takes around 100 hours on 200K data. For training PoE. and FineD-Eval, we follow the training procedures and hyperparameter settings introduced in their respective papers.
16

---

PoE
ChatGPT
1.0
0
0.70
1.00
0.76
0.78
0.75
0.75
0.73
0.71
0.64
0.75
0.55
1.00
0.61
0.56
0.48
0.43
0.39
0.37
0.36
0.48
8
0.79
0.76
1.00
0.87
0.84
76
0.75
0.74
0.74
0.84
0.66
0.61
1.00
0.66
0.56
0.51
0.46
0.45
0.45
0.54
0.79
0:79
0.87
1.00
0.82
0.79
0.76
0.77
0.73
0.93
0.05
0.59
0.06
1.00
0.56
0.43
0.49
0.40
0.45
0.51
0.6
0.6
0.80
0.75
0.84
1.00
74
0.77
0.79
0.75
0.81
0.63
0.48
0.56
0.56
1.00
0.59
0.49
0.53
0.56
0.66
0.70
0.77
0.51
0.45
0.52
0.4
0.4
Z
0.73
0.27
0.45
0.53
0.2
0.72
0.64
0.74
0.73
0.75
0.70
0.71
0.73
1.00
0.76
0.44
0.36
0.45
0.45
0.56
0.53
0.50
1.00
0.58
0.75
0.75
0.64
0.83
0.81
0.77
0.71
0.77
0.76
1.00
0.52
0.48
0.54
0.51
0.52
0.51
0.53
0.58
1.00
0.0
en
7h
85
de
ko
h
ar
8n
2h
65
de
jn
ko
hi
ar
2
LLaMA-7B
Alpaca-7B
1,00
0.65
0.70
0.73
0.58
0.60
0.52
0.46
0.49
0.75
1.00
0000
0.62
0.82
0.85
0.47
0.43
0.45
0.44
0.75
0.05
1.00
0.65
0.65
0.54
0.59
0.53
0.40
0.52
0.71
0.60
1.00
0.68
0.69
0.69
0.66
0.50
0.57
0.50
0.09
0.8
0.8
0.70
0.65
1.00
0.77
0.68
0.48
0.44
0.35
0.40
0.77
0.82
1.00
0.87
0.89
0.54
0.48
0.52
0.52
0.67
0.05
0.46
0.01
1.00
0.6
0.6
0.54
0.59
0.57
0.4
0.52
0.64
1.00
0.66
0:51
0.57
0.58
0:48
0.
68
0.63
0.56
0.46
0.48
0.35
0.46
0.46
0.65
0.66
1.00
0.58
0.52
0.45
0.57
0.52
0.52
0.53
0.62
0.68
1.00
0.68
0.55
0.49
0.52
0.40
0.42
0.40
0.55
0.51
0.50
1.00
0.51
0.44
0.50
0.52
0.50 0.53 0.64 0.63
0.60
1.00
0.58
0.75
0.71
0.77
0.81
0.71 0.61
0.57
0.52
0.51
1.00
0.75
0.69
0:82
0:83
0.82 0.57 0.56 0.55
0.58
1:00
en
7h
de
ko
hi
ar
0.0
7/
e5
de
fr
hi
0.0
BLOOM-7B
Phoenix-78
1.00
0:44
0.70
0.38
0.68
0.10
0.09
0.51
0.45
0:33
1.00
0.72
0:86
0.65
0.81
0.56
0.47
0.74
0.70
0.58
0.44
1:00
0.57
0.47
0.20
1.00
0.74
0.87
0.70
0.67
0.60
6.8
0.8
0.09
0.39
0.08
0.6
0.00
8
0.10
0.38
0.14
0.28
0.07
1.00
0:22
0.10
1.00
0.01
0.56
0.80
0.60
0.85
0.60
1.00
0.84
0.68
0.82
0.64
9
0:04
.0.03
0:00
-0
00
10
3.11
1.00
0:01
0.06
0.27
0.47
0.51
0.55
0
0.55
0.64
1.00
0.59
0.56
0.64
0.51
0.47
0.64
0.43
0.56
0.18
0.02
1.00
0.47
0.37
0.74
0.70
0.80
0.71
0.77
0.68
0.59
1.00
0.78
0.66
0.2
0.2
0.45
0.47
0.62
0.39
-0.04
0.47
1.00
0.35
0.70
0.67
0.59
6.76
1.00
0.69
0.20
1.00
0.69
1.00
hi
0.0
de
0.0
Figure 2: Inter-lingual Pearson correlations of different models on FED-Tur dataset.

---

FED-Dial
FED-Turn
10
10
FinelD-Fval
1:00
0.58
0:38
0.52
0.52
0.47
0.40
POF
1:00
0.34
0.29
0.61
0.26
0.67
0.40
ChatGPT
0.59
1,00
0.22
0.56
0.55
0.40
0.40
0.0
ChatGPT
0:34
1.00
0.41
0.51
0.36
0.35
0.48
LLaMA-7D
0.41
0.30
0.32
LLaMA-7B
0.53
0.45
0.26
0.43
Alpaca-78
BLOOM-7B
BLOOM:7B
Phoenix-7B 0.47
0.32
0.09
0.77
1.00
0.38
Phoenix-78 7B
0.67
0.35
0.26
0.63
0.40
1:00
0.43
Falcon-78
0.40
0.46
0.23
0.69
0.58
0.58
1.00
Falcon-7B
0.40
0.48
0.43
0.56
0.40
0.43
1.00
FineD-Eval
ChatGPT
Alpace 78
BLOOM-78 78
Phoenix-88
Felcon-79
ChatGPT
LL
Alpace 7B
BLOOM-78
Phoenit-78
falcon 7B
IEval-Dial
Dail
1.0
1.0
Hinel Eval
1:00
0.51
0.40
0.58
0.58
0.57
0.60
Pol
1:00
0.38
0.46
0.71
0.28
0.78
0.55
ChatGPT
0.51
1.00
0.41
0.64
0.48
0.42
0.50
0.0
ChatGPT
0:30
1.00
0.40
0.51
0.19
0.46
0.65
LLaMA-7D
0.40
0.56
0.30
0.27
LLaMA-7B
0.65
0.23
0.56
0.48
Alpaca-78
BLOOM-7B
BLOOM-7B
Phoenix-7B
0.57
0.42
0.22
0.60
0.73
1.00
0.61
Phoenix-7B
0.78
8.46
0.58
0.76
0.32
1.00
0.58
Falcon-7B
0.60
0.56
0.41
0.75
0.62
0.61
1.00
Falcon-78
0.55
0.46
0.48
0.50
0.27
0.58
1.00
FineD-Eval
ChatGPT
Alpaca 78
BLOOM-70 79
Phoenix-78
Felcon-79
Alpacaa 78
BLOOM-77 79
Phoenit-78
Falcon 79
Figure Inter-Metric Pearson correlations on different datasets.


Reviews:
[{"review_id": "mi82DXOt1Q", "reviewer": "Reviewer_sT3H", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**RQ1**. \"The practicality and effectiveness of such evaluations in the field of multilingualism are not as useful and effective as other evaluation tools that examine the overall capabilities of large models.\"\n\n- Although there are other tools evaluating the overall capabilities of LLMs, **none of them specifically focus on dialogue, nor evaluate their capabilities for automatic assessments of dialogues**. \n\n- In the dialogue community, **automatic dialogue evaluation remains an important and open problem [1-5]**. Table 6 in the limitation section reveals that even GPT-4  (by far the most powerful LLM), doesn't perfectly align with human judgment on the FED-Dial benchmark (> 0.8), indicating a need for further exploration in LLMs' dialogue evaluation capabilities.\n\n- **Our paper complements existing LLM evaluation works and provides an important analysis on how existing LLMs perform automatic dialogue evaluation. We also examine their abilities to evaluate dialogues in the multilingual settings**. The evaluation of multilingual dialogues is also attracting significant interest. For example, the recent DSTC11 challenge (https://dstc11.dstc.community/) has one shared task dedicated to this topic. **The resources we introduced are important contributions to the NLG community**.\n\n**RQ2**. \"The actual relevance of evaluating the multilingual ability of large models needs to be carefully considered, especially given that the multilingual ability of large language models is generally not poor nowadays\"\n\n- We want to emphasize that **our works target automatic evaluation of multi-lingual dialogue, which is not just examining the multilingual ability of LLMs**. We also test their capabilities in understanding the dialogue contexts in a multilingual setting thereby providing accurate assessment.\n\n- Additionally, even though the multilingual ability of large language models is generally not poor, if the reviewer carefully checks existing LLMs, especially the open-source LLMs, **most of them are targeting popular languages such as English or Chinese**, however their performance in other languages is under-explored. \n\n- Lastly, current dialogue evaluation datasets are english-oriented, in this paper **we have also made an important effort to have a unified resource in 10 different languages**.\n\n_References_:\n\n[1] Yi-Ting Yeh, Maxine Eskenazi, and Shikib Mehri. 2021. A Comprehensive Assessment of Dialog Evaluation Metrics. In The First Workshop on Evaluations and Assessments of Neural Conversation Systems, pages 15–33, Online. Association for Computational Linguistics.\n\n[2] Huynh, Jessica, Cathy Jiao, Prakhar Gupta, Shikib Mehri, Payal Bajaj, Vishrav Chaudhary, and Maxine Eskenazi. \"Understanding the Effectiveness of Very Large Language Models on Dialog Evaluation.\" arXiv preprint arXiv:2301.12004 (2023).\n\n[3] Gupta, Prakhar, Cathy Jiao, Yi-Ting Yeh, Shikib Mehri, Maxine Eskenazi, and Jeffrey P. Bigham. \"InstructDial: improving zero and few-shot generalization in dialogue through instruction tuning.\" In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 505-525. 2022.\n\n[4] Eric Smith, Orion Hsu, Rebecca Qian, Stephen Roller, Y-Lan Boureau, and Jason Weston. 2022. Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents. In Proceedings of the 4th Workshop on NLP for Conversational AI, pages 77–97, Dublin, Ireland. Association for Computational Linguistics.\n\n[5] Mehri, Shikib, Jinho Choi, Luis Fernando D'Haro, Jan Deriu, Maxine Eskenazi, Milica Gasic, Kallirroi Georgila et al. \"Report from the nsf future directions workshop on automatic evaluation of dialog: Research directions and challenges.\" arXiv preprint arXiv:2203.10012 (2022).\n\n[6] Mehri, Shikib, and Maxine Eskenazi. \"Unsupervised Evaluation of Interactive Dialog with DialoGPT.\" In 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, p. 225. 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is well organized and clearly written.", "reasons_to_reject": "This paper presents a benchmark for evaluating multilingual dialogues and creates a multi-round dialogue dataset in up to ten different languages, providing an excellent tool for assessing the multilingual ability of large models. However, in my opinion, the practicality and effectiveness of such evaluations in the field of multilingualism are not as useful and effective as other evaluation tools that examine the overall capabilities of large models. Therefore, the actual relevance of evaluating the multilingual ability of large models needs to be carefully considered, especially given that the multilingual ability of large language models is generally not poor nowadays.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper establishes a benchmark for evaluating multi-lingual, multi-turn dialogues, creating a dataset that covers various languages to address the current lack of evaluation benchmarks for language. The study also establishes strong self-supervised and multilingual baselines.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "c7HpWPCgLR", "reviewer": "Reviewer_SAJN", "summary": "", "questions": "A. How are the human annotations for evaluation obtained? What are the scores from the metrics compared against to compute the correlation?\n(After the rebuttal, please also write this clearly in the experimentation section)", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**RQ1**. “The proposed dataset is an aggregation of different dialogue datasets focussing on different properties of dialogue - yet the emphasis is more on “coherence” and excludes other properties of dialogue evaluation.”\n\n- In the literature, “coherence” at the dialogue level [1-3] and “relevance” at the response level [4-6] are the most evaluated dimensions. **The focus of our paper is to examine the evaluation capability of automatic metrics on multilingual dialogues rather than multi-dimensional dialogue evaluation [7-9]**. Hence, it is reasonable to select these two dimensions for analysis so that we can easily benchmark the performance of LLMs against previous state-of-the-art approaches.\n\n- Additionally, among the 18 English human-annotated dialogue datasets used in our paper, **coherence and relevance are the common dimensions shared by all the datasets**. On the other hand, collecting unified multi-dimensional annotations for all the 18 datasets and analyzing multi-dimensional evaluation capability by itself is a major problem, which we are also working on in progress. However, it is beyond the scope of the current paper.\n\n**RQ2**. “Further, the proposed dataset, despite being an evaluation benchmark for multilingual dialogue, uses automatic translation APIs and quality evaluation is done only on automatic metrics.”  and “It is disputed whether LLMs can be used for translation quality evaluation as zero-shot , further a lot of segment level evaluation metrics are not yet equipped to do quality evaluation (https://aclanthology.org/2023.acl-long.730/) . As it is a dataset, it is essential to have some human evaluation to assess the quality.”\n\n- First, we would like to emphasize that **we are working on multilingual automatic dialogue evaluation instead of machine translation**. MT is a tool for us to extend existing English-oriented dialogue data to other languages so that we can examine the multilingual dialogue evaluation capabilities of the BERT-based and LLM-based metrics. \n\n- Additionally, **we use a state-of-the-art commercial translation system** and the translation quality of our data is high as indicated by the different evaluation metrics. Also, **we use the most powerful LLM, GPT-4, for quality estimation. It has been shown to highly correlate with human evaluation on many NLP evaluation tasks [10-12]**.  Some researchers even use the output of GPT-4 to guide the development of NLG evaluation metrics [13].\n\n- **Besides GPT-4, we also use sacreBLEU, BERTScore, and BLEURT to assess the translation quality**. We compare the source sentence with the back-translated sentence using these metrics. As shown in Table 2 of the paper, all the metrics suggest that the translation quality of our dataset is good. **We perform additional quality estimation with Unbabel’s “wmt22-cometkiwi-da”**. Please find the results of all automatic metrics in the table below. We can see that **all the automatic metrics indicate high translation quality**, thereby suggesting that our benchmark is a valuable resource for both the dialogue and multilingual community.\n\n    | Metrics | EN-Zh | EN-ES | EN-DE | EN-FR | EN-JA | EN-KO | EN-HI | EN-AR | EN-RU |\n    |-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n    | WMT22-cometkiwi-DA ↑ | 0.827 | 0.845 | 0.834 | 0.842 | 0.854  | 0.846 | 0.793 | 0.835 | 0.825 |\n    | GPT-4 ↑ | 4.58 | 4.64 | 4.63 | 4.59 | 4.33 | 4.26 | 4.40 | 4.39 | 4.40 |\n    | BLEU (with back-translation) ↑ | 37.85 | 56.58 | 50.30 | 52.10 | 42.70 | 40.23 | 49.33 | 48.54 | 49.13 |\n    | BLEURT (with back-translation) ↑ | 0.773 | 0.820 | 0.817 | 0.811 | 0.786 | 0.782 | 0.797 | 0.794 | 0.804 |\n    | BERTScore (with back-translation) ↑ | 0.691 | 0.767 | 0.754 | 0.748 | 0.711 | 0.707 | 0.726 | 0.737 | 0.741 |\n\n- **Additional human evaluation is performed**. Specifically, we collaborated with a service provider to have native speakers, proficient in English and their native language, evaluate the quality of translations from English to their mother tongue. Each reviewer assessed 350 translation pairs, rating them on a 1-5 scale, where 1 is poor and 5 is excellent. These evaluations covered nine language pairs, totaling 3150 instances. A quality check was done on a random subset to ensure reliability. For the English-to-Chinese translations, three experts evaluated them. The inter-annotator agreement was 0.578, indicating medium agreement among experts. **We can see that the human evaluation results below reinforce that the quality of our translated data is good**.\n\n    | Metrics | EN-Zh | EN-ES | EN-DE | EN-FR | EN-JA | EN-KO | EN-HI | EN-AR | EN-RU |\n    |-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n    | Human ↑ | 4.75 | 4.68 | 4.59 | 4.53 | 4.39  | 4.32 | 4.17 | 4.41 | 4.71 |\n\n**RQ3**. “The human annotation collected for computing the correlation is not clear - without this information, it is difficult to judge the usability of the dataset.”  and “How are the human annotations for evaluation obtained? What are the scores from the metrics compared against to compute the correlation?”\n\n- We extend existing English dialogue evaluation benchmark datasets to other languages with machine translation. **The original human annotations provided in each dataset are reused for the same responses or dialogues in other languages**. \n\n- All the relevant works of the 18 existing English datasets, which contain the human-annotation information, are cited in our paper. In fact, **most English dialogue evaluation datasets used in our study are well-known and widely adopted for benchmarking automatic dialogue evaluation metrics in existing literature**. Hence, descriptions of the human annotation information of the 18 datasets are supplementary and are not the main text of our work. In this paper, we are trying to comprehensive analyze how state-of-the-art BERT-based and LLM-based metrics perform on multilingual automatic dialogue evaluation.\n\n- **For the reviewer’s reference, we provide the human annotations information for Persona-USR [8] and Topical-USR [8] as an example**. Persona-USR [8] and Topical-USR [8] contain 300 and 360 dialogue context-response pairs respectively. Each response is annotated by three dialogue experts along six quality dimensions: Understandable (0-1), Natural (1-3), Contextual Relevance (1-3), Interesting (1-3), Uses Knowledge (0-1), Overall Quality (1-5). Note that the numbers in the bracket are the Likert scales. The inter-annotator agreements for quality dimensions of both datasets are generally good (> 0.5). For instance, the correlation score of Persona-USR is computed between metric scores assigned to the 300 context-response pairs and the average “Contextual Relevance” human-annotated scores of those pairs. \n\n**RQ4**. “025 to 031 and 574 to 582, correlation scale ranges from -1 to 1, please check if the improvement can be quantified in absolute percentages”\n\n- **Yes, it is a common practice in dialogue evaluation literature to report absolute improvements in terms of correlation scores [4]**. Existing dialogue evaluation metrics generally exhibit a positive correlation with human evaluation. During the human annotation process, we often use higher likert scale to indicate better dialogue or response quality.\n\n**RQ5**. “I am slightly skeptic about the results produced by fine-tuning. If similar translation methods are used for training and evaluation, it is likely that the model is getting biased towards synthetic data. I wonder if this is causing artificial gain (See https://arxiv.org/pdf/2201.13405.pdf, https://aclanthology.org/2020.emnlp-main.618/) A quick check on the DuRecDial2.0 dataset may be helpful https://aclanthology.org/2021.emnlp-main.356/”\n\n- First, the finetuning process is to adapt the LLMs to better understand the dialogue context. If the reviewer carefully checks Table 4, 5 and section 6.2, you may find that **when finetuning on the English synthetic dialogue data only, Alpaca-7B and Phoenix-7B also see improvements in both English and other languages compared to zero-shot evaluation, especially under the turn-level evaluation setting. This suggests that the improvements are due to the capturing of high-level dialogue information rather than translation artifacts**.\n\n- Second, **we use different translation models (not similar translation methods) to translate the synthetic training data and the test data respectively**. The test data is translated with high-quality commercial models while the synthetic data is translated by the open-source NLLB-200-3.3B model. The translation quality of NLLB-200-3.3B is obviously worse than the commercial model. Yet, even with a worse translation, finetuning leads to substantial gains over zero-shot evaluation (Table 5), suggesting that **the finetuning is actually robust against translation errors**.\n\n- Lastly, in the supplementary (Table 14 and section E in the paper), we also show **the performance of finetuned LLMs on natural human-annotated Chinese dialogues (data originally in Chinese, not translated from English)**. We can observe that the finetuned LLMs also perform quite well. For example, **Alpaca-7B finetuned on the multilingual synthetic data performs much better than zeroshot evaluation with ChatGPT**. \n\n- With the above reasons, **it is unlikely that there are artificial gains due to biases in the translation process**. \n\n_References_:\n\n[1] Mesgar, Mohsen, Sebastian Bücker, and Iryna Gurevych. \"Dialogue Coherence Assessment Without Explicit Dialogue Act Labels.\" In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1439-1450. 2020.\n \n[2] Zhang, Chen, Yiming Chen, Luis Fernando D’Haro, Yan Zhang, Thomas Friedrichs, Grandee Lee, and Haizhou Li. \"DynaEval: Unifying Turn and Dialogue Level Evaluation.\" In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5676-5689. 2021.\n\n[3] Ghazarian, Sarik, Nuan Wen, Aram Galstyan, and Nanyun Peng. \"DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulations.\" In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 771-785. 2022.\n\n[4] Yi-Ting Yeh, Maxine Eskenazi, and Shikib Mehri. 2021. A Comprehensive Assessment of Dialog Evaluation Metrics. In The First Workshop on Evaluations and Assessments of Neural Conversation Systems, pages 15–33, Online. Association for Computational Linguistics.\n[5] Sai, Ananya B., Akash Kumar Mohankumar, Siddhartha Arora, and Mitesh M. Khapra. \"Improving dialog evaluation with a multi-reference adversarial dataset and large scale pretraining.\" Transactions of the Association for Computational Linguistics 8 (2020): 810-827. \n\n[6] Zhang, Chen, Luis Fernando D'Haro, Qiquan Zhang, Thomas Friedrichs, and Haizhou Li. \"PoE: A Panel of Experts for Generalized Automatic Dialogue Assessment.\" IEEE/ACM Transactions on Audio, Speech, and Language Processing 31 (2023): 1234-1250.\n\n[7] Zhang, Chen, Luis Fernando D’Haro, Qiquan Zhang, Thomas Friedrichs, and Haizhou Li. \"FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation.\" In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3336-3355. 2022.\n\n[8] Mehri, Shikib, and Maxine Eskenazi. \"USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation.\" In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 681-707. 2020\n\n[9] Mehri, Shikib, and Maxine Eskenazi. \"Unsupervised Evaluation of Interactive Dialog with DialoGPT.\" In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 225-235. 2020.\n\n[10] Liu, Yang, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. \"Gpteval: Nlg evaluation using gpt-4 with better human alignment.\" arXiv preprint arXiv:2303.16634 (2023).\n\n[11] Naismith, Ben, Phoebe Mulcaire, and Jill Burstein. \"Automated evaluation of written discourse coherence using GPT-4.\" In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pp. 394-403. 2023.\n\n[12] Rathje, Steve, Dan-Mircea Mirea, Ilia Sucholutsky, Raja Marjieh, Claire Robertson, and Jay J. Van Bavel. \"GPT is an effective tool for multilingual psychological text analysis.\" (2023).\n\n[13] Xu, Wenda, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Yang Wang, and Lei Li. \"Instructscore: Towards explainable text generation evaluation with automatic feedback.\" arXiv preprint arXiv:2305.14282 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Good choice of languages for the dataset. \n\nGood number and types of LLMs used for comparison. \n\nOpen domain multilingual conversations have received less interest and this work is a contribution towards that direction", "reasons_to_reject": "The proposed dataset is an aggregation of different dialogue datasets focussing on different properties of dialogue - yet the emphasis is more on “coherence” and excludes other properties of dialgoue evaluation.\n\nFurther, the proposed dataset, despite being an evaluation benchmark for multilingual dialogue, uses automatic translation APIs and quality evaluation is done only on automatic metrics.", "questions_to_authors": "A. How are the human annotations for evaluation obtained? What are the scores from the metrics compared against to compute the correlation?\n(After the rebuttal, please also write this clearly in the experimentation section)", "paper_topic_and_main_contributions": "This work proposes xDial-Eval, an evaluation benchmark for multilingual open domain dialogue by machine translating 18 English open domain dialogue datasets into 8 languages. To demonstrate the usability of the dataset, several metrics BERT-based/generation-based LLMs are evaluated on the benchmark. The work also suggests ensembling metrics to be better than commercial LLM systems \n\nThe contribution is primarily a resource contribution followed by NLP engineering experiment.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "A. It is disputed whether LLMs can be used for translation quality evaluation as zero-shot , further a lot of segment level evaluation metrics are not yet equipped to do quality evaluation(https://aclanthology.org/2023.acl-long.730/) . As it is a dataset, it is essential to have some human evaluation to assess the quality.  \n\nB. 025 to 031 and 574 to 582, correlation scale ranges from -1 to 1, please check if the improvement can be quantified in absolute percentages\n\nC. Please mention the languages in the abstract\n\nD. I am slightly skeptic about the results produced by fine-tuning. If similar translation methods are used for training and evaluation, it is likely that the model is getting biased towards synthetic data. I wonder if this is causing artificial gain (See https://arxiv.org/pdf/2201.13405.pdf, https://aclanthology.org/2020.emnlp-main.618/)\nA quick check on the DuRecDial2.0 dataset may be helpful\nhttps://aclanthology.org/2021.emnlp-main.356/"}}, {"review_id": "vhwXgW3KBC", "reviewer": "Reviewer_8yTD", "summary": "", "questions": "1. How long will it cost to run the whole evaluation of the proposed benchmark with the Ensemble metric such as LLama-7B on a typical GPU?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**RQ1**. “One of the concerns about this work is that the impact of the machine translation models on multiple parts. The datasets are translated and the synthetic datasets for fine-tuning are translated. If some sampled instances from the benchmark can be evaluated by human, it could be better.”\n\n- We want to clarify that the **impact of machine translation quality is not significant and it is not the main concern of the paper**, because we are targeting evaluating coherence and contextual relevance in multilingual dialogues. If the reviewer checks the correlation results in Table 4 and Table 5, we can observe that **even just finetuning the models on the original English synthetic data , which are free from machine translation errors, brings improvements in different languages**. This suggests that the finetuning does help the models to capture high-level dialogue representation rather than surface-level syntax. \n\n- Additionally, **we are using state-of-the-art commercial translation service to translate the English evaluation data, the quality of the translation is very good as indicated by multiple automatic evaluation measures**. As shown in the table below, the score range of GPT-4 is from 1 to 5, that of BLEU is from 0 to 100, and that of WMT22-cometkiwi-DA, BLEURT, and BERTScore is from 0 to 1. We can see that all automatic metrics’ scores are high.\n\n    | Metrics | EN-Zh | EN-ES | EN-DE | EN-FR | EN-JA | EN-KO | EN-HI | EN-AR | EN-RU |\n    |-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n    | GPT-4 ↑ | 4.58 | 4.64 | 4.63 | 4.59 | 4.33 | 4.26 | 4.40 | 4.39 | 4.40 |\n    | WMT22-cometkiwi-DA ↑ | 0.827 | 0.845 | 0.834 | 0.842 | 0.854  | 0.846 | 0.793 | 0.835 | 0.825 |\n    | BLEU (with back-translation) ↑ | 37.85 | 56.58 | 50.30 | 52.10 | 42.70 | 40.23 | 49.33 | 48.54 | 49.13 |\n    | BLEURT (with back-translation) ↑ | 0.773 | 0.820 | 0.817 | 0.811 | 0.786 | 0.782 | 0.797 | 0.794 | 0.804 |\n    | BERTScore (with back-translation) ↑ | 0.691 | 0.767 | 0.754 | 0.748 | 0.711 | 0.707 | 0.726 | 0.737 | 0.741 |\n\n- **For the reviewer’s reference, we have conducted additional human evaluation**. We collaborated with a service provider to have native speakers, proficient in English and their native language, evaluate the quality of translations from English to their mother tongue. Each reviewer assessed 350 translation pairs, rating them on a 1-5 scale, where 1 is poor and 5 is excellent. These evaluations covered 9 language pairs, totalling 3150 instances. A quality check was done on a random subset to ensure reliability. For the English-to-Chinese translations, three experts evaluated them. The inter-annotator agreement was 0.578, indicating medium agreement among experts. **The human evaluation results below agree with the automatic measures that the translation quality of the evaluation data is good**.\n\n    | Metrics | EN-Zh | EN-ES | EN-DE | EN-FR | EN-JA | EN-KO | EN-HI | EN-AR | EN-RU |\n    |-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n    | Human ↑ | 4.75 | 4.68 | 4.59 | 4.53 | 4.39  | 4.32 | 4.17 | 4.41 | 4.71 |\n\n- Lastly, it is true that high-quality translation helps better finetuning of the models. However, even finetuning on multilingual synthetic data, whose translation quality is worse than that of the evaluation data, we can still observe significant correlation improvements in table 5 across different languages. **This observation reinforces the idea that the impact of machine translation is not significant on multilingual dialogue evaluation**.\n\n**RQ2**. “Regarding the proposed metric which combines LLMs and FineD, one concern is the computational limitations of the open-source LLMs and the training of the BERT-style model, especially for 9 languages of 18 datasets.”\n\n- We want to clarify that **the models are not trained on the 18 evaluation datasets**. We are finetuning the models on a synthetic multilingual dialogue dataset, which is automatically constructed. Hence, our methods are based on self-supervised learning. After model finetuning, the same models will be applied to evaluate the 18 datasets spanning acorss 10 different languages.\n\n- As stated in our paper, the total size of the synthetic training data is 200K. The finetuning of RoBERTa model can be easily completed with a single NVIDIA 3090TI 24GB GPU while the low-rank adaptation (LoRA) finetuning of a LLaMA-7B model can be completed with a single NVIDIA 4090TI 40GB GPU. Additionally, as stated in section F of the supplementary, the time taken for LoRA finetuning of a 7B LLM takes roughly 100 GPU hours while the finetuning of RoBERTa is much faster, typically, it takes less than 24 GPU hours. **Hence, the computation costs are not expensive**. \n\n**RQ3**. “How long will it cost to run the whole evaluation of the proposed benchmark with the Ensemble metric such as LLama-7B on a typical GPU?”\n\n- We have 14930 x 10 = 149300 annotated turns and 8691 * 10 = 86910 annotated multi-turn dialogues in the benchmark. When inference with LLaMA-7B on a single NVIDIA 3090TI 24GB GPU, we set the batch size to 1. \n\n- For turn-level evaluation,  the inference speed of LLaMA-7B is 4.65 instances per second on average. Hence, it takes around $149300 / 4.65 / 3600  \\approx 8.92$ GPU hours to complete. For dialogue-level evaluation, the inference speed of LLaMA-7B is 3.17 instances per second on average. Note that the speed is slower than that of turn-level evaluation due to the longer dialogue contexts to process for multi-turn dialogues. Hence, it takes around $ 86910 / 3.17 / 3600 \\approx 7.62 $ GPU hours to complete. In total, it costs $ 8.92 + 7.62 = 16.54 $ GPU hours to finish inference on the entire benchmark.\n\n- For the RoBERT-based metric, the inference speed is much faster due to its much smaller parameter size and larger inference batch size. For example, if we use a batch size of 16, the inference speed for dialogue-level evaluation is 20.8 instances per second and it takes $  86910 / 20.8 / 3600 \\approx 1.16 $ GPU hours to complete the entire dialogue level evaluation. For turn-level evaluation, the inference speed is 44.8 instances per second and it takes $  149300 / 44.8 / 3600 \\approx 0.93 $ GPU hours to complete the entire turn level evaluation. The total cost will be $ 1.16 + 0.93 = 2.09 $ GPU hours to finish inference on the entire benchmark.\n\n- **In summary, the costs of ensemble the two metrics are much lower compared to inference with commercial LLMs, such as ChatGPT, in terms of both money and inference time**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper constructs xDial-Eval which contains 12 turn-level and 6 dialogue-level open-source datasets. The original English version datasets are translated into nine different languages with commercial machine translation models.\n- This paper assesses current discriminative and generative reference-free metrics on the proposed multilingual benchmark. The most recent LLMs are also evaluated in this paper.\n- This paper introduces an ensemble metric for the proposed multilingual benchmark by combining generative and discrimination models.", "reasons_to_reject": "- One of the concerns about this work is that the impact of the machine translation models on multiple parts. The datasets are translated and the synthetic datasets for fine-tuning are translated. If some sampled instances from the benchmark can be evaluated by human, it could be better.\n- Regarding the proposed metric which combines LLMs and FineD, one concern is the computational limitations of the open-source LLMs and the training of the BERT-style model, especially for 9 languages of 18 datasets.", "questions_to_authors": "1. How long will it cost to run the whole evaluation of the proposed benchmark with the Ensemble metric such as LLama-7B on a typical GPU?", "paper_topic_and_main_contributions": "- What is this paper about? This paper aims to provide a new multilingual open-domain dialogue evaluation benchmark including datasets, evaluation methods, and current metrics analysis.\n- what contributions does it make? This paper collects 9 languages versions of multiple dialogue datasets with machine translation, analyzes current reference-free metric performance on the proposed datasets, and proposes new metrics.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents xDial-Eval, an evaluation benchmark for multilingual open-domain dialogue systems. In addition to its contribution as a new evaluation resource, the paper also shared insights into applying LLMs to automatic dialogue evaluation. 

Two reviewers pointed out the need for human evaluation on the automatic machine translation quality during the construction of the multilingual dataset, to which the authors' rebuttal provided some further details on the conducted human evaluation results. However, given these new results were missing from the original paper submission, and somewhat in contradict with the description under section 3, 
they are not considered as sufficient to override to the original review criticism. 

One of the reviews was not taken into consideration for the AC recommendation due to the lack of details and no reviewer response during rebuttal period.|meta review
Decision: Accept-Findings