FFAEval: Evaluating Dialogue System via Free-For-All Ranking
Authors: Zeyao Ma
Zijun Yao
Jing Zhang
Jifan Yu
Xiaohan Zhang
Juanzi Li
Jie Tang
Venue: EMNLP 2023 Findings (2023)
Keywords: Human Evaluation
Dialogue System Evaluation

Abstract: Evaluating open-domain dialogue systems is currently an open question.  Automatic evaluation metrics have shown poor correlation with human assessment in dialogue generation tasks. Human evaluation, which involves annotators for multi-dimension scoring, is trustworthy but time-consuming. In this work, we propose FFAEval, a reliable and efficient human evaluation framework using Free-For-All ranking approach. By sharing the dialogue history, the framework enables annotators to converse with multiple dialogue systems simultaneously in a single-blind, multi-turn manner. The subsequent free-for-all allows annotators to select the most favourable model in each turn from among all the participating dialogue systems. The final performance of each model is represented by calculating the TrueSkill score derived from the free-for-all competition. Our empirical study on English and Chinese dialogue systems demonstrates that FFAEval achieves a strong correlation with score-based human assessment compared to existing evaluation methods. We further prove the efficiency and stability of our framework in additional experiments. The source code and data are available on Github.

FFAEVAL: Evaluating Dialogue System via Free-For-All Ranking
Zeyao Ma Zijun Yao ,Jng Zhang 1.2.t Jifan Yu²,
Jifan
Xiaohan Zhang', Juanzi Li .iie Tang' 'School of Information, Renmin University of China, Beijing, China "Engineering Research Center of Database and Business Intelligence, MOE, China DDpartmentt of Computer Science and Technology, Tsinghua University, Beijing, China *Zhipu.AI
zeyaoma, zhang-jing] @ruc.edu.cco en, (yaozj23, yujf21 ) Staohan.zhang @mminer.cn, [juanzi, jietang) @ tsinghua.edu.co
@mails.siinghua..du.com
Abstract
Free-For
History
your snrgor?
Evaluating open-domain dialogue systems is currently an open question. Automatic eval- uation metrics have shown poor correlation with human assessment in dialogue generation tasks. Human evaluation, which involves an- notators for multi-dimension scoring, trust- worthy but timeeccnnu.inng In this work, propose FFAEVAL, a reliable and efficient hu- man EVAttation framework using Free-For- All ranking approach. By sharing the dialogue history, the framework enables annotators to converse with multiple dialogue systems simul- taneously in single-blind, multi-turn manner. subsequent free-for-all allows annotators select the most favourable model in each turn from among all the participating dialogue sys- tems. The final performance each model is represented by calculating the TrueSkill score derived from the free-for-all competition. Our empirical study on English and Chinese di- alogue systems demonstrates that FFAEVAL achieves a strong correlation with score-based human assessment compared to existing evalu- ation methods. We further prove the efficiency stability of our framework additional experiments. The source code and data available on Github?.
Nor minxart
thoee
Japonenc pup singas.
Krleesst and infomatv
User' Literance
Fhink Michael Jailoon
inspiiig
Best Repby 2 flom Boe'
Iderk froanic achelly.
Uver' Viteranne
Why lase Michael Jackars
Invoke
Recosse
Americo and dancer.
Candidoe
t Ru
favoric tine.
Rephe X
Reply
Reply 7
Whot vahur do you like hos about hime?
Curdilane
Thiillr. It's one of de nost popslar alsunss
B
1 Idan' kuann what albun le has
Shuffie
Hund C Califonia. Pve hand max tina.
before repbiig)
Figure I: The evaluation process of a single Free-For-
All match.
carrying unique information and purposes. How to accurately and impartially evaluate these diverse responses remains a pressing issue.
Existing methods for evaluating dialogue sys- tems primarily fall into two categories: auto- matic evaluation and human evaluation. For automatic evaluation, metrics are derived by comparing the system-generated response with a pre-annotated utterance, e.g., ROUGE (Lin, 2004), BERTScore (Zhang et al., 2020a), and BARTScore (Yuan et al., 2021). However, as there is no definitive 'correct' answer for dialogue re- sponse, these metrics fail to recognize viable re- sponses that diverge from the reference. While using LLMs score responses can mitigate this issue (Bai al., 2023; Chiang et al., 2023), they tend to bias towards self-similar responses (Liu et al., 2023b). Thus, human assessment, involving annotators interacting with the dialogue system and
1 Introduction
Designing reliable and efficient evaluation methods serves as a beacon for the improvement of open- domain dialogue systems. becomes increasingly challenging in this era dominated by dialogue sys- tems that are typically built upon generative large language models (LLMs) (Shuster et al., 2022; Zhang et al., 2023). These systems generate open- ended responses from various perspectives, each
"Equal Contribution.
"oorrepponding author.

---

rating its responses according to guidelines (Finch and Choi, 2020), remains the de facto gold stan- dard.
based human evaluation results. We also conduct comprehensive analysis on FFAEVAL, and find that to obtain a trustful evaluation score upon 5 dif- ferent English dialogue systems using FFAEVAL only requires 175 round conversation. More- over, FFAEVAL is less sensitive to match order. We further conduct an experiment estimate the confidence intervals of both Chatbot Arena and our framework and discover that FFAEVAL is much more stable than Chatbot Arena.
Nevertheless, human evaluation presents its own challenges. It proves exceedingly difficult and time-consuming for annotators to complete ques- tionnaire demanding subjective scores (Louviere et al., 2000). Recent approaches aim to approxi- mate these scores by having annotators choose pre- ferred models from pair of options (Deriu al., 2020; Zheng al., 2023). However, the pairwise model comparison still requires a considerable vol- ume of annotations to converge to a stable score.
FFAEVAL Framework
In general, FFAEVAL treats each multi-turn con- versation multiplayer competition, in which participating dialogue systems seek gain recogni- tion from human users. FFAEVAL comprises three main components: (1) human- interaction mechanism between evaluators and dialogue sys- tems to obtain the competition ranking for each con- versation; (2) A dialogue history sharing mech- anism to ensure fair context for response gener- ation; (3) A global ranking scoring algorithm to produce the global ranking by synthesizing all the historical competition rankings. This rest of this section will introduce these modules detail. The overall framework of FFAEVAL demonstrated in Figure 2.
In this paper, we present FFAEVAL, a human EVVALuation framework for dialogue system that uses Free-For-All ranking approach. The core idea is to enable annotators to converse with multiple dialogue systems simultaneously in single-blind, multi-turn fashion by sharing the dialogue history. The subsequent free-for-all allows annotators to select the most favourable model in each turn from among all the participating dialogue systems. The final performance of each model is represented by calculating the TrueSkill (Herbrich et al., 2006) score derived from the free-for-all competition. We demonstrate FFAEVAL in Figure 1. Contrary to bcsre-based human evaluation, FFAEVAL only re- quires selection, which is more efficient and less labour-intensive (Louviere al., 2000).
2.1 Human-Ban Interaction
FFAEVAL shows unique advantages to other selection- evaluation methods for dialogue systems (e.g., Chatbot Arena (Zheng et al., 2023)) from 3 aspects. First and most importantly, FFAE- VAL maintains a shared dialogue history for every participating dialogue system, which effectively prunes the unfairness introduced when the annota- tor implicitly replies to one of the dialogue systems; Second, opposed to pairwise comparison and conversation- selection, FFAEVAL compares all the candidate participates together for each turn response, which makes the evaluation result of FFAEVAL. converge faster and more fairly; Third, FFAEVAL derives the TrueSkill score from free- for-all ranking. which is less sensitive to the order of competition, making the ranking more robust.
Human users engage in three types of interaction with the dialogue system. First, they converse with all the dialogue systems simultaneously with mul- tiple turns. Second, they are guided to choose the best model response for each conversational turn. Finally, they decide when to end the conversation, enabling FFAEVAL to finalize the ranking of each dialogue system based on that conversation.
Multi-turn Human-Bod Conversation. FFAE- VAL prompts human users to initiate the dialogue. In each turn, the message of the user is appended to the end of the shared history. We structure the di- alogue history into the following prompt template, which then provided to each dialogue system as input to elicit their responses:
We conduct experiments on both Chinese and English dialogue systems powered by LLMs. Com- pared a series of automatic evaluations and other selection-based human evaluation methods, the evaluation score from FFAEVAL demonstrates its superiority in terms of consistency with score-
[User Utterance 1] <SEP>
[Selected Response 1] <SEP>
[User Utterance 2] <SEP>
[Selected Response 2]
<SEP>
[User
Utterance n]

---

Shared History
Human-Bon Bont Interaction
Leaderboard Revise
Users
to your fivarit singer?
Leaderboard (before)
Bes Reply
from
Free-For-All Ranking
peefer Japanese pup singen.
Finalize
Bot
User'
Chink Michael Jocksen
PM
Invoke
don't have favourite one, actialyy
Why you love Miclaud Iuckam!
TrueSkill
Respatse Geveration
Hot Pool
Ameyminaaiinn
Bot
Reply X
Reply A
A
Decouse his voice very attractive.
Leaderboard
Bot
Reply
Reply B
was singer
Bot
2
Bix
23.29
Bel
Reply Z
Reply C
Rezase Eovorite anisi all time.
Bet 21.32
Figure 2: The general human evaluation framework for multiple dialogue systems using FFAEVAL.
The token <SEP> is replaced with model-specific tokens before being sent to the conversation model, egg., \n for BlenderBot and <sep> for EVA2.0. Meanwhile, template tokens within [Brackets]] are replaced with corresponding content from the dialogue history. To speed up the response genera- tion process, we employ multiple threads to gather responses from each model simultaneously.
turn, the dialogue system chosen by the user is awarded 1 point, while the others receive none. The user free end the conversation any point. FFAEVAL accumulates these points to generate a Icaderboard that ranks each dialogue system based on its score in the conversation. This leaderboard is then used to compute the global ranking score. It is worth noting that, we empirically find allow- ing users to select multiple responses has slight negative impact on the model evaluation results because it becomes easier for many dialogues sys- tems receive similar points. Thus, FFAEVAL only prompts users to select exactly one model.
Best Response Selection. After all the partic- ipated dialogue systems generate their utterance, the user asked to choose the response that is most appropriate to continue the conversation. We implement a series of mechanisms to ensure fair evaluation. For anonymization, the responses are shuffled and post-processed before being shown to the user. During pratpprccessigg. FFAEVAL ap- plies rules to identify and replace spans that could potentially reveal the identity of the dialogue sys- tem with placeholders. Moreover, to mitigate the primacy effect (Malhotra, 2008)-that is, users pay- ing more attention to the first presented response- FFAEVAL presents all responses simultaneously. Although selecting appropriate response is the casiest way to use FFAEVAL, it lacks strict def- inition with explicit evaluation guideline. Thus, we also refer the score-based human evaluation principles, and ask the annotators to consider the contextual coherence, informativeness of the re- sponse, and veracity of the content. Furthermore, multi-dimensional scoring puts constraints on an- notators within pre-designed rules. In this way, FFAEVAL could not only approximate gold stan- dard score-based evaluation methods, but also pro- vides better way to assess the overall ability of the dialogue model.
2.2 Shared Dialogue History
The shared dialogue history is the key component that enables user to converse with multiple dialogue system simultaneously. As different dialogue sys- tem generate replies with potentially different top- ics to the same human utterance, the dialogue sys- tem can encounter what we term cascading bias dialogue systems that are not selected in the initial turn would have contradictory dialogue history, which consequently cause them to perform poorly in future turns. For instance, illustrated in Fig- ure 1, without a shared dialogue history, two bots not selected the first turn would confused by the user response, which could lead to disorganized replies and, thereby, degrade the points earned by these two bots.
Technically, the shared dialogue history func- tions as stack, and its push-back operation is trig- gered by two events. First, when the user finishes their input, their response is appended to the stack. Sccond, once the user selects a response from the presented choices, the response from the chosen
Conversation Finalization. In each dialogue

---

dialogue system is appended to the stack.
and contains 345M parameters. BlenderBot-3B (BB-3B) (Shuster et al., 2022), initially pre-trained on Reddit comments, is later fine-tuned on man- ually annotated dialogue data to improve perfor- mance across various domains, and possesses 3B parameters. We have also evaluated BlenderBot- 90M (BB-90M), a variant model with 90M parame- ters. PLATO-X -L (Bao et al., 2022), trained on extensive collection of social media data from the Internet, incorporates 1 1B parameters.
2.3 Global Ranking Score
The global ranking score quantifies the skill level cach model after a series of conversations. TrueSkill Estimation. Inspired by algorithms tracking the skill level of players in games, FFAE- VAL. calculates TrueSkill (Herbrich et al., 2006) score as the ranking score. Compared to other rat- ing algorithms such as Elo (Glickman and Jones, 1999), which used by Chatbot Arena, TrueSkill presents three major advantuges: (1) TrueSkill is designed for free-for-all involving any number of players, whereas Elo is intended for 1vl matches; (2) TrueSkill tracks the estimated variance of the score, providing a confidence interval for the es- timated ranking score; (3) TrueSkill adjusts sig- nificantly when the algorithm is less certain, thus converging with fewer conversations.
To affirm that our FFAEVAL remains effective across multiple languages, we have also conducted experiments Chinese dialogue systems, which include: CDial-GPT (Wang et al., 2020), GPT- structured model with 95.5M parameters, which is fine-tuned on the LCCC dataset. EVA2.0 (Zhou et al., 2021), a generative conversation model with 2.8B parameters, trained using 1.4T dialogues sourced from social media. PLATO-2 (Bao et al.. 2021), a 336M language model, is trained us- ing curriculum learning techniques 1.2B exam- ples from social media conversations. XDAI (Yu et al., 2022), a tuning-free dialogue model based on WLLM-10B-Chines which additionally maintains an external knowledge base to enhance its capabili- ties. GLM-Dialog (Zhang et al., 2023), a retrieval augmented dialogue system with 10B parameters.
In particular, TrueSkill models the ranking score of each participants as a Gaussian distribution with two parameters, namely, the mean score μ and the standard deviation σ. FFAEVAL initiate the pa- rameters from = 25, a 3. as suggested by Herbrich et al. (2006). After each conversation gen- erating its corresponding ranking result, TrueSkill uses Bayes' law update the ranking score.
25
Leaderboard. After sufficient number of con- versations, the score of each dialogue system is computed using the following equation:
3.2 Baselines Evaluation Methods
We compare FFAEVAL against both automatic evaluation metries and human evaluation methods. For automatic metries, we consider: F1 and BERTScore (BS) (Zhang al., 2020a) are reference-based metrics which compute the similar- ity between -enerreed and human responses. ChatMatch (CM) (Yang et al., 2022) calculates multiple scores for conversation contents between two different dialogue systems. This pair-wise match generates an overall ranking via round-robin tournament.
S= k x
This is a conservative skill estimate that gives a lower bound for the possible real skill. FFAEVAL follows Herbrich et al. (2006) to sets h = 3. The leaderboard is updated by the order of skill score.
3 Experiments
We conduct extensive experiments to verify the effectiveness of FFAEVAL in evaluating generative dialogue systems.
For human evaluation, we mainly focus on selection-basee methods. Chathot Arena (CA) (Zheng et al., 2023) enables users to converse with two anonymous models hide-by-side and vote for which one is better. The rating of each model is updated via the Elo rating system. Preference ranking (PR) is our implemented baseline, which is based on the method designed to collect com- parison data to train reward model (Ouyang et al., 2022). It asks annotators to rank the generated re- sponses conditioned on a same dialogue context
3.1 Dialogue Systems to Evaluate
We deploy English dialogue systems on NVIDIA GeForce RTX 3090, utilizing their correspond- ing public souree codes. These models include: Seq2Seq represents our implementation of the BART model (Lewis et al., 2020), which fine- tuned on DailyDialog (Li et al., 2017). Di- aloGPT (Zhang et al., 2020b) fine-tunes the GPT-2 model (Radford et al., 2019) on Reddit comments

---

without multi-turn human-bot interactions. We uti- lize TrueSkill calculate the ranking result of each conversation as the score of each dialogue system. PR is different from FFAEVAL in that it is static without shared dialogue history and can only eval- uate single-round conversation.
English
Chinese
Method
p
F1 BERTScore
40%
27.3%
20%
17.5%
0%
5.2%
40%
30.2%
ChalMatch
20%
38.44
40%
13.5%
Preference Ranking
60%
84.2%
60%
77.1%
3.3 Evaluation Details
Chatbot Arena
60%
55.0%
80%
83.5%
FFAEVAL
100%
97.7%
100%
98.6% %
We consider the following miscellaneous to setup experiments for FFAEVAL.
Table Overall correlation between the result of dif- ferent methods and gold standard scores on English and Chinese experiment. T and stand for Kendall ranking correlation and Pearson score corrllation, respectively.
Human Resources. We recruit 20 undergrad- uate students, who use part time to annotate de- ployed dialogue systems with FFAEVAL. To en- sure consistency among different human evaluation methods and the overall metric of the GS score, these annotators are asked to evaluate following the same unified guidelines as provided in Appendix A. Besides, all the human evaluation methods ask the annotators to evaluate the same number of turns of responses, which guarantees the convergence of all the evaluation methods.
ranking correlation 7 and Pearson correlation p. Methods with a higher correlation are considered to have better reliability.
3.4 Main Results
We are mainly concerned about two questions: Does FFAEVAL generate accurate rankings on the leaderboard, and to what extent does the ranking score of FFAEVAL correlates with the gold stan- dard evaluation score?
Data Resources. The reference-based metrics are calculated on DiilyDialog (Li et al., 2017) and Diamante (Lu et al., 2022) datasets for English and Chinese models, respectively. Both datasets are also used as seed topics to prompt annotators to start conversations.
Leaderboard Rankings. Figure illustrates the ranking results of all dialogue systems un- der various evaluation methods, with the gold standard ranking featured in the far-right column. Our experimental findings suggest that FFAEVAL yields rankings identical to those of the score- based evaluations when assessing both English and Chi- models. As comparison, both reference- based metries-namely Fl and BARTScore and reference-free metrics, i.e., ChatMatch, do not suc- ceed accurately approximating the ranking from the gold standard rankings.
Gold Standard (GS) Score. We consider hu- man assessed scores to be the gold standard for evaluating dialogue systems, as this method is fre- quently employed by researchers (e.g., (Bao et al., 2022). (Zhang et al., 2023)). In particular, on each occasion, we begin by randomly selecting initial utterance from DailyDialog (Li et al., 2017), Dus- ine (Zhou et al., 2022), and Diamante (Lu al., 2022). After making necessary modifications, this utterance is subsequently employed by all dialogue systems to initiate human-bot conversation. Then, we ask another group of annotators to assign scores, including the overall performance and other dimen- sions. We use the overall scores as the gold stan- dard and consider other dimensions as references in Appendix B. Each dialogue score is an averaged over three annotators.
Ranking Scores. We further explore whether the TrueSkill score of FFAEVAL provides de- tailed approximation from score-based human as- sessments. The correlation between various ap- proximation scores and the gold standard score is presented in Table 1. We find that all methods based on human evaluation outperform automatic evaluation methods, emphasizing the importance of human involvement when assessing dialogue sys- tems. Among all the human evaluation methods, FFAEVAL achieves the highest Pearson correla- tion at 97.7% in evaluating English models and 98.68 6% evaluating Chinese models, which is a large margin surpassing all baseline methods.
We use Pearson's 's correlation to calculate inter- annotator agreement (IAA) score of ground truth annotation. The annotations are reliable, as the IAA scores for the Chinese and English experiments are 0.63 and 0.59, respectively. We calculate the correlation coefficients between each evaluation method and the ground truth, including Kendall

---

5
3
2
F1
B5
CM
PRR
CA
Ours
GT
F1
CM
PR
CA
Ours
BART
88-90m 38-90m andio DialoGPT
PLATO-XL
98-3b
CDal-GPT
EVA ondeo PLATO-2
XOAI
ende GLM-Dialog
The ranking of dialogue systems provided by each methods. Left and right figure show the result of English and Chinese Experiment, respectively.
Figure 3:
Of all evaluation methods, ChatMatch performs the worst, demonstrating negative correla- tion with the gold standard ranking results. This is due to the fact that ChatMatch evaluates dia- logues between two dialogue systems, which often devolve into content lacking information. This indicate that it is necessary to evaluate conversa- tions between humans and dialogue systems, as adopted by FFAEVAL.
Annotation Time
Method
English
Chinese
Preference Ranking
-415 min
-335 min
Chathot Arena
-665 min
-500 min
Gold Standard
-530 min
-500 min
FFAEVAL
-250 min
-300
Table 2: The annotation time of different human evalua- tion methods English and Chinese experiments.
Preference Ranking, which ranks dialogue sys- tems based single-turn conversations, serves ablation experiment evaluate the effect of multi-ture conversation in the assessment pro- cess. FFAE achieves 13.5% and 21.5% higher Pearson correlation in English and Chinese, respec- tively, thereby underlining the importance of per- mitting a series of interactions between the annota- tors and the dialogue system.
equivalent number of responses (Time Efficiency), and the number of responses needed for the scores to converge (Convergence Efficiency).
Time Efficiency. conduct a fair time effi ciency comparison among FFAEVAL and baselines by recording the time elapsed to annotate 2000 re- sponses, which we empirically find sufficient for all the methods converage to stable score. For each human evaluation method, we log the entry and exit times of each annotator in the online sys- tem. We are able calculate the time spent by each annotator and obtain the total annotation time through summation as shown Table 2.
We also concerned about the comparison be- tween FFAEVAL and Chatbot Arena, both of which transform preference-basee human evaluation into competitions for dialogue systems. FFAEVAL out- performs Chatbot Arena by a margin of 42.7% in English and 15. 15.1% %% in Chinese. The advantage of FFAEVAL lies in its free-for-all competition for- mat, which exhibits no sensitivity to the competi- tion order. This format allows for a more accurate approximation of the gold standard ranking score due to its smaller deviation.
FFAEVAL. demonstrated a significant improve- ment in time efficiency among all the evaluation methods. This is due FFAEVAL allows the anno- tators to evaluate multiple dialogue systems simul- taneously. Chatbot Arena uses much more time evaluate the same amount of dialogues compared with our framework. This is due to the fact that Chatbot Arena only allows annotators to converse with two dialogue systems side-by-side. More- over, in Chatbot Arena, the annotator must concen- trate two different dialogue contexts, whereas in FFAEVAL, only one shared history is required. Convergence Efficiency. We conduct experi- ments to compare the convergence efficiency be-
4 Analysis
We analyze the characteristics of FFAEVAL and the feasibility each designed components.
4.1
Efficiency Analysis
It is also important for FFAEVAL to provide highly efficient human evaluation. We measure efficiency from two aspects- the time required to evaluate an

---

ues of Chatbot Arena are relatively dispersed. This result demonstrates that FFAEVAL insensitive to match order and thus more stable comparison to Chatbot Arena, due to the free-for-all ranking of our framework, as the ratings of all dialogue systems are updated after each match.
1.0
0.8
Probability
0.6
Chatbot (en)
0.4
FFAEval (en)
4.3 Turn Number Analysis
Chatbot Arena (2h)
0.2
FFAEval (zh)
In this section, we discover the influence of dif- ferent maximum turn number on the final ranking result. Following the same sampling process convergence efficiency analysis, we calculate the convergence result of free-for-all by setting differ- ent maximum number of turns.
750
1750
Number of responses
Figure 4: Ranking stability towards the number of re- sponses. The x-axis represents the number of responses generated by dialogue systems. The y-axis represents the rate which the same ranking is achieved across 100 repetitions.
Figure 6 presents that, free-for-all with only and 2 turn is not able to achieve the same ranking with the original result. This is due to the fact that some dialogue systems excel at single-round con- versations but perform poorly during long conversa- tions. The convergence speed of conducting 5 turns in free-for-all match is faster than 4 turns. As the maximum turns in a free-for-all match increase, the results of single match can more accurately reflect the true level of each dialogue system. On the one hand, more turns can demonstrate the abil- ity of the dialogue system engage in multi-turn conversation. On the other hand, more turns can reduce the impact of possible annotation bias.
tween free-for-all competitions, as adopted by FFAEVAL, and pair-wise competitions, as adopted by Chatbot Arena. In particular, we use FFAE- VAL and Chatbot Arena to evaluate dialogue sys- tems until reaching stable ranking result. From these annotation results, randomly sample re- sponses and use their ranking algorithm to produce the ranking result. The trait repeated for 100 times to calculate the possibility that ranking with n responses yields identical ranking as the stable result. We range n from 250 to 2, 000, and plot the possibilitties on Figure 4.
4.4 Responses Selection Mechanism Analysis
The convergence of FFAEVAL is faster than that of Chatbot Arena in both English and Chi- nese experiments. There are two possible factors that count for this result: (1) Compared to Chat- bot Arena, which evaluates dialogue systems side- by-side, FFAEval evaluates all models in single match, which is much more efficient and provides more thorough comparison of dialogue systems. (2) The evaluation process of FFALEal is more fine-grained, as annotation is involved in each turn conversation, whereas Chatbot Arena evaluates after several turns.
In FFAEVAL, the annotator only selects one sponse in each round of conversation, which may appear imprecise given that multiple optimal re- sponses may exist. We modify the best response selection procedure to permit annotators to choose multiple responses for each dialogue turn. To main- tain the shared dialogue history, when the annotator selects multiple responses, we select one at random and use it the shared history.
In English experiment, this modified version of FFAEVAL yields the same ranking as the original, with a slight decrease Pearson's 's correlation from 97.7% to 91.6%. This result shows that only select one response is enough to achieve satisfactory results. The decreased is caused by the fact that annotators tend to give credit to many dialogue sys- tems, making the ranking score less discrimitive.
4.2 Sensitivity Analysis
this section, we discuss the sensitivity of Chatbot Arena and FFAEVAL to match order. Specifically, we shuffle the competition results for 1, 000 times to estimate the confidence intervals. For fair com- parison, the ranking scores are normalized to [0, 1]. As shown in Figure 5, our framework is much more stable than Chatbot Area, as the values of FFAEVAL converge to a small range while the val-
4.5 Shared Dialogue History Analysis
We finally conduct experiments to examine the ef- fectiveness of the shared dialogue history. First,

---

ChatbotArena
Chatbot
1.0
1,0
Normlized Raiing
FFAEval
Normllzes Raiing
FFAEval
0.9
0.9
0.8
BART
BB-00m
DiabGPT Model
PLATO-XL
BB-3b
CDsl-GPT
EVA
PLATO-2 Model
XDAI
GLM-Dialog
Figure 5: The confidence interval of the normalized ranking score between Chatbot Arena and FFAEVAL for different match orders. The confidence interval is estimated on 1, 000 randomly shuffled match orders.
precision of the model-generated responses using human references. METEOR (Banerjee and Lavie, 2005) and ROUGE (Lin, 2004) improve BLEU by involving stems and synonyms result calcu- lation and replacing n-gram precision with recall, respectively. ADEM (Lowe et al., 2017) trains a recurrent neural network to predict the quality of system responses. BERTScore (Zhang et al., 2020a) leverages BERT to extract the features of system response and reference, which are then used to compute cosine similarity.
3 turn
4 burn
Probability
0.6
5 turn
0.0
Number of Conversations
Reference-free automatic metrics have prolif- erated in recent years. FED (Mehri and Eskenazi. 2020a) uses DialoGPT and pre-designed questions to evaluate multiple dimensions of dialogue sys- tems. USR (Mehri and Eskenazi, 2020b) trains several models as sub-metrics to measure differ- ent aspects of dialogue. DynaEval (Zhang et al., 2021) leverages graph convolutional network to model the holistic quality of dialogue. ChatMatch (Yang et al., 2022) uses flexible scoring metrics to analyse dialogue records generated by bot-bot conversations, GPTScore (Fu et al., 2023) employs generative pre-trained models and a prompt frame- work to evaluate generated texts.
Figure 6: Ranking stability towards the turn number of conversation on English experiment. The x-axis repre- sents the number conversation. The y-axis represents the rate at which 100 repetitions yield the same ranking.
we remove the shared dialogue history compo- nent from FFAEVAL, resulting in independent di- alogue contexts for all dialogue systems. Then, we ask annotators to evaluate Chinese models in this modified framework. The result presents that the Kendall's correlation and Pearson's is correla- tion drops to 60% and 87.8%, respectively, which demonstrates the importance of shared dialogue his- tory in our framework. Furthermore, we conduct a t-test and demonstrate, with p = 0.0016, that the first-selected chatbot is significantly more likely to be credited in subsequent turns (i.e., cascading bias introduced in Section 2.2).
Human assessments involve annotators pro- vide subjective scores or preferences. Static eval- uation uses dialogue context to generate single re- sponses, which are then evaluated using multidi- mensional scoring (Galley et al., 2018) or prefer- ence selection (Li et al., 2016). Spot The Bot (De- riu et al., 2020) employs the concept of the Turing test to evaluate dialogue systems and uses survival analysis to obtain fine-grained features. Smith al. (Smith al., 2022) compare the sensitivity of vari- ous comparative methods for evaluating dialogue systems. Liu et al. (Liu et al., 2023a) evaluate mul- tiple dialogue systems in an interactive setup and
5 Related Work
Currently, the evaluation methods for dialogue sys- tems fall into three categories, which is described in detail below.
Reference-based automatic metrics require human-written references as a gold standard. BLEU (Papineni et al., 2002) calculates the n-gram

---

require annotators select one or more appropriate responses. Chatbot Arena (Zheng al., 2023) eval- uates LLMs side-by-side in a single-blind manner and calculates the leaderboard using the Elo rank- ing algorithm. Using Likert scales to rate chatbots after human-bot interaction is currently the stan- dard method (Venkatesh et al., 2018) (Dinan et al.. 2020) (Bao et al., 2022). However, this method places high cost dialogue generation and eval- uation. Existing comparative methods either evalu- ate using low-quality bot-bot dialogues or require a long time to reach a convergent ranking. Our framework expedites convergent processes while ensuring the reliability of evaluations.
collect release data containing personally identi- fiable information. Besides, fair compensation is paid to the annotators in accordance with the local market and their working hours.
A Acknowledgments
This work is supported by National Natural Sci- ence Foundation of China (62322214, 62072460. 621 62172424,6227667009 Beijing Natural Science Foundation (4212022); the Public Computing Cloud at Renmin University of China.
References
6 Conclusion
Yushi Bai, Jiahao Ying. Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li, and Lei Hou. 2023. Benchmarking foundation models with language-model-ne model-as an ee-axixieee. ArXiv preprint.
In this paper, we propose a reliable and efficient hu- man evaluation framework using free-for-all rank- ing, named FFAEVAL. This framework enables annotators to interact with multiple dialogue sys- tems simultaneously and make a preference selec- tion to credit the most favourable model in each turn of conversation. The final leaderboard is cal- culated using the results of free-for-all matches and the TrueSkill ranking algorithm. Our experiments demonstrate that FVAEVAL achieves a strong cor- relation with the gold standard score. Besides, we present that FFAEVAL outperforms other baselines in time efficiency and is stable towards match order.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACI. Workshop on Intrinsic and Ex- for
trinsic Evaluation Measures for Machine Translation and/or Summarization.
Siqi Bao, Huang He, Fan Wang. Hua Wu, Haifeng Wang, Wenquan Wu, Zhen Guo, Zhibin Liu, and Xinchao Xu. 2021. PLATO-2: Towards building an open-domain chatbot via curriculum learning. In Findings of the Association for Computational Lin- guistics: ACL-IJCNLP 2021.
In the future, we plan to introduce matchmak- ing process for the free-for-all to investigate the scalability and flexibility of our framework while handling large number of models.
Siqi Bao, Huang He, Fan Wang, Hua Wu, Haifeng Wang, Wenquan Wu, Zhihua Wu, Zhen Guo, Hua Lu, Xinx- ian Huang. Xin Tian, Xinchao Xu, Yingzhan Lin, and Zheng- Yu Niu. 2022. PLATO-XL: Exploring the large- scale pre-training of dialogue generation. In Findings of the Association for Computational Linguistics: AACL-IICNLP 2022.
Limitations
There are several possible limitations. First, al- though our framework approximates the gold stan- dard score to some extent, there is still a certain gap between them. Besides, FFAEVAL leverages pref- erence selection to rank dialogue systems, which makes it difficult to evaluate the ability of dialogue systems in a certain aspect (e.g., coherence, infor- mativeness), but rather at holistic level. Second, To guarantee a reliable result, annotators need to follow the instruction manual and be trained be- fore evaluating dialogue systems on our framework, which may result in decrease in efficiency.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing, 2023. Vicuna: open- source chatbot impressing gpt-4 with 90%* chattpt quality.
Jan Deriu, Don Tuggener, Pius von Däniken, Jon Ander Campos, Alvaro Rodrigo, Thiziri Belkacem, Aitor Soroa, Eneko Agirre, and Mark Cieliehak. 2020. Spot the bot: A robust and efficient framework for the evaluation of conversational dialogue systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Ethics Statement
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, lulian Serban, Ryan
All the human annotators involved in our paper are provided with informed consent, and we do not

---

Lowe, al. 2020. The second conversational in- telligence challenge (convai2). In The NeurlPS' 18 Comperition: From Machine Learning to Intelligent Conversations. Springer.
Jordan J Louviere, David A Hensher, and Joffre D Swait. 2000. Stated choice methods: analysis and applica- tions.
Ryan Lowe, Michael Noseworthy, Julian Vlad Serban, Nicolas Angelard-Gontic Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic Turing test: Learning to evaluate dialogue responses. In Proceed- ings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Sarah E. Finch and Jinho D. Choi. 2020. Towards uni- fied dialogue system evaluation: A comprehensive analysis of current evaluation protocols. In Proceed- ings of the 21th Annual Meeting of the Special Inter- est Group Discourse and Dialogue.
Hua Lu, Siqi Bao, Huang Hc, Fan Wang, Hua Wu, and Haifeng Wang. 2022. Towards boosting the open-domain chathot with human feedback. ArXiv preprint.
Jinlan Fu, See- Kiong Ng. Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate you desire. ArXiv preprint.
Michel Galley, Chris Brockett, Xiang Gao, Bill Dolan, and Jianfeng Gao. 2018. End-to-end conversation modeling: Moving beyond chitchat.
Neil Malhotra. 2008. Completion time and response or- der effects in web surveys. Public opinion quarterly.
Shikib Mehri and Maxine Eskenazi. 2020a. Unsuper- vised evaluation of interactive dialog with DialoGPT. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 225-235, 1st virtual meeting, Association for Computational Linguistics.
Mark E Glickman and Albyn Jones. 1999. Rating the chess rating system. CHANCE-BERIIN THEN NEW YORK-
Ralf Herbrich, Tom Minka, and Thore Graepel. 2006. TrueskillMM; a bayesian skill rating system. Advances in neural information processing systems.
Shikib Mehri and Maxine Eskenazi. 2020b. USR: An unsupervised and reference free evaluation metric for dialog generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681 681-707, Online. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. NeurIPS.
Jiwei Li. Michel Galley, Chris Brockett, Georgios Sp- ithourakis, Jianfeng Gao, and Bill Dolan. 2016. A persona-basee based neural conversation model. In Pro- ceedings of the 54th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers).
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DiilyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Nat- ural Language Processing (Volume 1: Long Papers).
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAl.
Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, Morteza Behrooz, William Ngan, Spencer Poff, Naman Goyal, Arthur Szlam, Y-Lan Boureau, Melanie Kambadur, and Jason Weston. 2022. Blenderbot 3: a deployed conversational agent that continually learns respon- sibly engage. ArXiv preprint.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out.
Sijia Liu, Patrick Lange. Behnam Hedayatnia, Alexan- dros Papangelis, Di Jin, Andrew Wirth, Yang Liu, and Dilek Hakkani-Tur, 2023a. Towards credible hu- man evaluation of open-domain dialog systems using interactive setup. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 37, pages 13264- 13272.
Eric Smith, Orion Hsu, Rebecca Qian, Stephen Roller. Y-Lan Bourcau, and Jason Weston. 2022. Human evaluation of conversations is an open problem: com- paring the sensitivity of various methods for eval- uating dialogue agents. In Proceedings of the 4th Workshop on NLP for Conversational Al, pages 77- 97, Dublin, Ireland. Association for Computational Linguistics.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023b. G-eval: NLG evaluation using GPT-4 with better human alignment. ArXiv preprint.

---

Anu Venkatesh, Chandra Khatri, Ashwin Ram, Fenfei Guo, Raefer Gabriel, Ashish Nagar, Rohit Prasad, Ming Cheng. Behnam Hedayatnia, Angeliki Metalli- nou, al. 2018. On evaluating and comparing open domain dialog systems. ArXiv preprint.
Zhuohan Li, Dacheng Li, Eric. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. 2023. Judging LLM--a-aa-udge with mt-hench and chathot arena.
Han Zhou, Xinchao Xu, Wenquan Wu, Zhengyu Niu, Hua Wu, Siqi Bao, Fan Wang, and Haifeng Wang. 2022. Sine: Service information augmented open- domain conversation. ArXiv preprint.
Yida Wang, Pei Ke. Yinhe Zheng. Kaili Huang. Yong Jiang, Xiaoyan Zhu, and Minlie Huang. 2020. A large-scale chinese short-text conversation dataset.
Hao Zhou, Pei Ke. Zheng Zhang, Yuxian Gu, Yinhe Zheng, Chujie Zheng. Yida Wang, Chen Henry Wu, Hao Sun, Xiaocong Yang. Bosi Wen, Xiaoyan Zhu, Minlie Huang, and Jie Tang. 2021. EVA: an open- domain chinese dialogue system with large-scale gen- erative pre-training. ArXiv preprint.
Ruolan Yang, Zitong Li. Haifeng Tang, and Kenny Zhu. 2022. ChatMatch: Evaluating chatbots by
tonomous chat tournaments. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers).
Jifan Yu, Xiaohan Zhang, Yifan Xu, Xuanyu Lei, Xinyu Guan, Jing Zhang, Lei Hou, Juanzi Li, and Jie Tang. 2022. Xdai: A auning-free framework for exploiting pre-trained language models in knowledge grounded dialogue generation. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text genera- tion. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Informa- tion Processing Systems 2021, NeurlPS 2021, De- cember 6-14, 2021, virtual.
Chen Zhang, Yiming Chen, Luis Fernando D' Haro, Yan Zhang, Thomas Friedrichs, Grandee Lee, and Haizhou Li. 2021. DynaEval: Unifying turn and di- alogue level evaluation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Confer- ence on Natural Language Processing (Volume Long Papers), pages 5676-5689, Online. Association for Computational Linguistics.
Jing Zhang, Xiaokang Zhang, Daniel Zhang-li, Jifan Yu. Zijun Yao, Zeyao Ma, Yiqi Xu, Haohua Wang, Xiao- han Zhang, Nianyi Lin, Sunrui Lu, Juanzi Li, and Jie Tang. 2023. Glm-dialog: Noise-toleran pre-training for knowledge-groundee dialogue generation. ArXiv preprint.
Tianyi Zhang, Varsha Kishore, Felix Wu. Kilian Q Weinberger, and Yoav Artzi. 2020a. Bertscore: Eval- uating text generation with BERT. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
Yizhe Zhang. Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020b. DIALOGPT: Large- scale generative pre-training for conversational re- sponse generation.
Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics: System Demonstrations.
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin,

---

Dimension
Score
Instruction
The response is irrelevant to context of the dialogue.
The response is simply restatement of the context.
There a contradiction between response the context.
Coherence
There a minor conffict between response the context. There slight logical conflict in the response.
2
3
The responses are contextual and consistent.
Thee responses are devoid of information.
The response merely restates the context and provides additional details.
Informativeness
The information the response conflicts with common sense.
2
The information in the response contains factual errors.
3
The response contains appropriate and correct information.
-
The response contains content that is harmful, biassed, or misleading.
Safety
-
The response may make people feel offended or uncomfortable.
The response safe.
The response failed to encourage the user to the next question.
Inspiration
The response inspires users to inquire about topies or related content.
3
The response motivates and enables them ask the next question immediatcly.
The has interest in continuing the conversation.
Engagingness
2
The conversation is somewhat dull, but it is still possible to continue talking.
3 - The user desires extended dialogue with the interlocutor.
- The user has no faith in the response of the interlocutor.
Faithfulness
2 - The user has some faith in the interlocutor's response.
3
- The user has faith in the interlocutor's response.
- The overall performance of the dialogue system is bad.
2
- The overall performance of the dialogue system is poor.
Overall
3
- The overall performance of the dialogue system is fair.
-
The overall performance of dialogue system is gond.
The overall performance of dialogue system excellent.
Table 3: Instructions for human annotators on different dimensions.
Annotation Manual
properties of dialogue systems.
Correlation with Different Dimensions
The instructions used for ground truth annotation are provided in Table 3. For the unified guideline of the three human evaluation methods, we require the annotators to focus first on the contextual relevance of models' responses in terms of coherence. Subse- quently, they evaluate the models' knowledge and conversational experience according to informative- ness. Finally, the annotrators consider their own subjective preferences, taking into account factors such as inspiration, engagingness, and faithfulness. The guideline is designed ensure fairness among three comparative human evaluation methods, and we can easily modify it to evaluate the specific
In addition to the gold standard score, we also cal- culated the Kendall ranking correlation and Pear- son correlation between various evaluation meth- ods and different sub-metrics of ground truth, shown in Table 4. FFAEVAL achieves high cor- relation with most sub-metrics. It makes sense, all
will be taken into account when making preference selections.
C Framework Implementation
Figure 7 shows the implementation of our online framework.

---

Cobe.
Info.
Safc.
Insp-
Enga.
Fait.
Method
7
7
English Experiment
Fl
40%
10.3%
60%
1.8%
20%
15.0%
60%
10.6%
40%
16.1%
40%
11.1%
0%
16.7%
20%
26.9%
20%
43.8%
20%
22.7%
0%
10.8%
0%
15.2%
CM
20%
33.3%
0%
30.1%
40%
18.6%
40%
45.3%
20%
38.0%
20%
32.9%
PR
60%
75.7%
80%
69.9%.
40%
56.7%
80%
71.7%
60%
78.1%
60%
76.2%
CA
60%
59.0%
40%
55.9%
0%
45.8%
80%
68.7%
60%
64.3%
60%
58.9%
Ours
100%
90.8%
80%
84.7%
40%
73.7%
80%
80.9%
100%
93 6%
100%
92.0%
Chinese Experiment
FI
32%
17.5%
40%
23.1%
60%
20.8%
20%
39.7%
20%
23.3%
20%
16.4%
BS
53%
28. 28.5%
60%
34.2%
80%
31.2%
40%
52.3%
40%
35.3%
40%
29.0%
CM
32%
2.0%
20%
4.6%
0%
5.5%
40%
43.6%
40%
18.5%
40%
15.0%
PR
53%
67.1 67.7%
40%
70.9%
20%
63.2%
60%
93.3%
60%
78.6%
60%
77 77.0%
CA
74%
70.0%
60%
73.3%
40%
68.2%
80%
91.6%
80%
79.8%
80%
82.6%
Ours
95%
96.3%
80%
95.9%
60%
88.6%
100%
97. 97.4%
100%
99.5%
100%
98.89
Table 4: Overall correlation between the result of different methods and scores of the different dimension of ground truth English and Chinese experiment. and p stand for Kendall ranking correlation and Pearson score correlation, respectively.
PE A:
BA
B1
B2
DG
PX
you music?
Of what kind music do you like
do! fan of the Beatles, Queen and Queen.
I do. listen lot of music. what kind of music you like?
Yes, do.
Yes, love music. listen to all the time. What kind of music do you like
Yesh, Hove them, too. Which songe do you like most?
&
love the baatles too! I'm not a big fan of thair music, but do love their music. what's your favorite song by them?
I'd have to say While My Guitar Gently Weeps is my favorite. Do you have favorite from them
The Beatles, Queen, and Queen.
Yes,
like Beatlemania songs
Rhapsady"
#ifite
Figure 7: A screenshot of our online framework.


Reviews:
[{"review_id": "pbyI5ET3FH", "reviewer": "Reviewer_H8nE", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We much appreciate your reviews and recognition of our work! Your concerns are addressed as follows:\n\n> While they clearly defined evaluation dimensions for each conversation, the study missed mentioning the sources that inspired these dimensions.\n\nThanks for your questions about the details of evaluation dimensions. We follow the evaluation process of some dialogue systems to design these dimensions. In particular, we evaluate the overall score following [1], coherence, informativeness, and engagingness following [2], and safety following [3]. In addition, we introduce two new dimensions, inspiration and faithfulness, because we believe that the existing dimensions do not adequately capture all aspects of the model's capabilities. Inspiration is a turn-level metric measuring how inspiring the response of chatbot is and faithfulness is a dialog-level metric measuring whether the annotator has faith in the chatbot.\n\n*Reference:*\n\n[1] Bao, Siqi, et al. \"PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable.\" Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.\n\n[2] Bao, Siqi, et al. \"PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generation.\" Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022. 2022.\n\n[3] Bao, Siqi, et al. \"PLATO-K: Internal and External Knowledge Enhanced Dialogue Generation.\" arXiv preprint arXiv:2211.00910 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Reasons:\n1. The study's outcomes demonstrated a strong match between their new metric and human evaluations. The metric's success in handling historical context was the key to this alignment.\n\n2. The authors effectively used statistical graphics to compare their model's confidence levels with Chatbot Arena, enhancing their analysis.\n\n3. Their investigation into the impact of multi-turn conversations on response quality provided valuable insights into the role of dialogue length.\n\n4. While they clearly defined evaluation dimensions for each conversation, the study missed mentioning the sources that inspired these dimensions.\n5. They mentioned ethical considerations for human annotation and privacy is maintained while collecting information.", "reasons_to_reject": "The write-up is skillfully crafted, and I couldn't identify any shortcomings that would warrant rejection.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The author introduced an automated evaluation metric that calculates Trueskill scores from diverse models. These scores are then employed to evaluate responses from a newly suggested model, selecting the optimal one. The study showcased a robust correlation between their metric and human-based scoring. This achievement was credited to the metric's prowess in managing history-based conversations, augmenting context-based learning reliability. Consequently, this enhancement led to a significant improvement in alignment with human evaluations. The paper additionally included a time comparison, substantiating time and convergence efficiency in comparison to human evaluation. A sensitivity analysis across various turns was conducted, contributing to a comprehensive assessment. Statistical representation further enhanced the clarity and impact of these analyses.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "\nIn the Appendix, there's a need for citations to provide the sources of inspiration for defining the dimensions within each category for evaluating dialogue."}, "extra_text": {}}, {"review_id": "NQGIRFzjke", "reviewer": "Reviewer_ojyE", "summary": "", "questions": "A. In order to get the GS score, Does all the models get a random conversation to begin? If all the models get a random conversation, how will that ensure fairness in comparison?\nB. What are the sub-metrics used? L357 just mentions various sub-metrics\nC. Please add more information as to why reference-free metrics do not succeed on this task.\nD. Are the results in Table 1 statistically significant? \nE. Did multiple annotators annotate the same conversation? What is the IAA score? How many groups of annotators were asked to annotate the GS score.\nF. How was the time for annotation calculcated? Were all the experiements to annotate done with the same samples?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable reviews! Your concerns about experiment details are meaningful for our work. We would like to address your questions in the following part and include these details in our next revision.\n\n> Comment A in Reason to Reject: Lack of information regarding the annotation guidelines provided for annotation deployed dialogue systems. What are the criteria the user is looking for during annotation?\n\nThank you for pointing out this. Due to page limitations, we supplement our annotation manual in Appendix A. To ensure consistency among the three human evaluation methods and the overall metric of GS score, the annotation process follows the same unified guidline as provided.\n\nIn particular, to better control variables during selecting the preferable response, we require the annotators to focus first on the contextual relevance of models' responses in terms of  coherence. Subsequently, they evaluate the models' knowledge and conversational experience according to informativeness. Finally, the annotrators consider their own subjective preferences, taking into account factors such as inspiration, engagingness, and faithfulness.\n\nTo avoid any potential misunderstandings, we would like to highlight this information in the main paper after revision and supplement more description on annotation details.\n\n> Comment B in Reason to Reject: The proposed system/framework might only capture which dialogue system is better for a users preference but does not provide insight into specific properties of the dialogue system, like whether has issues with coherency, or hallucinates. The framework feels incomplete without being able to understand the nuances of the each model.\n\nThanks for your comment! Similar to our response to your previous comment (Comment 1), annotators are required to use our framework following a unified guideline. So we capture not only the subjective user preference but also the overall performance of the dialogue system, including coherence, informativeness, etc.\n\nFor the evaluation of specific properties, as the framework is a type of comparative system, we only need to modify the annotation guidelines to match the definition of the property we want to evaluate.\n\n> Question A: In order to get the GS score, does all the models get a random conversation to begin? If all the models get a random conversation, how will that ensure fairness in comparison?\n\nThank you for your question. Indeed, the starting utterance for the conversation is an important control variable, which is taken into consideration in our experiments.\n\nIn particular, on each occasion, we begin by randomly selecting an initial utterance. This utterance is subsequently employed by all dialogue systems to initiate human-bot conversation. Therefore, the comparison is relatively fair, as the starting utterances of each model is the same.\n\n> Question B: What are the sub-metrics used? L357 just mentions various sub-metrics.\n\nWe apologize for any confusion that the term ``sub-metrics`` may cause. The ``sub-metrics`` are consistent with the ``other dimensions`` mentioned in L358, which refer to the dimensions except overall, including coherence, informativeness, safety, inspiration, engagingness, and faithfulness. A detailed introduction of these dimensions can be found in Appendix A.\n\n> Question C: Why reference-free metrics do not succeed on this task?\n\nThat's a good question and we will provide more supplements in our next revision. The reason for the poor performance, in our opinion, is twofold:\n\n* *Reference-free metrics, such as ChatMatch, may rely on evaluating bot-bot converstaions, while the quality degradation of dialogue between bots happens easily.* Although we generate 100 rounds of dialogue following the setting in the original paper, the content still lacks meaningful information on various topics, and duplicate responses frequently occur. This may prevent some powerful models from demonstrating their abilities in conversations. We provided an example of a conversation between DialoGPT and PLATO-XL:\n\n```\nDialoGPT: No problem, good luck with your future endeavors!\nPLATO-XL: Thanks, you too !\nDialoGPT: You're welcome :D\nPLATO-XL: Have a great day!\nDialoGPT: Thank you so much! You have an awesome day as well!\nPLATO-XL: You're welcome!\n```\n\n* ChatMatch employs multiple automatic metrics, such as perplexity and distinct-n, as well as rule-based algorithms such as the diversity scoring algorithm, to assess low-quality bot-bot dialogues. Unlike established reference-based metrics such as  F1 and BERTScore, which assess the bot's outputs by contrasting them with the high-quality human-written references, it is understandable that ChatMatch might yield relatively low performance.\n\n> Question D: Are the results in Table 1 statistically significant?\n\nThanks for your question. The results in Table 1 are statistically significant, please refer to official comment for more details.\n\n> Question E: Did multiple annotators annotate the same conversation? What is the IAA score? How many groups of annotators were asked to annotate the GS score.\n\nThanks for your questions about experiment details. As mentioned in L360, each conversation is assigned to three annotators for annotation. So three annotators annotate the same conversation.\n\nWe use Pearson's correlation to calculate inter-annotator agreement (IAA) score. The IAA scores for the English and Chinese experiments are 0.59 and 0.63, respectively. We believe the human evaluation is reliable, as the IAA score is around 0.6.\n\nThree annotators are organized into a single group. We evaluate the English model by four groups of annotators, and evaluate the Chinese model by another four groups. So we used a total of eight groups of annotators.\n\n> Question F: How was the time for annotation calculcated? Were all the experiements to annotate done with the same samples?\n\nThanks for your question. For the English experiment, we use the same set of annotators to independently assess different human evaluation methods. For each human evaluation method, such as FFAEval, we record the entry time of each annotator into the online system. Annotators are then required to focus exclusively on the annotation task until completion. Then, we record the exit times of annotators from the system. Therefore, we can calculate the time spent by each annotator and obtain the total annotation time through summation. The same annotation process is used for the Chinese experiment.\n\nIn particular, when designing the time efficiency experiment, we consider whether to record the time to annotate an equivalent number of conversations or to record the time requried to reach a stable score. We find that FFAEval attains a stable score with the fewest number of conversations, prompting us to employ this specific number of samples as the stopping criteria.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. Tackles an important problem of improving evaluation process for dialogue systems through an interesting framework\nB. Experiments conducted on multiple languages to demonstrate the effectiveness of the proposed evaluation framework.", "reasons_to_reject": "A. Lack of information regarding the annotation guidelines provided for annotation deployed dialogue systems. What are the criteria the user is looking for during annotation?  \nB. The proposed system/framework might only capture which dialogue system is better for a users preference but does not provide insight into specific properties of the dialogue system, like whether has issues with coherency, or hallucinates. The framework feels incomplete without being able to understand the nuances of the each model.\n ", "questions_to_authors": "A. In order to get the GS score, Does all the models get a random conversation to begin? If all the models get a random conversation, how will that ensure fairness in comparison?\nB. What are the sub-metrics used? L357 just mentions various sub-metrics\nC. Please add more information as to why reference-free metrics do not succeed on this task.\nD. Are the results in Table 1 statistically significant? \nE. Did multiple annotators annotate the same conversation? What is the IAA score? How many groups of annotators were asked to annotate the GS score.\nF. How was the time for annotation calculcated? Were all the experiements to annotate done with the same samples?", "paper_topic_and_main_contributions": "The authors propose a reliable and efficient human evaluation framework \"FFAEval\". The framework is used to evaluate and compare multiple open-domain dialogue systems through the usage of free for all ranking approach that allows annotators to pick the most favourable model in each turn and continue the conversation. \n\nThe proposed framework has been used to evaluate the existing model that have been trained on English and Chinese datasets. Results obtained using this framework shows higher level of correlation with human assessment compared to other evaluation methods.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "A. L209 add a citation for cascading bias."}, "extra_text": {}}, {"review_id": "VVw10MdDRM", "reviewer": "Reviewer_nRyd", "summary": "", "questions": "1. How does FFAEVAL deal with the case when none of the generated responses are appropriate [1]? \n\n2. Line 302, does comparison with reference-based metrics make sense? My understanding is that FFAEVAL is designed for the interactive evaluation setting. As the conversation evolves, the trajectory will diverge from the original human-human conversations.\n\n3. In Figure 1 and Table 1, considering that you are ranking just five dialogue models, how reliable or statistically significant is your findings? Additionally, if the number of models increases, will the reliability of the human user decrease given that they have to choose among a large pool of response candidates?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your reviews and address your concerns as follows.\n\n> Comment 1:  The design of FFAEVAL is not new. Refer to [1] in the missing reference, which contains two similar interactive evaluation setups as FFAEVAL: (1) multi-model Select-One-Best-from-All and (2) multi-model Select-All That-Apply.\n\nThank you for pointing out the missing reference. However, we notice that there are  differences between FFAEval and [1], as follows:\n\n* The ranking algorithm differs between [1] and FFAEval. Whereas [1] calculates the winning rate, FFAEval models the evaluation process as free-for-all matches using the TrueSkill ranking algorithm.  We change the ranking algorithm to winning rate in our evaluation method and find that the required chatbot response for evaluation convergence increases from 875 to 1250, illustrating the faster convergence by our ranking method.\n* Since we choose the TrueSkill, we initially generate the ranking result of a multi-turn conversation. Subsequently, we use this ranking result to update the overall leaderboard by TrueSkill. In contrast, [1] directly employs the result of a single turn to update the overall leaderboard.\n\nBesides the difference in framework design, it's crucial to highlight that [1] did not compare with any other evaluation baselines. Conversely, our approach comprehensively compares the proposed FFAEval with laterative evaluation baselines, which validates the reliability of this interactive evaluation framework.\n\n> Question 1: How does FFAEVAL deal with the case when none of the generated responses are appropriate [1]? \n\nThat's a good question and we have also considered this circumstance. To solve this problem, our online system enables annotators to regenerate dialogue system responses if none of them are appropriate.\n\n> Question 2: Does comparison with reference-based metrics make sense?\n\nThanks for your question. As our goal is to demonstrate the reliability of FFAEval in a fair and comprehensive manner, we believe it is essential to use a variety of evaluation methods as baselines. Since reference-based metrics are still frequently used as a reference for open-domain dialogue tasks, we think it makes sense to use them as baselines. Except for the reference-based method, we also compare FFAEval with reference-free metrics and various human evaluation methods, demonstrating the comprehensiveness of our experiment.\n\n> Question 3: How reliable or statistically significant is the result in Table 1? If the number of models increases, how to handle a large pool of response candidates?\n\nThanks for your question. The results in Table 1 are statistically significant, please refer to official comment for more details.\n\nIf we need to evaluate a large number of models, we can modify the framework to randomly select a fixed number of dialogue systems to be evaluated before a conversation begins. This modification can solve the difficulty of selecting the best response among a large number of candidates.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The clarity of the paper is good.\n\n2. The proposed method is technical sound and the analyses are comprehensive", "reasons_to_reject": "The design of FFAEVAL is not new. Refer to [1] in the missing reference, which contains two similar interactive evaluation setups as FFAEVAL: (1) multi-model Select-One-Best-from-All and (2) multi-model Select-All That-Apply.", "questions_to_authors": "1. How does FFAEVAL deal with the case when none of the generated responses are appropriate [1]? \n\n2. Line 302, does comparison with reference-based metrics make sense? My understanding is that FFAEVAL is designed for the interactive evaluation setting. As the conversation evolves, the trajectory will diverge from the original human-human conversations.\n\n3. In Figure 1 and Table 1, considering that you are ranking just five dialogue models, how reliable or statistically significant is your findings? Additionally, if the number of models increases, will the reliability of the human user decrease given that they have to choose among a large pool of response candidates?", "paper_topic_and_main_contributions": "Topic: The paper studies the topic of human evaluation of dialogue systems.\n\nResearch problem: Given the poor reliability of automatic metrics, human evaluation remains the gold standard in assessing dialogue systems. Yet, existing human evaluation setups, such as multi-dimensional scoring based on instruction manuals, pairwise comparisons of dialogue systems, are time-consuming. \n\nContributions: The authors propose a Free-For-All ranking approach to interactively evaluate the dialogue systems. Specifically, a human user will simultaneously chat with multiple dialogue models and select the best response among all the candidates generated by the models. The conversation continues with the selected response.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 428 - “analysis” -> “analyze”"}, "extra_text": {"Missing_References": "[1] Liu, Sijia, et al. \"Towards credible human evaluation of open-domain dialog systems using interactive setup.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 11. 2023.\n\n[2] Smith, Eric, et al. \"Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents.\" Proceedings of the 4th Workshop on NLP for Conversational AI. 2022."}}, {"review_id": "inC2CHe1jx", "reviewer": "Reviewer_giRW", "summary": "", "questions": "A. Have you ever tried some current dialogue systems and compare the ranking order with ChatBot Arena.\n\nB. What is the background of the human evaluators. Are they proficient in Chinese and English?\n\nC. Is there a data breach risk using the topics in the Daily Dialog dataset? Since some models may are trained on it but others may not.\n\nD. What is the backbone model of the BERTScore metric? And why use the F1 score instead of the BLEU score?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your reviews and address your concerns as follows.\n\n> Comment 1 in Reason to Reject: The proposed method may be less relevant to the authors' motivations in abstract section (automatic scores are not effective and human evaluation scores are not affordable). Since the proposed framework FFAEVAL and some similar framework like Chatbot Arena are used to do comparison between dialogue systems, I do not think it can be directly used to evaluate a single dialogue system, like give a fluency score or something like that. So these arena-based evaluation systems may not solve the problems of current score-based evaluation systems.\n\nThank you for your comment. FFAEval is indeed incapable of evaluating and scoring a single model. However, in most cases, the goal of the evaluation methods is to identify differences in capabilities between models to better guide researchers or users in discovering or selecting models that perform the best. Despite the fact that score-based metrics can be used to evaluate and score a single model, they are rarely employed for this purpose. As the goal of both score-based methods and arena-based methods is the same, we believe FFAEval can be an alternative to score-based methods to some extent.\n\n> Comment 2 in Reason to Reject: The claims of these paper lack solid experiments support. Dialogue systems evaluated in experiments are kind of out-dated, such as DialoGPT and PLATO-XL. The poor performance of old models may influence the evaluation process. Using current LLM-based chatbot like alpaca, vicuna may be more convincing. Besides, the experiments section lacks some details, like the number of test examples and the inference settings.\n\nThank you for your comment. Our goal is to verify the reliablity of FFAEval and we need a ground truth ranking as a reference. However, it's difficult for us to obtain a ground truth ranking for the current LLM as an acknowledged evaluation method is still under-investigated. Therefore, we use open-domain dialogue systems (e.g., PLATO-XL), as we can find an acknowledged and frequently used method (i.e., score-based human evaluation including overall, coherence, informativeness, etc.) by researchers (e.g., [1][2][3]).\n\nThanks for pointing out this. We will add more detailed information about the experimental setup in our next revision. We use 500 turns of conversation as test examples. To ensure fairness, we set the temperature to 1 for each dialogue system, and the length of input and output depends on the maximum length supported by the model.\n\n> Comment 3 in Reason to Reject: Although the authors state some difference between their framework and Chatbot Arena (a popular evaluation framework for comparing different dialogue systems), I think the difference between the two methods is not obvious. Chatbot Arena compares models two by two and then computes the elo score. FFAEVAL compares all models in one conversation and then computes the TrueSkill score.\n\nThank you for your comment. The difference between FFAEval and Chatbot Arena is as follows:\n\n* FFAEval maintains a shared dialogue history, while Chatbot Arena does not. Referring to chapter 4.5 in the paper, shared dialogue history is the key for FFAEval to achieve better performance by a large margin. This shared dialogue history serves as a critical component, ensuring a fair comparison between models during multi-turn conversations.\n* The annotation of FFAEval is more fine-grained. In Chatbot Arena, the annotator annotates one time after the conversation ends, while in FFAEval, the annotator annotates each turn and calculate the result based on this per-turn annotation. The evaluation process of FFAEval is more robust and reliable as the influence of incorrect annotation is less.\n* The ranking algorithm is different. FFAEval uses the TrueSkill algorithm to calculate the result of a free-for-all match, while Chatbot Arena uses the Elo algorithm to calculate the result of a 1v1 match. Thanks to the TrueSkill algorithm and free-for-all match, FFAEval converges faster and achieves better time efficiency. Besides, using TrueSkill mitigates the influence of model competition order, as demonstrated in Figure 5 of the paper.\n\n> Question A: Have you ever tried some current dialogue systems and compare the ranking order with ChatBot Arena.\n\nThanks for your question. According to the answer to Comment 2, our goal is to verify the reliability of FFAEval and it's difficult for us to obtain a ground truth ranking for the current LLM as an acknowledged evaluation method is still under-investigated. Therefore, we have not tried the current LLMs.\n\n> Question B: What is the background of the human evaluators. Are they proficient in Chinese and English?\n\nThanks for your question. Referring to L331, the annotators are all Chinese undergraduate students. Their native language is Chinese, and their English is at Chinese College Entrance Examination level. They are able to read and understand English conversations fluently. \n\n> Question C: Is there a data breach risk using the topics in the Daily Dialog dataset? Since some models may are trained on it but others may not.\n\nThanks for your question. We think the impact of data breach risk is minimal for the following two reasons:\n\n* The sampled starting utterance is not directly used for training. And we rewrite the sampled sentences to help annotators and chatbots create more meaningful conversations.\n* Although the initial starting utterances are sourced from the Daily Dialog dataset, it's important to note that in the following conversations, annotators are unlikely to engage with chatbots in accordance with the data distribution found in Daily Dialog training set.\n\n> Question D: What is the backbone model of the BERTScore metric? And why use the F1 score instead of the BLEU score?\n\nThanks for your question about experiment details. We use roberta-large and bert-base-chinese as the backbone of BERTScore for evaluating English and Chinese models, respectively. BLEU is a precision-focused metric, while F1 considers both values of precision and recall. Therefore, we use the F1 score as one of the representative methods of reference-based metrics.\n\n*Reference:*\n\n[1] Roller, Stephen, et al. \"Recipes for Building an Open-Domain Chatbot.\" Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 2021.\n\n[2] Bao, Siqi, et al. \"PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable.\" Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.\n\n[3] Yu, Jifan, et al. \"XDAI: A Tuning-free Framework for Exploiting Pre-trained Language Models in Knowledge Grounded Dialogue Generation.\" Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors propose to compare different dialogue system via a Free-For-All match.\n2. The proposed framework: FFAEVAL can compare multiple dialogue system at meantime, which can improve the efficiency of the evaluation system.\n3. Dialogue evaluation is an important topic in LLMs era. Besides, with the popularity of open domain dialog systems like ChatGPT and their need for human evaluation, it becomes more and more important to design an effective and efficient evaluation system.", "reasons_to_reject": "1. The proposed method may be less relevant to the authors' motivations in abstract section (automatic scores are not effective and human evaluation scores are not affordable). Since the proposed framework FFAEVAL and some similar framework like Chatbot Arena are used to do comparison between dialogue systems, I do not think it can be directly used to evaluate a single dialogue system, like give a fluency score or something like that. So these arena-based evaluation systems may not solve the problems of current score-based evaluation systems.\n2. The claims of these paper lack solid experiments support. Dialogue systems evaluated in experiments are kind of out-dated, such as DialoGPT and PLATO-XL. The poor performance of old models may influence the evaluation process. Using current LLM-based chatbot like alpaca, vicuna may be more convincing. Besides, the experiments section lacks some details, like the number of test examples and the inference settings.\n3. Although the authors state some difference between their framework and Chatbot Arena (a popular evaluation framework for comparing different dialogue systems), I think the difference between the two methods is not obvious. Chatbot Arena compares models two by two and then computes the elo score. FFAEVAL compares all models in one conversation and then computes the TrueSkill score.", "questions_to_authors": "A. Have you ever tried some current dialogue systems and compare the ranking order with ChatBot Arena.\n\nB. What is the background of the human evaluators. Are they proficient in Chinese and English?\n\nC. Is there a data breach risk using the topics in the Daily Dialog dataset? Since some models may are trained on it but others may not.\n\nD. What is the backbone model of the BERTScore metric? And why use the F1 score instead of the BLEU score?", "paper_topic_and_main_contributions": "This paper addresses the issue of comparing the performance of different open-domain dialogue systems. Automatic evaluation metrics have poor correlation with human evaluations in dialogue generation tasks. Human evaluations, involving multi-dimensional scores from annotators, are credible but time-consuming. Therefore, the authors propose a human evaluation framework called FFAEVAL, which adopts a Free-For-All Ranking method, aiming to provide a reliable and efficient evaluation method. \n\nBy sharing dialogue history, this framework allows evaluators to conduct single-blind, multi-turn dialogues with multiple dialogue systems at the same time. In the ensuing free-for-all competition, evaluators can choose the model they think is the best from all participating dialogue systems in each round. Finally, the performance of each model is represented by calculating TrueSkill scores.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel human evaluation framework, FFAEVAL, to compare the performance of various open-domain dialogue systems. Recognizing the limitations of automatic evaluation metrics and the time-consuming nature of traditional human evaluations, the authors propose a Free-For-All Ranking method that allows evaluators to conduct single-blind, multi-turn dialogues with multiple dialogue systems simultaneously. The paper offers a valuable contribution to designing effective and efficient evaluation methods for dialogue systems.
Decision: Accept-Findings