ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
Authors: Chi-Min Chan
Weize Chen
Yusheng Su
Jianxuan Yu
Wei Xue
Shanghang Zhang
Jie Fu
Zhiyuan Liu
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Model
Multi-Agent Debate
LLM evaluators
Primary area: generative models
TL;DR: We introduce ChatEval, a multi-agent debate framework for text evaluation. It outperforms traditional methods, aligning more closely with human assessment.

Abstract: Text evaluation has historically posed significant challenges, often demanding substantial labor and time cost. With the emergence of large language models (LLMs), researchers have explored LLMs' potential as alternatives for human evaluation. While these single-agent-based approaches show promise, experimental results suggest that further advancements are needed to bridge the gap between their current effectiveness and human-level evaluation quality.
Recognizing that best practices of human evaluation processes often involve multiple human annotators collaborating in the evaluation, we resort to a multi-agent debate framework, moving beyond single-agent prompting strategies.
In this paper, we construct a multi-agent referee team called $\textbf{ChatEval}$ to autonomously discuss and evaluate the quality of different texts. 
Our experiments on two benchmarks illustrate that ChatEval delivers superior accuracy and correlation in alignment with human assessment. Furthermore, we find that the diverse role prompts (different personas) are essential in the multi-agent debate process; that is, utilizing the same role description in the prompts can lead to a degradation in performance. Our qualitative analysis also shows that ChatEval transcends mere textual scoring, offering a human-mimicking evaluation process for reliable assessments.

Published as a conference paper at ICLR 2024
CHATEVAL: TOWARDS BETTER TORS THROUGH MULTI-AGENT DEBATE
EVALUA-
Chi-Min Chan', Weize Chen', Yusheng Su' Jianxuan Yu', Wei Xue",
Shanghang Z Jie Fu², Zhiyuan Liu
Tsinghua University
2 Hong Kong University of Science and Technology
3 Peking University
zorowin12 3 com
ABSTRACT
Text evaluation has historically posed significant challenges, often demanding substantial labor and time cost. With the emergence of large language models (LLMs), researchers have explored LLMs' potential as alternatives for human evaluation. While these single-agent--aaan basend approaches show promise, experimen- tal results suggest that further advancements are needed to bridge the gap between their current effectiveness and human-level evaluation quality. Recognizing that best practices of human evaluation processes often involve multiple human an- notators collaborating the evaluation, we resort multi-agent debate frame- work, moving beyond single-agent prompting strategies. In this paper, we build multi-agent referee team called ChatEval to autonomously discuss and evaluate the quality of different texts. Our experiments ChatEval delivers superior accuracy and correlation in alignment with human as- sessment. Furthermore, are essential in the multi-agent debate process; that is, utilizing the same role description in the prompts can lead degradation in performance. Our qualita- tive analysis also shows that ChatEval transcends mere textual scoring, offering a human-miin mimicking evaluation process for reliable assessments.
two benchmarks illustrate that
find that the diverse role prompts (different personas)
INTRODUCTION
Evaluating the quality of text generated by language models or written by humans has long been a challenging endeavor, consistently garnering substantial attention (Celikyilmaz et al.] 2020)_ Tra- ditional methodologics predominantly rely on human annotation of texts (Callison-Burr) approach considered overly demanding in terms of time and cost. Automatic evaluation metrics based on n- n-grrmms such as Rouge (Lin 2004), BLEU (Papineni et al. 2002), and METEOR (Baner- jee & Lavie 2005), have been proposed tackle this issue (Kondrak 2005). However, these methods have been shown to exhibit a relatively weak correlation with human judgments, partic- ularly in the context of tasks involving open-ended generation requiring domain-specific exper- tise NNvikkva el al. 2017).
2009),
In view of the impressive text understanding and instruction-followe a body of literature (Liu et al. 2023b Chiang & Lee, 2023 Gao et al. [2023] Shen et al., (023) has adopted LLM as an evaluator assess the quality of responses to open-ended questions or tradi- tional NLG tasks, including dialogue response generation and summarization. This methodology is dubbed aaj-uu-e (Zheng et al.) (003). Findings from these researches indicate that LLM can mimic human behavior and provide evaluations that correspond with human judgments, revealing potentially scalable and transparent alternative to costly and laborious human evaluations.
capabilities of recent LLMs,
While a single powerful LLM can already tackle various missions, emerging studies suggest that multiple L.MMs can further improve one another through debate and cooperation (Li al., 2023a Liang et al. 2023). By incorporating multiple LLMs into integrated group and designing specific
"Corresoondigg author. Email: 1 fuzzytaiing wwnttttinnguuu.. .ighua.ed..co

---

Published as a conference paper at ICLR 2024
Large Language Model (LLM) Based Agent
SinlleAgen method
Question How con improve my time management skils?
APter carefuly reviewing the respenses of both resporiees thrk ASSISTANT is letter
ASSISTANT improving your time maragement skils involves
Multi-Agent debate
ASSISTANT 2: Here are some tips to improve your time manggement Vie
APter discussing thoroughly with my co-wrrkrss we are convinced that ASSISTANT better on reason
Figure 1:
When several referees participate in the evaluation process, they can discuss with each other and finally give judgment that better aligned with human annotators.
interaction mechanisms, different L.MSs engage in proposing and deliberating unique responses and thought processes across several rounds. This approach leads to enhanced factuality of gen- erated responses (Du al. 2023) and improvement in the completion of arduous tasks (Li et al. 2023a Qian et al. (202)). Furthermore, the multi-agent group also addresses and mitigates the Degeneration-of--ne
(DOT) problem (Liang et al. 2023).
In the human evaluation processes, relying on a single perspective can introduce bias and instabil- ity in the results (Karpinska al.) 2021). Recognizing this, best practices often involve multiple human annotators collaborating in the evaluation (Van Der Lee et al., 2019). Drawing inspiration from this collaborative and iterative human evaluation approach, we propose ChatEval, system that enables each agent to employ varied communication strategies in collaborative discussion, working towards formulating final judgments. Furthermore, to enrich the evaluation dynamics, every agent within ChalEval is endowed with unique persona. This deliberate design ensures that each agent focuses on distinct perspectives or brings specific expertise to the table. By doing so, the collective evaluation benefits from more comprehensive lens, capturing nuances and subtleties that single perspective might overlook. Another underlying intuition of our work stems from renowned con- cepts in sociology and biology, including Collective DieeligenceeWWolllee et al. (010) and Cognitive Cnnergy[Luppi et al., 2022), where multiple cognitive processes or systems interact and cooperate way that produces a combined effect greater than the sum of their separate effects.
To summarize, the main contribution of our work is as follows:
1. We propose a culli-agent-beett based framework called ChatEval that aligns better with human preferences compared with single-agent aagett-based approaches depicted in Figure
2. We propose various communication strategies and demonstrate the necessity of diverse role prompts multi-agent debate scenarios.
3. In the qualitative study, we demonstrate that agents exhibit human-like behavior, cap- italizing on the richness and complexity of language interaction. This elevates ChatEval from being merely evaluation tool an embodiment of interactive natural language dia- logue.
METHODOLOGY
In this section, elaborate the principal components in ChatEval including debater agents, diverse role specification, communication strategy, and provide a detailed overview of each compo- nent's role and functionality.
2

---

Published as a conference paper at ICLR 2024
Nice
Bole
Cerol
Aice
Bole
Carol
Summerize izer
Nice
Bob
Card
round
round
(a) One-by-One
(b) Simultancous- Tall
(c) Simultancous -aluuss--Taa--aaaa -atk-with- wwith-Summarizer
The overall schematic diagram of our proposed three different kinds of communication strategy. The direction of the arrows represents the flow of information, meaning that what this person says will be appended to the chat history of the person pointed to by the arrow.
Figure 2:
Debater Agents. Debater agents are one of the most significant components in our framework. We treat each individual LLM as an agent and ask them to generate their response from the given prompt"" Responses from other agents are served as chat history which will be replaced in the prompt template. After configuring the agents, we then start the group debate where each agent autonomously receives responses from the others and, in turn, delivers its own responses to them. It should be noted that the whole process does not require human intervention.
Diverse Role Specification. As presented in Section ! diverse role specification is necessary for the framework as well. Although all the agents share a common prompt template, we substitute the role ddescription slot with diverse role prompts, specifying distinct personalities for different agents. We take inspiration from Wu al. (2023) and formulate an analogous role description.
Communication Strategy. How to maintain the chat history is another significant issue in ChatEval. In our work, we use a more intuitive term illustrate the maintenance of the chat history called communication strategy. In a nutshell, different communication strategies can be seen as different approaches to maintaining and manipulating their chat history. As is shown Figure 2 We primarily design three different communication strategies and the full description and algorithm of the above communication strategies can be found in Appendix
EXPERIMENTS
We evaluate ChatEval on two benchmarks. FairEval and Copical-Char which represent the cate- gories of open-ended question answer and dialogue response generation, respectively. For the LLMs and GPT-4 considering their strong capa- model such
in ChatEval, we choose to use ChaGGPTTGPT-33.--uuu bility shown in the past literature, we also test our frame work on smaller as Alama2-Chat-7B and Vicuna, the results of which are shown in Appendix D and the detailed settings for the experiments are discussed in Appendix
3.1 BENCHMARKS
The detailed introduction of different categories and benchmarks are listed follows:
Open-ended Question Answer is a key component within the field of NLP and generative AI. It necessitates an system to provide comprehensive, detailed, and human-like responses to questions that don't have predefined or fixed set of possible answers. The work of Chiang et al. (2023) encompasses a collection of 80 open-ender questions originating from a wide array of categorics, including common-sense counterfactual, coding, etc. We then take the human annotation results from Wu et al. 2023) to conduct the experiments in this paper. For each question, they direct three annotators to evaluate the replies given by Vicuna- 13B and ChatGPT through the given rules and finally derive the results by the majority votes among the annotators.
Dialogue Response Generation is task involves creating coherent and contextually appropriate response to a given input dialogue. We draw upon the Topical- Chat (Gooplakrishnan et al. 2019) dataset for study. We then take the human annotation results from Mehri & Eskenazi (2020) where they carry out the annotations on 60 dialogue contexts with each response generated by
'The full prompt template be found in Appendix

---

Published as a conference paper at ICLR 2024
different systems. Human evaluators analyzed these responses based natural, coherence, engag- ingness, groundedness, and understandable, where we take the first four dimensions for experiments in our paper following Zhong et (2022).
3.2 BASELINES
We evaluate ChatEval against following methods. As the main portion of our comparison, we pri- marily focuses on the single-agent-based method. Single-Agent means that we directly query an LLM to generate the response towards the evaluation Multi-Agent means that we employ multi- ple LLMs, either in an ensemble or through debate approach. When we do the ensemble, we apply the single-agent method multiple times using different role prompts across the same instance and then aggregate their results through averaging to derive the final outcome. By default, we configure the communication strategy to one-by-one, agent numbers to 2, and discussion turns to 2 in this sec- tion and employ position calibration techniques in both single-agent and multi- multi-agent settings. We will discuss more debate configurations in Section for completeness. For the open-ended question answer task, we also compare our method with a simple ensemble method and EvirEval (Wang et al. 2023b). They propose various strategies including Multiple Evidence Calibration (MEC) and Balanced Position Calibration (BPC). For the dialogue response generation task, we also compare our method with G-EVAL (Liu al.. 2023b They utilize CoT and probability-
improve the evaluation performance of LLM
summation for their method. Additionally, we include results from n-gram-based metrics, such as ROUGE (Lin) 2004). BLEU (Papineni et al., 2002) and embedding- based metrics such as BERTScore (Zhang et al. 2019).
3.3 RESULTS FOR OPEN-ENDED QUESTION ANSWERS
We adopt the same evaluation approach as Wang et al. 2023b) to assess the annotation results pro- duced by different methods and annotators. Specifically, we calculate the Accuracy (Acc.), which measures the proportion of correctly classified instances out of the total instances, and the Kappa correlation coefficient (Kap.) (McHugh] (012) which gauges the agreement between results from models and human annotators while taking into account the possibility of agreement occurring by chance. Both metrics provide insights into the reliability and consistency of the annotations. We take the human annotation results and FairEval's (Wang et al. (023b) best results from their paper. As is shown in Table D than any other LLM-based approach. Still, the average human annotations accuracy which is 71.7% shows there exists a certain degree of discrepancy among different unique individuals revealing that text evaluation is absolutely an arduous task. The second part and the third part of Table I show the results of FairEval's method and the results of our proposed method respectively. We find that (1) ChatEval can enhance the performance of the evaluation process, achieving higher alignment with human preference compared with single-agent evaluation. Specifically, the setti-ageet-bbaat method improves the accuracy by 6.2% for ChatGPT and 2.5% for GPT-4; (2) ChatEval surpasses FairEval's best results within both ChatGPT and GPT-4 settings showing the effectiveness of our proposed method; (3) Compared to ChatEval, basic ensemble fails to markedly improve the eval- uator's performance, highlighting the crucial role of natural language interaction in our framework.
different annotators can reach a relatively high agreement and perform better
3.4 RESULTS FOR DIALOGUE RESPONSE GENERATION
For the dialogue response generation benchmarks, we align the evaluation method with Zhong et al. (2022), calculating the tern-level Spearman and Kendall-Tau correlation in correspondence with hu- man judgments on four aspects (naturalness, coherence, engagingness and groundedness). Results can be found in Table 2 In the first part of Table 2 we demonstrate that i-gram-based metrics and embedding-base these methods can hardly reveal human preference. In the second part of Table 2 we show the results from the G-eval (Liu et al. 2023b) paper. They first ask the LLM to generate intermediate thought and finally calculate the weighted summation of the output scores based on the probabil- ity. The results show that their method outperforms previous traditional metrics depicting the fact
metrics perform overall poorly on all the aspects evaluated illustrating that
We use the same prompt template as our multi-agent debate settings in single-agent baseline except that we ignore some slot.
4

---

Published as a conference paper at ICLR 2024
Table Accuracy (Acc.) and Kappa correlation coefficient (Kap.) different methods on FairEval. We present our results with average and standard deviation by running the experiment five times.
Evaluator
Methods
Acc. (%)
Kap.
Human
Annotatorl
68.8
0.5
Annotator2
76.3
0.62
3nnotator3
70
0.5
FairEval
ChatGPT
MEC+BPC
58.7
0.31
GPT-4
MEC+BPC
62.5
0.37
Ours
ChatGPT
Single-Agent
53.7+1.4
0.27
ChatGPT
Multi-Agent (Ensemble)
55.5
0.29 0.201
ChatGPT
Multi-Agent (ChatEval)
60.0
0.30
GPT4
Single-Agent
60.8
0.36
GPT-4
Multi-Agent (Ensemble)
61.5400.
0.38
GPT-4
Multi-Agent (ChatEval) 63.8
0.40
Table Turn-level Spearman (p) and Kendall- Tau (T) correlations of different methods on Topical- Chat benchmark, SA means Single-Agent, EN means Multi-Agent (Ensemble) and MA means Multi-Agent (ChatEval). Our ChatGPT settings should be compared to G-EVAL-3.5, and GPT-4 settings should be compared to G-EVAL-4.
Naturalness
Coherence
Engagingness
Groundedness
Average
Metrics
p
T
p
p
7
7
0
7
ROUGE-L
0.146
0.176
0.203
0.193
0.300
0.295
0.327
0.310
0.244
0.244
BLEU-4
0.175
0.180
0.235
0.131
0.316
0.232
0.310
0.213
0.259
0.189
BERTScore
0.209
0.226
0.233
0.214
0.335
0.317
0.317
0.291
0.274
0.262
G-EVAL-3.5
0.539
0.532
0.544
0.519
0.691
0.660
0.567
0.586
0.585
0.574
G-EVAL-4
0.565
0.549
0.605
0.594
0.631
0.627
0.551
0.531
0.588
0.575
(hatGPT((A)
0.474
0.421
0.527
0.482
0.599
0.549
0.576
0.558
0.544
0.503
ChaiGPT(EN)
0.421
0.359
0.486
0.442
0.611
0.551
0.661
0.628
0.545
0.495
ChaIGPT(MA)
0.441
0.396
0.500
0.454
0.664
0.607
0.602
0.583
0.552
0.510
GPT-4(SA)
0.532
0.483
0.591
0.535
0.734
0.676
0.774
0.750
0.658
0.6IT
GPT-4(EN)
0.512
0.450
0.607
0.544
0.755
0.693
0.781
0.756
0.664
0.611
GPT-4(MA)
0.630
0.571
0.619
0.561
0.765
0.695
0.722
0.700
0.684
0.632
that the LLM--bsee evaluator is effective and reliable for evaluating the dialogue response genera- tion task. While their method delivers sound results, our proposed approach raises the bar in terms of performance for GPT-4. Specifically, ChatEval improves the average Spearman and Kendall- Tau correlation by 0.096 (16.3%) and 0.057 (10.0%) respectively. Additionally, compared with the single-agent method, ChatEval amplifies performance both for ChaiGPT and GPT-4, showing the effectiveness of our method which is aligned with the results in Section 3.3
4 ANALYSIS
In this section, we further explore the key components encompassed in ChatEval. We discuss the importance of diverse role prompts in Section 4.1 the effect of different communication strategies in Section 4.2 the impact of role numbers and discussion turns in Section 4.3 and carry out the qualitative study in Section 4.4 If not specified otherwise, we choose the EairEval benchmark and ChatGPT the backbone L.M for the analysis.
5

---

Published as a conference paper at ICLR 2024
4.1 THE IMPORTANCE DIVERSE ROLE PROMPTS
we demonstrate that ChatEval equipped with diverse role configura- tions can significantly improve the performance of evaluation. We further consider whether it is necessary to design diverse role prompts for the evaluation system. To answer so, we carry out the experiments by replacing all the role prompts with "You now Annotator, in the texl evaluation lask." and keeping other prompts unchanged. We experiment with the one-by- communication strategy and agents with 2 discussion turns. The results in Table B illustrate that ChatEval with the same role prompt design underperforms that with diverse role prompt design and cannot effectively enhance the performance compared with single-agent setting, highlighting the cruciality of diverse role prompt design in the multi-agent dehate framework.
Previously in Table and 2
of the referees
Based on the findings above, we are convinced that diverse role prompts are erucial for our frame- work. Furthermore, we delved deeper to study the effects of different portraits assigned to agents. To analyze this, we referred to the categories proposed by WWng et al.) (2023b). We designed specific roles for different groups and compared their evaluation quality with our default setting We selected four categories from which we could clearly derive specific roles for this experiment. Specifically, the four categories chose example, when we design the coding group, we recruit experts like Guido van Rossum by specifying "You Guido Rossum. You are the creator of the Python programmtigg language. [...]" in the role prompts. By designating different roles and backgrounds in the role prompts, we can assemble referee teams with specific expertise in various domains.
generic, coding, writing, and knowledge. As simple
As shown in Figure our specially designed knowledge, writing, and coding groups can outperform match the evaluations of other groups in corresponding categories. As for the generic group, found that it performs well overall compared to its counterparts. The results further underscore the effectiveness of the role prompts and reveal potential avenues for further optimizing the framework, such as using mechanisms like dynamic role prompt specification.
Table 3: Effect of diverse role specification on FairEval benchmark.
Evaluator
Methods
Acc. (%)
Kap.
ChatGPT
Single-Agent
53.8
0.27
ChatGPT
Multi-Agent (without Diverse Role Prompts)
53.8
0,25
ChatGPT Multi-Agent (with Diverse Role Prompts)
60
0.33
Generic Group
Knowladdge Group
Writing Group
Coding
Acc. (%6)
Generic
Knowledge
Writing
Coding
Figure 3: Evaluation quality of different groups on different categories.
which includes three different roles: general public, news author, and critic

---

Published as a conference paper at ICLR 2024
4.2 THE STUDY OF COMMUNICATION STRATEGIES
As shown in Figure] 2 we also design three different communication strategy termed as one-by- one, simultaneous-tall -takk, simultaneous sotiltanneoosttll-wwiiie formulations can be found in Appendix [ As depicted in Figure distinct communication strate- gies exhibit varying behaviors depending on the role-turn configuration. However, they peak at a similar performance range of 60.0%-62.5% in accuracy. Furthermore, the timultaneous-talk-wiiei summarizer strategy appears to offer slightly better scalability, as evidenced by its consistent upward trajectory with increasing role numbers and discussion turns. It is hypothesized that this is because the summarization-style history doesn't expand rapidly with the context length, thereby preserving the reasoning capability of the LLMs. Meanwhile, variations in performance among three differ- ent communication strategies underscore the influence of different strategies on the effectiveness of the evaluation quality, revealing the potential for further exploration and optimization of ChatEval. Thus, future studies could be aimed at more comprehensive understanding of different communi- cation strategies, and how they could be effectively employed to enhance performance, This could serve avenue for substantial improvements and novel insights in the multi-agent debate frame- work.
The detailed descriptions and formal
4.3 THE IMPACT OF ROLE NUMBERS AND DISCUSSION TURNS
We then study the impact of different role numbers and discussion turns. From Figure 4a 4b and 4c discernible trend is observed in the relationship between the role number and both Acc. and Kap- As the role number increases, there is a corresponding growth in performance, underscoring the effectiveness of incorporating diverse roles.
Conversely, no significant upward trend is detected with respect to the increase in discussion turns, as is shown in Figure 4d and 4e This observation aligns with the findings in Liang al. (2023)
Du et al. (2023). highlighting a consistent phenomenon where continual discussion often leads to stagnation
degradation of performance. As we mentioned before in Section 4.2 such trend may be attributed to issues associated with the rapidly ever-increasing context length, which nuanced understanding of
consequently diminishes the performance. These results prompt the balance needed between role-turn dynamics to optimize the performance of ChatEval.
Rale
(a) Results Role Numbers (One)
(b) Results Role Numbers (Sim)
(c) Results vs Role Numbers (wSum)
Micuttos
Biccusod
SAizus6a1 Rone
(d) Results vs Discussion Turns (One)
(c) Results vs Discussion Turns (Sim)
(f) Results vs Discussion Turns (wSum)
Figure 4: Comparison of different configuration of ChatEval on FairEval Benchmark. We use One, Sim and wSum to denote one-by me-bb--oee, simultaneously-tad -alk and simultaneously-taleee walk-----llw-lleeee wwwhwwitttariiie respectively.

---

Published as a conference paper at ICLR 2024
4.4 QUALITATIVE ANALYSIS
Figure S showcases the debate process towards the evaluation of two assistants' responses to the open-ended question "What are the most effective ways to deal with stress?" We can find that both of the responses produce similar strategies and equally compelling descriptions for dealing with stress, making it challenging to discern significant disparity in terms of quality. It is in this context of nuanced evaluation that the significance of the ChatEval process emerges.
We can pinpoint several human-like behaviors exhibited by the agents that can enrich our compre- hension of the evaluation process; (1) Opening Statement: Alice initiates the dehate with clear stance, establishing the foundational argument and guiding the trajectory of the subsequent dis- course. (2) Alternative Proposal: Bob introduces an alternative viewpoint, emphasizing the need to consider diverse interpretations. This not only broadens the discussion but also stimulates critical thinking. In the context of debate, the introduction of an alternative proposal prevents the stag- nation of thought, challenges pre-existing bias, and uncovers considerations that might otherwise be overlooked, ensuring that the discussions persistent adherence to her initial stance, even when faced with opposing views, exemplifies com- mitment and challenges other participants refine their perspectives. This behaviour ensures the conversation remains robust, focused, and continually evolving, driving all participants to higher level of engagement and critical thinking. a collective agreement amongst the participants, which is reached through mutual understanding and compromise, underlining the value of each presented viewpoint.
well-rounded.
(3) Stance Maintenance: Alice's
(4) Seeking Consensus: The discussion's 's climax reveals
In light of the above, ChatEval stands out not just as a tool for comparison but as embodiment of interactive natural language dialogue. By simulating human argumentative interactions, differen- tiates itself from statie, singlepreseeted opinions. This dynamic interaction showcases the richness and complexity of language, capturing nuances often missed in singular viewpoints. As such, Chat- Eval offers reliable evaluation process that not only mirrors human discourse but also highlights the transformative power of collaborative dialogue. This positions it uniquely, underscoring its sig- nificant potential to execute text evaluation tasks both reliably and effectively.
Question
Both Assistant and 2 provided comprehensive and detailed answers. [..]] However, Assislant 2 went a step further by mentioning mindhilness social support, techniques lkk progressive muscle re relaxaton, aromatheraoy. and massage Tharefore. would lean towards Ass stant 'or providing slightly more detailed responsel
ahet the most mosteffective ways to deal with stress?
rels
Assistant
effective ways deal with Exercise regularly: Physical activity can help reduce stress and improve moed.
Here
asseesments, However, would point out that Assistant Itss answer was more conciso and to Bhe paint theil ...] In your opinion, does the adcitional detail provided Assistant2 make response more helpful, or could the conciseneee Assislant 1's responie seen more beneficial?
agree
2. Practice relaxatinn te techniques: Deep breathing.
meditation, and yoga some effective ways relex and
reduce stress levels.
3. Get enough sleep: Lack of sleep can increase stress levels, Try to get least 7-8 hours of sleep each night [...]
In terms of cons stency with the question. both responses are equally consistent as they both provided effective to deal with strees, [..]] Therefore, believe both responses are equally valuable.
Assistant 2
Stress can be caused by variety of factors, including work, helationships, health and financial some effective ways to manage stress: 1. Exercie: Reqular exercie releases endorphins, which are natural stress-relievers. can be as simple as going for walk jog, Mindfulness: Windfulness practices, such as or deep breathing, can help to reduce stress and improve focus.
Here
I appreciate the points made by both you. W While Ass stant I's resporsie was indead concise. belleve that [...] Therefore. aill towards Assistant 2's response being slightlyl more helpful this moment
yoga,
fienes class.
Sociial S Support: Spending time with family and friends, seeking fupport mental heal professlonal, provide a sense of connectin and help reduce feelings of isolation and lonelines.
After discussing with my co-workers, belleve both Assistants performed equally well providing comprehensive and accurate responses.
[..]
Figure 5: An illustrative example from ChatEval presenting a discussion process involving partic-
ipants Alice Bob and Carol They initiate a group talk with the purpose of
determining the superior response from two assistants. Due spatial constraints within the layout, we use [...] to omit some redundant text.
8

---

Published as a conference paper at ICLR 2024
5 RELATED WORK
Automatic NLG evaluation In the landscape of NLG, evaluating the quality of texts represents a particularly arduous task. For significant period, evaluation was primarily dependent on human annotations that is labor-intensive and limited by scalability issues. Automatic NLG evaluation at- tempts to address these challenges by leveraging computational models assess the quality of texts. Previous work lies on the following categories: (1) n-gram-basee based metrics: ROUGE (Lin] (004) is a set of metrics that compute the amount of overlap between n-grams in the machine-generated sum- maries and the reference summaries. BLEU (Papineni et al. 2002) compare the generated texts with reference translations, based on the co-occurrence of n-grams in both texts. In spite of being easily and widely used, the above method is incapable of capturing syntactic and semantic similarity (Stent al., 2005). (2) embedding- -based metrics: Word embeddings are vector representations of words that capture their semantic properties. A bunch of work leverages word embeddings to evaluate the semantic similarity between two pieces of texts. BERTScore (Zhang et al.) 2019) use contextual ized word embeddings from transformer models like BERT (Devlin et al.) (018)) BLEURT (Sellam) et al. 2020) utilize supervised training data to enhance the performance. MoverScore (Zhao et al,
2019 combine contextualized word embeddings with Earth Mover's 's Distance (Rubner | 2000). (3) LLM-based metrics: Amidst the flourishing advancement of LLM which embodies a wealth of information derived from extensive training data, using L.LM
evaluator has experienced notable progress. GPTScore (Fu et al.] (023) utilize conditional probability to assign the texts a score representing its quality. (Wang et al. (2023a) explore the potential of utilizing ChatGPT as an NLG evaluator by prompting it to score texts directly. [Wang et al. (2023c) curate reliable dataset containing pairwise comparison and evaluation explanation which can be used train a foundation model making it a better evaluator. Bai et al. fairer evaluation results. G-EVAL (Liu et al. 2023b) propose beepability-weghhhi techniques to
(2023) propose decentralized evaluation to provide
calibrate the score given by single LLM.
Communicative Agents Most recently, significant attention has been dedicated to the development of communicative agents. These agents, often acted by LLMs like ChatGPT or GPT-4, are designed to interact and communicate effectively with other agents or human users using natural language. The primary goal is to facilitate more productive and efficient interaction and collaboration as differ- ent agents can autonomously communicate and negotiate to tackle more complex task collectively. Several studies have explored various aspects of communicative agents. [ et al. (2023a); Chen al. (203)) propose a cooperative agent framework enabling agents to autonomously cooperate to solve complex tasks. Paakk al. (2023) create a sandbox environment consisting of 25 individual virtual entities endowed with character description and memory system. Every intelligent agent is capa- ble of autonomously interacting with other agents and the environment simulating reliable human behavior. Dong et al. (2023); Qian et al. (2023) incorporate a waterfall model to manage a multi- agent virtual team towards software development. Liu al.] (2023a) utilize a sandbox environment to curate reliable datasets in better alignment with human preference and train a eocially-aligned LLM. Liang et al. (2023) and Du et al. (2023) also make use of the multi-agent debate framework in other scenarios such as translation and arithmetic problems resulting in better results. WWan et al. (203d) propose an alternative method called self-collaboration to enable the communication of agents by utilizing a single LLM prompted by multi-persona descriptions. Mandi al. (2023) pro- pose a novel framework designed for the collaboration of multiple robots, utilizing multiple LLMs to enhance coordination and strategic planning among the robots. Concurrent with our work, Li jet al.] (2023b); Zhang et al.) (2023) also propose a similar approach. However, they probe differ- ent dimensions of improving LLM-based evaluators and do not explore the effectiveness of natural language interaction.
CONCLUSION
In this paper, we present evidence that ChatEval contributes to improving the evaluation perfor- mance concerning text quality, aligning more closely with human preferences. We emphasize the necessity of the diverse role specification and propose distinct communication strategies integral components within ChatEval. Our qualitative analysis of the discussion process conveys insightful intuitions about how texts are evaluated by ChatEval and substantiates our approach's 's ability to sup- port comprehensive evaluations akin to human judgment, thereby demonstrating the reliability and efficacy of our framework.

---

Published as a conference paper at ICLR 2024
REFERENCES
Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozii Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, al. Benchmarking foundation models with language-model-a -am as-an-examiner arXiv preprint arXiv:2306.044 04180, 2023.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine transiation and/or summarization, pp. 65-72, 2005.
Chris Callison-Burch. Fast, cheap, and creative: Evaluating translation quality using amazon's mechanical turk. In Proceedings of the 2009 conference on empirical methods natural language processing, pp. 286-295, 2009.
Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. Evaluation of text generation: A survey. CoRR, abs/2006.14799 2020. URL http::ttp//..........t hatpps///axx....rr..//0006...../ 14799
Weize Chen, Yusheng Su. Jingwei Zuo, Cheng Yang. Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Agentverse: Facili- tating multi-agen collaboration and exploring emergent behaviors in agents, 2023.
Cheng-Han Chiang and Hung-yi Lee. Can large language models be an alternative to human evalu- ations? arXiv preprint arXiv:2305.01992 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See htlps://vicuna. Imsys. org (accessed 14 April 2023), 2023.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810. 18.0000005 2018.
Yihong Dong, Xue Jiang. Zhi Jin, and Ge Li. Self-collaboration code generation via chatgpt. arXiv preprint arXiv:2304.0753 075500, 2023.
Yilun Du, Shuang Li, Antonio Torralba, Joshua Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023.
Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire, arXiv preprint arXiv:2302..41111 2023.
Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. Human-like summarization evaluation with chatgpt. arXiv preprint arXiv:2304.0255 2023.
Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anushree Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tür. Topical-chat: Towards knowledge- grounded open-domain conversations. 2019.
Marzena Karpinska, Nader Akoury, and Mohit lyyer. The perils of using mechanical turk to evaluate open-ended text generation. arXiv preprint (XX.::2119006..3. 2021.
Grzegorz Kondrak. N-gram similarity and distance. In International symposium string processing and information retrieval, pp. 115-126. Springer, 2005.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large scale language model society. arXiv preprint arXiv:230..17766 2023a.
Ruosen Li, Teerth Patel, and Xinya Du. Prd: Peer rank and discussion improve large language model based evaluations. arXiv preprint arXiv:2307.0276. 2023b.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang. Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi- agent debate. arXiv preprint arXiv:2305. 19118, 2023.
10

---

Published as a conference paper at ICLR 2024
Chin- Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74, 2004.
Ruibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M Dai, Diyi Yang. and Soroush Vosoughi. Training socially aligned language models simulated human society, arXiv preprint arXiv:2305. 16960, 2023a.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303. 16633 2023b.
Andrea Luppi, Pedro AM Mediano, Fernando Rosas, Negin Holland, Tim D Fryer, John O'Brien, James B Rowe, David K Menon, Daniel Bor, and Emmanuel A Stamatakis. A synergistic core for human brain evolution and cognition. Nature Neuroscience, 25(6) 2022.
Zhao Mandi, Shreeya Jain, and Shuran Song. Roco: Dialectic multi-robot collaboration with large language models. arXiv preprint urXiv:2307.047388 2023.
Mary L McHugh. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276-282, 2012.
Shikib Mehri and Maxine Eskenazi. Usr: An unsupervised and reference free evaluation metric for dialog generation. arXiv preprint arXiv:2005.0000 00456, 2020.
Jekaterina Novikova, Ondłej Dušek, Amanda Cercas Curry, and Verena Rieser. Why we need new evaluation metries for NLG. In Proceedings of the 2017 Conference on Empirical Meth- ods in Natural Language Processing, pp. 2241-2252. Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-111 1238. URL https://
acl belanthology wwtp///1g111 1238
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic
evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Joon Sung Park, Joseph C O'Brien, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint
arXiv:2304.03442,, 2023.
Chen Qian, Xin Cong. Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. arXiv preprint arXiv:2307.07924 2023.
Yossi Rubner, Carlo Tomasi, and Leonidas Guibas. The earth mover's distance as a metric for image retrieval. International journal of computer vision, 00:99-121, 2000.
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. Bleurt: Learning robust metrics for text gener- ation. arXiv preprint arXiv:2004.04696, 2020.
Chenhui Shen, Liying Cheng, Yang You, and Lidong Bing. Are large language models good evalu- ators for abstractive summarization? arXiv preprint arXiv:2305, 13091, 2023.
Amanda Stent, Matthew Marge, and Mohit Singhai. Evaluating evaluation methods for genera- tion in the presence of variation. In International conference intelligent processing and computational linguistics, pp. 341-351. Springer, 2005.
Chris Van Der Lee, Albert Gatt, Emiel Van Miltenburg, Sander Wubben, and Emiel Krahmer. Best practices for the human evaluation of automatically generated text. In Proceedings of the 12th International Conference on Natural Language Generation, 355-368, 2019.
Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. Is chatgpt good nig evaluator? a preliminary study, arXiv preprint arXiv:2303. 04048, 2023a.
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint N.Vv:2305.17922 2023b.
11

---

Published as a conference paper at ICLR 2024
Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang. Cuuxiang Wang. Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. Pandalm: An automatic evaluation benchmark for I1m instruction tuning optimization. arXiv preprint arXiv: 2023c.
Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing cognitive synergy in large language models: A task-solving agent through malti-persooa self- collaboration. arXiv preprini arXiv:2307.05300 2023d.
Anita Williams Woolley, Christopher F Chabris, Alex Pentland, Nada Hashmi, and Thomas W Mal- one. Evidence for a collective intelligence factor in the performance of human groups, science, 33066004))686-688, 2010.
Ning Wu, Ming Gong, Linjun Shou, Shining Liang, and Daxin Jiang. Large language models are diverse role-players for summarization evaluation. arXiv preprint arXiv:2303.15078 2023.
Tianyi Zhang, Varsha Kishore. Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. arXiv preprint arXiv:1904. 190.099994 2019.
Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. Wider and deeper Ilm networks are fairer IIm evaluators, 2023.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. Mover- score: Text generation evaluating with contextualized embeddings and earth mover distance. arXiv preprint 1909:::909.00000 2019.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.0560
2023.
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. Towards a unified milti-dimeesional evaluator for text generation. arXiv preprint arXiv:2210.071 197, 2022.
A PROMPT TEMPLATE AND DIVERSE ROLE PROMPTS
The overall prompt template is shown in Table 4 we draw inspiration from Wu et al. (2023) and design several different tole descriptions as follows.
General Public You are now General Public, one of the referees in this task. You are interested in the story and looking for updates the investigation. Please think critically by yourself and note that it's your responsibility to choose one of which is the better first.
Critic You now Critic, one of the referees in this task. You will check fluent writing, clear sentences, and good wording in summary writing. Your job is to question others judgment to make sure their judgment is well-considered and offer alternative solution if two responses same level.
the
News Author You News Author, of the referees in this task. You will focus the consistency with the original article. Please help other people to determine which response is the better one.
Psychologist You are Psychologist, one of the referees in this task. You will study human behavior and mental processes in order to understand and explain human behavior. Please help other people determine which response is the better one.
Scientist You are Scientist, one of the referees in this task. You are professional engaged in systematic study who possesses strong background in the scientific method, critical thinking, and cooblem-solvinee abilities. Please help other people to determine which response is the better one.
B IMPLEMENTATION DETAILS
We choose to utilize models from OpenAl's GPT family as our LLMs in ChatEval, including GPT-4 and ChatGPT
and set the temperature to 0 to ensure reproducibility. The rationale
12

---

Published as a conference paper at ICLR 2024
[Question]
[source_text
[The Start of Assistant I's Answer]
(compared text )
[The End of Assistant I's Answer]
[The Start of Assistant 2's Answer]
(compared text. text.too)
[The End of Assistant 2's Answer]
[System]
We would like to request your feedback the performance of two AI assistants in response to the user question displayed above.
Please consider the helpfulness, relevance, accuracy, and level of detail of their responses. Each assistant receives an overall score on scale of to 10, where a higher score indicates better overall performance.
There are a few other referees assigned the same task, it's your responsibility to discuss with them and think critically before you make your final judgment. Here is your discussion history:
(chat history]
(role_desse pool_deecription
Now it's your time to talk, please make your talk short and clear, nagent.name)-
Table 4: The prompt template for FairEval Dataset. We replace the colored slot with real text before querying the LLMs. Note that we use the same template when conducting single-agent-basee experiments and ignore the chat history and role description slot.
behind this selection is the exceptional performance these models offer, being among the most ad- vanced and powerful in the world. Additionally, their accessibility and case of use through APIs enable us to directly call and interact with the models during our research, significantly simplifying the process. In current research, a given multi-agent group, all L.MMs belong to the same GPT family model, either all GPT-4 or all ChatGPT. We acknowledge the potential of heterogeneous groups for future research, which could provide fascinating insights into how strong models and weak models can cooperate in a multi-agent setting. Additionally, unlike previous work like Du al. (2023), we do not explicitly ask the debater agents to reach consensus at the end of the debate. In situations where the response format relies on direct comparison, we derive the final results from the majority vote among various annotators. Conversely, if the response format requires a direct score, we calculate the average score obtained from multiple annotators. This methodological approach ensures the impartiality and balance of our evaluation process.
focus homogeneous groups of L.LMs. That is, within
C FORMAL DEPICTION OF DIFFERENT COMMUNICATION STRATEGY
In this paper, we predominantly adopt the following three different communication strategies, which are:
1. One-By-One. During each round of the debate, the debater agents take turns in set order to generate their response based on the current observation. When it's time for a debater agent to respond, we directly concatenate what previous other agents have said into its chat history slot. Please refer to Algorithm
2. Simultancous-Tiil Talk. Unlike the one-by-one strategy, we carry out an alternative com- munication strategy called simultaneous-talk where debater agents are prompted asyn- chronously generate responses in each iteration of the discussion to nullify the impact of the speaking order. Please refer Algorithm
3. Simultaneous-Talke Talk-witth-Summrrirr The main difference between this strategy and simultaneous-tall is that we additionally employ another LLM as a summarizer. At the end of each iteration of the debate, we prompt this extra L.LM summarize the messages
13

---

Published as a conference paper at ICLR 2024
conveyed far and we replace the chat history slots of all debater agents with this sum- marization. Please refer to Algorithm
Algorithm 1: One-by--eee Oy-ebe
input agents number N, discuss turn T' group of debate agents [D1, DN chat history of each agent [H Hx], answer extracter (either majority vote or average score)
EXT
output: Final results for text evaluation ANS
for 1. 4- 0 to do
for n 1 to N do
hn f D,,(H,); // utilize agents to generate responses
for m 4 n to do
if > then
Hm + Hm + hn;
// concttenat current response to later agents' chat
history
end
end
end
10 end
ANS < "",
12 return ANS:
Algorithm 2: Simultaneous-Talk
input agents number N, discuss turn T, group debate agents [D..... Dx]. chat history of each agent [H,.... .Nv), wanwer.extraaee (either majority vote average score) EXT, buffer BUF
output: Final results for text evaluation ANS
for L + to T do
for n +- 1 to do hn (H,,): // utilize agents to generate responses buf +- buf hni
// add the responses in current turn to the buffer
end for
1 to do
H, - Hn + buf: // add the buffer to all agents' chat history
end
end
10 ANS + EX HNND:: HNN);:
return ANS:;
D GENERALIZATION TO SMALLER MODELS
Our primary focus in the paper is on the framework implemented with the most powerful LLMs such as ChatGPT and GPT-4. However, we also carry out the experiments on Llama2-Chat-77 and Vicuna-7b-v1.55 As indicated in the Table 5 Llama2-Chat achieves only marginal performance under single-agent CoT methods. Despite reaching an accuracy of 37.5% 37.5%, the negative kappa coeffi- cient suggests that the observed agreement is less than what would be expected by chance. However, is noteworthy that through the application of a multi-agens debate, Llama2-Cri -hha demonstrates modest improvement, albeit still marginally above chance levels.
14

---

Evaluator Published as a conference paper at ICLR 2024
Algorithm 3:
input agents number N, discuss turn T, a group debate agents [D1,.. ...... Dx]. chat history of each agent [H1 Hx], answer extracter (either majority vote average score) EXT, buffer BUF F, summarizer SUM S
output: Final results for text evaluation ANS
for t <- to T do
for n f- 1 to do
hn (HH))
// utilize agents to generate responses buf +- buf ha;
// add the responses in current turn to the buffer
end
for n 1 to do
Hn < Hn + SUM(BUF);
// add the summarized buffer to all agents' chat history
end
end
10 ANS + EXT(H, HNN): HNDD:
return ANS;
In contrast, Vicuna exhibits more robust performance improvement compared to Llama2-Chal ChatEval notably enhances its capabilities, achieving an accuracy of 52.3% and a kappa coefficient of 0.19, indicative of fair agreement beyond chance. While both models show limitations in their performance, these results demonstrate the effectiveness of ChatEval and underscore its positive impact.
Table 6: Average Cost on FairEval.
Table 5: Accuracy (Acc.) and Kappa correlation coefficient (Kap.) of smaller models on FairEval.
Cost
Time
Evaluator
Acc. (%)
Kap.
Human
$90
240min
GPT-4(SA)
$2.90
10min
Llama2-Chat-78
37.5+1.3
0.01
GPT-4(EN)
$8.70
10min
(SA)
GPT-4(MA)
$12.30
36min
Llama2-Chat-7B (MA)
40.0+1.1
0.05+0.01
ChatGPT(SA)
$0.11
3min
ChaiGPT(EN)
S0.33
3min
Vicuna-788 7B-v1. (SA)
45.041.5
0.12+0.01
ChatGPT(MA)
S0.41
I Immin
G-EVAL-3.5
$2.20
3min
Vicuna-7B-v1.5 (MA)
52.3
0.19
G-EVAL-4
$58
10min
FairEval-ChatGHP
$0.34
3min
FairEval-GPTS
$6.38
10min
AVERAGE COST
We release the average cost our framework and other methods against with employing human evaluators. In the Table 6 it is evident that the evaluator based on LLMs significantly reduces both time and financial expenditures. Although the ChatEval framework incurs higher costs than the SA and MA ensemble methods due to multiple inference rounds necessary for final judgment derivation, it remains more cost-effective than employing human evaluators. We consider this an acceptable trade-off for the benefits provided.
15


Reviews:
[{"review_id": "qUyQpABISB", "reviewer": "Reviewer_2Bwn", "summary": "This paper focuses on the task of automatic text evaluation and highlights the limitations of existing n-gram metrics in correlating with human judgments. The authors propose ChatEval, a framework leveraging LLMs for automatic text evaluation. Instead of prompting a single LLM to assess the generation quality, ChatEval integrates multiple LLMs, facilitating debates among them to enhance the robustness and human-like quality of evaluation results.", "questions": "1. As mentioned in section 3.2, two agents are employed in the implementation of ChatEval. Can they always reach an agreement at the end of debates?\n\n2. Regarding Table 2, does \"MA\" in the table represent multiple agents employed in the debating way. Are there corresponding results for multiple LLMs used in an ensemble manner, aligning with the methods in Table 1?\n\n3. Considering the performance gap between G-EVAL-3.5 and ChatGPT(MA) in Table 2 for dialogue response generation, it appears that the effectiveness of the proposed framework is influenced by the chosen LLM. Has there been an evaluation of ChatEval's generalization ability?\n\n4. It seems that the G-Eval framework can be easily adapted to open-ended QA evaluation. From the performance differences in Table 2 between G-Eval and ChatEval, I wonder whether the multiple-agent debating framework really works better than the single-agent COT framework?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using LLMs for automatic text evaluation is intriguing and holds potential.\n2. The paper is well-structured and easy to follow. The inclusion of informative figures and tables enhances clarity.", "weaknesses": "1. The paper may benefit from providing more in-depth technical details about the ChatEval framework. While a prompt template is present in the appendix, the determination of LLM roles and the potential impact of varying roles and orders remain unclear. In addition, the framework relies on prompting LLMs, however, the paper lacks sufficient information on the design of prompts and their robustness.\n2. The evaluation results present a challenge in assessing whether the multi-agent debate framework outperforms existing LLM-based evaluators in general cases. Addressing questions related to the listed questions would contribute to a more informed judgment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KlCILgMECN", "reviewer": "Reviewer_asDf", "summary": "In this work, the authors propose a new framework called ChatEval that contains multiple debater agents to evaluate the quality of the text generated.  Each debater agent is an LLM aimed to generate responses based on prompts and there are 3 communication strategies followed between the multiple debater agents.\nThe authors evaluate the proposed approach on two benchmarks Fair Eval and Topical Chat and find that the proposed framework aligns closely with the human preferences.", "questions": "1. In Tables 1 and 2 why is only Multi Agent (ChatEval) using the strategy of One-by-one mentioned? Why are results from other strategies not provided in the table?\n2. In Section 3.3, it is claimed that Chateval surpasses Fair Eval's best results but this does not seem to be the case with regards to the Kappa correlation coefficient. \n3. How effective is the framework at detecting issues on hallucination which is quite common along LLMs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses an important problem of evaluating textual generations by using LLMs and trying to reduce the shortcomings faced with traditional human-based evaluations.\n2. Results demonstrate the framework's effectiveness on the FairEval and Topical Chat dataset by comparing it against various other frameworks on this dataset.", "weaknesses": "1. In the abstract, the authors mention that one of the drawbacks of using humans in the pipeline is the time and labor cost. The analysis of the proposed framework would benefit significantly if there is any analysis in terms of time spent by humans for annotation vs LLM.\n2. The framework seems to be suited only for short conversations based on the Analysis in Section 4.3. This would be an issue when extending this framework for evaluating longer dialogue conversations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rO2PAjOJpC", "reviewer": "Reviewer_zL85", "summary": "The paper introduces a new way of using LLMs for text evaluation through multi-agent discuss and evaluation. Instead of using the LLM to generate one round of evaluation, the authors propose to design multiple LLM agents with diverse roles, and let them go through several rounds of discussion to make the final evaluation. Results on one open question answering dataset, and one dialogue dataset demonstrates better alignment with human judgement for the proposed multi-agent framework. The authors also conduct additional experiments to examine individual components of the system, including the design of communication pattern among agents, diverse role of agents, number of agents involved and number of discussion rounds, which are helpful for future study.", "questions": "Please see weakness above. In particular the clarification on implementation details/experiments and discussion on the limitation.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The authors carefully study the different components of the multi-agent system, including the communication pattern, agent role design, number of agents and discussion rounds. The additional experiments for individual components and detailed discussion are very helpful for understanding the design choices and inform future research.\n2. Results on two benchmarks showcase improvement of using multi-agent with both ChatGPT and GPT-4.", "weaknesses": "1. The experiments are conducted only on two datasets that are relative small. On Topical-Chat the win/loss of single-agent and multi-agent is mixed on different rubrics, and ChatGPT and GPT-4 seem to have different behaviors. Some more analysis on each rubric will be helpful rather than simply compare the improvement on average.\n2. The experiments are conducted only with ChatGPT and GPT-4, thus unclear whether the proposed method could generalize to other LLMs, or limiting to GPT models. Evaluation on more models of different capacities could help understand when the multi-agent framework would work, is there emergent ability that require model with decent performance to have constructive discussion and evaluation.\n3. There is no discussion on limitation of the proposed method. For example, the inference speed/cost will get affected given that multi-agent requires more rounds of generation with the LLM.\n4. Some implementation details could be clarified. \n    1. In section 3.2, it is mentioned that the default results are obtained with 2 agents, what are these 2 agents? Appendix describes the implementation of 5 agents, while in section 4.1 it is mentioned that there are three different roles.\n    2. In section 4.1, the authors mentioned comparison between specific roles and default setting, but seems Figure 3 only includes results of specific roles.\n    3. What is the specific version of ChatGPT and GPT-4 used in the paper, since that will affect model performance as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mGWuDKYxlf", "reviewer": "Reviewer_rxGE", "summary": "This paper proposes ChatEval, a multi-agent framework for text evaluation that simulates human collaborative and argumentative dialogue. \nChatEval demonstrates superior performance on two benchmarks, FairEval and Topical-Chat, compared to single-agent and other LLM-based methods.", "questions": "1. About fairness comparison. The setting of ChatEval is 2 agents with 2 discussion turns. It is not clear what the setting of Multi-Agent (Ensemble) is like.\n\n2. More detailed about the Ensemble method.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work has implications for the field of text evaluation, which aligns better with human preferences.\n\n2. The authors have conducted extensive experiments and provided a thorough analysis of the results. The proposed method outperforms single-agent and other LLM-based methods on two benchmarks, demonstrating its effectiveness.\n\n3. The work also highlighted the importance of diverse role prompts, which is a valuable insight for future research.", "weaknesses": "1. Figure 4 does not show whether simultaneous-talk-with-summarizer can outperform one-by-one. Although the chart has an upward trend, it is still necessary to further increase the Role Numbers and Discussion Turns to prove the author's point of view.\n\n2. The specific examples of three different communication strategies are lacking, which can be put into the appendix.\n\n3. The resource cost of ChatGPT/GPT-4 is a problem that needs to be considered, and the paper does not compare the resource consumption of previous methods with the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VuFZtwCgpq", "reviewer": "Reviewer_XZLb", "summary": "This paper presents ChatEval, a multi-agent debate framework that utilizes multiple LLMs to debate and evaluate the quality of machine-generated texts and improve the quality of the final text as a result of the process. The experiments consider two benchmarks: (1) FairEval along with human annotation results from Wu et al. (2023) – containing 80 open-ended questions with three annotators’ annotations on two model outputs and (2) Topical-Chat along with human annotation results from Mehri & Eskenazi (2020) – containing 60 dialogue contexts with human annotation on six model outputs. Based on the findings, the authors claim that ChatEval “delivers superior accuracy and correlation in alignment with human assessment.” In addition, they find that using diverse role prompts (using different personas) helps improve performance.", "questions": "Minor suggestions\n* In the abstract, I’d encourage the authors to include actual numerical results, as opposed to the textual description “superior accuracy and correlation”.\n* In the abstract, it would be good to mention specific tasks or contexts that these claims are made.\n* In Figure 1, if possible, showing actual (full) texts or abbreviated versions of texts (but longer than what it’s currently shown) would help readers understand how different model outputs (infused with different personas) contribute to overall performance boost.\n* In the experiments section, the authors can clarify whether they use one model for all roles or use different models for different roles (it can be inferred from the results, but it’s better to clarify it before presenting the results).\n* In Figure 4, the range on the y-axis should match in order to facilitate easy comparison between configurations.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "* Note that I’ve listed both strengths and weaknesses here as they are topically grouped together.\n* The work provides a good list of related work and contextualizes itself in relation to prior/concurrent work. In doing so, however, it is unclear how the main goal of the proposed approach mirrors or differs from prior work (other than the proposed method itself; scoping the contribution/types of tasks more tightly can help here) and what kind of pros and cons or trade-offs there are in comparison to these other works that also leverage multiple LLMs. I’d encourage the authors to provide more information on this aspect both in introduction and related work, as I find the current explanation to be a bit hard to understand in the related work (i.e., “Concurrent with our work, Li et al. (2023b); Zhang et al. (2023) also propose a similar approach. However, they probe different dimensions of improving LLM-based evaluators and do not explore the effectiveness of natural language interaction.” In this sentence, what are “different dimensions” and what does “natural language interaction” mean here?)\n* I think the idea of exploring diverse role specifications and communication strategies has a lot of potential. However, the current version that’s explored in the paper lacks depth and justification. For instance, there is one line justification for how the diverse role specification was formulated (“We take inspiration from Wu et al. (2023) and formulate an analogous role description.”), but it is unclear what other alternatives there are, why this is a good baseline to default to, what potential limitations are with this approach. Same thing applies to the design of communication strategies. The authors designed three strategies, but it is unclear how the design decisions were made and what factors are accounted for.\n* Regarding the experiments, I have some concerns about the authors’ main claim. In Table 1, it’s unclear whether the difference between different methods is meaningful since it’s smaller than the difference between human annotators. More discussion on the inter-annotator agreement and how we should interpret the results would be necessary to make the claim about “superior accuracy and correlation”.  To account for a small number of instances, I’d highly encourage the authors to include standard deviations or errors on all tables and figures. On the other hand, I do appreciate that the authors delve into the importance of diverse role prompts, communication strategies, and the impact of role numbers and discussion turns. However, the qualitative analysis lacks nuance when reporting the finding; I’d be very cautious to call model behaviors “human-like” and “not just as a tool [...] but as an embodiment of interactive natural language dialogue” solely based on the four patterns observed in the experiments.\n* Overall, I find the paper to have much room for improvement in terms of writing, need for more justification and explanation for design choices for the proposed method, and need more rigor in analyzing and reporting the experiment results.", "weaknesses": "Provided above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)