IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answering

Authors: ruosen  li
ruochen  li
barry  wang
xinya  du
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 22:45:38

Abstract: To evaluate Large Language Models (LLMs) for question answering (QA), traditional methods typically focus on directly assessing the immediate responses generated by the models based on the given question and context. In the common use case of humans seeking AI assistant’s help in finding information, these non-interactive evaluations do not account for the dynamic nature of human-model conversations, and interaction-aware evaluations have shown that accurate models are not necessarily preferred by humans Lee et al. Recent works in human-computer interaction (HCI) have employed human evaluators to conduct interactions and evaluations, but they are often prohibitively expensive and time-consuming to scale. In this work, we introduce an automated evaluation framework IQA-EVAL to Interactive Question Answering Evaluations, more specifically, we introduce LLM-based Evaluation Agent (LEA) that can: (1) simulate human behaviors to generate interactions with IQA models; (2) automatically evaluate the generated interactions. Moreover, we propose assigning personas to LEAs to better simulate groups of real human evaluators. We show that: (1) our evaluation framework with GPT-4 (or Claude) as the backbone model achieves a high correlation with human evaluations on the IQA task; (2) assigning personas to LEA to better represent the crowd further significantly improves correlations. Finally, we use our automated metric to evaluate five recent LLMs with over 1000 questions from complex and ambiguous question answering tasks, which would cost $5k if evaluated by humans.

Main paper: 
 introduction: 
 The advent of Large Language Models (LLMs) has significantly advanced the field of natural language processing (NLP), enabling systems to perform a wide range of tasks with remarkable proficiency [Zhao et al., 2023; Wei et al., 2022a; Yang et al., 2024; Du, 2024b; Jing et al., 2024] . Among these tasks, question answering (QA) has emerged as a critical and representative goal-oriented application, demonstrating the potential of LLMs to generate informative responses as an assistant [Biancofiore et al., 2024] . Multiple methods have been proposed to enhance the faithfulness and explainability of generated information [Wei et al., 2022b; Long, 2023; Li and Du, 2023; Du, 2024a] . Beyond developing these methods, rigorous evaluation of the generated outputs is also crucial.Accurate and consistent evaluation helps researchers understand existing LLM's capacities and emerging human-LLM QA interactions [Chang et al., 2023; Lin and Chen, 2023; Chang et al., 2023] . Traditionally, automatic metrics such as accuracy have been used to evaluate models based on the quality of their direct answers to specific questions. However, as interactions between humans and LLMs grow more complex and nuanced, these traditional metrics often fail to capture the full spectrum of a model's capabilities (e.g. helpfulness and fluency), particularly in interactive QA settings [Liu et al., 2016; Deriu et al., 2021] , whereas interactions are crucial for user experience and system effectiveness, yet remains overlooked in traditional evaluation paradigms. Recent works such as [Lee et al., 2023 ] evaluated human-LLM interactions, a process that involves human participation and annotation. Although human evaluations for these interactions provide a closer approximation to real-world use cases, this approach is significantly costly and time-consuming. Recognizing the need for automatic evaluation, works like G-Eval [Liu et al., 2023b] , LLM-Eval [Lin and Chen, 2023] , FaithScore [Jing et al., 2023] , and PRD [Li et al., 2024] proposed to automate the assessment of non-interactive LLM responses using LLMs as evaluators.Drawing insights from (a) using LLM for automatic evaluation and (b) literature of LLM-agents research Wang et al. [2024] ; Deshpande et al. [2023] , we propose IQA-EVAL framework to autoevaluate the performance of IQA models (i.e. LLMs) with LLM-based Evaluation Agent (LEA) to simulate and then evaluate interactions. By additionally incorporating personas, our experiments on a well-annotated dataset show that our methods align well with human judgments and provide a more comprehensive evaluation of LLMs in interactive settings than traditional metrics. Finally, we benchmark recent LLMs with new complex and ambiguous questions, and demonstrate that the accuracy of answers does not always transfer to the corresponding ranking of models on their capability of achieving good human-model interactions. See an overview of our system in Figure 1 .Our contributions are as follows:• We propose the first LLM agent-based automatic evaluation framework IQA-EVAL designed specifically to generate and then evaluate interactions in IQA. Our results demonstrate a strong correlation with human evaluations. • We propose persona-based LLM evaluation agents to better assess how models adapt to different user preferences and interaction styles. • We experiment with IQA-EVAL framework to benchmark the most recent LLMs on the IQA task, demonstrating the strength and general effectiveness of our framework on fully automated evaluation. 
related work: 
 Evaluating Interactions Traditional methods for human-model dialogue evaluation have often been centered around single-turn pairwise evaluation [Vinyals and Le, 2015; Li et al., 2016] . Some methods with multi-turn Likert scores emerge [Venkatesh et al., 2017; Zhang et al., 2018; See et al., 2019] but require time-consuming and costly collection of human-model conversations. In response, Ghandeharioun et al. [2019] suggests a self-play scenario where a dialog system engages in conversation with itself, employing a set of proxy measures for evaluation.Acute-eval [Li et al., 2019] and LLM-Eval [Lin and Chen, 2023 ] have advanced multi-turn evaluation frameworks, reflecting the increasing demand for sophisticated techniques that holistically capture human-AI interactions. Further studies in specific interaction domains like task completion [Liu et al., 2023a] , code generation [Yang et al., 2023] , and collaborative problem-solving [Lee et al., 2023; Huang et al., 2023; Fu et al., 2023] emphasize the need for evaluations that consider both environmental and human elements [Wang et al., 2023b] . Our work differs from these methods by introducing an automated approach that emphasizes interaction quality and significantly reduces the reliance on human annotations.LLM-based Agent for Simulation Recently, LLMs rises to demonstrate human-like intelligence, as evidenced in various studies [Xiao et al., 2023; Rao et al., 2023; Li et al., 2023; Jiang et al., 2023; bench authors, 2023; Brown et al., 2020; Touvron et al., 2023; Chowdhery et al., 2022] . The integration of LLMs into agents that simulate complex human behaviors and social interactions is an area of growing research interest [Maes, 1995; Wooldridge and Jennings, 1995; Xi et al., 2023] . For example, Park et al. [2022] and Gao et al. [2023] employ these agents to simulate human emotions, attitudes, and behaviors in social networks, while Park et al. [2023] leverages in-context learning to simulate human behaviors in sandbox world. Moreover, Argyle et al. [2022] utilizes "algorithmic bias" inherent in GPT-3 to reflect the response patterns of different human subgroups. Horton [2023] utilize LLMs in experimental setups of behavioral economics experiments to facilitate pilot studies. Additionally, Hämäläinen et al. [2023] and Wang et al. [2023a] investigate LLM-based agents in recommender systems to simulate and collect data on user behavior. These studies show the broad applicability and potential of LLM-based agents in simulating human behaviors and interactions across diverse applications. Our work utilizes LLM-based evaluation agents (LEAs) to fully automate interactive quality assessments, handling both interaction generation and evaluation, to enhance the evaluation of IQA models in realistic scenarios.Personas in NLP Personas are constructed profile prompts that represent key traits of a group of users, as defined in the HCI field, reflecting their characteristics, behaviors, and goals to guide the design of technologies that are well-suited to user needs [Cooper et al., 2014] . This approach enhances relevance and personalization in NLP applications [Nargund et al., 2022; Bamman, 2015; Sheng et al., 2021; Zhong et al., 2020] , offering significant potential for customizing engagement and improving the effectiveness of conversational agents [Li et al., 2016; Zhang et al., 2018; Chan et al., 2019; Madotto et al., 2019; Zheng et al., 2019] . Li et al. [2016] introduces a persona-based neural conversation model to enhance dialogue personalization and coherence. Zhang et al. [2018] develops personalized dialogue agents that incorporate user-specific details to enhance interaction. In our work, persona settings enable our framework to tailor interactions and assessments, aligning more closely with the specific characteristics and preferences of different user groups.3 IQA-EVAL: Evaluating Interactive Question Answering (IQA)In this section, we introduce our IQA-EVAL framework for automatically evaluating Interaction Question Answering Models (IQA models) with LLM-based Evaluation Agents (LEA). LEAs are used to simulate humans in the following two stages: (1) generating interactions with IQA models; and (2) evaluating interactions. Lastly, we discuss the use of personas to for LEAs. 
interaction generation with lea (stage 1): 
 Inspired by peer discussions Lee et al. [2023] ; Wang et al. [2024] , we prompt LEAs to simulate human behaviors for effective interaction generation with IQA models. The structured prompt includes three key components: (1) a role description;(2) a task description; and (3) instructions for the discussion.Role Description outlines the people that the LEA model will simulate during interactions. For example, the description for a standard persona could be: You are mimicking a human.Task Description briefly describes the action that LEA model needs to perform in the task. For example, in a multi-choice question answering task, the prompt could be structured as follows: You are trying to choose the correct answer for the given question. Both role and task descriptions can be adjusted based on the persona, as discussed in Section 3.3.Discussion instruction guides LEAs on their subsequent steps by providing detailed descriptions to facilitate progress in interactions. It comprises two essential components: (1) the actions to take; and(2) the detailed procedures to follow. For example, in a question answering task, the prompt specifies:You can ask an assistant questions for help. Please ask sub-questions to approach answers. In each turn, please only ask one sub-question to interact with the assistant.In the sub-question, please include all necessary information in the original question, such as the question and all options. If you know the answer, please output "So, the answer is: A, B, C, or D".At the start of an interaction, the LEA receives a system prompt that includes all three components above, along with the specific question to be addressed. As the LEA interacts with the IQA model, it generates sub-questions to request clarification of unknown entities, definitions, or particular aspects of the original question. Then, the IQA Model takes the questions as the input and output responses.After receiving responses, the LEA continues to pose further questions until it determines the final answer. The full prompt structure and interaction details are provided in Appendix C.2.Inspired by G-eval [Liu et al., 2023b] , which demonstrates that evaluations by GPT models align closely with human assessments in NLG tasks, we propose utilizing LEAs for interaction evaluation. LEAs assess interactions generated by LEAs and IQAs in Stage 1. The module takes task details, such as questions or articles, and interactions as the input, and output evaluation scores. The prompt contains three parts: (1) role and task description;(2) metrics definition; and (3) evaluation instruction.Role and task description instructs LEA to conduct evaluation. The role description acts the same as the role description in the previous stage. Moreover, it briefly describes the evaluation task. The general prompt looks like: You are a helpful and precise evaluator who checks the quality of the AI assistant's responses in interactions.Metrics definitions describes the criterion that LEA needs to follow in the evaluation process. They can customized for different tasks. For the question answering task, we add the following prompt to define the "helpfulness" metric:Helpfulness (5-point Likert): How helpful was having access to the AI Assistant compared to not having access?Both the aforementioned parts can be tailored according to the persona that the LEA simulates in Stage 1, as detailed in Section 3.3.Evaluation instruction outlines the specifics of the evaluation task and the required output format. This section may appear as a separate part of the prompt. For example, the instruction within the prompt might be structured as follows:Please evaluate the above interactions between user and AI assistant by using the following metrics: <Metric definitions> Please output each of the above metrics line-by-line.Finally, all evaluation scores for metrics are calculated by averaging the results of multiple runs. The complete prompt is available in Appendix C.1, and further details about our implementation can be found in Section 4.Both aforementioned evaluation stages typically use a default persona. While this constitutes a somewhat neutral baseline in knowledge, language proficiency, and beliefs to bo baseline, individual users often exhibit diverse personal preferences and characteristics, making a one-size-fits-all evaluation less effective. Moreover, the persona distribution of the target user group significantly impacts the performance of IQA models in real-world applications. For instance, if 20% of human users prefer brief interactions and 70% prefer detailed information, applying a general LEA to simulate this group of persons is likely to result in poor correlation with downstream users.To better simulate the diversity of the groups of people and provide individualized evaluations, we assign personas to LEAs. This affects prompts of both interaction generation and interaction evaluation processes.For example, when the LEA is assigned with the "Critical-Seeker" persona (definition in C.3) for interaction generation, we adapt the default role and task description (in Stage 1) to:You prefer interactions rich in critical information. You need help from an assistant and try to get critical information from it to answer the following questions. For interaction evaluation, the default role and task description prompt changes toThe AI Assistant should provide straightforward, simple, and concise answers to aid users in deducing solutions. Additionally, the definition of metrics is also adjusted to align with this persona, with further details available in Appendix C.3. 
meta-evaluation of iqa-eval framework: 
 To measure how our framework provides trustworthy IQA evaluations that align with human preferences, we conduct meta-evaluations experiments and report correlation scores.Dataset and Evaluation Metrics We apply our evaluation method on the annotated dataset from the study by Lee et al. [2023] . This dataset consists of 3641 interactions from 331 annotators. Questions in the dataset are multi-choice and are derived from the MMLU dataset [Hendrycks et al., 2020] (example question in Figure 1 ). The construction of MMLU requires each worker to participate in 11 random conversations with one of the following three IQA models: TextDavinci (text-davinci-001),TextBabbage (text-babbage-001), and Davinci (davinci-001). At the end of conversations, fluency and helpfulness scores are annotated by annotators. The number of queries and accuracy for each IQA model can be easily deduced from annotations. In this work, We adjust the four metrics to evaluate generated interactions:• Fluency (5-point Likert): How clear (or fluent) were the responses from the AI Assistant?• Helpfulness (5-point Likert): Independent of its fluency, how helpful was having access to the AI Assistant compared to not having access? • Number of Queries: Counts the number of interaction turns in the conversation. This metric helps assess the efficiency of the AI in resolving queries within a minimal number of interactions. • Accuracy: Quantifies how accurately the AI's responses match the golden answers. This is critical for evaluating the correctness of the AI's knowledge and its application in practical scenarios.LEA Models To evaluate the effectiveness of IQA-EVAL framework, we experiment with different LEA models on the above-mentioned three LLMs IQA models in MMLU. For LEA that conducts both interaction generation and interaction evaluation, we use ChatGPT (GPT-3.5-turbo-1106), GPT4 (GPT-4-1106-preview), Claude (Claude-1).We report Pearson correlations as the measrue of agreement between the Human evaluations and LEA evaluations of IQA models.According to Table 2 , all models, including GPT4, GPT3.5, and Claude, show high correlation with human evaluations. GPT4 aligns most closely with human judgments in both "Helpfulness" and "Fluency" metrics and the highest overall correlation score. This indicates that these models are capable of effectively performing IQA-EVAL framework as LEA models.In Table 1 , ChatGPT scores closest to human judgments, particularly in the "Helpfulness" metric. Conversely, GPT4 and Claude score lower on "Helpfulness" than human evaluations, because they tend to produce inaccurate and repetitive responses that lack coherence and do not directly address Claude 4.13 3.03 3.00 4.47 3.47 3.23 2.20 2.67 2.07 0.67 0.53 0.57 IQA-EVAL-GPT3. 5 4.30 3.87 3.93 4.47 3.67 3.97 1.57 1.77 2.00 0.63 0.47 0.53 user queries, as indicated by their generated explanations. IQA-Eval scores on "Fluency" are close and highly correlated to human judgments. Both scores given by humans and LEA models show that IQA models provide fluent outputs. Furthermore, according to the "# Queries" metric, most models conclude conversations more quickly than humans, except for Claude, which requires more turns, potentially due to its non-OpenAI origins that it needs more turns to adapt the conversational style and understand responses. Notably, GPT4 achieves the highest accuracy among all models. Moreover, we consider the impact of self-enhancement bias and conduct more experiments. Details are in Section 6.4. Results are in shown Figures 2a and 2b . The Pearson correlation coefficients for fluency and helpfulness between human judgments and LEA evaluations show distinct patterns. ChatGPT demonstrates a stronger correlation with human ratings, recording a correlation score of 0.424 for fluency and a correlation score of 0.306 for helpfulness. In contrast, Claude shows slightly lower correlations (0.281 for fluency and 0.287 for helpfulness). This shows that ChatGPT aligns better with human compared to Claude in these specific metrics.Moreover, for interactions between humans and IQA models (Figure 2 ), LEA evaluations moderately correlate with human evaluations. However, for interactions between LEAs and IQA models (Table 2 ), which is the main focus of our paper, LEA evaluations highly correlate (around 0.6) with human evaluations. This indicates that LEA models are helpful when participating in the whole evaluation process, including both the interaction and evaluation. However, when evaluating interactions between humans and IQA models, LEAs focus differently from humans, which causes moderate correlations.Thus, the results show that (1) both models' evaluations moderately correlate with human evaluation;(2) ChatGPT's evaluation is closer and related to human evaluation than Claude's.Apart from the metrics above, we also prompt LLM-based Evaluation Agent (LEA) to explain the reason for generating scores in the free-form text format. We find that: 1) ChatGPT Generates more human-like, positive reviews: ChatGPT evaluations are generally more positive, frequently using terms like "helpful", "relevant", and "useful" -words not always noted by workers in their annotations. Despite this, ChatGPT often identifies similar issues as human raters, such as the provision of irrelevant and repetitive information. Overall, ChatGPT's assessments align well with human evaluations; 2) Claude flags more Issues: Claude is more strict and critical to IQA models in interactions.In the free-text feedback, Claude tends to highlight more issues with model responses rather than acknowledging positive aspects, especially for Davinci. For one question, after interacting with the IQA model/assistant (ChatGPT in this case), human and two LEAs provide the following feedback:Human: When rephrasing questions well, the answers could be found in the AI's response.ChatGPT: The AI assistant was helpful in providing relevant information, but there were issues with the accuracy.The AI assistant's responses were not very helpful. The responses were often vague, repetitive, or did not directly answer the question.5 Effect of Assigning Persona to LLM Evaluation Agent (LEA) 
persona definitions: 
 As discussed in Section 3.3, we assign personas to LEAs to simulate different groups of humans for diverse human alignments. We investigate assigning the following personas to LEAs, which are defined based on the crowdworker survey results in Lee et al. [2023] .• Expert: knowledgeable; quickly learns new concepts and applies them in the reasoning process to answer questions. • Critical-Thinker: people who prefer critical information rather than redundant or detailed responses.• Adaptability-Seeker: people who prefer assistants can understand their questions even if they are not precise. • Clarity-Seeker: people who prefer clear explanations from assistants.Based on the survey results on the distributions of personas of workers Lee et al. [2023] , for each persona P , we split the crowdworkers into two groups: "persons with persona P ", and "normal persons without specific persona P ". For the first group, we initialize the role prompt in Section 3.3 and use it for the LEAs. For the second group, we utilize default prompting for LEAs. The LEA model in this section is ChatGPT(GPT-3.5-turbo-1106).In Table 3 , we show the results of model (with personas) evaluations. To accurately simulate persona distribution, each interaction is executed multiple times, with different personas (including the standard one) assigned to LEA based on their distribution proportions. This method ensures that each persona's influence and characteristics are proportionally represented in the simulation, reflecting their respective prevalence within the overall distribution. The final score for a persona is an average of all experiment results.The "Expert" persona decrease LEA query counts as "Expert" already possesses relevant knowledge and only needs key explanations. The "Clarity-Seeker" requires the most interaction turns among all personas for comprehension, but achieves the highest accuracy with TextBabbage and Davinci through detailed understanding of questions."Critic-Thinker" and "Adapability-Seeker" in Tables 3 and 4 rarely surpass upon the standard persona's human-preference aligment. We hypothesize that these personas are less reflected within the overall distributions of human preferences. In Table 3 , the varying "# Queries" across personas reveals their significant influence on LEA interaction strategies. Accuracy remains consistent after adding personas, showing no performance degradation.Together, these results indicate that assigning specific personas steer LEAs to perform IQA-EVAL in a more fine-grained and human-aligned way. 6 Benchmarking LLMs with IQA-EVAL on more Types of QuestionsQuestion: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Yes. Code and data are publicly available as shown in the first page footnotes.Guidelines:• The answer NA means that paper does not include experiments requiring code. • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
datasets: 
 To evaluate the robustness and generalizability of our evaluation framework, we conduct benchmarking across different models on two distinct question answering datasets, each offering unique challenges and complexities requiring advanced reasoning. AmbigQA Min et al. [2020] is a collection of 14,042 annotated questions sourced from the NQ-OPEN benchmarks Kwiatkowski et al. [2019a] , an open-domain QA dataset. It focuses on questions with inherent ambiguities, reflecting the complexity encountered in real-world queries. These ambiguities often involve diverse aspects such as events, entity references, and answer types, resulting in multiple plausible answers for each question.HotpotQA Yang et al. [2018] comprises 113,000 question-answer pairs sourced from Wikipedia, which require multi-hop reasoning spanning multiple documents. It contains a rich array of intricate questions that demand the synthesis of information from various texts to determine accurate answers.In this benchmark, we select 500 questions from each dataset to form a dataset containing 1,000 complex multi-hop and ambiguous questions. The IQA evaluation benchmarks are presented in Table 5 . We divide IQA Models into two categories: weak IQA Models (TextDavinci, TextBabbage, and Davinci) and strong IQA Models (GPT3.5, GPT4, Claude, Llama2, and Zephyr). Weak IQA Models can assist the LEA with answering HotpotQA questions, but due to their knowledge limitations, they cannot help much with AmbigQA questions. Zephyr achieves the lowest performance compared to other strong IQA Models. On the HotpotQA dataset, Zephyr's accuracy performance is only comparable to the strongest one among weak IQA Models, TextDavinci.Most "Helpfulness" and "Fluency" scores are high (exceeding 3 out of 5), especially for strong IQA Models like GPT4. For the "# of queries", it is uncommon for interactions to extend beyond two turns.As on HotpotQA, most interactions conclude at the beginning of the second turn, as IQA models have effectively guided users to reach the answers. For AmbigQA, some conversations last longer whereas LEA spends additional turns on clarifying ambiguous entities before approaching the final answer. Additional benchmarking results on the Natural Question dataset are in Appendix D.LLMs are shown to demonstrate self-favouring behaviours [Panickssery et al., 2024] , and no verified or accessible mitigations to this issue exist to the best of our knowledge. This issue is particularly concerning when the LEA models evaluating the IQA models share the same underlying model. In this section, we discuss our two of our attempts to assess the effects of this bias.Following Zheng et al. [2023] and Furniturewala et al. [2024] , we included some empirically useful debiasing instructions as follows:Please act as an impartial and unbiased judge. In your evaluation, please be objective and do not include any bias or your preference.In Table 5 , scores on the row of GPT3.5 is vulnerable to self-enhancement bias. However, with the above debiasing prompt, in Table 7 shows that the results of new prompts are highly similar to the original prompts. Similarly, Table 6 shows that the results of modified and original prompts are differ only lightly.We also designed a second methodology to mitigate self-enhancement bias. In this experiment, multiple LEA models evaluate the performance of IQA models during each interaction ("Multiperspective"). In other words, we introduced third-party evaluations, where various LEA models assess the IQA models' performance instead of relying solely on the LEA model itself involved in the interaction. After evaluation, we use the average score from all LEA models as the final score. The results of IQA-Eval-Multi-perspective look as in Table 8 . The correlations between IQA-Eval-Multi-perspective and human evaluations are in Table 9 .We believe that that the self-preference bias has limited impact on IQA-Eval. Stronger IQA models require fewer turns in interactions. On the more challenging AmbigQA, the stronger model, GPT4, typically requires only one turn to assist LEA in solving questions with high accuracy. In contrast, less capable models like Llama2 and GPT3.5 need more turns to clarify ambiguous entities and have lower QA accuracies. A similar trend is observed on the HotpotQA.We obtain a similar model ranking with a much lower cost. Compared to Chatbot Arena4 , our accuracy-based ranking of IQA Models follows a similar trend: GPT4 > Claude > GPT3.5 > Llama2 > Zephyr. In addition, our evaluation method, IQA-EVAL, is fully automated. Our method makes it a cost-effective alternative for large-scale evaluations.Evaluation of interaction performance does not always match Non-Interaction performance.In interaction evaluations, accuracy on final results is not the only metric to show IQA Models' performance. The quality of intermediate responses is a significant aspect. On both "helpfulness" and "fluency" metrics, Claude is always the best IQA Model on HotpotQA questions, while on AmbigQA, Llama2 and GPT3.5 outperform GPT4. IQA Model rankings on these two aspects differ from those in Chatbot Arena (non-interaction).The performance of IQA Models largely affects the final performance. The accuracies in both Table 5 and Table 10 show a consistent trend. The Pearson correlations of the accuracy between the tables are 0.77 and 0.87 on both datasets, respectively. A strong IQA model, such as GPT4, can lead the LEA to finish tasks and largely improve the LEA's performance on those tasks. Weak assistants may drag down the LEA's performance, such as the performance of the LEA on both datasets decreases after interacting with Zephyr. 
conclusion: 
 To conclude, we introduced IQA-EVAL, a novel approach for evaluating interactive questionanswering systems using large language models. Our methodology achieves automatic interaction generation and evaluation with LEA, and enhances the evaluation process by assigning personas to LEA for better matching diverse groups of people. We show that our approach aligns closely with real human interactions and judgment, indicating that a scalable, automatic IQA-EVAL process can be achieved. We providing insights on recent LLM's capability in conducting IQA with IQA-EVAL which would cost $5,000 for human evaluations. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We specified in the appendix. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: We do conform with above code.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Evaluation works bear little risk for negative societal impacts. We acknowledge that a potential bias issue could arise from LLM's self-enhancement behaviors when acting as evaluators. We made efforts to show this bias should have limited impacts on the results and our claims are in Section 6.4, Appendix E, and Appendix G.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]  
 
 Appendix: In conducting this study, certain limitations have influenced our scope and findings. First and foremost, LLMs are shown to demonstrate self-favouring behaviours [Panickssery et al., 2024] , and no verified or accessible mitigations to this issue exist to the best of our knowledge. We discuss some of our attempts to address this in Section 6.4, but this limitation necessitates future research.Our methodology was applied exclusively to multi-choice question-answering tasks due to constraints imposed by the datasets used. Moreover, do not investigate how allowing prompt editing could affect the results. This choice limits the generalizability of our findings across the wider array of question answering formats that exist in both academic research and practical applications.We advocate for subsequent research efforts to extend the application of our proposed evaluation strategies to a more diverse set of question answering tasks, beyond the multi-choice format.Furthermore, there is a significant opportunity to test these methods with a broader spectrum of LLMs, including those at the cutting edge of the field. Such expansions would not only validate the versatility and robustness of our approaches but also potentially uncover additional insights into the nuances of LLM interaction and performance in varied contexts.In our study, we meticulously crafted each persona to mitigate bias, ensuring they do not adversely impact the IQA-Eval process, as evidenced by the results presented in the table above. However, our personas represent only a limited range. There exists the potential for negative effects from other personas that may be inadequately designed or deliberately biased to achieve specific outcomes. Thus, we oppose irresponsible persona designs that result in biased evaluation results. The principle of persona definition and design should be thoroughly studied in future works. We hope our work will help facilitate future research into better automatic interaction evaluations aligning with crowds.Both C.1 and C.2 are adapted from Lee et al. [2023] . All the following prompts are from the view of LEA in interactions.We follow the setting in the data and evaluate conversations worker-wise. In other words, we combine conversations for each worker and send all of them to evaluators. Please evaluate the above conversations between user and AI assistant by using the following metrics: Fluency (5-point Likert): How clear (or fluent) were the responses from the AI Assistant? Helpfulness (5-point Likert): Independent of its fluency, how helpful was having access to the AI Assistant compared to not having access? Helpfulness (free-form): Why did you find the AI Assistant helpful or unhelpful? Please output each of the above metrics line-by-line.Since this is a multi-choice question answering task, the full prompt for models is as follows:You are mimicking a human. You are trying to choose the correct answer to the given question.Please ask an assistant sub-questions for help approaching answers.In each turn, please only ask one sub-question to interact with an assistant. In the sub-questions, please include all necessary information, such as the question and options, in the original question. If you know the answer, please output "So, the answer is: A, If the current turn reaches the maximum number we set, the system prompt before "{Question}" looks as follows:Please choose the correct answer to the given question. Please output "So, the answer is: A, B, C, or D."The question prompt for multi-choice questions in MMLU is as follows: For HotpotQA and AmbigQA datasets, the question prompt only contains a question, such as: <question>We design distinct prompts for each persona. Both prompts in meta-evaluation and model interaction modules change with personas.In model interaction prompts, we only modify the first sentence based on personas. See all persona prompts in Table 11 . We benchmark IQA models in another dataset called Natural Questions (Kwiatkowski et al. [2019b] ). This dataset comprises authentic questions posed by users about Wikipedia articles, demanding true multi-turn dialogues for resolution, akin to the setup in multi-turn conversational QA dataset QuAC (Choi et al. [2018] ). The experiment results are as in Tables 12 and 13 . All numbers of queries in the two tables are around 3, and each response to a query from IQA models contains an average of 2 sentences.Given the number of sentences in each IQA model's response in Table 14 , the non-interactive outputs are roughly equivalent to about two interaction turns, less than three turns in interactive outputs. Thus, The interaction process of IQA-EVAL involves not only reasoning processes but also simulating genuine interactive multi-turn conversations. This suggests that the performances shown in tables 12 and 13 above are driven more by multi-turn interactions than by reasoning processes. Furthermore, these interactions lead to enhanced accuracy, as demonstrated by the superior results in the first two tables compared to those in the last table (non-interactive).We conduct two new experiments to study the effects of changing persona assignments. Results indicate that IQA-EVAL is sensitive to incorrect persona assignments. When the persona distribution 15 ), the performance of IQA-EVAL shows a lower correlation with human evaluations.Moreover, the last two lines in Table 15 describe the correlation between human evaluations and IQA-EVAL within a sub-group only containing pure experts. The correlation results in line "IQA-EVAL (Pure Expert)" represent that (1) our personas accurately represent the pure expert group, as its correlation with the line "Human (Pure Expert)" remains nearly consistent with those in line "IQA-EVAL (Expert)" and (2) given this completely correct persona distribution, our IQA-EVAL correlates well with human evaluations.F Accurate QA models are preferred by humans in IQA-EVALThe quote from our cited paper Lee et al. [2023] is "[...] perception of helpfulness is not necessarily reflected in the overall interaction accuracy." It describes the conclusion of multiple tasks in that paper (e.g. text summarization, social dialogue, QA). However, in the QA settings, Table 3 in Lee et al. [2023] shows that humans prefer accurate models on the QA task. We also conducted experiments (1) using LEA to evaluate interactions between LEAs and IQA models (interactive) and ( 2) using LEA to evaluate direct answers generated by IQA models (non-interactive).Our experiments in Tables 16 and 17 show that LEA models prefer accurate models, which aligns well with the conclusion from human annotations.We follow the method proposed by Sheng et al. [2021] and conduct a new experiment to evaluate the offensiveness and harmfulness of our personas using the RealToxicityPrompts dataset (Gehman et al. [2020] ) on our LEA models. The results are in the table 18 . The values in the table above represent the success rates (higher is better) for each bias metric, persona, and LEA model (GPT3.5 and Claude). Scores labeled "None" are consistently lower than those for all personas, indicating that our personas do not increase offensiveness or harmfulness in conversations. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard 


Decision: The authors propose an interactive evaluation framework that uses an LLM to pose multi-turn questions to QA models under evaluation. The reviewers in general appreciate the usefulness of the framework, the fact that it is multi-turn and the results are correlated with human judgement. There are some concerns raised (including ethical concerns), such as the danger of self-bias during evaluation, the novelty of the work, and several technical details and questions. The authors conduct further experiments to support their rebuttal, addressing most concerns adequately.

Reviews: review: paper_summary: paper_summary: This research addresses the evaluation methodology for multi-turn conversation using Large Language Models (LLMs), a topic of active research recently. The study proposes an evaluation framework called IQA-Eval, which consists of the target model (IQA model) and an agent (LEA model) that engages in conversation with the IQA model and evaluates each turn. The primary challenge in multi-turn evaluation is to 1) interactively converse with the IQA model and 2) evaluate each generated turn in a cost and time-efficient manner. This research aims to overcome this challenge by leveraging LLMs that consider personas to facilitate both conversation and evaluation.
Through various experiments, the study confirms that applying LLMs for conversation and evaluation results in a high correlation with Human Evaluation. Additionally, it includes an analysis of the differences in the number of turns due to the varying capabilities of each IQA model.
strength_weakness: strength_weakness: - The evaluation dataset consists entirely of multiple-choice questions, making it unsuitable for generating and evaluating multi-turn conversations. This is evident in Tables 3 and 5, where the average length of conversation turns is less than 2. Such a situation suggests that the process might be more about reasoning rather than simulating genuine interactive multi-turn conversations. It is necessary to analyze whether the performance shown in Table 6 is due to multi-turn interactions or the influence of the reasoning path. Alternatively, an evaluation using datasets that assume genuine multi-turn dialogue scenarios is needed.
- The verification process for personas is lacking. There is no substantial evidence to consider that the performance differences by persona shown in Table 4 are reflective of the actual characteristics of those personas.
- A self-preference issue appears to have arisen in Table 5, and there is no control in place to address this.
questions: questions: - How does the Free Form Feedback in this study differ from the explanations provided by existing LLM-based evaluation methods (such as LLM-Eval and G-Eval)?
- What is the total number of repetitions for the experiments mentioned in section 5.2? Could you provide the variance values for each experiment?
- Why are the results for AmbigQA not included? Even if the performance of the IQA models is not optimal, the results should still be presented.
limitations: limitations: - The limitations are stated in the Appendix, clearly outlining the limitations of the experiments.
paper_summary: paper_summary: The authors introduce a novel method to simulate a human conversation when evaluated on an interactive question answering (IQA) and evaluate the simulated interaction according to some predefined metrics.
strength_weakness: strength_weakness: When using LLMs to evaluate the outputs from LLMs, recent research has shown LLMs to be biased in preferring their own generations compared to generations from other models, even if their generation wasn’t better [0]. This may bias the IQA-EVAL metrics if the same model is used for both IQA-EVAL and the IQA model.

The authors provided limited evidence for their claims in section 4.3. Could the authors provide a more robust justification of why “ChatGPT’s assessments align well with human evaluations”, and other claims made in this section?

The authors did not recognise that this method could cause negative societal impacts, instead stating “Evaluation works bear little risks for negative scoeital impacts”. LLMs have been repeatedly shown to exhibit strong biases as a result of their training procedure. Using LLMs for evaluation opens the risk of reinforcing these biases, whether within training of future models, or benchmark answers, or whatever is downstream of this method.

[0] https://arxiv.org/abs/2404.13076
questions: questions: See Weaknesses
limitations: limitations: The authors addressed the limitations.
paper_summary: paper_summary: This paper introduces IQA-EVAL, a framework for automatically evaluating interactive question-answering (IQA) systems using large language models. The authors propose using LLM-based Evaluation Agents (LEAs) to simulate human behavior in both generating interactions with IQA models and evaluating those interactions. The framework also incorporates persona assignments to LEAs to better represent diverse user groups. The authors demonstrate that IQA-EVAL achieves high correlation with human judgments and use it to benchmark several recent LLMs on complex question-answering tasks.
strength_weakness: strength_weakness: - Limited novelty, as using LLMs as judges is quite common in evaluation tasks. This paper mainly focuses on a relatively new setting - the interactive QA setting.
- The considered interaction scenarios could be more diverse to better reflect real-world user behaviors
questions: questions: - Line 207~213: can you provide more explanations into why ChatGPT sometimes outperformed GPT-4 in your experiments? This seems counterintuitive given GPT-4's generally stronger capabilities.
- Have you considered incorporating more diverse interaction patterns, such as clarification questions, ambiguous queries, or follow-up questions? How might this affect the evaluation results?
- Have you explored how different persona distributions might impact the evaluation results? How sensitive is the framework to changes in persona assignments?
limitations: limitations: The authors have made some effort to address limitations, such as acknowledging potential biases in LLM-based evaluation and discussing the impact of persona assignments. However, the paper would benefit from discussing the limited diversity of interaction patterns considered.
paper_summary: paper_summary: This paper proposes an automated evaluation framework for Interactive Question Answering tasks based on LLM-based agent. The authors utilize LLMs to simulate the interaction between human and IQA models and use them to evaluate the interactions automatically. Additionally, they assign predefined personas to LLMs to better simulate the interaction characteristics of different groups. Their evaluation framework achieves a stronger correlation with human evaluations and eliminates the high costs of manual evaluation.
strength_weakness: strength_weakness: The weakness of this paper is the lack of technical contributions and novelty, and its experimental analysis is not strong enough. They only used prompts to implement their methods, without any prompt engineering approach. Furthermore, their idea of using LLMs as evaluators is similar to some previous research papers, such as G-Eval [1].

[1] Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. CoRR, abs/2303.16634, 2023.
questions: questions: Major questions:
1. The paper utilizes LLMs as evaluators for Interactive Question Answering. However, according to some previous research papers [2], a systematic bias may be introduced by LLM evaluators and the results of evaluation may be influenced. Did you consider the impact of bias on the results in your research and try any methods to eliminate it?
2. You mentioned that accurate models are not necessarily preferred by humans in your paper, but you did not demonstrate this result in your experiments. Could you add a control group in your experiment?
3. There exists some errors in your analysis of LEA for stage 2. You stated that models’ evaluations highly correlate with human evaluation, while the Pearson correlation coefficient between 0.2 and 0.5 only indicates a weak or moderate correlation. 
4. In your analysis of Table 1, you concluded that model gave higher scores for the “Fluency” metric than human because of the clarity and grammatical correctness of the response generated by IQA models. Could you please provide a detailed explanation of why you concluded that the model understands the concept of “Fluency” better than humans? 

Minor questions:
1. Could you please point out the LEA model used in the experiments shown in Table 3, like in Table 1 and 2?

[2] P. Wang, L. Li, L. Chen, D. Zhu, B. Lin, Y. Cao, Q. Liu, T. Liu, and Z. Sui. Large language models are not fair evaluators. CoRR, abs/2305.17926, 2023.
limitations: limitations: Yes

Comments: title: Reply to Ethics Review
Response to the authors
More discussions?
More discussions?
Reviewer please respond
Reviewer please respond
Reviewer please respond
More discussions
More discussions
shall we have more discussions
Shall we have more discussions?
shall we have more discussions?
shall we have more discussions
Additional global response: new methodology to further mitigate self-favoring bias
More Responses to Weakness 1:
More clarification on the reply to question 3
comment: We follow the method proposed by Sheng et al. [1] and conduct a new experiment to evaluate the offensiveness and harmfulness of our personas using the RealToxicityPrompts dataset [2] on our LEA models. The results are in the table below:

| Persona             | Offensiveness IQA-EVAL-GPT3.5 | Offensiveness IQA-EVAL-Claude | Harmful IQA-EVAL-GPT3.5 | Harmful IQA-EVAL-Claude |
| ------------------- | ----------------------------- | ----------------------------- | ----------------------- | ----------------------- |
| None                | 89.5                          | 91.7                          | 62.5                    | 72.5                    |
| Expert              | 95.5                          | 97.3                          | 67.8                    | 75.5                    |
| Critical-Thinker    | 93.3                          | 95.5                          | 65.4                    | 73.7                    |
| Adaptability-Seeker | 94.5                          | 95.5                          | 62.8                    | 74.6                    |
| Clarity-Seeker      | 95.0                          | 94.0                          | 62.5                    | 73.0                    |

The values in the table above represent the success rates (higher is better) for each bias metric, persona, and LEA model (GPT-3.5 and Claude). Scores labeled "None" are consistently lower than those for all personas, indicating that our personas do not increase offensiveness or harmfulness in conversations.

# Ethics Statement

In our study, we meticulously crafted each persona to mitigate bias, ensuring they do not adversely impact the IQA-Eval process, as evidenced by the results presented in the table above. However, our personas represent only a limited range. There exists the potential for negative effects from other personas that may be inadequately designed or deliberately biased to achieve specific outcomes. Thus, we oppose irresponsible persona designs that result in biased evaluation results. The principle of persona definition and design should be thoroughly studied in future works. We hope our work will help facilitate future research into better automatic interaction evaluations aligning with crowds.

References:
1. Sheng, Emily, et al. "Revealing persona biases in dialogue systems." arXiv preprint arXiv:2104.08728 (2021).
2. Gehman, S.; Gururangan, S.; Sap, M.; Choi, Y.; and Smith, N. A. 2020. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2020, 3356– 3369. Online: Association for Computational Linguistics
Sorry for the late reply! Thank you for your reply, my concern has been addressed. I have raised my score.
Dear Reviewer,

The discussion period is ending.

Do you have any other questions of interest? In our previous rebuttal, we have added relevant explanations and several experiments, and we believe that these can address your concerns.

We are eagerly looking forward to your response!
Dear Reviewer,

The discussion period is ending.

Do you have any other questions of interest? In our previous rebuttal, we have added relevant explanations and several experiments, and we believe that these can address your concerns.

We are eagerly looking forward to your response!
Thanks authors' for the detailed responses. While some of my questions have been addressed with more explanations, my overall opinions remain the same. I still think the contribution "augment LLM-agent with personas for eval" is a bit marginal, I tend to keep my score unchanged.
Dear reviewer,

Thank you for your efforts in reviewing this paper. Now that the authors have provided their response, do you have any further comments?

Thank you,
AC
Dear reviewer,

Thank you for your efforts in reviewing this paper. Now that the authors have provided their response, do you have any further comments?

Thank you,
AC
Dear reviewer,

Thank you for your efforts in reviewing this paper. Now that the authors have provided their response, do you have any further comments?

Thank you,
AC
I am grateful for the further analysis presented by the authors. Given their sound arguments, I am going to raise my score to a 7.
Dear Reviewer,

The discussion period is ending. 

Do you have any other questions of interest? In our previous rebuttal, we have added relevant explanations and several experiments, and we believe that these can to some extent address your concerns.

We are eagerly looking forward to your response!
Dear Reviewer,

The discussion period is ending. 

Do you have any other questions of interest? In our previous rebuttal, we have added relevant explanations and several experiments, and we believe that these can to some extent address your concerns.

We are eagerly looking forward to your response!
Dear Reviewer,

The discussion period is ending. 

Do you have any other questions of interest? In our previous rebuttal, we have added relevant explanations and several experiments, and we believe that these can to some extent address your concerns.

We are eagerly looking forward to your response!
Dear Reviewer, 

Do you have any other questions of interest? In our rebuttal, we have added relevant explanations and several experiments, and we believe that these can to some extent address your concerns.

We are eagerly looking forward to your response!
Dear Reviewer, 

do you have any other questions of interest? In our previous rebuttal, we have added relevant explanations and several experiments, and we believe that these can to some extent address your concerns.

We are eagerly looking forward to your response!
Dear Reviewer, 

do you have any other questions of interest? In our previous rebuttal, we have added relevant explanations and several experiments, and we believe that these can to some extent address your concerns.

We are eagerly looking forward to your response!
We designed a new methodology to further mitigate self-favoring bias. In this experiment, multiple LEA models evaluate the performance of IQA models during each interaction (“Multi-perspective”). In other words, we introduced third-party evaluations, where various LEA models assess the IQA models' performance instead of relying solely on the LEA model itself involved in the interaction. After evaluation, we use the average score from all LEA models as the final score. The results of IQA-Eval-Multi-perspective look as follows: (**Bold** numbers indicate they are the closest to human results.)

|                                   | Helpfulness |          |          | Fluency  |          |          | # Queries |          |          | Accuracy |          |          |
| --------------------------------- | ----------- | -------- | -------- | -------- | -------- | -------- | --------- | -------- | -------- | -------- | -------- | -------- |
| LEA models                        | TDA         | TB       | DA       | TDA      | TB       | DA       | TDA       | TB       | DA       | TDA      | TB       | DA       |
| Human                             | 4.60        | 3.84     | 3.52     | 4.35     | 3.84     | 3.22     | 1.78      | 2.57     | 2.66     | 0.69     | 0.52     | 0.48     |
| IQA-EVAL-GPT4-Multi-perspective   | **4.32**    | **3.70** | **3.53** | 4.57     | **3.74** | 3.68     | **1.57**  | 2.27     | **2.3**7 | 0.87     | 0.83     | 0.67     |
| IQA-EVAL-Claude-Multi-perspective | 3.96        | 3.13     | 3.10     | **4.29** | 3.51     | **3.22** | 2.20      | **2.67** | 2.07     | **0.6**7 | **0.53** | 0.57     |
| IQA-EVAL-GPT3.5-Multi-perspective | 3.98        | 3.23     | 3.04     | **4.41** | 3.67     | 3.59     | **1.57**  | 1.77     | 2.00     | 0.63     | 0.47     | **0.53** |

----

The correlations between IQA-Eval-Multi-perspective and human evaluations are as follows:

|  | Helpfulness | Fluency   | Overall   |
| ---------- | ----------- | --------- | --------- |
| IQA-EVAL-GPT4-Multi-perspective | **0.702**   | 0.601     | **0.624** |
| IQA-EVAL-Claude-Multi-perspective | 0.663       | **0.613** | 0.602     |
| IQA-EVAL-GPT3.5-Multi-perspective | 0.641       | 0.552     | 0.533     |

The overall result shows that GPT-4 is the best model aligning with human evaluations.
Both datasets, HotpotQA and AmbigQA, used in Table 5 are not multi-choice datasets. Their inputs and outputs are texts. HotpotQA requires LEA modes to output a short text including an answer to the given question and contexts. AmbigQA requires LEA models to disambiguous questions first and then answer that question by giving a short text.

As you’ve suggested, we also benchmark IQA models in another dataset called Natural Questions [1]. This dataset comprises authentic questions posed by users about Wikipedia articles, demanding true multi-turn dialogues for resolution, akin to the setup in QuAC [2]. The experiment results are as follows:

----

LEA: Claude-3

| IQA models | Helpfulness | Fluency | # Queries | Accuracy |
| ---------- | ----------- | ------- | --------- | -------- |
| GPT3.5     | 4.86        | 4.88    | 2.82      | 0.42     |
| Claude     | 4.88        | 4.90    | 3.02      | 0.38     |
| Llama2     | 4.90        | 4.84    | 3.18      | 0.34     |
| Zephyr     | 4.84        | 4.90    | 3.02      | 0.28     |

----

LEA: GPT-4

| IQA models | Helpfulness | Fluency | # Queries | Accuracy |
| ---------- | ----------- | ------- | --------- | -------- |
| GPT3.5     | 4.12        | 5.00    | 2.76      | 0.44     |
| Claude     | 4.02        | 5.00    | 2.76      | 0.40     |
| Llama2     | 3.20        | 4.84    | 3.08      | 0.32     |
| Zephyr     | 3.30        | 4.86    | 2.92      | 0.36     |

----

All numbers of queries in the first two tables are around 3, and each response to a query from IQA models contains an average of 2 sentences.

Similar to Table 6 in the paper, we conduct non-interactive experiments on the following IQA models.

| IQA models | # Sentences | Accuracy |
| ---------- | ----------- | -------- |
| GPT3.5     | 4.66        | 0.38     |
| Claude     | 3.16        | 0.34     |
| Llama2     | 5.21        | 0.30     |
| Zephyr     | 4.68        | 0.24     |

Given the above number of sentences in each IQA model’s response, the non-interactive outputs are roughly equivalent to about two interaction turns, less than three turns in interactive outputs.

Thus, **The interaction process of IQA-EVAL involves not only reasoning processes but also simulating genuine interactive multi-turn conversations. This suggests that the performances shown in the first two tables above are driven more by multi-turn interactions than by reasoning processes.** Furthermore, these **interactions lead to enhanced accuracy**, as demonstrated by the superior results in the first two tables compared to those in the last table (non-interactive).

If you think we misunderstood your question and if you have further questions, please let us know.

Reference:
1. Kwiatkowski, Tom, et al. "Natural questions: a benchmark for question answering research." Transactions of the Association for Computational Linguistics 7 (2019): 453-466.
2. Choi, Eunsol, et al. "QuAC: Question answering in context." arXiv preprint arXiv:1808.07036 (2018).
We are sorry for the minor inaccuracy in the original reply to question 3. We expect that IQA-EVAL is sensitive to incorrect persona assignments.

We conduct two new experiments to study the effects of changing persona assignments. When the persona distribution is incorrect (such as 20% Expert in the table below), the performance of IQA-EVAL shows a lower correlation with human evaluations.

Moreover, the last two lines in the following table describe the correlation between human evaluations and IQA-EVAL within a sub-group only containing pure experts. The correlation results in line “IQA-EVAL (Pure Expert)” represent that (1) our personas accurately represent the pure expert group, as its correlation with the line “Human (Pure Expert)” remains nearly consistent with those in line “IQA-EVAL (Expert)” and (2) given this completely correct persona distribution, our IQA-EVAL correlates well with human evaluations.

| 5-point Likert                | Helpfulness |      |      |       | Fluency |      |      |       |
| ----------------------------- | ----------- | ---- | ---- | ----- | ------- | ---- | ---- | ----- |
| LEA models                    | TDA         | TB   | DA   | ρ     | TDA     | TB   | DA   | ρ     |
| Human                         | 4.60        | 3.84 | 3.52 |       | 4.35    | 3.84 | 3.22 |       |
| IQA-EVAL (Expert)             | 4.17        | 3.08 | 3.12 | 0.756 | 4.47    | 3.84 | 3.40 | 0.787 |
| IQA-EVAL (20% <br><br>Expert) | 4.31        | 3.26 | 3.44 | 0.708 | 4.62    | 4.09 | 3.65 | 0.741 |
| IQA-EVAL (40% <br><br>Expert) | 4.21        | 3.14 | 3.23 | 0.751 | 4.49    | 3.88 | 3.44 | 0.779 |
| IQA-EVAL (60% <br><br>Expert) | 4.11        | 3.01 | 3.00 | 0.725 | 4.43    | 3.77 | 3.34 | 0.734 |
| IQA-EVAL (80% <br><br>Expert) | 4.02        | 2.90 | 2.79 | 0.680 | 4.30    | 3.56 | 3.12 | 0.703 |
| Human (Pure Expert)           | 4.69        | 4.00 | 3.73 |       | 4.36    | 3.96 | 3.26 |       |
| IQA-EVAL (Pure Expert)        | 4.37        | 3.57 | 3.33 | 0.778 | 4.20    | 3.40 | 2.97 | 0.786 |

If you think we misunderstood your question, please let us know and feel free to ask any follow-up questions.