Efficient Evaluation of LLMs via Branching Preference Learning

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 08:40:06

Abstract: Large language models (LLMs) have made significant advances across various generative tasks, progressing toward achieving near-human levels of intelligence. However, in many scenarios, LLMs face the challenge of insufficient human evaluation or even the inability to evaluate reliably.  Particularly, in complex dialogue scenarios involving diverse and intricate user intents, LLMs as evaluators of AI responses exhibit a substantial gap compared to humans. Moreover, due to the scarcity of high-quality evaluation data, LLMs exhibit deficiencies in their evaluation capabilities. In this work, we conceptualize the evaluation process as a decision tree, where each node represents an evaluation action, and each path from the root to a leaf node represents a trajectory of evaluation reasoning. We demonstrate that within a limited search space, there exist better decision-making behaviors that facilitate the model in making reasonable and accurate judgments. Specifically, we propose a tree-based data sampling method to generate supervised data and preference pairs derived from the evaluation tree. Furthermore, we introduce preference learning based on the DPO algorithm, which empowers the fine-grained evaluation model to explore and learn better branching strategies within budget-limited scenarios. Our model significantly reduces the dependency on labeled data and demonstrates strong performance across three different evaluation settings: in-distribution, out-of-distribution, and transfer evaluation. Experiments indicate that our model can reduce inference costs by 90\% compared to conducting searches across the entire evaluation tree, thereby significantly enhancing efficiency.

Main paper: 
 introduction: 
 Dialogue evaluation capability [6] is one of the fundamental abilities of human social interaction, involving the comprehension and interpretation of user intentions, as well as providing reasonable judgments on the correctness of different responses. Automated evaluation can assist humans to supervise powerful LLMs and is an essential component for superalignment and weak-to-strong generalization techniques [4] . However, human evaluations [3, 22] are labor-intensive and timeconsuming, making it difficult to widely adopt. Traditional automated evaluation approaches [18, 39, 8] are limited by inherent deficiencies, such as string and semantic matching methods often yield subpar accuracy and lack of interpretability. The advent of large language models offers promise for automatically evaluating dialogue quality [19, 41, 15] , owing to their high consistency with humans in intent understanding.Nevertheless, automated evaluation remains a challenging issue due to the diversity of tasks and scenarios it may encounter. The user queries often encompass multiple intentions [38] , which cannot typically be addressed using a single evaluation criterion. However, related research [35, 42] often attempts to treat evaluation as a simplistic 'one-step' reasoning problem, causing even the most Figure 1 : The agreement between human judgment and LLMs in Eval-P benchmark in out-ofdistribution evaluation. Auto-J serves as the "one-step" evaluation baseline, while Fennec is the "multi-step" baseline. The Initial, SFT, and DPO models were trained using our generated data. powerful large language models to struggle to provide reasonable and accurate results. It is essential for the evaluation model to adapt to different scenarios and provide critical evaluation criteria.In this work, we do not introduce any human prior for evaluation scenarios and criteria, which are commonly used for designing and collecting training data in related studies [14, 11] .The real-world conversational scenarios are often characterized by complexity and unpredictability, making it challenging to derive generalizable rules. Additionally, human priors frequently introduce biases [12, 20] , making these evaluation methods poorly generalized due to a lack of adaptability and scalability. Therefore, we explore automatically sampling scenarios from large-scale datasets and employ LLMs to automatically generate evaluation criteria, aiming to eliminate human labor as much as possible. Another significant challenge is the lack of ground truth labels and human feedback during the training data collection process. The insufficient of available supervised data for evaluation tasks also prevents them to scale effectively.Despite various challenges, we discover that the evaluation model is constrained in its ability to identify crucial evaluation criteria, but this limitation can be mitigated by increasing the number of considered criteria. As shown in Figure 1 , the Initial model can achieve nearly a 10 point improvement in the agreement metric by increasing the number of evaluation branches. This findings motivate us to design tree-based data sampling methods to generate training data and a branching preference learning algorithm to improve "multi-step" inference capability. Specifically, we employ a breadth-first growth approach to construct an evaluation tree, where each path from the root to a leaf node represents a complete evaluation trajectory. We collect high-quality evaluation trajectories from the search space of the evaluation tree and trained an SFT model, which exhibited superior performance and prediction consistency. Furthermore, we refine these evaluation trajectories and train a DPO model [24] , which can effectively prioritize and output crucial evaluation criteria, thereby enhancing the model's inference effectiveness.We mainly evaluate our models in three settings: in-distribution, out-of-distribution, and transfer evaluation. Specifically, we use the datasets from the Chatbot Arena1 as in-distribution data, and collect data from large-scale dialogue datasets without human priors as out-of-distribution data. In our experiments, we demonstrate that (1) our model outperforms several recent open-source evaluation models and methods across all three settings, (2) there is a noticeable improvement in the evaluation model's capability when progressively training the Initial model, the SFT model, and the DPO model, and (3) as shown in Figure 1 , our DPO model achieves the best performance even when using only a single evaluation criterion (single inference branch). 
related works: 
 Automated dialogue evaluation [6] has long been a significant challenge in the field of generative AI.Recent work [10, 7, 35, 41] has demonstrated that LLMs can act as automated evaluators, serving as alternatives to human judges. However, LLMs still exhibit issues such as positional bias and prediction inconsistency [34, 40] . Many studies have relied heavily on human priors [14, 11] , thereby neglecting to explore the model generalization capabilities. In contrast, our research focuses on examining the performance with different data distributions and investigates how to bridge this gap.We consider automated evaluation as a complex reasoning task and aim to improve model performance by optimizing reasoning trajectories. When handling such tasks, LLMs typically utilize decision trees [37, 23] to model the reasoning process. They often employ search algorithms like A* [21, 13] or Monte Carlo Tree Search (MCTS) [29, 31] to identify the optimal reasoning path within the candidate decision. However, these methods generally rely on deterministic reward signals or feedback, which are absent in our settings. We demonstrate that the ensemble boundary of the evaluation branches provides a feasible reward signal to verify the accuracy of the reasoning trajectories. Based on this, we can guide the model to generate a substantial amount of high-quality data.Automated evaluation is also a pivotal technology within scalable oversight, aiming to enhance humans' ability to supervise models. For example, humans may ask models to critique the outputs of other models [9, 28] or use models to help decompose a problem into simpler subproblems [17] . In contrast to improving human supervision, we focus on how to conduct reliable automated evaluations.Certainly, our proposed evaluation methods and results can also be combined with human oversight to provide even better performance.We collect 5M large-scale dialogue data and extracted 7K samples from it to serve as out-of-distribution (OOD) training data. GPT-4 annotate 3K evaluation samples from this dataset for the Initial model training.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The limitations of this study are discussed in Section 6.1.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem setup: 
 In this work, our primary focus is on evaluating AI responses, particularly in analyzing query and response pairs within given datasets to determine which response is better2 . Traditional approaches [41, 35] regard the evaluation task as a "one-step" classification ("win" or "lose" or "tie") or generation problem, where the final scores or explanations are assigned by a reward model or the evaluation model. However, with complex reasoning tasks or scenarios, a given query may involve multiple intents, whether explicit or implicit [38] , yet the generated responses by AI often overlook some of these intents, constrained by the model's capabilities. Therefore, multiple evaluation criteria are required [19] to verify whether the responses address the query requirements and align with user intentions. Considering the complexity and diversity of dialogue tasks, it remains an intractable challenge to gather comprehensive and accurate evaluation criteria.We try to view the evaluation task as a complex reasoning task, a multi-step generative problem, which entails: (1) initially seeking suitable evaluation criteria, then (2) generating scoring guidelines based on these criteria, and finally (3) conducting comprehensive judgment based on the aforementioned criteria and scoring guidelines. Formally, given a dialogue X , we will use an evaluation model to sequentially obtain the criterion C, scoring guideline S, and judgment J :EQUATIONwhere π θ represents the evaluation policy, the prompt please refer to Appendix A.3. Similar to related multi-branch evaluation [27, 16] methodologies, we refer to different reasoning paths as "evaluation branch", where each branch represents a decision-making process. Unlike previous methods [19] that relied on enumerating criteria, our goal is for the evaluation model to automatically generate crucial and high-priority criteria.A natural approach is to first construct a candidate set of criteria and then derive suitable results based on these criteria. To address this task, we focus on the following two challenges:• How to construct an appropriate candidate set? Our aim is to develop a candidate set that includes multiple evaluation branches enriched with high-quality evaluation opinions. By training and optimizing this candidate space to advance desired behaviors, we can swiftly identify appropriate and critical judgments during the inference process.• How to rank the judgments? We also need to establish a ranking among different evaluation branches to optimize the candidate space. In contrast to recent studies [13] , our evaluation dataset lacks ground truth labels or environmental feedback to act as reward signals. The cost of obtaining these signals is prohibitive, requiring not only expensive human labor but also facing issues of low consistency among humans in many ambiguous problems. Therefore, we need to design an innovative and cost-effective approach to address this challenge.We use 3K OOD training data (which includes evaluation data) and 2K ID dialogue data (which did not include evaluation data) to train the transfer model.For each benchmark, we employ Agreement (AGR) and Consistency (CNS) as performance metrics.Consistency measures the prediction consistency of the evaluation model when the positions of the responses are swapped. Agreement quantifies the proportion of evaluations that meet the criteria for swap consistency and align with human judgments. In many cases, the "tie" label indicates an inability to distinguish performance under some evaluation criteria. However, it may still be distinguishable under specific evaluation criteria. Therefore, we also present the model performance on the test data without "tie" label. For more details, please refer to Appendix A.2Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
method: 
 Figure 2 illustrates an overview of our method, which involves three stages for model training: First, we train the Initial model to construct the evaluation tree; Then, we sample different evaluation branches as supervised data to train the SFT model, enhancing branch prediction consistency; Finally, we collect preference data to train the DPO model, ensuring rapid sampling of critical branches.Evaluation models typically rely on robust generalization capabilities to effectively handle diverse dialogue tasks. Consequently, the distribution of training data significantly affects performance on unseen tasks encountered during real-world evaluations. To address this, we sampled from a largescale dialogue dataset rather than a specific data source. We then apply the K-Means algorithm [2] to cluster the data. Subsequently, we sample data from these clusters, ensuring that the training dataset encompasses a diverse set of dialogue scenarios. More details refer to Appendix A.1We aim to construct a dataset from scratch for evaluation, consisting of dialogues paired with their corresponding evaluation trees. Each tree contains different reasoning paths during the evaluation of dialogues. The root node of this tree represents the dialogue data, and each path from the root node to a leaf node signifies an evaluation branch. Each evaluation branch comprises three decision-making behavior nodes: criterion C, scoring guideline S, and judgment J . To simulate this decision process, we introduce a multi-branch training approach [16] to train an LLM as the initial policy π Initial . We employ GPT-4 (gpt-4-0125-preview) [1] to generate corresponding multi-branch training data to enhance quality. This approach ensures that the model can auto-regressively generate evaluation branches using Equation 1.We expand the branch candidates sampled from the policy π Initial using the breadth-first growth, thereby including as many high-quality evaluation paths as possible. Due to the different paradigms of SFT and DPO, we employ consistency pruning to split the sampling space to obtain training data:• Breadth-first Growth: The evaluation tree contains two distinct growth manner: for criterion C node, we use LLM's brainstorming capability to generate k relevant criteria; for scoring guideline S and judgment J node, we use sampling method by adjusting the LLM's temperature and top-p parameters. To simplify, we utilize the Initial model π Initial to generate a complete binary tree for each subtree with a criteria node as its root. Furthermore, since the evaluation task requires testing the model's consistency by swapping response positions, we can obtain k × 8 different evaluation branches. Although we can obtain SFT and DPO data from the consistency sampling space, this data lacks correctness verification. Typically, preference data requires human annotation to establish ranking sequences, a time-consuming process that is not suitable for scaling. Therefore, we propose two alternative approaches to label each evaluation branch with its correctness:• Branch Ensemble: Considering that there are only three final labels for judgment ("win" or "lose" or "tie"), we use an ensemble result of evaluation branches to obtain the consensus label. The ensemble method provides a lower bound of judge error without incurring additional costs. For SFT data, we filter out data that is inconsistent with the ensemble results. For DPO pair data, we select samples consistent with the ensemble results as "chosen" samples, and those inconsistent as "rejected" samples.• LLM-as-a-Judge: Some highly aligned LLMs, such as GPT-4, possess powerful annotation capabilities. Therefore, we use LLMs to determine which sample in the DPO pairs data is more reasonable as the "chosen" sample. In our experiments, we found that this method has only a 20% disagreement rate compared to the Branch Ensemble method. We analyze this method in Section 5.4As shown in Figure 3 , we combine consistency pruning and automated labeling to collect the corresponding preference data. Through the labeling of judgments, we can also obtain preference information for criterion C and scoring guideline S based on the final judgment J decisions. Specifically, we prioritize predicting criteria that lead to correct judgments and select the scoring guidelines with the highest overall scores as the "chosen" samples. Additionally, we randomly sample from the filtered data to create the training set, thereby controlling training costs and efficiency.We use the Initial model as the starting point to train the SFT model π SFT using supervised learning, which reduces inconsistent predictions compared to the initial policy. Then, we take the SFT model as the initialization to train the DPO model π DPO using Direct Preference Optimization, which can learn the decision priorities of different branches, with the objective:EQUATIONwhere the (x, y) represents data pair of different decision tasks in Equation 1, y c represents the "chosen" sample, and y r represents the "rejected" sample.During the inference process, we create a single branch for each criterion to conduct evaluation, and control the number of generated branches k to adjust the inference efficiency. Since the DPO model employs sampling optimization, it usually achieves optimal performance with only a few branches.  
experiments: 
 As the most popular LLM evaluation platform recently, Chatbot Arena demonstrates high alignment with human judgments in pairwise response evaluations. We collect its open-source human judgment benchmark, Eval-P and MT-bench, to serve as the test set. We gather training data comprising both dialogue data and evaluation data for the following three evaluation scenarios:1. In-distribution evaluation: We apply the Fennec [16] training data to train the Indistribution (ID) model, which included 3K dialogue data from Auto-J [14] , along with evaluation data annotated by GPT-4. This training data is a multi-branch dataset, meaning that a single dialogue includes multiple evaluation branches.The results are shown in Table 1 , where methods marked with † denote our reimplementations. Since the Initial model leverages the Fennec training data for initialization, its performance can be A more comprehensive analysis of these instability problems is provided in Section 5.7.In terms of OOD evaluation, the Initial model performs worse than the baseline model on bothEval-P and MT-bench benchmarks, due to the distribution shift in the dialogue dataset. With RLHF [22] training, the SFT model significantly surpasses the Initial model in consistency rate and also enhances the agreement rate. Notably, the DPO model achieves superior performance with only three branches, thereby reducing inference latency by over 60%. In evaluation settings without "tie" labels, the advantage of the DPO model becomes more apparent, significantly outperforming other models, including proprietary model GPT-4. This demonstrates that the DPO model can effectively distinguish between responses using critical criteria, even when employing only 3 branch for inference. Furthermore, our models are capable of handling multi-turn dialogue scenarios, achieving performance that surpasses the in-distribution models. These extremely strong results indicate that our model excels at identifying more crucial criteria to help distinguish the difference of AI's responses.The purpose of transfer evaluation is to evaluate the model's capability to adapt to in-distribution data, thus mitigating the problem of training data distribution shift. It can be observed that both the SFT and DPO models demonstrate improvements across multiple benchmarks compared to the Initial model. Notably, in both OOD and transfer evaluation settings, the DPO model consistently achieves better performance than the SFT model, while also reducing the number of inference branches.Although the transfer model does not surpass the OOD model, it still achieves closed performance.In Section 5.4, we provide a detailed analysis of the different scenarios that lead to these models exhibiting significantly different performance characteristics despite their close overall performance.To investigate the impact of scenario categories distribution in the training data, we need to analyze the scenarios within the OOD, ID training sets, and the Eval-P test set. For this purpose, we employ the scenario classifier trained by Auto-J, which effectively categorizes dialogue data into 58 different scenarios. Figure 4 presents the distribution of scenarios. It can be observed that Auto-J's training set is well-balanced across the predefined scenarios, closely matching the distribution of the Eval-P test set. In contrast, within the OOD data, the "Others" category exceeds 30%, and "GeneralCommunication" surpasses 50%. The significant differences in scenario distributions between the OOD data and the test set can lead to performance variations in test cases. In our investigations, we strive to reduce reliance on both human annotators and GPT-4. Specifically, in the current work, we trained an Initial model using annotation data generated by GPT-4 without any addi- 4 , the model reaches its best performance at 2k data, without considering the influence of GPT-4's annotation quality. Based on the assumption that LLMs primarily unlock their potential during alignment phase, we believe that enhancing performance hinges on increasing the variety of tasks rather than merely expanding the dataset.The Direct Preference Optimization (DPO) algorithm [24] aims to optimize the selection of various 6 Discussion 
limitations: 
 Currently, our model faces some limitations: (1) It cannot handle cases where all AI responses are incorrect, which should not be labeled as a "tie". (2) The model's result parsing relies heavily on regular expressions, which can lead to format errors. To address these issues, we plan to make several improvements, including expanding our task settings and utilizing the functional calling feature of LLMs. Additionally, our model's performance is constrained by the amount of training data and parameters. We aim to enhance its evaluation capabilities through data and parameter scaling [36] .Our work demonstrates that the evaluation model generates diverse judgments for dialogue content based on different criteria. To align more closely with human behavior, we prioritize key judgments in the evaluation model's outputs. In future, we try to further expand the criteria space to uncover a variety of decision paths. Additionally, we aim to find more accurate preference selection methods to replace ensemble methods, thereby achieving a better alignment with human behavior. 
broader impact: 
 Our work focuses on the task of automatic evaluation, specifically exploring how to learn better evaluation strategies from an evaluation tree. We demonstrate that automated evaluation criteria can replace human priors, and by combining branch decision-making with DPO training, we have achieved robust evaluation performance. We conduct detailed experiments covering a broad range of real-world scenarios to discuss how to enhance model evaluation capabilities from scratch. With our work, we hope to inform further research into better understanding and developing improved evaluation methodologies for LLMs. Here, ♣ represents the dialogue data collected from Fennec, ♠ represents the dialogue data collected from large-scale open-source dialogue data, and ♢ represents the evaluation data annotated using GPT-4. In the ID settings, the same dialogue data is used across different training stages, while the different dialogue data is used for other setups. Given the diverse nature of dialogue tasks, the assumption of data distribution is highly influenced by the collection strategies and data deduplication methods employed. These processes inherently vary, and it is challenging to guarantee that each dataset comprehensively represents distinct domains or tasks. We utilize Zephyr-7BScenarios Initial SFT DPO ID 3k ♣♢ 3k ♣ 3k ♣ OOD 3k ♠♢ 2k ♠ 2k ♠ Transfer 3k ♠♢ 1k ♣ 1k ♣Chat 10 [33] as the backbone to train our evaluation model. We employ DeepSpeed [26] library, Zero Redundancy Optimizer (ZeRO) [25] Stage 3, FlashAttention [5] , and the bfloat16 (BF16) and tfloat32 (TF32) mix computation precision on 8 NVIDIA A100 GPUs. The number of gradient accumulation steps is 32, the learning rate of the initial model and SFT model is 1e-5, and the learning rate of the DPO model is 5e-7. The number of epochs is 1 in each training stage. We set β to 0.1when training the DPO model.Table 7 -10 shows different prompts. Table 7 shows the prompt for response correction, and Table 8 elaborates on the prompts that GPT-3.5 and GPT-4 models use to generate the testing results. Table 9 is employed for preference generation powered by gpt-4-0125-preview. ---SYSTEM MESSAGE---Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below. You should choose the assistant that follows the user's instructions and answers the user's question better. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Begin your evaluation by comparing the two responses and provide a short explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible. After providing your explanation, output your final verdict by strictly following this format: " Step Content[[A]]" if assistant A is better, "[[B]]" if assistant BFor evaluating human satisfaction with responses from an AI assistant based on a [User Query], we need to brainstorm and establish ten [Evaluation Criteria] directly linked to the user's query. These criteria play a crucial role in objectively assessing response content, with higher priority and greater evaluation weight. *** As an illustration: 1. Relevance: Evaluate whether the response is directly related to the user's query. Table 10 : Prompts for multi-step evaluation. The criteria in Scoring Guidelines and Pairwise-eval is regularly extracted from the output of the first step. The scoring guidelines in Pairwise-eval are the output of the second step.Discuss antitrust laws and their impact on market competition. Compare the antitrust laws in US and China along with some case studies."Response A (from Alpaca-13B) Response B (Preferred, from Vicuna-13B-v1.2)Antitrust laws are designed to promote competition in markets and protect consumers from anticompetitive practices. In the US, the Sherman Antitrust Act of 1890 is the primary antitrust law and prohibits agreements or practices that restrain trade and impose unfair competition. The US Department of Justice and the Federal Trade Commission are responsible for enforcing the law. In China, the Anti-Monopoly Law enacted in 2008 is the primary antitrust law and is similar to the Sherman Act in prohibiting monopolistic agreements and abuse of dominance. Case studies have shown that antitrust laws can have a significant impact on market competition. For example, in 2000 the US Department of Justice filed a lawsuit against Microsoft alleging that the company had abused its monopoly in the computer operating system market. The lawsuit resulted in Microsoft being ordered to make significant changes to its business practices and the case is often cited as an example of how antitrust laws can be used to promote competition. In China, ... . Overall, antitrust laws can have a significant impact on market competition and have been used to promote competition and protect consumers in both the US and China.Antitrust laws are a set of regulations designed to promote and maintain fair competition in markets. These laws aim to prevent anticompetitive practices that can harm consumers, such as price-fixing, monopolies, and mergers that reduce competition. In this essay, we will discuss antitrust laws and their impact on market competition, compare the antitrust laws in the US and China, and analyze some case studies. Antitrust laws have a significant impact on market competition. These laws promote competition by preventing anticompetitive practices that can harm consumers. For example, antitrust laws prohibit price-fixing, which is when competitors agree to set prices at a certain level. This practice can lead to higher prices for consumers and reduced competition. Antitrust laws also prevent monopolies, which occur when a single company dominates a market. Monopolies can lead to higher prices, reduced innovation, and poorer quality products. Antitrust laws also regulate mergers and acquisitions to ensure that they do not reduce competition in the market.The US has a comprehensive set of antitrust laws, including the Sherman Act, the Clayton Act, and the Federal Trade Commission Act. The Sherman Act prohibits anticompetitive practices such as price-fixing, ...Both assistants address the core aspects of antitrust laws, their impact on market competition, and provide comparisons between the US and China along with case studies. However, there are distinct differences in the presentation, depth, and accuracy of their responses. Assistant A Pros: 1. Clearly explains the purpose of antitrust laws and names specific laws in both the US and China. Justification: The key experiment results are summarized in the end of the introduction.principal contributions are clearly stated in the end of the abstract.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We include the information of computational resources in Appendix A.2.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the impacts on Section 7.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [No] Justification: This study poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] Justification: We properly cited all benchmarks used in this study.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.https://chat.lmsys.org/Here, "better" is defined as aligning with human preferences and values.https://github.com/huggingface/alignment-handbook 
 
 Appendix:  


Decision: In this paper, the authors conceptualize the NLG evaluation process as a decision tree, where nodes are evaluation actions (criteria, scoring guidelines, or judgements) and paths are evaluation trajectories consists of a sequence of actions. To fine-tune an LM to generate NLG evaluation following such procedure, the authors propose to collect data by sampling branches from such trees while applying some pruning heuristics (self-consistency and positional-consistency). The authors show that LMs fine-tuned with such data (with SFT and DPO) outperform baseline systems on benchmarks such as Chatbot Arena and RewardBench. 

All reviewers agree that the idea of framing LLM evaluation as a decision tree is novel and interesting; they acknowledge that reducing the dependency on human-labeled data by generating supervised data and preference pairs from such evaluation tree could be empirically very useful; most reviewers agree that the authors did a good job on their experiment design, they have included multiple settings (in-distribution, out-of-distribution, and transfer evaluation) and provided a thorough assessment of the proposed method against many baselines systems. 

Reviewers have mixed opinions regarding the presentation of the work, YdUv and sKgh find it easy to read but cKwh and RFGW find the method description less straightforward and suggest the authors to provide a figure demonstrating concretely how an LM evaluation is conducted using the proposed framework. 

During the author-reviewer discussion period, the authors have successfully addressed many concerns, below are some remaining concerns that keep reviwers hesitant to recommend acceptance:
- Reviewer cKwh fails to be convinced about the use of positional-consistency. While acknowledging its precedence among other heuristics in prior work (as argued by the authors), they remain considering this design choice arbitrary. 
- Reviewer GhxJ had concerns about the in-distribution results in Table 1, it was unclear why the SFT/DPO results are not better than other approaches using the same dataset. The authors made the effort running additional experiments that mixing the in-distribution data with new synthetic ID data and this achieves consistently better results. While applauding to that response, the reviewer believes further investigation is required to fully understand this, e.g., an ablation study on how much the performance improvement was from original data vs from the synthetic data.

I share some concerns with the reviewers as following:
- I partly agree with Reviewer cKwh that self-consistency and positional-consistency are a bit arbitrary. I acknowledge that both are necessary properties that a good evaluation path should have, and the authors adopt these properties by following prior works, but in my opinion consistency properties are far from suffiicient. Do the authors mention in the paper/response about the possibility that the LM fine-tuned with such data (with SFT and DPO) could fall into some local optima that overly-concern about consistency metrics but hurts on some other dimensions (especially on the semantics of an evaluation chain)?
- As also mentioned by Reviewer GhxJ, I have concerns about the diversity of the sampled trees. To my understanding, the diversity between branches could affect a lot the quality of the sampled data. As described in the paper (Section 4.3), the multiple criteria are generated with a brainstorming prompt, and the guidelines and judgements are sampled by playing with temperature and top-p parameters. I fail to find how they try to diversify the tree branches nor how to measure the diversity. In the authors' reponse to Reviewer GhxJ they say "only a minimal chance exists for the model to output semantically similar criteria" then refer us to Table 10 and 11 for examples. I fail to be convinced without seeing quantitative evidence; I fail to find qualitative evidence from Table 11 either (and actually the table does not show exmaple of guidelines?).

Overall this submission is around the borderline in terms of scores, the authors are doing a good job making their work stronger during the discussion period thank to some good suggestiong by the reviewers. I suggest the authors to continue improving their work especially in the line of better understanding its in-distribution behavior, as well as to measuring/encouraging diversity of the fine-tuning data.

Reviews: review: paper_summary: paper_summary: In this work, the authors propose a tree-based data sampling method to conceptualize the evaluation process as a decision tree, where each node represents an evaluation action, and each path from the root to a leaf node represents a trajectory of evaluation reasoning. The proposed method involves generating supervised data and preference pairs derived from the evaluation tree for SFT and DPO training. This approach aims to reduce the dependency on labeled data and improve the performance of the evaluation model across in-distribution, out-of-distribution, and transfer evaluation settings. Experimental results demonstrate that the proposed model can enhance evaluation efficiency and performance.
strength_weakness: strength_weakness: 1. Potential Biases --- The initial multi-branch training data is generated using only GPT4, which could introduce bias to the training data. Moreover, the branch ensemble method could also introduce bias to the training data. If the training data is biased or unrepresentative, the model's evaluations may also be biased. The authors should consider labeling a small annotation set to validate the branch ensemble approach.
questions: questions: 1. In table 1, what is the number of branches generated by Initial, SFT, and DPO for evaluation?

Typos:
1. Line 36: single quote format error
2. Line 256: "" -> ``''
3. Figure 4: font too small, hard to read
limitations: limitations: Yes, limitation discussed in the paper.
paper_summary: paper_summary: The paper investigates how to improve the quality of automated evaluation through fine-tuning (SFT and DPO). The main algorithm proposed by the paper is to construct an search tree which consists of node of (criterion, scoring guide, and judgment). This tree is later pruned and modified and the different paths serve as fine-tuning data for SFT and DPO.

My current rating is tentative. If the authors can kindly clarify the details of the paper, I'm happy to raise the score.
strength_weakness: strength_weakness: I don't think this paper has substantial weaknesses.

1. There are some imperfections of text -- mostly just need to be clarified. Missing notation definitions, etc. 
2. The performance improve over Auto-J on AGR is minor (55.13 -> 57.18). OOD evaluation, Zephyr-7B AGR is 56.75 and GPT-4 is 62.28 (which is close, but not quite close). CNS however is beating GPT-4. This would be very helpful for me to understand a bit more about what CNS is, and whether beating GPT-4 on this metric is meaningful or not (see Q6).
questions: questions: I do however have a few questions:

1. It seems that no human preference/judgment label is used. The initial policy is GPT-4. Then the creation of data for SFT and DPO are through the tree construction. Is my understanding correct? If so, this is a type of self-improving/bootstrapping method, which I find quite interesting. It also has greater impact on other fields of LLM.
2. Sec 4.2, consistency pruning. I'm not sure I understand self-consistency -- In Figure 3 left most tree, how did you obtain different versions of J1? The figure label says "red dot is judgment result by swapping the response positions" -- isn't it "positional-consistency" if swapping positions is involved? Please clarify.
3. When you collect preference label (Sec 4.4), for Branch Ensemble, how do you create the ensemble?
4. LLM-as-a-Judge. The writing says additional analysis is in Section 5.4. Sec 5.4 talks more about Auto-J performance in different scenarios. Where is the result of using Auto-J as a judge to create labels for DPO?
5. Table 2 Auto-J dagger (second row). What is this!? The caption did not explain what this means. It might be in the text but I couldn't find it. 
6. Can you expand on why AGR and CNS are good evaluation metrics? Appendix A.2 only has training details. 
7. I appreciate the honesty in Sec 5.7 -- can you offer some explanation on why DPO is unstable for in-distribution training? It's often hard to explain empirical behaviors -- it's perfectly ok if you don't have an explanation.
limitations: limitations: Yes
paper_summary: paper_summary: They present an approach to improving LM evaluation by having models first generate an evaluation criteria, then a scoring guideline, and then finally a final judgement. They then develop a procedure for collecting training data corresponding to these three steps by applying branching/pruning approach (sample multiple criteria, from each sample multiple guidelines, etc...). They then use the generated data to train a DPO and SFT model. They find that their method outperforms baseline evaluation approaches according to correlation with human judgement on dialogue evaluation.
strength_weakness: strength_weakness: * The paper is honestly pretty hard to follow. There's a lot of moving parts and it's not explained in an easy to digest way.
* The specific method presented seems a little bit ad-hoc, and could be justified better in the paper (e.g. why use criteria, then guideline, then judgement, why not some other sequence of steps?).
* Looking at Figure 1, it doesn't seem that their method improves all that much over the baseline

Nits:
* The related work seems pretty sparse. There's lots of work on improving LM evaluation in math reasoning settings that isn't discussed.
* Figure 4, the text is really small and hard to read.
questions: questions: * The description of the growth / pruning part is pretty confusing and in general a better explanation of what exactly is happening there would be really helpful.
* "Agreement quantifies the proportion of evaluations that meet the criteria for swap consistency and align with human judgments.": why both of these criteria need to be met? Isn't the swap part already covered by the consistency evaluation?
limitations: limitations: They do a good job of discussing the limitations. I would also note that it is unclear how effective this is when applied to more challenging tasks like mathematical reasoning (e.g. MATH benchmark) as a limitation.
paper_summary: paper_summary: This paper proposes a novel approach to efficiently evaluate LLMs using branching preference learning. The authors conceptualize the evaluation process as a decision tree, where each path represents an evaluation reasoning trajectory. They introduce a tree-based data sampling method and preference learning based on the DPO algorithm to improve evaluation capabilities. The method is tested in three settings: in-distribution, out-of-distribution, and transfer evaluation. The authors claim their model significantly reduces dependency on labeled data and demonstrates strong performance across different evaluation settings while reducing inference costs by 90% compared to searching the entire evaluation tree.
strength_weakness: strength_weakness: - The biggest concern I have is that in-distribution performance is not better than other baselines it compares to. This begs the question of where the improvement gain is from. If in-distribution evaluation performance is mediocre but out-of-distribution does better, then doesn't the most gain come from a better dataset? 
- Another concern is the unnaturalness of using evaluation criteria as individual nodes. How to ensure the coverage of those criteria across different nodes. Are they overlapping each other or completely different? The paper is quite vague on this.
- Why is each criteria subtree only a binary tree? If using a tree structure, it seem like it can be easily extend to multiple nodes rather than just 2 at each layer.
questions: questions: - It is unclear how to sample multiple evaluation path? By using a high temperature? Or is each individual criterion a criterion node?
- What is the Eval-P benchmark? It is not cited in the paper.
limitations: limitations: Yes
paper_summary: paper_summary: In this work, the authors conceptualize the evaluation process as a decision tree, where each node represents an evaluation action, and each path from the root to a leaf node represents a trajectory of evaluation reasoning. The authors demonstrate that within a limited search space, there exist better decision-making behaviors that facilitate the model in making reasonable and accurate judgments.
strength_weakness: strength_weakness: 1. The authors missed a lot of key related works, including close-ended benchmarks such as MMLU, MMLU-pro, MixEval, GSM8k, GSM1k, etc; open-ended benchmarks such as Arena-Hard, AlpacaEval, WildBench, Chatbot Arena, etc.
2. I think the writing needs improvement. Now it's not easy for a reader to get what you are focusing on. If you are doing evaluation, then try to use some pipeline figures and comprehensive captions to describe the core idea. Besides, all captions in this paper is misleading, not telling the reader about what is happening in the table or figure; also, there lacks some key sections such as conclusion.
3. How to measure the quality of the proposed evaluation? I think just evaluating 5-6 models is far from enough. Beyond that, how is the model rankings related with Chatbot Arena or some other popular benchmarks such as MMLU?
questions: questions: Is it necessary to consider evaluations specifically for dialogue settings? I think all LLM evaluations are in a dialogue settings, the difference is just one-turn or multi-turn evals. Just calling it open-ended eval will be better aligned with the LLM research community.
limitations: limitations: Yes

Comments: title: Further Discussion
Further Discussion
Further Discussion
comment: Hi Reviewer sKgh, do you have any comments regarding the authors' rebuttal?
Hi Reviewer cKwh, do you think the authors' new response has addressed your concerns?
Hi Reviewer GhxJ, do you think the authors' new response has addressed your concerns?
Hi Reviewer RFGW, do you think the authors' new response has addressed your concerns?
**Thank you very much for your suggestions, which have helped us improve the quality of our work.** Following your advice, we have conducted new experiments and  to provide the following clarifications：

+ We have never denied the contribution of data, especially the additional unlabeled data (in the OOD setting) used in our approach. However, it is important to note that this data is unlabeled, making it widely accessible without increasing extra costs.

+ These additional data are simply dialogue data. The purpose of the evaluation tree is to generate the criteria, scoring guidelines, and judgments required to train the evaluation model. Therefore, including only dialogue data is insufficient for training an evaluation model. Hence, we cannot consider the new unlabeled dialogue data as the key to improving performance. Our approach requires combining these dialogue data and constructing SFT and DPO data.

+ **Most importantly**: Our proposed method is based on the observation in Figure 1, which shows that increasing the number of branches can enhance evaluation performance. This means that even without the additional data, the initial model can achieve good evaluation performance (**though it may require > 40 branches**). The SFT and DPO methods are designed to enable the model to efficiently reach the final result (**< 3 branches**). This is also why we employed branch ensemble in Section 4.4, as branch ensembles generally yield better decision results and provide a higher upper bound on performance. The purpose of designing the evaluation tree is to allow the model to more quickly identify the key evaluation paths, which can be derived from the branch ensemble results (as referenced in Section 4.4). **Thus, the evaluation tree is largely aimed at accelerating the evaluation process by sampling fewer branches (yet more crucial)**. I believe our method's design and experimental results are well-supported by the substantial evidence mentioned above.

+ We verified whether model collapse was the reason for the impact on ID performance. By mixing the original training data with the newly synthesized data when training the SFT model according to your advise (thanks again), **we observed significant performance improvement, which supports our initial hypothesis in Section 5.7**. Continuously training the SFT model and DPO model with the same data affects model convergence. However, compared to the OOD setting, there is still a lack of new unlabeled data, making direct comparisons still unfair. If we obtain more unlabeled domain-specific data, we believe our methods can achieve even better results. Considering the rebuttal time limitation, we may not be able to conduct additional experiments. However, our experiments, along with the common response experiments on the Reward Bench, should be sufficient to address your concerns.


|                                    | branch | Eval-P w/tie    | Eval-p w/o Tie  |
| ---------------------------------- | ------ | --------------- | --------------- |
| Ours ID SFT (copy from paper)      | 1      | 56.68/86.64     | 70.76/89.11     |
|                                    | 5      | 55.96/86.57     | 72.91/88.13     |
| Ours ID DPO (copy from paper)      | 1      | 55.24/84.26     | 69.87/86.95     |
|                                    | 5      | 57.18/85.63     | 74.88/88.52     |
| Ours OOD SFT (copy from paper)     | 1      | 54.59/87.14     | 70.56/88.52     |
|                                    | 5      | 55.10/87.86     | 73.69/89.99     |
| Ours OOD DPO (copy from paper)     | 1      | 55.89/89.44     | 75.76/90.67     |
|                                    | 5      | 56.75/90.37     | 77.23/92.24     |
| Ours ID SFT + Mix Trainng  （new） | 1      | 56.87/87.92     | 70.92/88.99     |
|                                    | 5      | 56.32/87.86     | 75.17/89.79     |
| Ours ID DPO + Mix Trainng  （new） | 1      | 57.54/90.66     | 78.60/93.32     |
|                                    | 5      | **58.41/91.52** | **79.69/93.72** |

Caption：Our Mix Training setup differs from that in the paper by using both original and synthetic data to train the model, whereas the paper only used synthetic data for training the sFT model. As a result, we achieved an Agreement of 58.41 vs. 55.80 , and a Consistency of 79.69 vs. 74.14 compared to Fennec.
Thanks for your review comments!

> If you could add some more clarity in the paper about how the AutoJ and Fennec settings differ from yours:

We outline the settings and core challenges, highlighting the differences:

+ Auto J's approach evaluates two different AI responses using a one-step evaluation process. The dialogue data in its training dataset was human created to match the distribution of the test set, as shown in Figure 4. During training, only the SFT method was used, with no DPO training included.

+ Fennec employs a multi-step evaluation, using criteria and scoring guidelines to alleviate the complexity of the evaluation task. The training also involved human selected dialogue data and included only SFT training, with no DPO training.

+ In our approach: 1）There is no human selection or labeling cost—both the sampling of dialogue data and preference labels are fully automated. 2）We use a multi-step evaluation process to simplify the difficulty of complex evaluation tasks. 3）Most importantly, we incorporate DPO optimization, which helps the model achieve better performance with fewer branches, enhancing evaluation effectiveness.

+ Our method demonstrates that even a 7B parameter evaluation model has significant potential compared to the 70B model used in [1] for evaluation tasks. This advantage is largely due to our task decomposition (similar to Chain of Thought), which breaks down the evaluation process into three steps: generating evaluation criteria, scoring guidelines, and final judgements results. These steps and their intermediate outputs simplify the task, making the 7B model more capable in complex scenarios.

> Your method has a large description length, which generally goes against my prior, and makes more skeptical that every aspect of the method is necessary and also muddles the true contribution of the work.

+ If possible, could you please point out the details that might be unclear to you? This will help us explain our work more clearly.

+ We will also include a new Figure to illustrate how the evaluation task is conducted: The evaluation model receives a user query and two AI responses, and it only needs to assign one of three labels (win, lose, or tie) to indicate which AI response is better, or if they are equal.

+ In the Methods section (Section 4), we cover the following: 1) Section 4.1 explains how we collect the dialogue dataset. 2) Section 4.2 shows how we train an initial model. 3) Section 4.3 details how we construct the evaluation tree. 4)Section 4.4 describes how we collect training data based on the evaluation tree. 5) Section 4.5 demonstrates how we train the SFT and DPO models. Among these, only Section 4.2 follows a process similar to other related work. The remaining sections introduce new approaches developed in our work, which may explain some discrepancies with your prior understanding. Even compared to the latest research [1], our work offers novel contributions: We demonstrate that it's possible to improve evaluation model performance without relying on human labeled data, and we also showcase the potential of a 7B model, which is popular in both academia and industry.

> I think positional consistency is a pretty random heuristic. I agree that LMs might be sensitive to position and whatnot, but LMs are sensitive in all kinds of ways to prompts, so why focus on just position

+ The issue of positional consistency was not proposed by us; please refer to references [2] and [3].

+ Given that works [3], [4], and [5] all consider positional consistency and use Agreement and Consistency as evaluation metrics, I do not agree this as a flaw in our work.

+ We fully agree with your point that LLMs exhibit more bias beyond positional consistency. However, since positional consistency is the most direct and easily improvable, I believe it is necessary to address these biases during training. In fact, with pirwise evaluation, if positional consistency significantly decreases when positions are swapped, it would be unacceptable for downstream applications such as reward models or model corrections.

Considering that both Reviewer YdUv and Reviewer sKgh also agree our presentation to be clear, we hope to engage in further discussion to address any concerns for you.

[1] Self-Taught Evaluators （Meta 2024,8）

[2] Large language models are not fair evaluators

[3] Generative judge for evaluating alignment

[4] PROMETHEUS 2: An Open Source Language Model Specialized in Evaluating Other Language Models

[5] OffsetBias: Leveraging Debiased Data for Tuning Evaluators
I appreciate your response. If you could add some more clarity in the paper about how the AutoJ and Fennec settings differ from yours, that would be great.

Regarding the confusing parts: I think it would be less confusing with a clear example or an LM evaluation using your framework somewhere in the main text (e.g. a concrete example in a figure somewhere). Your method has a large description length, which generally goes against my prior, and makes more skeptical that every aspect of the method is necessary and also muddles the true contribution of the work. I think positional consistency is a pretty random heuristic. I agree that LMs might be sensitive to position and whatnot, but LMs are sensitive in all kinds of ways to prompts, so why focus on just position; you could invent so many other arbitrary heuristics that would probably be just as good and provide no real insight whatsoever. Pair these criticisms with the fact that one of the evaluation metrics is less standard (e.g. consistency) (essentially created to show that this paper's method works), I would have a hard time accepting the paper as it stands. That being said I think the reward bench evaluation you added looks promising, so I am willing to raise by 1 point.
Thanks for your review comments. Seems you have some misunderstandings about our paper.

1. **For concern 1**:

+ We have ensured that the majority of relevant works, particularly those related to preference evaluation with LLMs, have been cited [1-11].

+ The related works you mentioned on close-ended benchmarks are not focused on preference evaluation, while open-ended benchmarks, such as Chatbot Arena, are focus on model ranking. It’s important to note that model ranking is not the same as preference evaluation, even though it is conducted through a preference evaluation approach. Preference evaluation requires to evaluate diverse dialogue data, not just scoring different models (Because Chatbot Arena may not include some specialized user queries, please refer to [9]).

+ The Chatbot Arena and AlpacaEval that you referred to are part of the series of works cited as [1] and [11]. The only ones not cited are MMLU, GMS8k, and WildBench etc. We have already clarified in our common response that MMLU is used for accuracy evaluation. Our preference evaluation serves as a complement, primarily focused on evaluating human preferences. 

+ We will include these missing citations in the next version.

2. **For concern 2**:

+ We do not consider *The correlation with Chatbot Arena* to be an indispensable metric. Please refer to recent works on preference evaluation [12] and [13], where most do not use Chatbot Arena correlation as a comparative experiment. Recent studies have centered their experiments around **Reward Bench**. We have added Reward Bench experiments in the common response, and even compared to the latest works, we maintain the novelty. Unlike [12], which used a large amount of human labeled data, and [13], which generated synthetic evaluation data using 70B LLMs without human labels, our approach consistently emphasizes "improving evaluation performance without using human labeled data." Additionally, we decomposed task difficulty using an evaluation tree, enabling a 7B model to comparable with a 70B model, which addresses a gap in [13] where small LLMs struggle with evaluation.

+ We have also provided the correlation coefficients between 20+ models and Chatbot Arena: Spearman correlation = 0.854.

|                         | Arena Score | Our Score |
| :---------------------- | :---------- | :-------- |
| gpt-4                   | 1162        | 230.13    |
| mpt-30b-chat            | 1045        | 227.79    |
| vicuna-13b              | 1042        | 226.79    |
| gpt-3.5-turbo           | 1068        | 222.26    |
| zephyr                  | 1041        | 222.0     |
| claude-v1               | 1149        | 219.53    |
| wizardlm-13b            | 1058        | 218.93    |
| Llama-2-70b-chat        | 1093        | 217.80    |
| vicuna-7b-v1.3          | 1005        | 212.46    |
| Llama-2-13b-chat        | 1063        | 210.86    |
| Koala-13b               | 964         | 205.13    |
| oasst-sft-4-pythia-12b  | 893         | 203.5     |
| mpt-7b-chat             | 927         | 201.8     |
| Chatglm-6b              | 924         | 200.16    |
| RWKV-4-Raven-14B        | 921         | 197.66    |
| palm-2-chat-bison-001   | 1003        | 192.99    |
| Llama-2-7b-chat         | 1037        | 192.33    |
| Alpaca-13B              | 901         | 183.06    |
| stablelm-tuned-alpha-7b | 840         | 180.20    |
| Dolly-V2-12B            | 822         | 165.13    |
| LlaMA-13B               | 799         | 160.93    |
| fastchat-t5-3b          | 868         | 154.2     |

[1] Judging llm-as-a-judge with mt-bench and chatbot arena

[2] Gptscore: Evaluate as you desire

[3] Tigerscore: Towards building explainable metric for all text generation tasks

[4] Generative judge for evaluating alignment

[5] Branch-solve-merge improves large language model evaluation and generation.

[6] Self-critiquing models for assisting human evaluators

[7] Large language models are not fair evaluators

[8] Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization

[9] Evaluating large language models at evaluating instruction following

[10] Judgelm: Fine-tuned large language models are scalable judges.

[11] Alpacafarm: A simulation framework for methods that learn from human feedback

[12] Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluation （DeepMind 2024,7）

[13] Self-Taught Evaluators （Meta 2024,8）
Hi AC, 

The authors have made a comprehensive rebuttal, which indeed clarified some concerns. 
However, I still feel many concerns cannot be simply solved by rebuttal–the first draft are only allowed to contain minor issues.

I feel there are two main flaws of this paper:
1. Detached from the literature. Many works are lost in the introduction and related work section–discussion of the chatbot arena are missing; as a method for llms-as-a-judge evals, authors didn't explicitly discussed the recent llms-as-a-judge evals. This could reflects that this work could have been done without seeing sufficient research context.
2. The experiments demonstrating the effectiveness of their method are not convincing. The most important metrics, such as the correlation with chatbot arena requires at least 25 models to be statistically significant. And the original draft didn't contain such discussions, and the rebuttal also didn't show such correlation scores.
Thank you for the response and addressing my questions. I am still concerned about the ID setting shown in Table 1. On the same dataset, compared to Auto-J and Fennec the proposed method is not really doing better. It is sometimes worse. This is evidence that the methodology is not the reason behind performance gain. Thank you for bringing up model collapse. One can definitely avoid model collapse even during iterative data synthesis [1]. The main problem here is that external data seems to be helping the most rather than the evaluation tree process.

[1] https://x.com/RylanSchaeffer/status/1816535790534701304
Hi Reviewer RFGW, could you provide more insights on why / on which point you don't think the authors have sufficiently addressed your concerns (given that you decide to keep your score)? Because there is still one day until the end of the reviewer-author discussion period, they authors could still have chance to clarify and to provide more information to address your concerns.
Thank authors for your responses. I keep my original score unchanged.
Thanks to the authors for the responses. I will keep the current scores that I believe reflects my overall assessment of the paper.