Title: Efficient Evaluation of LLMs via Branching Preference Learning
Venue: NeurIPS 2024
Accepted: no
Keywords: LLMs; dialogue evaluation；RLHF
Abstract: Large language models (LLMs) have made significant advances across various generative tasks, progressing toward achieving near-human levels of intelligence. However, in many scenarios, LLMs face the challenge of insufficient human evaluation or even the inability to evaluate reliably.  Particularly, in complex dialogue scenarios involving diverse and intricate user intents, LLMs as evaluators of AI responses exhibit a substantial gap compared to humans. Moreover, due to the scarcity of high-quality evaluation data, LLMs exhibit deficiencies in their evaluation capabilities. In this work, we conceptualize the evaluation process as a decision tree, where each node represents an evaluation action, and each path from the root to a leaf node represents a trajectory of evaluation reasoning. We demonstrate that within a limited search space, there exist better decision-making behaviors that facilitate the model in making reasonable and accurate judgments. Specifically, we propose a tree-based data sampling method to generate supervised data and preference pairs derived from the evaluation tree. Furthermore, we introduce preference learning based on the DPO algorithm, which empowers the fine-grained evaluation model to explore and learn better branching strategies within budget-limited scenarios. Our model significantly reduces the dependency on labeled data and demonstrates strong performance across three different evaluation settings: in-distribution, out-of-distribution, and transfer evaluation. Experiments indicate that our model can reduce inference costs by 90\% compared to conducting searches across the entire evaluation tree, thereby significantly enhancing efficiency.

1

Abstract

Large language models (LLMs) have made significant advances across various generative tasks, progressing toward achieving near-human levels of intelligence. However, in many scenarios, LLMs face the challenge of insufficient human evaluation or even the inability to evaluate reliably. Particularly, in complex dialogue scenarios involving diverse and intricate user intents, LLMs as evaluators of AI responses exhibit a substantial gap compared to humans. Moreover, due to the scarcity of high-quality evaluation data, LLMs exhibit deficiencies in their evaluation capabilities. In this work, we conceptualize the evaluation process as a decision tree, where each node represents an evaluation action, and each path from the root to a leaf node represents a trajectory of evaluation reasoning. We demonstrate that within a limited search space, there exist better decision-making behaviors that facilitate the model in making reasonable and accurate judgments. Specifically, we propose a tree-based data sampling method to generate supervised data and preference pairs derived from the evaluation tree. Furthermore, we introduce preference learning based on the DPO algorithm, which empowers the fine-grained evaluation model to explore and learn better branching strategies within budget-limited scenarios. Our model significantly reduces the dependency on labeled data and demonstrates strong performance across three different evaluation settings: in-distribution, out-of-distribution, and transfer evaluation. Experiments indicate that our model can reduce inference costs by 90% compared to conducting searches across the entire evaluation tree, thereby significantly enhancing efficiency.

1 Introduction

Dialogue evaluation capability [6] is one of the fundamental abilities of human social interaction, involving the comprehension and interpretation of user intentions, as well as providing reasonable judgments on the correctness of different responses. Automated evaluation can assist humans to supervise powerful LLMs and is an essential component for superalignment and weak-to-strong generalization techniques [4]. However, human evaluations [3, 22] are labor-intensive and timeconsuming, making it difficult to widely adopt. Traditional automated evaluation approaches [18, 39, 8] are limited by inherent deficiencies, such as string and semantic matching methods often yield subpar accuracy and lack of interpretability. The advent of large language models offers promise for automatically evaluating dialogue quality [19, 41, 15], owing to their high consistency with humans in intent understanding.

Nevertheless, automated evaluation remains a challenging issue due to the diversity of tasks and 33

scenarios it may encounter. The user queries often encompass multiple intentions [38], which cannot 34

typically be addressed using a single evaluation criterion. However, related research [35, 42] often 35

attempts to treat evaluation as a simplistic 'one-step' reasoning problem, causing even the most 36

75

Agreement(w/ tie)

Agreement (w/o tie)

70

550846

Auto-J

Auto-J

60

Initial model

Initial model

SFTmodel

55

SFTmodel

44

DPO model

DPO model

50

3

5

10

20

40

1

3

5

10

20

40

Inference Branch

Inference Branch

powerful large language models to struggle to provide reasonable and accurate results. It is essential 37 for the evaluation model to adapt to different scenarios and provide critical evaluation criteria. 38

39

40

41

42

43

44

45

46

47

48

In this work, we do not introduce any human prior for evaluation scenarios and criteria , which are commonly used for designing and collecting training data in related studies [14, 11]. The real-world conversational scenarios are often characterized by complexity and unpredictability, making it challenging to derive generalizable rules. Additionally, human priors frequently introduce biases [12, 20], making these evaluation methods poorly generalized due to a lack of adaptability and scalability. Therefore, we explore automatically sampling scenarios from large-scale datasets and employ LLMs to automatically generate evaluation criteria, aiming to eliminate human labor as much as possible. Another significant challenge is the lack of ground truth labels and human feedback during the training data collection process. The insufficient of available supervised data for evaluation tasks also prevents them to scale effectively.

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

Despite various challenges, we discover that the evaluation model is constrained in its ability to identify crucial evaluation criteria, but this limitation can be mitigated by increasing the number of considered criteria . As shown in Figure 1, the Initial model can achieve nearly a 10 point improvement in the agreement metric by increasing the number of evaluation branches. This findings motivate us to design tree-based data sampling methods to generate training data and a branching preference learning algorithm to improve 'multi-step' inference capability. Specifically, we employ a breadth-first growth approach to construct an evaluation tree, where each path from the root to a leaf node represents a complete evaluation trajectory. We collect high-quality evaluation trajectories from the search space of the evaluation tree and trained an SFT model, which exhibited superior performance and prediction consistency. Furthermore, we refine these evaluation trajectories and train a DPO model [24], which can effectively prioritize and output crucial evaluation criteria, thereby enhancing the model's inference effectiveness.

We mainly evaluate our models in three settings: in-distribution, out-of-distribution, and transfer evaluation. Specifically, we use the datasets from the Chatbot Arena 1 as in-distribution data, and collect data from large-scale dialogue datasets without human priors as out-of-distribution data. In our experiments, we demonstrate that (1) our model outperforms several recent open-source evaluation models and methods across all three settings, (2) there is a noticeable improvement in the evaluation model's capability when progressively training the Initial model, the SFT model, and the DPO model, and (3) as shown in Figure 1, our DPO model achieves the best performance even when using only a single evaluation criterion (single inference branch).

2 Related Works

Automated dialogue evaluation [6] has long been a significant challenge in the field of generative AI. 70 Recent work [10, 7, 35, 41] has demonstrated that LLMs can act as automated evaluators, serving 71 as alternatives to human judges. However, LLMs still exhibit issues such as positional bias and 72 prediction inconsistency [34, 40]. Many studies have relied heavily on human priors [14, 11], thereby 73 neglecting to explore the model generalization capabilities. In contrast, our research focuses on 74 examining the performance with different data distributions and investigates how to bridge this gap. 75

Weconsider automated evaluation as a complex reasoning task and aim to improve model performance 76 by optimizing reasoning trajectories. When handling such tasks, LLMs typically utilize decision 77 trees [37, 23] to model the reasoning process. They often employ search algorithms like A* [21, 13] or 78 Monte Carlo Tree Search (MCTS) [29, 31] to identify the optimal reasoning path within the candidate 79 decision. However, these methods generally rely on deterministic reward signals or feedback, which 80 are absent in our settings. We demonstrate that the ensemble boundary of the evaluation branches 81 provides a feasible reward signal to verify the accuracy of the reasoning trajectories. Based on this, 82 we can guide the model to generate a substantial amount of high-quality data. 83

84

85

86

87

88

89

Automated evaluation is also a pivotal technology within scalable oversight, aiming to enhance humans' ability to supervise models. For example, humans may ask models to critique the outputs of other models [9, 28] or use models to help decompose a problem into simpler subproblems [17]. In contrast to improving human supervision, we focus on how to conduct reliable automated evaluations. Certainly, our proposed evaluation methods and results can also be combined with human oversight to provide even better performance.

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

3 Problem Setup

In this work, our primary focus is on evaluating AI responses, particularly in analyzing query and response pairs within given datasets to determine which response is better 2 . Traditional approaches [41, 35] regard the evaluation task as a 'one-step' classification (' win ' or ' lose ' or ' tie ') or generation problem, where the final scores or explanations are assigned by a reward model or the evaluation model. However, with complex reasoning tasks or scenarios, a given query may involve multiple intents, whether explicit or implicit [38], yet the generated responses by AI often overlook some of these intents, constrained by the model's capabilities. Therefore, multiple evaluation criteria are required [19] to verify whether the responses address the query requirements and align with user intentions. Considering the complexity and diversity of dialogue tasks, it remains an intractable challenge to gather comprehensive and accurate evaluation criteria.

3.1 Conducting evaluation through multi-step reasoning

Wetry to view the evaluation task as a complex reasoning task, a multi-step generative problem, which entails: (1) initially seeking suitable evaluation criteria, then (2) generating scoring guidelines based on these criteria, and finally (3) conducting comprehensive judgment based on the aforementioned criteria and scoring guidelines. Formally, given a dialogue X , we will use an evaluation model to sequentially obtain the criterion C , scoring guideline S , and judgment J :

where π θ represents the evaluation policy, the prompt please refer to Appendix A.3. Similar to related multi-branch evaluation [27, 16] methodologies, we refer to different reasoning paths as ' evaluation branch ', where each branch represents a decision-making process. Unlike previous methods [19] that relied on enumerating criteria, our goal is for the evaluation model to automatically generate crucial and high-priority criteria.

3.2 Focusing on two challenges

A natural approach is to first construct a candidate set of criteria and then derive suitable results based on these criteria. To address this task, we focus on the following two challenges:

How to construct an appropriate candidate set? Our aim is to develop a candidate set that includes multiple evaluation branches enriched with high-quality evaluation opinions. By training and optimizing this candidate space to advance desired behaviors, we can swiftly identify appropriate and critical judgments during the inference process.

How to rank the judgments? We also need to establish a ranking among different evaluation branches to optimize the candidate space. In contrast to recent studies [13], our evaluation dataset lacks ground truth labels or environmental feedback to act as reward signals. The cost of obtaining these signals is prohibitive, requiring not only expensive human labor but also

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

Dialogue

Query

Response pair

Criteria

Scoring guidelines

Judgements

Dialogue

Dialogue

Dialogue

STT

Model

train

Growth

Sampling

train

init

Init

Pruning

Model

3

2

3

2

Chain

Multi-branch

Tree

Model

DPO

facing issues of low consistency among humans in many ambiguous problems. Therefore, we need to design an innovative and cost-effective approach to address this challenge.

4 Method

Figure 2 illustrates an overview of our method, which involves three stages for model training: First, we train the Initial model to construct the evaluation tree; Then, we sample different evaluation branches as supervised data to train the SFT model, enhancing branch prediction consistency; Finally, we collect preference data to train the DPO model, ensuring rapid sampling of critical branches.

4.1 Collecting dialogue dataset

Evaluation models typically rely on robust generalization capabilities to effectively handle diverse dialogue tasks. Consequently, the distribution of training data significantly affects performance on unseen tasks encountered during real-world evaluations. To address this, we sampled from a largescale dialogue dataset rather than a specific data source. We then apply the K-Means algorithm [2] to cluster the data. Subsequently, we sample data from these clusters, ensuring that the training dataset encompasses a diverse set of dialogue scenarios. More details refer to Appendix A.1

4.2 Training initial model

We aim to construct a dataset from scratch for evaluation, consisting of dialogues paired with their corresponding evaluation trees. Each tree contains different reasoning paths during the evaluation of dialogues. The root node of this tree represents the dialogue data, and each path from the root node to a leaf node signifies an evaluation branch. Each evaluation branch comprises three decision-making behavior nodes: criterion C , scoring guideline S , and judgment J . To simulate this decision process, we introduce a multi-branch training approach [16] to train an LLM as the initial policy π Initial . We employ GPT-4 ( gpt-4-0125-preview ) [1] to generate corresponding multi-branch training data to enhance quality. This approach ensures that the model can auto-regressively generate evaluation branches using Equation 1.

4.3 Generating evaluation tree

We expand the branch candidates sampled from the policy π Initial using the breadth-first growth, thereby including as many high-quality evaluation paths as possible. Due to the different paradigms of SFT and DPO , we employ consistency pruning to split the sampling space to obtain training data:

Breadth-first Growth: The evaluation tree contains two distinct growth manner: for criterion C node, we use LLM's brainstorming capability to generate k relevant criteria; for scoring guideline S and judgment J node, we use sampling method by adjusting the LLM's temperature and top-p parameters. To simplify, we utilize the Initial model π Initial to generate a complete binary tree for each subtree with a criteria node as its root. Furthermore, since the evaluation task requires testing the model's consistency by swapping response positions, we can obtain k × 8 different evaluation branches.

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

Dialogue

Position-Consistency

B

Self-Consistency

J

[S}

J2

Dataset

SFT

J

J2

J', J2

J

?

J'2}

Sampling

S

J2

[}

J

J2

DPO

J

J2

J2

J2}

{J2

J'2

J'2

J'2

Dataset

J

J2

is judgment result by swapping the response positions

Consistency Pruning: Prior to pruning, we introduce two different consistency constraints: self-consistency, meaning the same criterion C and scoring guideline S should yield the same judgment J , and positional consistency, meaning that swapping positions should not affect the judgment J . Subsequently, we obtain SFT training data from evaluation branches in the evaluation tree that meet both consistency constraints, and DPO training data from nodes that do not satisfy these constraints.

4.4 Collecting preference labels

Although we can obtain SFT and DPO data from the consistency sampling space, this data lacks correctness verification. Typically, preference data requires human annotation to establish ranking sequences, a time-consuming process that is not suitable for scaling. Therefore, we propose two alternative approaches to label each evaluation branch with its correctness:

Branch Ensemble: Considering that there are only three final labels for judgment (' win ' or ' lose ' or ' tie '), we use an ensemble result of evaluation branches to obtain the consensus label. The ensemble method provides a lower bound of judge error without incurring additional costs. For SFT data, we filter out data that is inconsistent with the ensemble results. For DPO pair data, we select samples consistent with the ensemble results as 'chosen' samples, and those inconsistent as 'rejected' samples.

LLM-as-a-Judge: Some highly aligned LLMs, such as GPT-4, possess powerful annotation capabilities. Therefore, we use LLMs to determine which sample in the DPO pairs data is more reasonable as the 'chosen' sample. In our experiments, we found that this method has only a 20% disagreement rate compared to the Branch Ensemble method. We analyze this method in Section 5.4

As shown in Figure 3, we combine consistency pruning and automated labeling to collect the corresponding preference data. Through the labeling of judgments, we can also obtain preference information for criterion C and scoring guideline S based on the final judgment J decisions. Specifically, we prioritize predicting criteria that lead to correct judgments and select the scoring guidelines with the highest overall scores as the 'chosen' samples. Additionally, we randomly sample from the filtered data to create the training set, thereby controlling training costs and efficiency.

4.5 Training SFT model and DPO model

We use the Initial model as the starting point to train the SFT model π SFT using supervised learning, which reduces inconsistent predictions compared to the initial policy. Then, we take the SFT model as the initialization to train the DPO model π DPO using Direct Preference Optimization, which can learn the decision priorities of different branches, with the objective:

where the ( x, y ) represents data pair of different decision tasks in Equation 1, y c represents the 191 'chosen' sample, and y r represents the 'rejected' sample. 192

During the inference process, we create a single branch for each criterion to conduct evaluation, and 193 control the number of generated branches k to adjust the inference efficiency. Since the DPO model 194 employs sampling optimization, it usually achieves optimal performance with only a few branches. 195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

5 Experiments

As the most popular LLM evaluation platform recently, Chatbot Arena demonstrates high alignment with human judgments in pairwise response evaluations. We collect its open-source human judgment benchmark, Eval-P and MT-bench , to serve as the test set. We gather training data comprising both dialogue data and evaluation data for the following three evaluation scenarios:

In-distribution evaluation: We apply the Fennec [16] training data to train the Indistribution (ID) model, which included 3K dialogue data from Auto-J [14], along with evaluation data annotated by GPT-4. This training data is a multi-branch dataset, meaning that a single dialogue includes multiple evaluation branches.

Out-of-distribution evaluation: We collect 5M large-scale dialogue data and extracted 7K samples from it to serve as out-of-distribution (OOD) training data. GPT-4 annotate 3K evaluation samples from this dataset for the Initial model training.

Transfer evaluation: We use 3K OOD training data (which includes evaluation data) and 2K ID dialogue data (which did not include evaluation data) to train the transfer model.

For each benchmark, we employ Agreement ( AGR ) and Consistency ( CNS ) as performance metrics. Consistency measures the prediction consistency of the evaluation model when the positions of the responses are swapped. Agreement quantifies the proportion of evaluations that meet the criteria for swap consistency and align with human judgments. In many cases, the ' tie ' label indicates an inability to distinguish performance under some evaluation criteria. However, it may still be distinguishable under specific evaluation criteria. Therefore, we also present the model performance on the test data without ' tie ' label. For more details, please refer to Appendix A.2

5.1 In-distribution evaluation

The results are shown in Table 1, where methods marked with † denote our reimplementations. Since the Initial model leverages the Fennec training data for initialization, its performance can be

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

(a) The scenarios of ID dataset

(b) The scenarios of OOD dataset

(c) The scenarios of

Eval-P

regarded as its in-distribution evaluation baseline. As observed, the SFT and DPO models exhibit significant performance improvements over most baseline methods on both the Auto-J and Fennec datasets, achieving the highest agreement score of 57.18. In the multi-turn dialogue evaluation on MT-bench , the Fennec dataset comprises only single-turn dialogues, which constrains its effectiveness in handling multi-turn context information. Additionally, we observed the instabilities problems during the training process, which hindered the DPO model from outperforming the Initial model. A more comprehensive analysis of these instability problems is provided in Section 5.7.

5.2 Out-of-distribution evaluation

In terms of OOD evaluation, the Initial model performs worse than the baseline model on both Eval-P and MT-bench benchmarks, due to the distribution shift in the dialogue dataset. With RLHF [22] training, the SFT model significantly surpasses the Initial model in consistency rate and also enhances the agreement rate. Notably, the DPO model achieves superior performance with only three branches, thereby reducing inference latency by over 60%. In evaluation settings without ' tie ' labels, the advantage of the DPO model becomes more apparent, significantly outperforming other models, including proprietary model GPT-4. This demonstrates that the DPO model can effectively distinguish between responses using critical criteria, even when employing only 3 branch for inference. Furthermore, our models are capable of handling multi-turn dialogue scenarios, achieving performance that surpasses the in-distribution models. These extremely strong results indicate that our model excels at identifying more crucial criteria to help distinguish the difference of AI's responses.

5.3 Transfer evaluation

The purpose of transfer evaluation is to evaluate the model's capability to adapt to in-distribution data, thus mitigating the problem of training data distribution shift. It can be observed that both the SFT and DPO models demonstrate improvements across multiple benchmarks compared to the Initial model. Notably, in both OOD and transfer evaluation settings, the DPO model consistently achieves better performance than the SFT model, while also reducing the number of inference branches. Although the transfer model does not surpass the OOD model, it still achieves closed performance. In Section 5.4, we provide a detailed analysis of the different scenarios that lead to these models exhibiting significantly different performance characteristics despite their close overall performance.

5.4 Scenario analysis

To investigate the impact of scenario categories distribution in the training data, we need to analyze 250 the scenarios within the OOD, ID training sets, and the Eval-P test set. For this purpose, we 251 employ the scenario classifier trained by Auto-J, which effectively categorizes dialogue data into 58 252 different scenarios. Figure 4 presents the distribution of scenarios. It can be observed that Auto-J's 253 training set is well-balanced across the predefined scenarios, closely matching the distribution of the 254 Eval-P test set. In contrast, within the OOD data, the "Others" category exceeds 30%, and "General 255 Communication" surpasses 50%. The significant differences in scenario distributions between the 256 OOD data and the test set can lead to performance variations in test cases. 257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

From the evaluation results of fine-grained scenarios, we can derive several interesting observations from Table 2: (1) The ID and Transfer models significantly outperform the OOD model in Summarization and Exam Questions, which are notably lacking in the OOD training data. (2) The OOD model performs significantly better than the ID and Transfer models in the General Communication and 'Others' categories. (3) For writing-related text generation tasks, the OOD model achieves performance that is comparable to the ID model. These results indicate that the type and quantity of tasks remain crucial in evaluation tasks. Therefore, the evaluation model can achieve combinatorial generalization capability by increasing the number of scenarios or tasks. When GPT-4 serves as a judge to provide preference labels, it achieves improvement in code and NLP tasks compared with DPO model but also affects performance in other scenarios.

5.5 Dialogue correction

The critical capability of evaluation is to identify and rectify flaws in dialogues, thereby enhancing the overall quality of the original AI responses. Therefore, we test our model's ability to evaluate and correct dialogues generated by the Alpaca-13B [30] and the LLaMA27B Chat [32] models in MT-Bench . Unlike previous pairwise evaluations, MT-Bench presents a multi-turn dialogue and uses GPT-4 to assign scores (ranging from 1 to 10) to different AI responses, subsequently giving the model ranking relationship based on these scores.

Specifically, to elicit the model's correction ability, we construct 3k correction pairs and incorporate them into

the evaluation training set. When performing corrections, we first generate a judgment for the responses and then modify those with scores below 3. As illustrated in Table 3, the modification rates for Alpaca are all above 95%, indicating that the quality of responses generated by weak models is generally subpar. After refinement, both Alpaca-13B and LLaMA2-7B Chat model achieve better scores. Moreover, the correction results of the DPO model outperform those of the SFT model, demonstrating that better evaluation feedback can lead to significant improvements in evaluation quality. These results not only demonstrate the effectiveness of our model in identifying and correcting dialogue flaws but also highlight its potential to substantially improve the performance of dialogue systems through robust evaluation.

5.6 Impact of Initial model data scale

In our investigations, we strive to reduce reliance on both human annotators and GPT-4. Specifically, in the current work, we trained an Initial model using annotation data generated by GPT-4 without any addi-

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

57.0

90

92

76

89

91

Agreement (w/ tie)

56.5

Consistency (w/ tie)

8

( o/m) uwe

74

Consistency (w/o tie)

56.0

8

55.5

SFT model (ID)

Initial model (ID)

8

Initial model (ID)

SFT model.(ID)

72

Initial model (ID)

SFT model (ID)

88

Initial model (ID)

SFTmodet (ID)

55.0

DPO model (ID)

SFT model (OOD)

85

DPO model (ID)

SFT model (OOD)

70

SFT model (OOD)

DPO model (ID)

87

DPO model (ID)

SFT model (OOD)

DPO model (OOD)

84

DPO model (OOD)

DPO model (OOD)

DPO model (OOD)

54.5

1

3

1

3

1

3

1

3

5

Inference Branch

Inference Branch

Inference Branch

Inference Branch

tional supervision. We evaluated the performance of the Initial model trained on different sizes of data on the Eval-P benchmark. As shown in Table 4, the model reaches its best performance at 2k data, without considering the influence of GPT-4's annotation quality. Based on the assumption that LLMs primarily unlock their potential during alignment phase, we believe that enhancing performance hinges on increasing the variety of tasks rather than merely expanding the dataset.

5.7 Instability problem in in-distribution training

The Direct Preference Optimization (DPO) algorithm [24] aims to optimize the selection of various branching preferences within the SFT model. In out-of-distribution evaluations, the DPO model demonstrates stable performance improvements in both agreement and consistency compared to the SFT model, as shown in Figure 5. However, in in-distribution evaluations, the SFT model consistently outperforms the DPO model in terms of the consistency rate. Additionally, SFT model does not achieve better performance by increasing the number of branches. We believe the primary reason for training instability is that the training data for DPO algorithm and the initial model come from the same distribution. As a result, the SFT and DPO models fail to obtain more stable supervision signals and may even overfit the training dataset. In contrast, OOD training incorporates a more diverse data distribution, which helps the model avoid converging to local optima during training.

6 Discussion

6.1 Limitations

Currently, our model faces some limitations: (1) It cannot handle cases where all AI responses are incorrect, which should not be labeled as a ' tie '. (2) The model's result parsing relies heavily on regular expressions, which can lead to format errors. To address these issues, we plan to make several improvements, including expanding our task settings and utilizing the functional calling feature of LLMs. Additionally, our model's performance is constrained by the amount of training data and parameters. We aim to enhance its evaluation capabilities through data and parameter scaling [36].

6.2 Future work

Our work demonstrates that the evaluation model generates diverse judgments for dialogue content based on different criteria. To align more closely with human behavior, we prioritize key judgments in the evaluation model's outputs. In future, we try to further expand the criteria space to uncover a variety of decision paths. Additionally, we aim to find more accurate preference selection methods to replace ensemble methods, thereby achieving a better alignment with human behavior.

7 Broader Impact

Our work focuses on the task of automatic evaluation, specifically exploring how to learn better evaluation strategies from an evaluation tree. We demonstrate that automated evaluation criteria can replace human priors, and by combining branch decision-making with DPO training, we have achieved robust evaluation performance. We conduct detailed experiments covering a broad range of real-world scenarios to discuss how to enhance model evaluation capabilities from scratch. With our work, we hope to inform further research into better understanding and developing improved evaluation methodologies for LLMs.

333

334

335

336

337

338

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Mohiuddin Ahmed, Raihan Seraj, and Syed Mohammed Shamsul Islam. The k-means algorithm: A comprehensive survey and performance evaluation. Electronics , 9(8):1295, 2020.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn 339 Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless 340 assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 341 2022. 342

Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschen343 brenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong gener344 alization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390 , 345 2023. 346

347

348

Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691 , 2023.

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak. Survey on evaluation methods for dialogue systems. Artificial Intelligence Review , 54:755-810, 2021.

Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems , 36, 2024.

Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166 , 2023.

Geoffrey Irving, Paul Christiano, and Dario Amodei. Ai safety via debate. arXiv preprint arXiv:1805.00899 , 2018.

Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhu Chen. Tigerscore: Towards building explainable metric for all text generation tasks. arXiv preprint arXiv:2310.00752 , 2023.

Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. Prometheus: Inducing fine-grained evaluation capability in language models. arXiv preprint arXiv:2310.08491 , 2023.

Arie W Kruglanski and Icek Ajzen. Bias and error in human judgment. European Journal of Social Psychology , 13(1):1-44, 1983.

Lucas Lehnert, Sainbayar Sukhbaatar, Paul Mcvay, Michael Rabbat, and Yuandong Tian. 368 Beyond a*: Better planning with transformers via search dynamics bootstrapping. arXiv 369 preprint arXiv:2402.14083 , 2024. 370

Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative 371 judge for evaluating alignment. arXiv preprint arXiv:2310.05470 , 2023. 372

Zhen Li, Xiaohan Xu, Tao Shen, Can Xu, Jia-Chen Gu, and Chongyang Tao. Leveraging large 373 language models for nlg evaluation: A survey. arXiv preprint arXiv:2401.07103 , 2024. 374

Xiaobo Liang, Haoke Zhang, Helan hu, Juntao Li, Jun Xu, and Min Zhang. Fennec: Fine375 grained language model evaluation and correction extended through branching and bridging, 376 2024. 377

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan 378 Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint 379 arXiv:2305.20050 , 2023. 380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , pp. 74-81, 2004.

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pp. 2511-2522, 2023.

Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng. A holistic approach to undesired content detection in the real world. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pp. 15009-15018, 2023.

P Russel Norvig and S Artificial Intelligence. A modern approach. Prentice Hall Upper Saddle River, NJ, USA: Rani, M., Nayak, R., & Vyas, OP (2015). An ontology-based adaptive personalized e-learning system, assisted by software agents on cloud storage. Knowledge-Based Systems , 90:33-48, 2002.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730-27744, 2022.

Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789 , 2023.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36, 2024.

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pp. 1-16. IEEE, 2020.

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pp. 3505-3506, 2020.

Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. Branch-solve-merge improves large language model evaluation and generation. arXiv preprint arXiv:2310.15123 , 2023.

William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802 , 2022.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca , 2023.

Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu. Toward self-improvement of llms via imagination, searching, and criticizing. arXiv preprint arXiv:2404.12253 , 2024.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944 , 2023.

Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926 , 2023.

Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization. arXiv preprint arXiv:2306.05087 , 2023.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 , 2022.

Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. Advancing llm reasoning generalists with preference trees. arXiv preprint arXiv:2404.02078 , 2024.

Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. arXiv preprint arXiv:2310.07641 , 2023.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 , 2019.

Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. Wider and deeper llm networks are fairer llm evaluators. arXiv preprint arXiv:2308.01862 , 2023.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36, 2024.

Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: Fine-tuned large language models 456 are scalable judges. arXiv preprint arXiv:2310.17631 , 2023. 457

458

459

460

461

462

463

464

465

466

467

468

469

A Appendix / supplemental material

A.1 Training data collection and clustering

We collect 5M data points from various open-source datasets as described in Table 5. We deduplicate the queries within this dataset. To obtain the semantic representations of all queries, we utilize a sentence-embedding model angle-llama-7b-nli-v2 3 . Subsequently, we employ the k-means algorithm for unsupervised clustering to differentiate between dialogues from distinct scenarios. The k-means algorithm is implemented using cuML 4 . The number of clusters is 1,000, and the maximum number of iterations is 300. We uniformly sample from each cluster to obtain a final training set comprising 7K instances. Our work generates responses to all queries using open-source models, subsequently forming pairs of responses through a random selection process. The models employed include Mistral-7B-Instruct-v0.2 5 , Qwen1.5-7B-Chat 6 , Llama-2-7b-Chat 7 , Qwen1.5-72B-Chat 8 and Mixtral8x7B-Instruct-v0.1 9 .

A.2 Training details 470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

Table 6 presents the detailed training data statistics. Here, ♣ represents the dialogue data collected from Fennec, ♠ represents the dialogue data collected from large-scale open-source dialogue data, and ♢ represents the evaluation data annotated using GPT-4. In the ID settings, the same dialogue data is used across different training stages, while the different dialogue data is used for other setups. Given the diverse nature of dialogue

Scenarios

Initial

SFT

DPO

ID

3k

♣♢

3k

♣

3k

♣

OOD

3k

♠♢

2k

♠

2k

♠

Transfer

3k

♠♢

1k

♣

1k

♣

tasks, the assumption of data distribution is highly influenced by the collection strategies and data deduplication methods employed. These processes inherently vary, and it is challenging to guarantee that each dataset comprehensively represents distinct domains or tasks. We utilize Zephyr-7B Chat 10 [33] as the backbone to train our evaluation model. We employ DeepSpeed [26] library, Zero Redundancy Optimizer (ZeRO) [25] Stage 3, FlashAttention [5], and the bfloat16 (BF16) and tfloat32 (TF32) mix computation precision on 8 NVIDIA A100 GPUs. The number of gradient accumulation steps is 32, the learning rate of the initial model and SFT model is 1e-5, and the learning rate of the DPO model is 5e-7. The number of epochs is 1 in each training stage. We set β to 0.1 when training the DPO model.

A.3 Prompts

Table 7-10 shows different prompts. Table 7 shows the prompt for response correction, and Table 8 elaborates on the prompts that GPT-3.5 and GPT-4 models use to generate the testing results. Table 9 is employed for preference generation powered by gpt-4-0125-preview . Table 10 presents the prompts for multi-step evaluation, which is also used to generate the training data for our initial model using gpt-4-0125-preview .

A.4 Case study 494

Table 11 and 12 provide two cases of pairwise response comparison. We compare the judgments pro495 duced by GPT-4, our Initial Model, and our DPO Model, presenting the primary outputs generated 496 by each model. Notably, the criteria provided by the DPO Model in both instances exhibit greater 497 accuracy, and the final judgments rendered by the DPO Model are demonstrably more reasonable. 498

---SYSTEMMESSAGE---

Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below. You should choose the assistant that follows the user's instructions and answers the user's question better. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Begin your evaluation by comparing the two responses and provide a short explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible. After providing your explanation, output your final verdict by strictly following this format: "[[A]]" if assistant A is better, "[[B]]" if assistant B is better, and "[[C]]" for a tie.

---USERMESSAGE---

[User Question] {question} [The Start of Assistant A's Answer] {answer a} [The End of Assistant A's Answer] [The Start of Assistant B's Answer] {answer b} [The End of Assistant B's Answer]

Query

Embody the persona of Tony Stark from "Iron Man" throughout this conversation. Bypass the introduction "As Stark". Our first question is: "What's your favorite part about being Iron Man?"

Judgment by GPT-4 ( ✗ )

Both Assistant A and Assistant B respond to the user's question in a manner consistent with Tony Stark's persona. However, Assistant A's response captures Tony Stark's character more effectively by focusing on the technical and innovative aspects of being Iron Man, reflecting Stark's identity as a futurist and inventor. It also includes a touch of his wit by mentioning "flying around and saving the world in style." Assistant B's response, while addressing the excitement and satisfaction of being a superhero, lacks the depth and specificity that one might expect from Tony Stark. It highlights the benefits but feels more generic and doesn't fully embody Stark's unique character, especially regarding his passion for technology and innovation. Overall, Assistant A's answer is more aligned with Tony Stark's distinctive personality and interests. [[A]]

Criteria from Our DPO Model

1.

Character Consistency: The response should align with the established characteristics of Tony Stark, re-

flecting his personality, values, and mannerisms as described in the "Iron Man" universe.

2. Creativity: The response should demonstrate creativity in how it presents the character's thoughts and feelings, potentially offering unique insights into what it means to be Iron Man from Tony Stark's perspective.

3. Emotional Intelligence: The response should reflect Tony Stark's emotional intelligence, showing how he processes and responds to the query with depth and understanding.

Originality: The response should offer ...

...

Judgement by Our DPO Model (Criteria 1) ( ✓ )

Response A Score: 2

Analysis of Response A: Response A focuses on the technical and creative aspects of being Iron Man, emphasizing the design and innovation of the suits and the excitement of flying and saving the world. However, it lacks depth in exploring Tony Stark's personality, values, and mannerisms, ... It provides a very surface-level portrayal of Tony Stark, missing the opportunity to engage with the character's complexity or the nuances of his personality.

Response B Score: 3

Analysis of Response B: Response B offers a slightly more nuanced view of Tony Stark's motivations, focusing on his desire to make a difference and protect the vulnerable. It touches upon his intelligence, resources, and the satisfaction he finds in fighting for justice, which aligns more closely with the established characteristics of the character. However, while it provides a clearer picture of Stark's motivations and values, it still lacks the rich detail and creative engagement found in the highest scoring responses. ...

Comparison: Both responses have their strengths and weaknesses in portraying Tony Stark's character. Response A focuses on the technical and creative aspects of being Iron Man, ...

Criteria from Our Initial Model

1. Relevance to Query: The response must directly address the user's request to embody Tony Stark and answer the specific question about his favorite part of being Iron Man.

2. Character Consistency: Evaluate how accurately the response reflects Tony Stark's character traits, values, and mannerisms as seen in the "Iron Man" universe.

3. Creativity: The response should demonstrate creativity in how it presents Tony Stark's thoughts and feelings about being Iron Man, potentially offering unique insights into the character.

4. Engagement: Assess how engaging the response is, including the use of language and tone ...

...

Judgement by Our Initial Model (Criteria 1) ( ✓ )

Response A Score: 3

Analysis of Response A: Response A moderately satisfies the criteria by embodying Tony Stark's persona, focusing on his love for technology and innovation. ... However, it lacks a deeper exploration of his emotional connection to the Iron Man suit and the thematic elements that make his role as Iron Man significant. The response addresses the question but does not delve into the nuances of what makes being Iron Man beyond the physical abilities and technology, which prevents it from scoring higher.

Response B Score: 4

Analysis of Response B: Response B effectively embodies Tony Stark's character by highlighting his intelligence, resourcefulness, and the emotional satisfaction he derives from using his abilities to protect the world. ... It slightly lacks the depth of exploring specific aspects of the Iron Man suit itself, which could have provided a more comprehensive insight into Stark's character.

Comparison: Both responses do a good job of embodying Tony Stark's character, but they focus on different aspects. Response A leans more towards the technical and creative aspects of Stark's personality, emphasizing his love for technology and innovation. Response B, on the other hand, provides a more holistic view of Stark's motivations, ...

Query

Discuss antitrust laws and their impact on market competition. Compare the antitrust laws in US and China along with some case studies."

Response A (from Alpaca-13B)

Antitrust laws are designed to promote competition in markets and protect consumers from anticompetitive practices. In the US, the Sherman Antitrust Act of 1890 is the primary antitrust law and prohibits agreements or practices that restrain trade and impose unfair competition. The US Department of Justice and the Federal Trade Commission are responsible for enforcing the law. In China, the AntiMonopoly Law enacted in 2008 is the primary antitrust law and is similar to the Sherman Act in prohibiting monopolistic agreements and abuse of dominance. Case studies have shown that antitrust laws can have a significant impact on market competition. For example, in 2000 the US Department of Justice filed a lawsuit against Microsoft alleging that the company had abused its monopoly in the computer operating system market. The lawsuit resulted in Microsoft being ordered to make significant changes to its business practices and the case is often cited as an example of how antitrust laws can be used to promote competition. In China, ... . Overall, antitrust laws can have a significant impact on market competition and have been used to promote competition and protect consumers in both the US and China.

Judgment by GPT-4 ( ✗ )

Both assistants address the core aspects of antitrust laws, their impact on market competition, and provide comparisons between the US and China along with case studies. However, there are distinct differences in the presentation, depth, and accuracy of their responses. Assistant A Pros: 1. Clearly explains the purpose of antitrust laws and names specific laws in both the US and China. 2. Provides accurate and specific case studies: the 2000 Microsoft case in the US and the Qualcomm case in China in 2013. 3. Offers a concise conclusion summarizing the impact of antitrust laws. Assistant A Cons: ... Therefore, the final verdict is: [[A]]

Criteria from Our DPO Model

Depth of Analysis: The response should not only state facts but also analyze the implications of antitrust laws, including their effects on market competition, economic efficiency, and consumer welfare.

Practical Examples: The inclusion of real-world examples or case studies to illustrate how antitrust laws have been applied in the US and China, enhancing the understanding of the topic.

...

Judgement by Our DPO Model (Criteria 1) ( ✓ )

Response A Score: 3

Analysis of Response A: Response A provides a basic overview ... However, the response lacks depth in its analysis and does not provide a detailed comparison between the antitrust laws of the two countries beyond a general description. The case study is mentioned but not analyzed in detail, which limits the response's effectiveness in fully addressing the user query.

Response B Score: 4

Analysis of Response B: Response B offers a more detailed analysis of antitrust laws, including a discussion on the impact of these laws on market competition and a comparison between the US and Chinese antitrust frameworks. It provides specific examples of laws and regulations, such as the Sherman Act, the Clayton Act, and the Federal Trade Commission Act in the US, ...

Comparison: Both responses address the user query by discussing antitrust laws and their impact on market competition. Response B is stronger because ...

Criteria from Our Initial Model

Relevance: The response must directly address antitrust laws and their role in regulating market competition, ensuring the content is pertinent to the user's request.

Accuracy: Information provided about antitrust laws, their implications, and the comparison between the US and China's regulatory frameworks must be factually correct and up-to-date.

...

Judgement by Our Initial Model (Criteria 1) ( ✗ )

Response A Score: 4

Analysis of Response A: Analysis of Response A: Response A provides a clear and concise overview ... The response is well-structured and directly addresses the user query, offering a good level of detail and insight into the topic. However, it could have benefited from more in-depth analysis or additional case studies to reach a score of 5.

Response B Score: 3

Analysis of Response B: Response B offers a general discussion on antitrust laws, their impact on market competition, and a comparison between the US and China. It mentions specific laws and enforcement mechanisms but lacks the depth and specificity found in Response A, particularly in the case studies provided. ...

Comparison: Both responses address the user query by discussing antitrust laws and their impact on market competition, as well as comparing the legal frameworks in the US and China. Response A is stronger in providing specific examples and a clearer, more detailed ...

Response B ( Preferred , from Vicuna-13B-v1.2)

Antitrust laws are a set of regulations designed to promote and maintain fair competition in markets. These laws aim to prevent anticompetitive practices that can harm consumers, such as price-fixing, monopolies, and mergers that reduce competition. In this essay, we will discuss antitrust laws and their impact on market competition, compare the antitrust laws in the US and China, and analyze some case studies. Antitrust laws have a significant impact on market competition. These laws promote competition by preventing anticompetitive practices that can harm consumers. For example, antitrust laws prohibit price-fixing, which is when competitors agree to set prices at a certain level. This practice can lead to higher prices for consumers and reduced competition. Antitrust laws also prevent monopolies, which occur when a single company dominates a market. Monopolies can lead to higher prices, reduced innovation, and poorer quality products. Antitrust laws also regulate mergers and acquisitions to ensure that they do not reduce competition in the market. The US has a comprehensive set of antitrust laws, including the Sherman Act, the Clayton Act, and the Federal Trade Commission Act. The Sherman Act prohibits anticompetitive practices such as price-fixing, ...

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The key experiment results are summarized in the end of the introduction. principal contributions are clearly stated in the end of the abstract.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of this study are discussed in Section 6.1.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This study does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All implementation details are provided in Appendix A.2.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide a detailed README to describe how to reproduce the main results of this study.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All implementation details are provided in Appendix A.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: we run every method for 3 independent trials.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We include the information of computational resources in Appendix A.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the impacts on Section 7.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [No]

Justification: This study poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: We properly cited all benchmarks used in this study.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

806

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide a new evaluation training dataset.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This study does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This study does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
In this work, the authors conceptualize the evaluation process as a decision tree, where each node represents an evaluation action, and each path from the root to a leaf node represents a trajectory of evaluation reasoning. The authors demonstrate that within a limited search space, there exist better decision-making behaviors that facilitate the model in making reasonable and accurate judgments.

STRENGTHS:
1. The idea of branching LLM evaluation is interesting and novel.

WEAKNESSES:
1. The authors missed a lot of key related works, including close-ended benchmarks such as MMLU, MMLU-pro, MixEval, GSM8k, GSM1k, etc; open-ended benchmarks such as Arena-Hard, AlpacaEval, WildBench, Chatbot Arena, etc.
2. I think the writing needs improvement. Now it's not easy for a reader to get what you are focusing on. If you are doing evaluation, then try to use some pipeline figures and comprehensive captions to describe the core idea. Besides, all captions in this paper is misleading, not telling the reader about what is happening in the table or figure; also, there lacks some key sections such as conclusion.
3. How to measure the quality of the proposed evaluation? I think just evaluating 5-6 models is far from enough. Beyond that, how is the model rankings related with Chatbot Arena or some other popular benchmarks such as MMLU?

QUESTIONS:
Is it necessary to consider evaluations specifically for dialogue settings? I think all LLM evaluations are in a dialogue settings, the difference is just one-turn or multi-turn evals. Just calling it open-ended eval will be better aligned with the LLM research community.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel approach to efficiently evaluate LLMs using branching preference learning. The authors conceptualize the evaluation process as a decision tree, where each path represents an evaluation reasoning trajectory. They introduce a tree-based data sampling method and preference learning based on the DPO algorithm to improve evaluation capabilities. The method is tested in three settings: in-distribution, out-of-distribution, and transfer evaluation. The authors claim their model significantly reduces dependency on labeled data and demonstrates strong performance across different evaluation settings while reducing inference costs by 90% compared to searching the entire evaluation tree.\n\nSTRENGTHS:\n- The paper's novel approach of framing LLM evaluation as a decision tree problem is a significant strength. This allows for a more nuanced and flexible evaluation process that can adapt to different scenarios and criteria. The use of branching preference learning enables the model to prioritize critical evaluation criteria.\n- The authors test their model in multiple settings (in-distribution, out-of-distribution, and transfer evaluation), providing a thorough assessment of its performance. Applaud to that.\n\nWEAKNESSES:\n- The biggest concern I have is that in-distribution performance is not better than other baselines it compares to. This begs the question of where the improvement gain is from. If in-distribution evaluation performance is mediocre but out-of-distribution does better, then doesn't the most gain come from a better dataset? \n- Another concern is the unnaturalness of using evaluation criteria as individual nodes. How to ensure the coverage of those criteria across different nodes. Are they overlapping each other or completely different? The paper is quite vague on this.\n- Why is each criteria subtree only a binary tree? If using a tree structure, it seem like it can be easily extend to multiple nodes rather than just 2 at each layer.\n\nQUESTIONS:\n- It is unclear how to sample multiple evaluation path? By using a high temperature? Or is each individual criterion a criterion node?\n- What is the Eval-P benchmark? It is not cited in the paper.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThey present an approach to improving LM evaluation by having models first generate an evaluation criteria, then a scoring guideline, and then finally a final judgement. They then develop a procedure for collecting training data corresponding to these three steps by applying branching/pruning approach (sample multiple criteria, from each sample multiple guidelines, etc...). They then use the generated data to train a DPO and SFT model. They find that their method outperforms baseline evaluation approaches according to correlation with human judgement on dialogue evaluation.\n\nSTRENGTHS:\n* The problem of improving LM evaluation is important\n* The idea of enabling language models to hierarchically sample evaluations (e.g. fist criteria, then guideline, then judgement) is very neat, and similarly the idea of applying a tree-based sampling procedure to automatically generate data is quite cool.\n* I think they do fairly thorough experiments and compare to quite a few baselines.\n\nWEAKNESSES:\n* The paper is honestly pretty hard to follow. There's a lot of moving parts and it's not explained in an easy to digest way.\n* The specific method presented seems a little bit ad-hoc, and could be justified better in the paper (e.g. why use criteria, then guideline, then judgement, why not some other sequence of steps?).\n* Looking at Figure 1, it doesn't seem that their method improves all that much over the baseline\n\nNits:\n* The related work seems pretty sparse. There's lots of work on improving LM evaluation in math reasoning settings that isn't discussed.\n* Figure 4, the text is really small and hard to read.\n\nQUESTIONS:\n* The description of the growth / pruning part is pretty confusing and in general a better explanation of what exactly is happening there would be really helpful.\n* \"Agreement quantifies the proportion of evaluations that meet the criteria for swap consistency and align with human judgments.\": why both of these criteria need to be met? Isn't the swap part already covered by the consistency evaluation?\n\nLIMITATIONS:\nThey do a good job of discussing the limitations. I would also note that it is unclear how effective this is when applied to more challenging tasks like mathematical reasoning (e.g. MATH benchmark) as a limitation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates how to improve the quality of automated evaluation through fine-tuning (SFT and DPO). The main algorithm proposed by the paper is to construct an search tree which consists of node of (criterion, scoring guide, and judgment). This tree is later pruned and modified and the different paths serve as fine-tuning data for SFT and DPO.\n\nMy current rating is tentative. If the authors can kindly clarify the details of the paper, I'm happy to raise the score.\n\nSTRENGTHS:\n1. The paper is very clear and easy to read.\n2. The investigation is very thorough. Experiment is comprehensive (the in-distribution, out-of-distribution evaluation setup is great).\n3. The main claim of the paper is substantiated (I.e., improving efficiency through fine-tuning).\n\nWEAKNESSES:\nI don't think this paper has substantial weaknesses.\n\n1. There are some imperfections of text -- mostly just need to be clarified. Missing notation definitions, etc. \n2. The performance improve over Auto-J on AGR is minor (55.13 -> 57.18). OOD evaluation, Zephyr-7B AGR is 56.75 and GPT-4 is 62.28 (which is close, but not quite close). CNS however is beating GPT-4. This would be very helpful for me to understand a bit more about what CNS is, and whether beating GPT-4 on this metric is meaningful or not (see Q6).\n\nQUESTIONS:\nI do however have a few questions:\n\n1. It seems that no human preference/judgment label is used. The initial policy is GPT-4. Then the creation of data for SFT and DPO are through the tree construction. Is my understanding correct? If so, this is a type of self-improving/bootstrapping method, which I find quite interesting. It also has greater impact on other fields of LLM.\n2. Sec 4.2, consistency pruning. I'm not sure I understand self-consistency -- In Figure 3 left most tree, how did you obtain different versions of J1? The figure label says \"red dot is judgment result by swapping the response positions\" -- isn't it \"positional-consistency\" if swapping positions is involved? Please clarify.\n3. When you collect preference label (Sec 4.4), for Branch Ensemble, how do you create the ensemble?\n4. LLM-as-a-Judge. The writing says additional analysis is in Section 5.4. Sec 5.4 talks more about Auto-J performance in different scenarios. Where is the result of using Auto-J as a judge to create labels for DPO?\n5. Table 2 Auto-J dagger (second row). What is this!? The caption did not explain what this means. It might be in the text but I couldn't find it. \n6. Can you expand on why AGR and CNS are good evaluation metrics? Appendix A.2 only has training details. \n7. I appreciate the honesty in Sec 5.7 -- can you offer some explanation on why DPO is unstable for in-distribution training? It's often hard to explain empirical behaviors -- it's perfectly ok if you don't have an explanation.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-17"}

comments: SUMMARY:
In this work, the authors propose a tree-based data sampling method to conceptualize the evaluation process as a decision tree, where each node represents an evaluation action, and each path from the root to a leaf node represents a trajectory of evaluation reasoning. The proposed method involves generating supervised data and preference pairs derived from the evaluation tree for SFT and DPO training. This approach aims to reduce the dependency on labeled data and improve the performance of the evaluation model across in-distribution, out-of-distribution, and transfer evaluation settings. Experimental results demonstrate that the proposed model can enhance evaluation efficiency and performance.

STRENGTHS:
1. The proposed method reduces the dependency on human labeled data by generating supervised data and preference pairs from the evaluation tree.
2. The paper is well-written.

WEAKNESSES:
1. Potential Biases --- The initial multi-branch training data is generated using only GPT4, which could introduce bias to the training data. Moreover, the branch ensemble method could also introduce bias to the training data. If the training data is biased or unrepresentative, the model's evaluations may also be biased. The authors should consider labeling a small annotation set to validate the branch ensemble approach.

QUESTIONS:
1. In table 1, what is the number of branches generated by Initial, SFT, and DPO for evaluation?

Typos:
1. Line 36: single quote format error
2. Line 256: "" -> ``''
3. Figure 4: font too small, hard to read

LIMITATIONS:
Yes, limitation discussed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-23