Direct Judgement Preference Optimization

Authors: PeiFeng Wang, Austin Xu, Yilun Zhou, Caiming Xiong, Shafiq Joty
Primary area: generative models
Keywords: LLM-as-judge, generative judge, auto-evaluation

Abstract:
Auto-evaluation is crucial for assessing response quality and offering feedback for model development. Recent studies have explored training large language models (LLMs) as generative judges to both evaluate model responses and generate natural language critiques. However, existing models have been trained almost exclusively with supervised fine-tuning (SFT), often only on a small number of datasets, resulting in poor generalization across different evaluation settings and tasks. In this paper, we investigate how learning from both positive and negative data with direct preference optimization (DPO) enhances the evaluation capabilities of LLM judges across three evaluation tasks: pairwise, single ratings, and binary classification. We achieve this by creating three forms of DPO data from a diverse collection of human and synthetic judgements on contemporary model outputs, with the goal of training our model to generate meaningful critiques, make accurate judgements, and understand what constitutes good and bad responses for a given user input. To demonstrate the effectiveness of our method, we train judge models of three sizes: 8B parameters, 12B, and 70B, and conduct a comprehensive study over 13 benchmarks (7 pairwise, 4 single rating, and 2 classification), measuring agreement with human and GPT-4 annotations. Our models exhibit the best aggregate performance, with even our 8B model outperforming strong baselines like GPT-4o and specialized judge models, such as OffsetBias-8B, Auto-J-13B, Prometheus-2-8x7B, and Skywork-Critic-70B, in pairwise benchmarks. Further analysis shows that our judge model robustly counters biases such as position and length bias, flexibly adapts to practitioner-specified evaluation protocols, and provides helpful language feedback for improving downstream generator models.

Decision: reject
Reviewer scores: 3, 3, 8, 6

Reviews and discussion:
<thread>
creation date:
1729135910950

thread_type:
official_review

<thread_object>
creator: reviewer_ay6r

<content>
summary:
The paper explores the use of preference learning algorithms (DPO) to enhance the evaluation/critique capabilities of LLMs. By relying on human-annotated judgment scores to automatically assess the quality of model-generated critiques, they are classified into chosen or rejected categories, facilitating the preference learning. Additionally, the authors designed three training methods: Chain of Thought (CoT), standard judgment, and response deduction to optimize the model, achieving impressive experimental results on 13 benchmarks, covering pairwise, point-wise, and binary classification evaluation protocols.

strengths:
* This paper thoroughly analyzes the contribution of three different training examples to enhancing model evaluation and reflection capabilities: CoT, standard judgment, and response deduction. 
* The authors conducted extensive testing on 13 benchmarks covering three types of evaluation protocols: pairwise, pointwise, and classification, with experimental results demonstrating superiority over existing baselines. 
* The proposed method maintains good reliability in terms of bias in LLM-as-a-judge.

weaknesses:
### 1. Some Prior Works Need to Be Discussed

Using preference learning for optimization sounds impressive, but some prior work have already utilized preference learning to enhance the evaluation/critique capabilities of LLMs, such as CriticGPT  [1] and Themis [2]. 

1. CriticGPT employs a standard RLHF workflow to improve the evaluation/critique capabilities of LLM in the code generation task, relying heavily on cost human annotations. 

2. Similarly, Themis also utilizes DPO to enhance the evaluation capabilities of language models. Themis also automatically construct the preference data by checking if the model's judgment scores are consistent to human-annotated judgment scores. In terms of motivation and implementation details, there is not much difference between this paper and Themis. Of course, the authors introduce three different preference examples to enhance the capabilities of LLMs. 

However, from my perspective, these two works need to be carefully compared and analyzed in this paper. Although the authors can compare the Themis with proposed method (CriticGPT is a closed-sourced model), I do not recommend doing so because I believe there may be numerous variables that could affect the experimental results, such as the size of the dataset and the optimization techniques (Themis re-designs the DPO loss).

Beyond above comments, this paper, Themis, and CriticGPT heavily rely on human-annotated judgment labels to automatically construct preference datasets. Therefore, the scalability of these methods is limited. I hope to receive some detailed discussion and analysis from the authors on this aspect.



### 2. Flexibility of Evaluation Critiera

Similar to models in the Prometheus family, the current model's input includes an additional evaluation protocol except for the user query and the evaluated response. Therefore, during real-world evaluation setting, additional craft of evaluation criteria is required, as demonstrated in lines 328-329. Although Section 5.4 demonstrates that the impact of the crafted evaluation protocol on evaluation is limited, both this paper and Prometheus-like models require the input of an evaluation protocol (no matter the human-annotated or automatically generated by GPT-4). Therefore, when performing evaluation in real-world scenarios, this severely limits the scalability of the method because it may require additional supervision of humans and advanced models (From my perspective).

### 3. Coarse-grained Evaluation Aspects

As demonstrated in Section 4.2 (line 231 tp line 234), the evaluation aspects for building training dataare general quality, like facuality, helpfulness, safety, etc, which is limited and coarse-grained, compared to previous works like Auto-J and Prometheus.
For example, Auto-J introduces 58 tasks and corresponding fine-grained evaluation criteria. Prometheus generate the score-rubrics-like customized evaluation criteria for specific user queries.

### 4. Without the evaluation of  textual critques quality (Chain-of-Thought)
This paper only evaluate the quality of judgment scores, while neglect the evaluation of textual critique quality (the quality of chain-of-thought). Previous works have demonstrated that even if the judgment score is consistent with human-annotated, their chain-of-thought may contains several flaws [3]. This will introduce numerous bias and drawbacks for application of the fine-tuned critique model in some important reseach fields, like the self-improvement of LLMs. 

---


[1] LLM Critics Help Catch LLM Bugs: https://arxiv.org/abs/2407.00215

[2] Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability: https://arxiv.org/abs/2406.18365

[3] The Critique of Critique: https://arxiv.org/abs/2401.04518

questions:
1. This paper, Themis, and CriticGPT heavily rely on human-annotated judgment labels to automatically construct preference datasets. Therefore, the scalability of these methods is limited. 

2.  When performing evaluation in real-world scenarios, requiring the input of evaluation criteria limits the scalability of the method because it may require additional supervision of humans and advanced models.

I hope to receive some detailed discussion and analysis from the authors on these aspects.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730654099456

thread_type:
official_review

<thread_object>
creator: reviewer_azff

<content>
summary:
The paper explores enhancing large language models (LLMs) as generative judges for evaluating model responses and generating critiques. It introduces direct preference optimization (DPO) using diverse human and synthetic judgments to improve evaluation across pairwise, single ratings, and binary classification tasks. The trained models, especially the 8B parameter model, outperform strong baselines and specialized judge models, showing robust performance and adaptability across various benchmarks.

strengths:
1. The problem of automated evaluation is important and well studied in recent years. Thus, the work is well motivated.
2. Use of DPO using positive and negative pairs to improve LLM eval is novel. Response deduction is also a nice idea.
3. The framework is comprehensive -- works on 3 eval setttings -- Single Rating, Pairwise Comparison, and Classification.
4. Comprehensive eval on seven pairwise comparison benchmarks, four single rating evaluation benchmarks, and two classification benchmarks. The results are convincingly better compared to our SOTA models and even gpt4o.
5. Bias evaluation results, ablations, flexible prompting strategies, downstream model improvements -- are all nice additions in the paper.

weaknesses:
1. It will be interesting to see how easy vs hard negatives can help. Given the current way of generating negatives, it is unclear if they were hard enough and if a mix of easier and hard negatives could help improve further.
2. Choice of 70%:15%:15% for DCoT, DStd and DDed sounds somewhat arbitrary. Can you provide further reasoning behind this?
3. Broadly a comparison to evaluation systems with multiple models is not expected as such, but still it will be nice to compare and check how does your method works compared to methods like reconcile.

Typos:
1. citep vs citet should be used properly. E.g. line 41 should have "GPT-4 (OpenAI, 2023)" and not "GPT-4 OpenAI (2023)".
2. Notations M_t vs M_{teacher} or M'_{teacher} do not match between Section 3 and Fig. 2. Same for student model.

questions:
1. Is it possible to quantify hardness of negatives and does that correlate with evaluation accuracy?
2. Choice of 70%:15%:15% for DCoT, DStd and DDed sounds somewhat arbitrary. Can you provide further reasoning behind this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730780792341

thread_type:
official_review

<thread_object>
creator: reviewer_xfuc

<content>
summary:
This paper curates the preference data to train an LLM as an evaluation judge using DPO, and demonstrates the effectiveness of the trained models in different sizes (e.g., 8B, 12B, 70B) on three types of evaluation tasks (e.g., pairwise comparison, single rating and classification). The contribution lies in the method of creating preference data for three evaluation tasks using a collection of labeled data from diverse data sources (annotated by humans or other models).

strengths:
**Originality v.s. Significance**: This paper proposes to process existing annotated data into the preference data format, and train an LLM as a judge using DPO on the curated data. It’s a straightforward application of DPO to the task of training an LLM for evaluation. The novelty lies in the way of constructing preference data from existing labeled datasets for three evaluation tasks — but see the novelty concerns in the weakness section below. The technical novelty is weak, compared to the significance of releasing a good evaluation model to the community. However, I don’t find the statement of releasing the model checkpoints and the curated data from the paper. It’d be better to clarify this explicitly. 

**Quality**: Extensive experiments and analyses have been conducted on seven pairwise comparison benchmarks, four single rating benchmarks, and two classification benchmarks.

weaknesses:
**Novelty of data curation**: As the main contribution of this paper is data curation, it’s better to clarify the difference between this work and existing studies on data curation for LLM evaluation. The key differences are still unclear even after reading the description in Lines 136-150. From Table 1, existing studies have explored the three evaluation tasks (Vu et al., 2024), and have used both SFT and DPO (Want et al, 2024c). This paper seems incremental to combine what has been done in existing studies at a larger scale of training data. 

**Presentation**: It’s hard to interpret the results and the evaluation with other baseline models (see details in the question section below). The tables and figures are far away from their corresponding text descriptions, making it hard to read them at the same time.

questions:
1. In Line 315, what do you mean by evaluating models trained for single ratings on classification? How do you use a model trained for single ratings to evaluate classification tasks?
2. In Line 314, “All baselines are evaluated only on the tasks they are trained for”.  Can you report which tasks these baselines are trained for? How can you get the results for the tasks that these baselines are not trained for in Table 1 & 2? Do you get the baseline models’ results from their papers or run the baseline models by yourself? Do you test your models on those benchmarks without further fine-tuning your models on the benchmark data at all?
3. In Table 5, what are the 6 types of biases? And How do you measure these biases? I understand the position bias can be measured by the consistency of choosing responses even if the order is swapped. But what about the other biases?
4. Line 464 claims that “we plot the average performance across all three evaluation tasks when removing each training task”, but Fig 4 may miss the SFT+DPO w/o CoT critique.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731141661999

thread_type:
official_review

<thread_object>
creator: reviewer_mylg

<content>
summary:
The paper "Direct Judgement Preference Optimization" improves large language models (LLMs) used as generative judges by introducing Direct Preference Optimization (DPO), which enhances evaluation by learning from both positive and negative examples. The authors train models on pairwise comparison, single ratings, and classification tasks using a mix of human and synthetic data, including Chain-of-Thought (CoT) critiques and response deduction. Their models outperform baselines like GPT-4o across 13 benchmarks, mitigate biases, and adapt to various evaluation protocols, offering more robust and flexible performance while also providing valuable feedback for improving other models.

strengths:
**Diverse Evaluation Tasks**: The paper proposes a comprehensive evaluation framework, including pairwise comparisons, single ratings, and binary classification. This makes the model adaptable to different evaluation needs​.

**Model Performance**: The paper demonstrates that their judge models outperform existing baselines, such as GPT-4o and specialized judge models, across multiple benchmarks, including pairwise comparison, single rating, and classification tasks​.

**Bias Mitigation**: The model robustly counters common biases, such as position and length bias, offering improved generalization across evaluation protocols.

weaknesses:
**Content organization **: The organization of this paper could be further improved, for example, structurize some long paragraphs.


**Design philosophy**: The paper could be strengthened by providing a more detailed explanation of the design philosophy, including the rationale for including each component, their individual functions and relationships, and how they relate to the background presented in Section 2. An ablation study or discussion of the impact of removing individual components would also be valuable.

questions:
See the Weaknesses.


The authors could enhance the paper by clarifying the relationship between Direct Judgement Preference Optimization and Direct Preference Optimization, highlighting key similarities and differences. Additionally, they could more explicitly state the main takeaways of the paper and explain how their use of three types of data for DPO differs from and improves upon the original DPO approach.

What are the takeaway messages of this paper? It seems like the authors use three types of data for DPO rather one type in the original DPO?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731962505469

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thank you to all reviewers!

comment:
We thank the reviewers for their helpful comments and suggestions. Overall, we are delighted that reviewers found our work “well-motivated” and “novel” (Reviewer azFf). Reviewers found “The novelty [...] in the way of constructing preference data” (Reviewer xfuc) and our auxiliary training task of response deduction as a “nice idea” (Reviewer azFf). We are also delighted that all reviewers found our evaluations to be “comprehensive” (Reviewer MyLG) and “extensive” (Reviewer xfuc), with our results being “convincingly better compared to our SOTA models” (Reviewer azFf) and “demonstrating superiority over existing baselines” (Reviewer Ay6r). Reviewers also found our judge showed “good reliability” in the face of bias (Reviewer Ay6r) and found the additional analysis with prompt flexibility, downstream model improvement, and training task ablations as “all nice additions in the paper” (Reviewer axFf).

We have updated our manuscript with additional experimental results showing the effects of hard negative preference training pairs (Appendix C.6) and evaluating the quality of critiques generated from our models (Appendix C.7). We have also re-written portions of the main text for clarity in response to reviewer questions. All major additions or revisions are marked in blue text.

We respond to each reviewer’s comments separately, and hope to have a productive discussion period!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposed to apply DPO to train a model for evaluation. This works collected and constructed and mixes preference data from multiple sources and trained evaluation models. Experiments demonstrated the effectiveness of the proposed method.

Strengths:
1. Strong empirical results across various tasks.

Weaknesses:
1. As one reviewer pointed out, this work mainly applies DPO for training an evaluation model, and the contribution mainly lies in constructing preference data and empirically training the judge models, and similar evaluation data has been collected by previous works such as Vu et al., 2024, Want et al, 2024c, and criticGPT.

Overall, despite the strong empirical results, this work mainly applies DPO for training judge models, and the main contribution is mainly in constructing preference data. Considering the review scores, I'm recommending rejection, but I wouldn't mind if the paper gets accepted.