Learning Generative Judge from Preference Data

## Abstract

Abstract Learning from preference feedback is a common practice for aligning large language models (LLMs) with human value.
Conventionally, preference data is learned and encoded into a scalar reward model that connects a value head with an LLM to produce a scalar score as preference or reward.
However, scalar models lack interpretability and are known to be susceptible to biases in datasets.
This paper investigates leveraging the generation capability of LLMs to address both limitations in one shot.
Specifically, we prompt the pre-trained LLM to generate positive and negative judgments, both supported with rationales in natural language form.
The self-generated contrastive judgment pairs are used to train the generative judge with Direct Preference Optimization (DPO).
This proposal of training the generative J udge using self-generated Con trastive judgments (Con-J) ensures natural interpretability due to the generated rationales together with the judgments, as well as high robustness against bias without the need for an additional reward head.
Experimental results show that the performance of Con-J is comparable to the scalar reward model trained on the same collection of preference data, and demonstrate its superior interpretability and robustness in encoding human preferences.

## 1 Introduction

As Artificial Intelligence (AI) systems advance with the emergence of Large Language Models, it is crucial to ensure they align with human instructions, values, and ethics.
LLMs alignment is generally achieved by learning from preference data that compares pairs of responses to a question (Rafailov et al., ; Christiano et al., ; Liu et al., ).
However, collecting high-quality human preference data is both time-consuming and costly.
In practice, the construction of preference datasets often involves scaling with a combination of human and AI-generated feedback Lee et al. (); Hou et al. ().
An additional advantage of AI feedback is its ability to be incorporated in real-time, which facilitates online algorithms such as iterative-DPO Xiong et al. (); Xu et al. () and online-DPO Guo et al. ().
Therefore, it is crucial to develop an efficient and accurate AI-based preference model that aligns with human values.

Figure: Figure 1: Top: Examples of a preamble, a question, a pair of answers, and the corresponding judgment (see the detailed version in Table 5). Bottom: Illustrations of a scalar reward model and the proposed Con-J for preference judgment.
Refer to caption: x1.png

To obtain such preferences, industrial practices have used scalar models Hou et al. () that concatenate the pre-trained LLM with a value head to generate scalar scores for the responses.
However, the scalar model suffers from limitations, particularly in the following aspects:
(i) Lack of interpretability:
Beyond scalar scores, additional rationales are crucial to enhance the reliability of judgments and facilitate human involvement in the evaluation loop.
(ii) Susceptibility to bias:
It is prone to capturing the biases present in the preference dataset rather than human values.
For example, when the majority of positive answers in preference datasets are longer sentences, the learned LLM will likely favor more verbose answers (Huang et al., ).

To address the above limitations, we propose Con-J, which trains a generative Judge using its self-generated Contrastive judgments (see Figure 1).
Con-J leverages the LLM’s pre-existing judgment abilities Zelikman et al. () and bootstraps its capacity to generate more accurate judgments.
As shown in Figure 2, Con-J consists of three steps:
(Judgment Sampling) Sample several judgments from a pre-trained LLM by prompting it with a query and a pair of candidate answers.
(Judgment Filtering) Leverage the true preference annotations to construct contrastive judgment pairs, i.e., judgments with correct or incorrect preference.
(Training) Train Con-J from the pre-trained LLM based on these contrastive judgments using Direct Preference Optimization (DPO).

The design of Con-J differs from existing methods for enhancing the capabilities of generative judges (or LLM-as-a-judge) (Li et al., ; Kim et al., ; Park et al., ).
These methods typically depend on external models (particularly GPT-4) or algorithmic schemes to produce high-quality instruction-tuning datasets.
In contrast, Con-J directly learns from preference data using a self-bootstrapping approach similar to that of scalar reward models.
As LLMs become more powerful, aligning them with high-quality judgments becomes more difficult since humans may not always be able to write superior judgments.
Instead, Con-J offers a new way by eliciting what the LLM already knows, supervised by human preferences, which are much easier to obtain than high-quality judgments.

We train and evaluate Con-J on self-built commercial datasets across three domains: Text Creation, Math, and Code and a series of publicly available datasets and benchmarks.
Our findings indicate that Con-J not only significantly outperforms the scalar model in the Text Creation task and achieves comparable performance in Math and Code but also that its performance, when trained on domain-specific data, will significantly surpass that of GPT-4o.
Additionally, Con-J trained on publicly available datasets achieves comparable performance with GPT-4o and surpasses a series of existing open-source models.
As a generative judge, Con-J can generate rationales to support its preference prediction.
We evaluate the correctness of these rationales and find that as the accuracy of preference predictions improves, the correctness of the rationales also increases.
Additionally, in a synthetic experiment, we found that Con-J is less susceptible to dataset biases, which we attribute to its generative training target and its feature to generate rationales simultaneously.
To facilitate further research and development within the community, We release the training process and model weights of Con-J trained on publicly available datasets at [https://huggingface.co/ZiyiYe/Con-J-Qwen2-7B](https://huggingface.co/ZiyiYe/Con-J-Qwen2-7B).

To summarize, our contributions are:

- 1.
We propose Con-J, an approach that trains a generative judge using a self-bootstrapped technology to learn from preference data.
- 2.
We show that Con-J can offer more accurate rationales during preference learning.
We also provide theoretical motivation and empirical evidence showing that Con-J can be more robust to dataset biases by training with rationales.
- 3.
We test the performance of Con-J in commercial datasets and publicly available benchmarks.
Con-J outperforms the scalar models and a series of existing generative judges.

## 2 Preliminary

### 2.1 Task definition

Given a question or prompt $q$ and a pair of assistant responses $a^{1}$ and $a^{2}$, the task is to judge the preference between $a^{1}$ and $a^{2}$.
To accomplish this, we train the model using an existing preference dataset $D=\{(q,a^{-},a^{+})_{i}\}_{i=1}^{N}$, where $a^{+}$ is a preferred answer compared to $a^{-}$.
The model’s performance is subsequently evaluated on a separate, non-overlapping preference dataset by measuring the accuracy of its preference judgments.

### 2.2 Scalar Model

The most common practice for getting the preference judgment is to use a scalar model (SM) similar to the reward model in the RLHF stage (Hou et al. ()).
The SM predicts numerical scores $r(q,a)$ for $a\in\{a^{1},a^{2}\}$ and judges the preference by comparing $r(q,a^{1})$ and $r(q,a^{2})$.
It is typically initialized by concatenating a pre-trained LLM with a randomly initialized shallow MLP head.
The most widely used training objective for the SM follows the Bradley-Terry model, which maximizes the probability of $a^{+}$ being preferred:

$$ $P(a^{+}\succ a^{-}|q)=\frac{\exp(r(q,a^{+}))}{\exp(r(q,a^{+}))+\exp(r(q,a^{-}) )}=\sigma\big{(}r(q,a^{+})-r(q,a^{-})\big{)}$ (1) $$

where $\sigma$ is the sigmoid function.

The above-mentioned SM utilizes the prompt $q$ and a single answer $a$ as input, which we denote as a pointwise SM.
In addition, existing research has investigated a pairwise variant that uses a pair of candidates as input, i.e., $r(q,a^{1},a^{2})$ (Jiang et al., ).
The pairwise vanilla reward formalizes the preference probability of $a^{+}$ as:

$$ $P(a^{+}\succ a^{-}|q)=\sigma(r(q,a^{+},a^{-})).$ (2) $$

To train the above-mentioned pointwise and pairwise SM $r$, we maximize the log-likelihood of the preferences by minimizing the following loss function:

$$ $\ell_{R}(r)=-\sum_{(x,a^{+},a^{-})}\log p_{r}(a^{+}\succ a^{-}\mid x)=\begin{ cases}-\sum_{(x,a^{+},a^{-})}\log\sigma(r(x,a^{+})-r(x,a^{-}))&\text{( pointwise)}\\ -\sum_{(x,a^{+},a^{-})}\log\sigma(r(x,a^{+},a^{-}))&\text{(pairwise)}\end{cases}$ (3) $$

## 3 Improving Generative Judge by Training on Contrastive Judgments

Figure: Figure 2: The steps for constructing Con-J with a preference dataset that includes preference annotations for a pair of answers to a question. $S_{1}$: Prompt Con-J to generate multiple judgments for a pair of answers by repeated sampling and hint-driven sampling. $S_{2}$: Bootstrap contrastive judgment pairs by filtering with true preference labels. A contrastive judgment pair consists of one judgment with the correct preference and the other either with an incorrect preference or does not explicitly indicate a preference. $S_{3}$: Train Con-J using the DPO loss on contrastive judgments and the SFT loss on positive judgments.
Refer to caption: x2.png

Instead of using a scalar model for preference judgment, we propose to leverage the LLM itself to make preference judgments (Guo et al. (); Lee et al. ()).
Given the question $q$ and a pair of answers $a^{1}$ and $a^{2}$, we construct a prompt $p$ by concatenating a preamble with $q$, $a^{1}$, and $a^{2}$.
The preamble is an instruction that describes the task and asks an LLM $\pi$ to act as a judge (see examples in Fig. 2 and Appendix 5).
Then the LLM generates natural language judgments $j=\pi(p)$, which contains the judgment as well as the rationale in a JSON style: a key named "rationale" includes a step-by-step explanation and verification of the answers, and another key named "better_answer" indicates the LLM’s binary judgment.

As shown in Figure 2, the construction of Con-J consists of three steps: judgment sampling, judgment filtering, and Con-J training.

Judgment sampling: We construct contrastive judgment pairs by prompting the LLM to generate multiple judgments.
As shown in Figure 2, this is achieved by (1) repeated sampling and (2) hint-driven sampling.

Repeated sampling prompts the LLM to generate multiple outputs from the same prompt, each utilizing a different random seed during the generation process.
However, the LLM may produce only one-sided judgments (i.e., all judgments preferred $a^{1}$ or $a^{2}$) across all repeated samples.
In such cases, we cannot construct contrastive judgment pairs with repeated sampling.
Therefore, we propose hint-driven sampling to compel the LLM to generate judgments that favor specific answers.
Essentially, the LLM is provided with an explicit indication of which answer is better, and is instructed to generate the judgment accordingly in the same JSON format as above.
The prompt template for hint-driven sampling is provided in Table 6.
By manipulating the hint, we can get a contrastive judgment pair for any prompt input.

Judgment filtering:
We denote the outputs from repeating sampling as $M(p)$.
$M(p)$ can potentially include both “positive” and “negative” judgments.
A positive judgment indicates the judgment corresponding to the keyword "better_answer" is correct ($j^{+}$), while a negative judgment ($j^{-}$) indicates the judgment is incorrect ($j^{i}$) or the model doesn’t explicitly indicate its preference ($j^{n}$).
Contrastive judgment pairs $\{(j^{+},j^{-})\}$ are hence constructed as the direct product of the positive judgment set $M(p)^{+}$ and negative judgment set $M(p)^{-}$.
We set the number of repeated samplings to 8, allowing for the construction of up to 4 pairs (in the optimal case, there exist 4 positive and 4 negative judgments among the 8).
For hint-driven sampling, we prompt the LLM with one correct and one incorrect hint and construct one pair from them.
The detailed sampling and filtering process is outlined in Algorithm 1.

Con-J training: Based on the constructed judgment pairs $D^{J}=\{(q,a^{+},a^{-},j^{+},j^{-})_{i}\}_{i=1}^{K}$, we train the LLM $\pi$ with a direct preference optimization (DPO) loss function:

$$ $\ell^{\text{DPO}}=-\sum_{(p,j^{+},j^{-})}\log\sigma\left[\eta\log\frac{\pi({j} ^{+}|p)}{\pi_{0}({j}^{+}|p)}-\eta\log\frac{\pi({j}^{-}|p)}{\pi_{0}({j}^{-}|p)}\right]$ (4) $$

where $\pi_{0}$ is the reference model initialized as the base LLM and remains untrained.
Following existing practice Liu et al. (); Hong et al. (); Pal et al. (), the DPO also fuses a small weight of supervised fine-tuning (SFT) loss to help mitigate the overoptimization issue, which can be formulated as:

$$ $\ell^{\text{SFT}}=-\sum_{(p,j^{+})}\log\pi({j}^{+}|p)$ (5) $$

Then we linearly combine the DPO loss and the SFT loss with a small weight $\alpha$:

$$ $\ell^{\text{final}}=\ell^{\text{DPO}}+\alpha*\ell^{\text{SFT}}$ (6) $$

DPO training promotes distinguishing between answers.
Existing open-source generative judges are generally trained using supervised fine-tuning (SFT) (Kim et al., ; Zhang et al., ; Li et al., ) to imitate correct judgments.
However, we empirically find that only SFT is insufficient (see Setion 4).
The intuition is that LLMs should identify the more important aspects of a judgment, rather than patterns that may appear in both positive and negative judgments (Park et al., ).
For example, both “answer 1 has logical errors, so the better answer is 2” and “answer 2 has logical errors, so the better answer is 1” could be positive judgments for different prompt inputs, even though they are opposite in meanings.
When LLM is trained with SFT loss, it may primarily imitate the common pattern that appears in both answers rather than developing the ability to make judgments based on the prompt.
Similar studies (^1^11https://github.com/LLaMafia/SFT_function_learning) indicate that the likelihood of generating negative output might even surpass that of positive output during SFT training.

Rationales bring robustness against bias.
The proposed Con-J can generate rationales in addition to the binary preference prediction.
We suggest that training the model to generate rationales can impart a regularization effect and help avoid potential biases in the datasets.
Here we provide a theoretical motivation for this effect.
We decompose a judgment $j$ into $j_{r}$ and $j_{y}$, representing the rationale and the binary preference prediction, respectively.
Adding the rationales as training targets can be formalized by introducing an intermediate variable $j_{r}$ influencing the conditional probability $P_{\theta}(j_{y}\mid p)$:

$$ $P_{\theta}(j_{y}\mid p)=\sum_{j}P_{\theta}(y\mid j_{r},p)\,P_{\theta}(j_{r} \mid p)$ (7) $$

By including rationales, the bias in preference data is distributed between $j_{y}$ and $j_{r}$, reducing its direct impact on $j_{y}$.
We can formalize the loss function as:

$$ $\ell(\theta)=-\sum_{(p,j)\in D}\log P_{\theta}(j_{y}\mid p)-\sum_{(p,j)\in D} \log P_{\theta}(j_{r}\mid p)$ (8) $$

The loss $P_{\theta}(j_{r}\mid p)$ encourages the model to find representations that are also effective for predicting $j_{r}$, as it is less influenced by the bias compared $P_{\theta}(j_{y}\mid p)$, which solely depends on the dataset’s preference annotation.

Generative judge resists bias with a better prior.
SM modifies the LLM’s architecture with a classification head and uses a discriminative training target.
On the contrary, Con-J uses an architecture consistent with the pertaining process and generative training objectives (Zhang et al. ()).
We refer to Erhan et al. () and assume the parameter inherited from the pre-trained LLM as adding an infinite penalty:

$$ $\ell(\theta)=\ell_{data}(\theta)+\frac{\lambda}{2}||\theta-\theta_{0}||^{2}$ (9) $$

where $\lambda$ is the regularization strength.
We make an ideal hypothesis that there exists an optimal $\theta^{*}$ which fully encodes human values and consistently makes true judgments.
Such an $\theta^{*}$ must have sufficient world knowledge beyond what preference datasets can provide during LLM post-training.
Hence we assume that the parameters obtained during the pre-training phase are closer to $\theta^{*}$ than a random distribution.
This analysis suggests Con-J gets a smaller penalty term for optimization towards $\theta^{*}$.
In contrast, SM adopts a different training objective and introduces a randomly initialized head, making the regularization effect less significant.
Hence, SM can be good at encoding the knowledge reflected in the preference dataset but is also more sensitive to its bias than Con-J.

## 4 Experiments

### 4.1 Experimental Setup

Datasets.
We train scalar models and Con-J on three datasets within different vertical domains: Creation, and Math, then evaluate their performance in terms of the accuracy of preference predictions.
The datasets are self-built commercial datasets consisting of approximately 120,000, 50,000, and 50,000 preference samples for Creation, Math, and Code, respectively.
The Creative dataset involves tasks on text creation such as writing poetry or crafting headlines, whereas Math and Code datasets concentrate on math problem-solving and code writing, respectively.
The datasets cover diverse sources, ranging from data generated by a commercial ChatBot, data generated by ChatGPT, and data from open-source datasets like HH-rlhf Bai et al. () and Infinity-Instruct(^2^22https://huggingface.co/datasets/BAAI/Infinity-Instruct).
The preference annotation for these datasets is gathered from human annotators, with each annotation being performed by one annotator and subsequently verified by another.
In addition to the self-built datasets, we train Con-J on a publicly available dataset
Skywork-Reward-Preference-80K-v0.1(^3^33https://huggingface.co/datasets/Skywork/Skywork-Reward-Preference-80K-v0.1) and test its performance on public benchmarks including Infinity-Preference(^4^44https://huggingface.co/datasets/BAAI/Infinity-Preference), UltraFeedback Cui et al. (), PKU-SafeRLHF Ji et al. (), and Reward-Bench Lambert et al. ().
We ensure that no identical prompts appear in both the training and test sets by filtering them out of the training set.
This version of Con-J is publicly released in anonymized link.

Model.
We select Qwen2-7B-Instruct Yang et al. () as the base model to train both the scalar model (named SM) and the proposed generative judge (named Con-J).
SM includes both pairwise and pointwise variants.
Additionally, we included the original pre-trained Qwen2-7B-Instruct as an untrained variant of Con-J.
In addition, we compare Con-J with a range of generative judges, including GPT-4o(^5^55https://openai.com/index/hello-gpt-4o/) and two generative models (Auto-J Li et al. () and Prometheus 2 Kim et al. ()) trained by SFT, Llama series (Llama3.1-8B, and Llama3.1-70B), and Qwen series (Qwen2-7B, Qwen2.5-72B).

Hyper parameters.
We train SM and Con-J from with the DeepSpeed library Rasley et al. (), Zero Redundancy Optimizer (ZeRO) Stage 3 Rajbhandari et al. (), gradient-checkpointing Chen et al. () and FlashAttention Dao et al. ().
We use bfloat16 (BF16) and tfloat32 (TF32) mix computation precision.
We set a peak learning rate of $9e-6$ with 3% warmup steps and cosine scheduler, and a maximum sequence length of 4,096.
The batch sizes of SM and Con-J are set to 128 and 24, respectively.
For Con-J, we linearly combine the SFT loss and the DPO loss with $\alpha=1e^{-6}$.

Sampling and inference strategy for Con-J.
We use VLLM Kwon et al. () for the inference for Con-J.
During repeated sampling and hint-driven sampling, we employ greedy sampling with top-p set at 0.9 and top-k at 20, with a maximum output length set as 512, a temperature of 1.0, and a repetition penalty of 1.2.
During the evaluation, we set top-p at 1.0 and a temperature of 0.0.

### 4.2 Main Results

**Table 1: Judgment accuracy of GPT-4o, SM, and Con-J. $*$ indicates the performance difference between Con-J is significant at $p<0.05$ using a pair-wise t-test.**
| Model | Creation | Math | Code |
| --- | --- | --- | --- |
| GPT-4o | $55.6^{*}$ | $74.8^{*}$ | $68.1^{*}$ |
| SM (point-wise) | $69.4^{*}$ | 84.8 | 69.4 |
| SM (pair-wise) | $69.2^{*}$ | 84.6 | 69.6 |
| Con-J | 72.4 | 85.0 | 70.1 |

Table 1 presents the results of SM and Con-J on the self-built commercial datasets across three vertical domains.
It can be observed that
(i) there is no significant difference between the pointwise and pairwise variants of SM.
Although existing research suggests that concatenating the list of responses improves performance for scoring the responses Jiang et al. (), we do not observe this effect on our datasets.
(ii) Both Con-J and SM outperform off-the-shelf GPT-4o, indicating that small models trained on domain-specific data can effectively reflect domain-related preferences.
(iii) On the same preference datasets, Con-J consistently outperforms SM across all tasks with a significant gap on the Text Creation task.
This indicates that Con-J is more effective at acquiring accurate judgment abilities than SM.

We carry out an ablation study to investigate the variants of Con-J.
Given that current methodologies often employ Supervised Fine-Tuning (SFT) to train generative judges Li et al. (); Kim et al. (), we developed an SFT variant of Con-J, trained exclusively on positive judgments using SFT loss.
As illustrated in Table 3, Con-J trained with our proposed framework outperforms its variant without DPO loss across all datasets.
This observation demonstrates the effectiveness of training from contrastive judgments.
In addition, Con-J outperforms its variant without hint-driven sampling, which relies solely on repeated sampling and may be infeasible to construct contrastive judgment pairs for some prompts.
Similar findings have been observed when using self-taught techniques to improve LLMs Zelikman et al. ().
Additional variants of Con-J were also tested, which are detailed in the Appendix.

**Table 2: Accuracy of generative judges on the test sets of four benchmarks: Infinity-Preference, UltraFeedback, PKU-SafeRLHF, and Reward-Bench. Results in bold are the best among all models and results with underline are the second-best.**
|  | Infinity-<br>Preference | Ultra-<br>Feedback | PKU-<br>SafeRLHF | Reward-Bench |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | Chat | Chat-H | Safety | Reasoning |  |  |  |
| Llama3.1-8B | 59.0 | 62.9 | 66.4 | 80.7 | 49.8 | 64.0 | 68.1 |
| Llama3.1-70B | 64.0 | 71.4 | 67.6 | 97.2 | 70.2 | 82.8 | 86.0 |
| Qwen2-7B | 59.0 | 64.5 | 67.2 | 91.3 | 44.8 | 73.6 | 69.0 |
| Qwen2.5-72B | 70.0 | 66.0 | 58.7 | 86.6 | 61.4 | 74.5 | 90.7 |
| Auto-J | 69.0 | 63.9 | 66.9 | 93.0 | 40.0 | 65.5 | 50.5 |
| Prometheus 2 | 68.0 | 63.3 | 63.0 | 85.5 | 49.1 | 77.1 | 76.5 |
| GPT-4o | 75.0 | 72.2 | 69.6 | 95.3 | 74.3 | 87.6 | 86.9 |
| Con-J (ours) | 81.0 | 73.0 | 68.4 | 91.3 | 79.6 | 88.0 | 87.1 |

**Table 3: Judgment accuracy of Con-J and its variants. $*$ indicates the performance difference between Con-J is significant at $p<0.05$ using a pair-wise t-test.**
| Model | Creation | Math | Code |
| --- | --- | --- | --- |
| Con-J untrained | $53.6^{*}$ | $63.4^{*}$ | $61.7^{*}$ |
| Con-J w/o Hint | $61.3^{*}$ | $77.4^{*}$ | 68.2 |
| Con-J w/o DPO | $54.6^{*}$ | $64.2^{*}$ | $63.5^{*}$ |
| Con-J | 72.4 | 85.0 | 70.1 |

For a fair comparison with other generative judge models, we trained an open-source version of Con-J using the publicly available dataset Skywork-Reward-Preference-80K-v0.1, and evaluated its performance on public benchmarks, as shown in Table 2.
Con-J outperforms existing publicly available large language models (LLMs) in the vast majority of benchmarks, including commercial instruction-tuned models such as Llama series, Qwen series, and a series of LLMs trained specially for preference judgments such as Auto-J and Prometheus 2, except in the Chat and Reasoning sub-task of Reward-Bench.
Additionally, Con-J achieves comparable performance with the closed-source model GPT-4o in all benchmarks.
As of September 18, 2024, Con-J ranks first on the reward-bench leaderboard among all models within 7B parameters, as well as second among open-source generative models.

### 4.3 Preference learning yields meaningful and useful rationales.

We select 5 checkpoints trained on different numbers of contrastive judgment pairs, i.e., 2k, 4k, 8k, 16k, and 50k.
Then we prompt GPT-4o with the question, corresponding pair of answers, and the judgment generated by Con-J (see the prompt in Table 9).
GPT-4o is instructed to score the judgment ranging from 1 to 5 for the rationale’s correctness and from 1 to 3 for the rationale’s consistency with its predicted preference.
Additionally, GPT-4o is tasked as a meta-judge to evaluate whether Con-J makes correct preference predictions.
In this process, if the judgments of GPT-4o conflict with the dataset’s true preference annotations, we exclude these questions for further analysis, as these questions may exceed GPT-4o’s capabilities.

Experimental results are presented in Figure 3.
From Figure 3(a), we observe that the correctness of the rationales improves when Con-J is trained with more data and achieves increased judgment accuracy.
However, we find that the consistency between Con-J’s preference prediction and its rationales decreases with the increase in judgment accuracy, as shown in Figure 3(b).
These observations indicate that Con-J’s abilities to make binary preference predictions and generate correct rationales both improve with training.
However, the increase in inconsistency indicates that these improvements may not be balanced.
We suspect that because the supervision from preference datasets focuses solely on predicted binary preferences, the enhancement in Con-J’s binary prediction ability is more pronounced, leading to inconsistency.

Figure: (a)
Refer to caption: extracted/5922893/figures/rationale_score_correctness.png

We further test whether the improved rationales can be used to help a weak judge make better judgments.
We use the untrained Con-J as the weak judge and prompt it with the rationales generated by the stronger model.
As shown in Figure 3(c), the weak judge yields more accurate preference prediction with the rationales provided by a stronger model.
Additionally, we find that rationales generated by GPT-4o can also improve the weak judge, with performance comparable to the rationales from the strongest checkpoint of Con-J.
This indicates that Con-J not only surpasses GPT-4o in preference prediction performance but also generates rationales with similar effectiveness.

### 4.4 Rationales help Con-J become less susceptible to biases in the dataset.

Scalar models are known to be susceptible to biases in datasets, resulting in preference judgments that reflect biases in the pattern of data rather than true human values.
To investigate the susceptibility of Con-J, we conduct a synthetic experiment that injects artificial bias into the data.
We define the degree of bias in training set $\mathcal{D}_{\text{train}}$ as $\gamma$.
This indicates that the $\gamma$ proportion of the data contains unexpected biases that don’t necessarily reflect human value, while the rest $1-\gamma$ of the data are randomly sampled from the original training set.

Figure: (a) Format (Adversarial)
Refer to caption: extracted/5922893/figures/format.png

We consider two widely studied biases: format bias and verbosity bias Park et al. (); Singhal et al. (); Shen et al. ().
For format bias, we use two different prompts to rewrite the pair of answers (see prompt instructions in Table 8).
Specifically, we rewrite the good answer with a more verbal format and the bad answer with a more formal format with GPT-4o.
For verbosity bias, we follow existing practice Zheng et al. () by asking GPT-4o to rephrase the good answer without adding any new information and insert the rephrased text at the end of the original answer.

All experiments are conducted on a filtered Math dataset, where the filtering ensures that the solution in the preferred answer is correct, while incorrect in the less preferred answer.
In this scenario, the format and verbosity of the answers should not be considered in the preference judgments as the correctness of the solution is prioritized.
We investigate the performance of Con-J, SM, and Con-J trained and evaluated without generating the rationales (see the prompt examples in Table 7) under the injections of the addressed bias.
We trained them on a biased dataset and tested their performance on an original, test set $\mathcal{D}_{\text{test}}^{\text{general}}$ without any modification, and a test set constructed by injecting a contrary bias $\mathcal{D}_{\text{test}}^{\text{adversarial}}$.
The contrary bias involves applying the bias for preferred answers from the training set to less preferred answers in the testing phase, and vice versa.
The training used the same learning rate as mentioned earlier.
All models achieved an accuracy of over 99% on the biased training dataset when $\gamma=1$, indicating the models are thoroughly trained on the data and are strongly influenced by the bias.

We present the experimental results in Figure 4(a-d).
From Figure 4(a-d), we have the following observations:
(i) Con-J outperforms SM when trained with biased data.
Intuitively, Con-J does not train a value head from scratch; instead, it leverages the text generation capabilities of pre-trained LLMs.
This approach helps retain global knowledge about the task from pretraining and allows it to extract useful judgment standards even from biased data.
(ii) When tested on the adversarial datasets, Con-J without rationales significantly underperforms Con-J when $\gamma>0.2$ and $\gamma>0.33$ for the format bias and the verbosity bias, respectively.
The above observations indicate that Con-J becomes more robust at learning from biased data through training with rationales.
(iii) When tested on general test sets, Con-J without rationales demonstrates comparable performance to Con-J with rationales.
Some existing research suggests that rationales or critics generated ahead of preference judgments can facilitate Chain-of-Thought (CoT) reasoning and help the LLM make better judgments Lee et al. (); Ankner et al. (); Ye et al. (); Zhang et al. ().
A possible explanation for the lack of a similar CoT effect in our data is that the CoT process is often already embedded in the responses, making the CoT procedure for judgment potentially unnecessary.

### 4.5 Case Analysis of Con-J

**Table 4: Example of rationales and binary preference generated by Con-J. We prompt GPT-4o to annotate the correctness (ranging from 1 to 5) of the rationales and consistency between the predicted preference and the rationale (ranging from 1 to 3).**
| Prompt input | Rationale | Preferred |  |
| --- | --- | --- | --- |
| Creation (1)XX | Question: On the theme of comprehensive rural revitalization, propose a paper title.<br>Answer 1: Research on Strategic Pathways for Rural Revitalization from a Multi-Collaborative Perspective.<br>Answer 2: Comprehensive Rural Revitalization: Deep Integration of Strategies and Practices. | The second answer provides a broader applicability and depth of description, with a smoother and more natural context than the first answer.<br>Correctness: 5, Consistency: 3 | Answer 2<br>(correct) |
| Math (1)XXX | Question: Today is August 12, 2021. What was the date 30 days ago?<br>Answer 1: According to the given information, today is August 12, 2021, so 30 days ago was July 13, 2021. Therefore, 30 days ago was July 13, 2021.<br>Answer 2: July 12, 2021. | The second answer does not provide a detailed explanation, whereas the first answer offers a detailed reasoning process and uses correct date subtraction logic.<br>Correctness: 2, Consistency: 3 | Answer 1 (incorrect) |
| Math (2)XXXX | Question: Soldiers built a 2000-meter line. They averaged 350 meters per day for the first 3 days. What is the required daily average for the remaining 2 days?<br>Answer1: $(2000-350\times 3)\div 2=1400\div 2=700$ meters.<br>Answer: An average of 700 meters needs to be set up each day for the last two days.<br>Answer 2: $(2000-350\times 3)\div 2=(2000-1050)\div 2=950\div 2=475$ meters.<br>Answer: 475 meters for the last two days. | The first answer is straightforward and accurate, taking into account the total distance that needs to be set up and the known information; the second answer offers a slightly simplified expression and the result is correct.<br>Correctness: 2, Consistency: 1 | Answer 2<br>(correct) |
| Code (1)XXXXXXX | Question: Reverse the elements in an array and make each character in the elements lowercase. planet_arr = [‘Earth’, ‘Saturn’, ‘Mars’];<br>Answer 1: def reverse_array(arr):; return arr[::-1]; def lowercase_array(arr):; return [x.lower() for x in arr]; planet_arr = [‘Earth’, ‘Saturn’, ‘Mars’]; print(reverse_array(planet_arr));print(lowercase_array(planet_arr));<br>Answer 2: planet_arr = [‘Earth’, ‘Saturn’, ‘Mars’]; planet_arr.reverse(); planet_arr = [p.lower() for p in planet_arr]; print(planet_arr); This code will first reverse the order of the elements in the ‘planet_arr‘ array, and then lowercase each character in each element. The resulting array will be [‘mars’,‘saturn’,‘earth’]. | The first answer provides a clearer step-by-step solution to the problem by achieving array reversal and lowercase conversion through independent functions, which makes the code more modular. The second answer is more concise and accurately meets the question’s requirements by first reversing the array and then converting it to lowercase.<br>Correctness: 3, Consistency: 1 | Answer 1 (incorrect) |

We present example judgments generated by Con-J in Table 4, covering three domains and illustrating both correct and incorrect preference predictions.
For that cases of Creation (1) and Math (1), there is high consistency between the rationale and the binary preference (the consistency scores are 3).
We observe that both the rationale and the preference prediction are correct in Creation (1) while incorrect Math (1).
Conversely, the consistency in the cases of Math (2) and Code (1) is relatively low.
In these cases, we find that the rationale does not reflect the most direct support for preference prediction.
For example, in the third example, the answers are compared based on their format, while the key difference is that one of the answers involves an incorrect calculation and is not involved in the rationales.
This indicates that even if the model can make a correct judgment, it may not necessarily be based on the correct rationale.

## 5 related work

LLM alignment.
The initial approach developed for aligning LLMs with human values was reinforcement learning from human feedback (RLHF) (Christiano et al., ; Liu et al., ).
This technique involves training a scalar reward model (RM) and then using reinforcement learning (RL) to optimize a policy according to the RM.
In recent years, a series of direct alignment from preference (DAP) works, such as DPO (Rafailov et al., ), SiLC (Zhao et al., ), and IPO Azar et al. (), have gained popularity.
Unlike RLHF, DAP methods directly update the LLM using pairwise preference data, making the alignment simpler, and more stable.
To scale the preference datasets, it is common to train an external machine model from existing preference datasets (Hou et al. (); Wu et al. ()).
This online and scalable construction process enables DPO to be deployed in an iterative setting (Xiong et al. (); Xu et al. ()) or online setting (Guo et al. ()).
Therefore, how to build an accurate external model for preference judgment is an important problem.

LLM-as-a-judge.
Instead of training a scalar model for preference judgment, employing LLMs as a generative judge has been a promising alternative (Zheng et al., ; Ye et al., ).
Efforts have been made to train language models specialized in evaluations.
For example, Li et al. (); Kim et al. () proposes constructing an instruction-tuning dataset by prompting GPT-4 and using supervised fine-tuning (SFT) to train a pre-trained LLM as a generative judge.
Zhang et al. () propose training the LLM by minimizing the SFT loss with a single “Yes” or “No”, along with a rationale generated either by prompting Gemini 1.0 Pro or through algorithmic construction.
Our contribution to the existing research is that Con-J uses self-sampled contrastive judgments under the supervision of preference data, allowing for more efficient data construction while achieving better performance.

## 6 Discussions and Conclusions

We introduced Con-J, a novel approach that trains a generative judge by self-bootstrapped learning from preference data.
Con-J addresses the limitations of scalar reward models, including lack of interpretability and susceptibility to dataset bias.
Our experiments on commercial datasets across Text Creation, Math, and Code domains, as well as publicly available benchmarks, demonstrate the effectiveness of Con-J.
Moreover, we show that the correctness of the rationales generated by Con-J improves during learning from preference data.
This enables Con-J not only to make accurate judgments but also to provide reasonable explanations, potentially facilitating human-in-the-loop supervision of LLM alignment.
Finally, we found that Con-J is less susceptible to biases in datasets compared to its variants without rationales and the scalar models.

As AI systems become more powerful, many suggest that they will reach the point at which human are unable to easily and reliably assess the quality of their outputs (Casper et al., ).
To address this issue, using another AI to supervise itself is a viable solution; however, researchers suggest that these methods may fail without human involvement (Shumailov et al., ).
This paper contributes to addressing this issue in two ways.
On the one hand, Con-J can be used to supervise LLMs by acting as a judge.
At the same time, Con-J produces an explanation of its output that is legible to humans or another trusted system.
This indicates that we can spot any possible errors made by Con-J.
On the other hand, the training and construction of Con-J rely solely on preference data, which is easier to acquire from human annotators than high-quality instruction tunning data.
Furthermore, in many cases humans often find it difficult to provide verbal reasons for their preference, the training of Con-J could be integrated with human preference annotations, thereby enhancing the transparency of the entire annotation process.

Several limitations of this work guide future directions including:
(i) We demonstrate that preference learning can enhance the model’s ability to generate correct rationales.
Another unresolved and intriguing question is whether enhancing the quality of rationales during the sampling process could also improve the model’s preference prediction abilities.
It is an important problem to enhance the model’s ability not only to make accurate preference predictions but also to base those judgments on correct reasoning.
(ii) We demonstrated that Con-J can more effectively resist bias than SM in an adversarial experiment.
However, further analysis is needed to understand why Con-J outperforms SM on complex, realistic datasets, and whether this is also related to bias.
(iii) We suggest that Con-j can potentially facilitate human collaboration through interpretable preference judgments for LLM training.
The design of such a pipeline is another interesting and valuable direction.

## Appendix A Appendix

**Table 5: An example of a prompt fed to the LLM to generate preference reward judgments, consisting of a preamble (introduction and instructions describing the task), a question, and a pair of candidate answers. The preamble is neutral and does not indicate which answer is better.**
| Preamble | As an evaluation expert, given a question and its two possible answers, please choose which answer best aligns with coherence, accuracy, coverage, and overall quality. Output your judgment in JSON format, where ”rationale” is your explanation, and ”better_answer” is an integer type of 1 or 2, for example, {“rationale”: “your explanation”, “better_answer”: 1}. Below are the question and its candidate answers: |
| --- | --- |
| Question | The volume of a rectangular prism is 96 cubic decimeters, and the base area is 16 square decimeters. What is its height in decimeters? |
| Answer 1 | We can use the following formula to calculate the height h of the rectangular prism: V = S \times h. Substitute the known values into the formula: 96 = 16 \times h; h = 6. Therefore, the height of the rectangular prism is 6 decimeters. |
| Answer 2 | 5. Height = Volume / Base Area = 96 / 16 = 5 decimeters. |
| Prompt | As an evaluation expert, given a question and its two possible answers, please choose which answer best aligns with coherence, accuracy, coverage, and overall quality. Output your judgment in JSON format, where ”rationale” is your explanation, and ”better_answer” is an integer type of 1 or 2, for example, {“rationale”: “your explanation”, “better_answer”: 1}. Below are the question and its candidate answers:<br>Question: The volume of a rectangular prism is 96 cubic decimeters, and the base area is 16 square decimeters. What is its height in decimeters?<br>Answer 1: We can use the following formula to calculate the height h of the rectangular prism: V = S \times h. Substitute the known values into the formula: 96 = 16 \times h; h = 6. Therefore, the height of the rectangular prism is 6 decimeters.<br>Answer 2: 5. Height = Volume / Base Area = 96 / 16 = 5 decimeters. |

**Table 6: Prompt template with preamble using correct or incorrect hints, where $\alpha,\beta$ are the ID of correct and incorrect answers, respectively, $\{\alpha,\beta\}=\{1,2\}$, {{Question}}, {{Answer 1}}, {{Answer 2}} are the text content of the question, answer 1, and answer 2, respectively. When the LLM does not output a valid JSON format as expected (e.g., “rationale”: “your explanation”, “better_answer”: $\alpha$), we use an alternative prompt (rows 3-4) to prompt it again and insert its output as the rationale into the template.**
| Prompt with preamble_correct | As an evaluation expert, given a question and its two possible answers, please choose which answer best aligns with coherence, accuracy, coverage, and overall quality. Below are the question and its candidate answers:<br>Question: {{Question}}<br>Answer 1: {{Answer 1}}<br>Answer 2: {{Answer 2}}<br>Given that answer $\alpha$ is better than answer $\beta$, please provide the rationale. Output your judgment in JSON format, where ”rationale” is your explanation, and ”better_answer” is an integer type of $\alpha$, for example, {“rationale”: “your explanation”, “better_answer”: $\alpha$}. |
| --- | --- |
| Prompt with preamble_incorrect | As an evaluation expert, given a question and its two possible answers, please choose which answer best aligns with coherence, accuracy, coverage, and overall quality. Below are the question and its candidate answers:<br>Question: {{Question}}<br>Answer 1: {{Answer 1}}<br>Answer 2: {{Answer 2}}<br>Given that answer $\beta$ is better than answer $\alpha$, please provide the rationale. Output your judgment in JSON format, where ”rationale” is your explanation, and ”better_answer” is an integer type of $\beta$, for example, {“rationale”: “your explanation”, “better_answer”: $\beta$}. |
| Prompt with preamble_correct (alternative) | As an evaluation expert, given a question and its two possible answers, compare the answers according to their coherence, accuracy, coverage, and overall quality. Below are the question and its candidate answers:<br>Question: {{Question}}<br>Answer 1: {{Answer 1}}<br>Answer 2: {{Answer 2}}<br>Given that answer $\alpha$ is better than answer $\beta$, please provide the rationale: |
| Prompt with preamble_incorrect (alternative) | As an evaluation expert, given a question and its two possible answers, compare the answers according to their coherence, accuracy, coverage, and overall quality. Below are the question and its candidate answers:<br>Question: {{Question}}<br>Answer 1: {{Answer 1}}<br>Answer 2: {{Answer 2}}<br>Given that answer $\beta$ is better than answer $\alpha$, please provide the rationale: |

**Table 7: Prompt template for forcing the generative LLM outputs only the binary judgment without any rationales. {{Question}}, {{Answer 1}}, {{Answer 2}} are the text content of the question, answer 1, and answer 2, respectively.**
| Prompt | As an evaluation expert, given a question and its two possible answers, please choose which answer best aligns with coherence, accuracy, coverage, and overall quality. Output your judgment in JSON format in which ”better_answer” is an integer type of 1 or 2, for example, {“better_answer”: 1}. Do not include any additional explanations. Below are the question and its candidate answers:<br>Question: {{Question}}<br>Answer 1: {{Answer 1}}<br>Answer 2: {{Answer 2}} |
| --- | --- |

**Table 8: Prompt template for transforming the answers into different formats.**
| Prompt for rewriting the answer into a more verbal format | You are someone who works on popularizing mathematical knowledge.<br>Please restate the following content in simpler, more accessible language without changing the original meaning, affecting its length, or adding extra information.<br>Below is the input:<br>{{Answer}} |
| --- | --- |
| Prompt for rewriting the answer into a more formal format | You are a researcher in the field of mathematics.<br>Please restate the following content using precise mathematical language without changing the original meaning, affecting its length, or adding extra information.<br>Below is the input:<br>{{Answer}} |
| Prompt for rewriting the answer to be more verbose | Please summarize the input by listing the key points in a numbered format.<br>Below is the input:<br>{{Answer}} |

**Table 9: Prompt template for scoring the rationales and judgments of Con-J.**
| ##Background<br>Given below is a question and two corresponding answers:<br>Question: {{Question}}<br>Answer 1: {{Answer 1}}<br>Answer 2: {{Answer 2}}<br>A judge has assessed these two answers and judged which one is better.<br>Here are its judgment and the corresponding rationales:<br>{{Judgment}}<br>##Workflow and Scoring<br>Please analyze whether the judgment is correct and evaluate the rationale by scoring them on (i) correctness and (ii) consistency with the binary judgment:<br>Correctness:<br>1: Completely incorrect, with obvious erroneous arguments.<br>2: Mostly incorrect, with some correct arguments.<br>3: Partially correct, with almost no errors.<br>4: Mostly correct.<br>5: Completely correct.<br>Consistency:<br>1: The emotional tone of the reasons is inconsistent with the final judgment.<br>2: The reasons have no clear emotional tone.<br>3: The emotional tone of the reasons is consistent with the final judgment.<br>##Formatting<br>Please return the results in JSON format, for example: “Judgment Correctness”: “Correct”, “Rationale Correctness”: 4, ”Rationale Consistency”: 3, where “Judgment Correctness” can be either “Correct” or “Incorrect”. |
| --- |

Figure: Algorithm 1 Constructing contrastive judgment pairs for Con-J

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see weakness section.
soundness: 3
strengths: * Very interesting and relevant approach that uses synthetic data to effectively propose a good empirical solution.
* Generating rationales would be very useful to verbalize and detect biases in the preference data and can help provide better data quality.
* I like that the authors train a model on publicly available datasets and report numbers on widely used benchmarks instead of just relying on the internal test sets.
summary: The paper proposes a generative approach to build reward models for preference learning. They generate judgements with explanations for both positive and negative samples in preference pairs and then use DPO to align a generative reward model. They find that this contrastive approach leads to improved performance on internal and public benchmarks when compared against scalar reward models and other generative methods.
weaknesses: * The approach feels like it would add a lot of overhead in alignment training - RLHF is already extremely computationally intensive and waiting for models to generate rationales before providing the judgement feels like it would not be a feasible approach to use for cutting-edge model alignment.
* I would like to see some qualitative analysis of the rationales - there seems to be minimal information on how good they are and what they might be useful for. Has there been any human evaluation of the rationales when inconsistencies have been detected?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How exactly does Con-J without rationale differ from the scalar model (Figure 4)?
2. One can also imagine a debiasing effect from using pre-trained LMs as rationale generators -- they are likely to generate rationales that align with preferences from their pre-trained data. While this may be ok if the preferences are generally in line with the pre-trained data, there may be issues with aligning to preferences with fundamentally different values or principles. Is there any investigation of whether the LM likes to generate the same set of values in its rationales?
3. Are there overall statistics from section 4.5 about how consistent / correct the rationales tend to be? Either automated, or human annotated (perhaps on a subset).
soundness: 2
strengths: The experimental results demonstrate that Con-J is an effective method, compared to scalar reward models. The results section is quite comprehensive, with baselines across multiple models and datasets. The analysis section is informative with lots of useful takeaways: including studies of how Con-J can be robust in the presence of bias, and where LMs can generate rationales that fail to support the answer judgment. Overall, the claims are well-supported.
summary: This paper introduces Con-J, a new way of learning preferences by augmenting human pairwise preferences with LM-generated rationales. First, a LM is prompted to generate rationale pairs ($j^+$, $j^-$) for either element of the preference pair ($a^+$, $a^-$). The base LM is then trained with DPO to optimize $j^+$ over $j^-$ (vs. DPO with scalar rewards, which is simply trained to optimize $a^+$ over $a^-$). Empirical studies are performed on three domains (Creation, Math, Code), which find that Con-J outperforms scalar reward models at providing the correct judgment, that it outperforms other generative judges (Auto-J and Prometheus 2), and and that it is less susceptible to bias and spurious correlations in the reward function.
weaknesses: 1. The empirical results (Table 10, Figure 4(b)(d)) seem to indicate that the main contribution to Con-J's good results are not the rationales, but (presumably) the regularization in eq. 9 instead (It's unclear to me what exact parts Con-J without rational removes...). Experimentally, what happens when this regularization is ablated? Furthermore, the claims in the abstract/introduction seem to suggest that rationales themselves are the reason why Con-J succeeds. While experimentally it's demonstrated that the rationales do improve robustness to bias, it seems misleading to claim this when the main results actually suggest that the rationales do not improve judgment accuracy at all. It may be good to do a deeper analysis of what is actually inducing this improvement.
2. Furthermore, the introduction highlights interpretability as a benefit of rationales (and puts it on the same priority level as robustness to bias). While intuitively rationales are indeed more interpretable, the paper never demonstrates the utility of this interpretability. What happens if you do have humans (or even other LMs) check the rationales and throw out inconsistent ones? (As far as I understand, the filtering step simply checks the correctness of the final answer, but not the rationale itself.) Is this interpretability actually useful for constructing better / more useful reward models? Will it be better to align downstream models with rationales?
3. The method has many moving parts. While the figure is useful, parts of the text and figure could be clearer. Having a concrete example with a LM-generated rationale in the main body or the figure could be useful for visualizing the method. Furthermore, it may be good to list out all the moving parts, esp for the ablations it is unclear what parts are preserved and what parts are not (e.g. how exactly does Con-J without rationales differ from the scalar model).
4. Note -- quite a few generative reward model methods have come out at about the same time this work (see https://arxiv.org/abs/2408.15240, https://arxiv.org/abs/2410.12832). While these can be considered concurrent work, perhaps a discussion of the tradeoffs of this method compared to other methods could be useful somewhere in the paper.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: Questions
- [Q1] Can you clarify (or point to a URL) how you were able to obtain a test split for Ultrafeedback? The HuggingFace version (openbmb/Ultrafeedback) only contains a train split.

Comments/Nits:
- [C1] Line 259-261. The “Code” vertical domain is not mentioned
- [C2] Some LaTeX citation commands were used incorrectly. For example, \citet is often used in place of \citep, breaking the flow of writing. I highly recommend the authors to revisit these parts and update accordingly.
soundness: 3
strengths: - [S1] The novelty of the paper’s contribution is clear. I appreciate how Con-J’s adjustments of the typical generative RM set-up was motivated.
- [S2] The ablations are also reasonable, especially on the different variants of the contrastive pairs (e.g., positive judgments only, etc.).
summary: The paper introduces Con-J, an approach for training a generative judge using contrastive judgments for preference learning in order to address the problem of (i) lack of interpretability and (ii) susceptibility to biases in scalar reward models. Con-J involves creating contrastive judgments from a set of query-answer pairs sampled from a pre-trained LLM. These contrastive pairs are then used for preference finetuning, for example, direct preference optimization. Finally, they were able to show that Con-J performs better on several benchmarks such as RewardBench.
weaknesses: Overall, I lean towards accepting the paper on the merit of its contribution's novelty (new way of framing the generative RM setup), but there are some sections--particularly in the analysis--that are not fully convincing and needs revisiting (see [W1]). 

[W1] I’m not entirely convinced with the analyses of Consistency and Correctness in Sections 4.3 and 4.5. I think it’s helpful if the paper first defines what “consistency” and “correctness” measure. In addition, it will be helpful to show that the GPT-4o scorer is reliable. Suggestion to the authors: define and motivate the aspects of consistency and correctness in the beginning of the paper.
- The prompt template in Table 9 speaks about the emotional tone, which I think measures a different aspect than consistency. Perhaps that’s also the reason why the results  in Figure 3b are surprising (I would expect the consistency of the rationales would also improve) and aren’t consistent with the cited literature on chain-of-thought (CoT).
- The scoring for Correctness is also confusing: the rubric for 3 can be confused for 2 and 4.  It’s also important to describe what correctness means for a subjective task such as Creation. I wonder what the findings would be if the rubric is more distinct.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. The paper seeks to address a crucial issue in preference (or reward learning), where the reward model may be prone to bias and is not easily understood from the human interpretability angle. However, the learning algorithm (objective) is not very novel and recent work [1,3-5] has already leveraged such DPO + SFT type losses have leveraged such as contrastive learning formulation. However, I could not find these works being cited unless I am missing something. 

2. While the idea is simple, my primary concern is that the generated rationales can be biased or be of low-quality if the chosen base model is not strong enough. Unless I am missing something, the authors do not conduct any human evaluation of the rationales being sampled.  How does one make sure that the rationales are at least surpass some *baseline* reasoning quality, regardless of the preference relation without a manual  verification component?

 

3. How do you control for the length of the judgements being sampled? From my perspective, if the intention in the Con-J methodology is better/explainable reward learning with compute-efficiency, then would you not have to control for judgement length pre-inference? In that case, for questions that would naturally require a more thorough reasoning process (CoT), wouldn’t the preference judgement be partial or biased in such cases? If judgement-length is controlled, how can shortened or partial/biased judgements’ affect on preference learning be mitigated in light of the inconsistency of rationale vs binary decisions observed in section 4.3?


4 .How was the alpha value chosen? [1] suggests that a larger $\alpha$ (compared to this work) of 0.01 for text-regularization is optimal for generalizable reward learning, including testing on benchmarks like rewardbench. Also, [2,3] provides empirical evidence that DPO loss can lead to reduction in likelihoods of the winning responses. It’s reasonable to expect the SFT component helps in ensuring $\pi_ (j+ / p)$ stays relatively high during optimization, but authors here use an $\alpha$ of $1e^{-6}$ which is significantly lower than the optimal $\alpha$ in [1]. Clearly, this affects SFT-based regularization, but was this $\alpha$ value empirically determined or were there any particular motivations for choosing this value??
soundness: 3
strengths: The paper is clearly-written and the motivation is clear that it attempts to bridge preference modeling with a generative component to reduce bias as well as to improve interpretability. Section 3 is pretty clear as in the math formulation and what the crucial components are in eq.4 and 5.  The empirical evaluation also seems reasonable with a variety of LLMs chosen and data domains being selected. I also find the illustrative examples and prompt-structures to be quite helpful for reproducibility and cross-comparisons.
summary: The paper introduces Con-J, a generative judge trained via self-bootstrapped learning from preference data to overcome limitations of scalar reward models, such as lack of interpretability and dataset bias. This model is trained with a combination of direct preference optimization (DPO) loss for contrastive judgement learning with an additional SFT term over positive judgements. Experiments across various domains (Text Creation, Math, Code) and benchmarks show that Con-J performs effectively, improving rationale correctness during training. This enables accurate judgments and meaningful explanations, supporting human-in-the-loop LLM alignment. Con-J also demonstrates reduced susceptibility to dataset bias compared to scalar and rationale-free models.
weaknesses: I find this work to be a commendable engineering effort but with moderate novelty. In particular, is it not quite evident why do we need DPO’s formulation since the argument about similarity with pretraining objective and DPO’s policy being an optimal solution to the KL-constrained optimization problem is not directly addressed. Furthermore, SFT+ DPO combination for reward model training has been quite popular but I find the motivation for adding the SFT component to be less clear, especially the authors downweight this component with a relatively small weight compared to previous works. The authors argue that adding SFT avoids the reward overoptimization issue but I don't see an empirical evaluation of how this affects the log-likelihoods of pos and neg judgements in their Con-J training.

Meta-review: This paper introduces Con-J, a method for aligning LLMs with human values that uses a generative LLM to produce contrastive judgments with natural language rationales, addressing the limitations of traditional scalar reward models. Training this "Judge" with Direct Preference Optimization results in improved interpretability and robustness against biases, outperforming scalar models and other generative LLMs in experiments. The authors release their code and trained models. Most reviewers appreciate the novelty of the approach (one reviewer disagrees), the clarity of presentation, as well as the evaluation and results. However, the reviewers also raise some concerns, primarily regarding the rationales and interpretability (the analysis and conclusions is not convincing) and the analysis of consistency and correctness. The authors respond to all concerns and questions and one reviewer increases their score; not all reviewers engaged in discussion.

Decision: Accept (Poster)