Learning LLM-as-a-Judge for Preference Alignment
Authors: Ziyi Ye
Xiangsheng Li
Qiuchi Li
Qingyao Ai
Yujia Zhou
Wei Shen
Dong Yan
Yiqun LIU
Venue: ICLR 2025 Poster (2025)
Keywords: LLM alignment
Generative judge
LLM-as-judge
Reward model
Primary area: generative models
TL;DR: We devised a generative judge that learns from preference data, demonstrating better interpretability and robustness compared to traditional scalar models for encoding preferences.

Abstract: Learning from preference feedback is a common practice for aligning large language models (LLMs) with human value. Conventionally, preference data is learned and encoded into a scalar reward model that connects a value head with an LLM to produce a scalar score as preference. However, scalar models lack interpretability and are known to be susceptible to biases in datasets. This paper investigates leveraging LLM itself to learn from such preference data and serve as a judge to address both limitations in one shot. Specifically, we prompt the pre-trained LLM to generate initial judgment pairs with contrastive preference in natural language form. The self-generated contrastive judgment pairs are used to train the LLM-as-a-Judge with Direct Preference Optimization (DPO) and incentivize its reasoning capability as a judge. This proposal of learning the LLMas-a-Judge using self-generated Contrastive judgments (Con-J) ensures natural interpretability through the generated rationales supporting the judgments, and demonstrates higher robustness against bias compared to scalar models. Experimental results show that Con-J outperforms the scalar reward model trained on the same collection of preference data, and outperforms a series of open-source and closed-source generative LLMs. We open-source the training process and model weights of Con-J at https://github.com/YeZiyi1998/Con-J.

Published as a conference paper at ICLR 2025
LEARNING LLM-AS-A-JUDGE FOR PREFERENCE ALIGNMENT
Ziyi Yel, Xiangsheng Li², Qiuchi Li Qingyao ; Yujia Zhou', Wei Shen?, Dong Yan Yiqun Liu'
'Department of Computer Science and Technology. Tsinghua University
Baichhan AI *University of Copenhagen
ye-zy20@ mails. tangsutsinguua.c.. edu..n. lixshó@gmail.com qiuchi.i@di.ku..k aiqy @ssinghua.cdu.cn, zhouyujia @ @mail. maiil.tsinhhi tsinghua.ndee edu.en, shenwei09176 @ 126.com, weyshionen gmail.com. yiqunliu @ttngguu..edii enu.ccn
ABSTRACT
Learning from preference feedback is a common practice for aligning large lan- guage models (LLMs) with human value. Conventionally, preference data is learned and encoded into a scalar reward model that connects value head with LLM produce scalar score preference. However, scalar models lack interpretability and are known to be susceptible to biases in datasets. This paper investigates leveraging LLM itself learn from such preference data and serve as judge to address both limitations in one shot. Specifically, we prompt the pre-trained LLM to generate initial judgment pairs with contrastive preference in natural language form. The self-generated contrastive judgment pairs are used train the L.LM- with Direct Preference Optimization (DPO) and in- centivize its reasoning capability as judge. This proposal of learning the I.I.-- as-a-Judge using self-generated Contrastive judgments (Con-J) ensures natural interpretability through the generated rationales supporting the judgments, and demonstrates higher robustness against bias compared to scalar models. Experi- mental results show that Con-J outperforms the scalar reward model trained the same collection of preference data, and outperforms a series of open-source source and closed-source generative LLMs. We open-source the training process and model weights of Con-J at httpps//gtthub.co//////llll
INTRODUCTION
As Artificial Intelligence (Al) systems based on Large Language Models (LLMs) are increasingly used various applications, it is crucial to ensure they align with human instructions, values, and ethics. LLMs alignment is generally achieved by learning from preference data that compares pairs of responses to a question (Rafailov et al. 2024 Christiano et al. 2017 Liu et al. 2020). However, collecting high-quality human preference data both time-consuming and costly. In practice, the construction of preference datasets often involves a combination of human and Al-generated feed- back (Lee et al.) 2023 Hou al. 2024). Therefore, it crucial to develop efficient and accurate Al model for preference prediction that aligns with human values.
To obtain such preferences, industrial practices have used scalar models (Hou et 2024) that con- catenate the pre-trained LLM with a value head to produce scalar scores as preference. However, the scalar model suffers from limitations, particularly in the following aspects: (i) Lack of inter- pretability: The scalar model does not provide any support or explanation for its judgment. This hinders human involvement for its evaluation and verification. (ii) Susceptibility to bias: It is prone to capturing the biases present in the preference dataset rather than human values. For example, when the majority of positive answers preference datasets are longer sentences, learned LLM will likely favor more verbose answers (Huang et al. 2024b).
This work is supported by Quan Cheng Laboratory (Grant No. QCLZD2U2301). "Correponndig author.

---

Published as a conference paper at ICLR 2025
Instruction
As an evaluation expert, given question and its two possible answers, please choose which answer best aligns with coherence, accuracy, coverage, and overall quality. Output your judgment in JSON format, where "rationale" is your explanation, and "better_and ansswrr" is integer type of 1 2. for example, ("rationale": "your explanation", "better_anse ansswrr:: 1). Below are the question and its candidate answers:
Preamble
Question
The volume of rectangular prism 96 cubic decimeters, and the base area is 16 square decimeters. What is its height in decimeters?
following formula calculate
Height Volume Base Area 96 decimeters. Therefore, the height of rectangular prism decimeters.
Answer
height: \times Substitute the known values into the formula: \times
Answer
Scalar reward model
Generative judge: Con-J
("rationale": "Answer provides complete solution that involves basic methad and logical reasoning, Answer contains calculation error", "better_a aasserr"s"""" 1)
LLM +
Scalar score
LLM
Value head
Answer
Answer
Rationale
Preference prediction
Figure I: Top: Examples of a preamble, a question, a pair of answers, and the corresponding judg- ment (see the detailed version in Table 5). Bottom: Illustrations of scalar reward model and the proposed Con-J for preference judgment.
To address the above limitations, propose Con-J. which learns LLM-as-a-Judge using its self- generated Contrastive judgments (see Figure Con-J leverages the LLM's pre-existing judgment ability and bootstraps this ability for more accurate preference prediction. As shown in Figure 2 Con-J consists three steps: (Judgment Sampling) Sample several judgments from a pre-ttriied LLM by prompting it with query and pair of candidate answers. (Judgment Filtering) Leverage the true preference annotations to construct contrastive judgment pairs, i.e., judgments with correct or incorrect preference. (Contrastive Training) Train Con-J from the pre-trained LLM based on these contrastive judgments using Direct Preference Optimization (DPO).
The design of Con-J differs from existing methods for learning the LLM-as-a-judge (or generative judge) 'Zhang et al. (2024); Kim et al. y: Park et al. (022). These methods typically depend on external models (particularly GPT-4) or algorithmic schemes to produce high-quality instruction- tuning datasets. In contrast, Con-J directly learns from preference data using seff-bootstrapping approach without supervised fine-tuning on human instructions. ful, aligning them with high-quality instruction data becomes more difficult since humans may not always be able to write superior instructions. Instead, Con-J offers new way by eliciting what the LLM already knows, supervised by human preferences, which are much easier to obtain than high-quality human instructions.
L.MS become more power-
We train and evaluate Con-J self-built commercial datasets across three domains: Creation, Math, and Code. We observe that Con-J not only outperforms the scalar model but also significantly surpasses GPT-40 across three domains. At the same time, Con-J generate rationales support its preference prediction. We conduct human and machine evaluations on the rationales terms of their correctness, i.e.. whether the rationale provides prediction, and consistency, whether the rationale expresses the same preference as Con-J's final preference prediction. Experimental results show that preference learning improves the correctness of Con-J's rationales. Additionally, we found that Con-J is less susceptible to dataset biases, which we attribute the regularizing effect inherent in Con-J's generative training target. To facilitate further research and development within the community, we train and release public version of Con-J on publicly available datasets, which outperforms series of existing open-source L.LMs.
accurate analysis supporting the preference
To summarize, our contributions are: (1) We propose Con-J, an approach that trains an LLM-as-a- judge using a self-bootstrapped technology to learn from preference data. (2) We show that Con-J can offer not only more accurate preference prediction but also more accurate rationales, unrav- eling a self-evolution process during preference learning. We also provide theoretical motivation and empirical evidence showing that Con-J is more robust to dataset biases due its generative training target. (3) We test the performance of Con-J in commercial datasets and publicly available benchmarks. Con-J outperforms the scalar models and series of existing LLMss.
2

---

Published as a conference paper at ICLR 2025
2
PRELIMINARY
2.1 TASK DEFINITION
Given a question prompt and pair of assistant responses a' and a², the task is to judge the preference between a' and a². To accomplish this, we train the model using an existing preference dataset D {(g. a a ,a+)) where a is preferred answer compared a The model's per- formance is subsequently evaluated on separate, non-overlapping preference dataset by measuring the accuracy of its preference judgments.
2.2 SCALAR MODEL.
The most common practice for getting the preference judgment is to use scalar model (SM) similar the reward model the RLHF stage (Hou al. (2024)). The SM predicts numerical scores r(q, a) for E {a) and judges the preference by comparing r(q, a') and r(g. (²g.a"). It is typically initialized by concatenating pre-trained LLM with a randomly initialized shallow MLP head. The most widely used training objective for the SM follows the Bradley-Terry model, which maximizes probability of at being preferred:
exp(r(q,at))
P(a+ > lg) exp(r(q,at)) + exp(r(q,a-)) (((q.
(1)
where a is the sigmoid function.
The moove-mentioned SM utilizes the prompt ( and single answer as input, which we denote as pointwise SM. In addition, existing research has investigated pairwise variant that uses pair of candidates as input, i.e., r(g, a²)a²) (Jiang et al. 2023). The pairwise vanilla reward formalizes the preference probability of at as:
P(a > aq) a(rlg, )).
(2)
To train the tlove-mentioned pointwise and pairwise SM r, previous studies maximize the log- likelihood of the preferences by minimizing the following loss function:
lagp-(a* x a (2) {
Σ (aatta a-) log (r(z, a+) -r(xx (pointwise) (pairwise)
(R(T) = Σ
(xaataa>) log ar(e,at
aa)))))aaa)))))))))))))))))))))) (orr
(x.at
(3)
2.3
LLM-AS-A-JUDGE
Instead of using scalar model for preference judgment, existing literature also leverages the LLM itself as a generative judge to make preference judgments (Guo et al. (2024); Zheng et al. (203)) : Li
al. 2024b); Gu et 20244: Li al. (2024a)). Given the question g and a pair of answers at and a², a prompt instruction that describes the task and asks LLM π to act as a judge (see examples Appendix (5).
constructed by concatenating a preamble with a and a² The preamble is an
Then the LLM generates natural language judgments j = π (p). Next, regular expression matching adopted extract the preference prediction of the judgment j.
Some of the existing LLMs-as au-ugesss will generate supporting rationales before making the pref- erence prediction (Li et al. 2023), while others directly output judgments according to template (Shiwen et al.) 2024). In our experiment, we construct the preamble by modifying the prompt from the existing research (Lee et al. 2023). The preamble requires the L.LMs to output the judgment in JSON style: key named "rationall includes step-by-step reasoning and explanation, and another key named "better
answer" "betterranswer" indicates the LLM's binary preference.
IMPROVING LLM-AS-A-JUDGE BY TRAINING ON CONTRASTIVE JUDGMENTS
As shown in Figure the construction Con-J consists of three steps: judgment sampling, judg- ment filtering, and contrastive training.

---

Published as a conference paper at ICLR 2025
Preference Dataset
5, Judgments Sampling
S Judgments Filtering
S₁ Contrastive Training
Instruction
repeateddhintt driven sampling
correct
J
i'E (1) E (J)u (*)
DPO Loss
Preamble Question
Judgment
Answer Answer
Pre- trained LLM
Rationale
wrong
J
Contrastive
LLM Post-
Hint
Preference
prediction
J
Judgments
train
Preference
Preference
Loss
Answer
based filter
unknown
The steps for constructing Con-J with preference dataset that includes preference an- notations for a pair of answers to question. Sy: Prompt Con-J to generate multiple judgments for a pair of answers by repeated sampling and hint-driven sampling. a preference prediction and supporting rationale (see Figure | for example). Sy: Bootstrap contrastive judgment pairs by filtering with true preference labels. S3: Train Con-J using the DPO loss on contrastive judgments and the SFT loss positive judgments.
Figure 2:
judgment composed of
(Si) Judgment sampling: We construct contrastive judgment pairs by prompting the LLM gen- erate multiple judgments. As shown in Figure 2 this is achieved by (1) repeated sampling and (2) hint-driven sampling. Repeated sampling prompts the LLM to generate multiple outputs from the same prompt, each utilizing different random seed. However, the LLM may produce only one-sided judgments (i.e., all judgments preferred a a²) across all repeated samples, making impossible construct contrastive judgment pairs. Therefore, we propose hint-driven sampling to compel the I.LM to generate judgments that prefer specific answers. Essentially, the L.M is pro- vided with an explicit indication of which answer is better, and is instructed to generate the judgment accordingly in JSON format as above. The prompt template for hint-driven sampling is provided in Table 6 By manipulating the hint, we can get contrastive judgment pair for any prompt input.
(S2) Judgment fiterin:: We denote the outputs from repeating sampling as M (p). M (p) can poten- tially include both "positive" and "negative" judgments. A positive judgment indicates the prefer- ence prediction corresponding to the keyword "better answer" is correct (i+), while a negative judgment (j indicates the preference prediction is incorrect (j*) the model doesn't explicitly indicate its preference (j Contrastive judgment pairs {(i+,] }}>) are then constructed as the direct product of the positive judgment set M (p)+ and negative judgment set (p) We set the number of repeated samplings to 8, allowing for the construction of up to 4 contrastive pairs (in the optimal case, there exist positive and negative judgments among the 8). For hint-driven sampling, we prompt the LLM with one correct and one incorrect hint and construct one pair from its judgments. The detailed sampling and filtering process is outlined in Algorithm
(Sa) Contrastive training: Based on the contrastive judgment pairs D = {(g. at a
{ we train the LLM with direct preference optimization (DPO) loss function:
x(j(lp) log TTj (p)
DPO
Σ log a log (i+ (f+|p))
(4)
To(j 1p)
(p.ft
where To is the reference model initialized as the base L.M and remains untrained. Following existing practice (Hong et al., [2244] Pal al., [2024] Yang et al.) 2024b) Huang al. 22024a] Cen et al., (024), the DPO also fuses a small weight of supervised fine-tuning (SFT) loss to help mitigate the overoptimization issue (Liu al. 2024, Fisch al. 2024), which can be formulated as:
SFT
log m()+(p) (P.+)
(5)
Then linearly combine the DPO loss and the SFT loss with small weight
ffinll
DPO + a SFT
(6)
DPO training enables LLM to better distinguish different answers. Existing methods to im- prove the judgment accuracy of LLM generally based on supervised fine-tuning (SFT) (Kim) et al. 2024; Zhang et al.) 2024 Li et al. 2023) to imitate correct judgments. However, we empir- ically find that SFT only insufficient (see Setion 4.. Intuitively, LLMs should identify the more

---

Published as a conference paper at ICLR 2025
important aspects of judgment, rather than patterns that may appear in both positive and negative judgments (Park et al. 2024). For example, both "answer has logical errors, so the better answer is 2" and "answer 2 has logical errors, the better answer is 1" could be positive judgments for different prompt inputs, even though they are opposite in meanings. When L.M is trained with SFT loss, it may primarily imitate the common pattern that appears in both answers rather than devel- oping the ability to make judgments based on the prompt. Existing evidence | even suggests that the likelihood of generating negative output might even surpass that of positive output during SFT training.
Rationales bring robustness against bias. The proposed Con-J can generate rationales in addition to the binary preference prediction. We suggest that training the model to generate rationales can impart a regularization effect and help avoid potential biases in the datasets. Here we provide a theoretical motivation for this effect. We decompose judgment j into j. and jy. representing the rationale and the binary preference prediction, respectively. Adding the rationales as training targets can be formalized by introducing intermediate variable j. influencing the conditional probability Poliy p):
Paliy 1p) - E jr, Po(ir p)
(7)
By including rationales, the bias in preference data is distributed between jy and jr, reducing its direct impact on Jy. We can formalize the loss function as:
((0)
Σ (D.iED
log Polju fr.
Σ (p.i)ED
log Po p)
The loss Po(rr p) encourages the model to find effective representations for predicting jr. exerting a regularizing effect compared to directly predicting the preference jy-
LLM-as-a-Judge resists bias with LLM's prior Icarning from pretraining. SM modifies the LLM's architecture with classification head and uses discriminative training target. On the contrary. Con-J uses an architecture consistent with the pertaining process and generative training objectives (Zhang al. (2024)). We refer to Erhan al. (209) and assume the parameter inherited from the pre-trained LLM as adding an infinite penalty:
> 2 0011
(()) = Eatata (0)
where À is the regularization strength. We make an ideal hypothesis that there exists an optimal 0* which fully encodes human values and consistently makes true judgments. Hence we assume that the parameters obtained during the pre-training phase are closer 8 than a random distribution. This analysis suggests Con-J gets a smaller penalty term for optimization towards 0*. In contrast, SM adopts different training objective and introduces a randomly initialized head, making the regularization effect less significant (smaller and partially randomly initialized 0o). Hence, SM can be good at encoding the knowledge reflected in the preference dataset but is also more sensitive to its bias than Con-J.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. We train scalar models and Con-J on three datasets within different vertical domains: Cre- ation, and Math, then evaluate their performance in terms of the accuracy of preference predictions. The datasets are self-built commercial datasets consisting of approximately 120,000, 50,000, and 50,000 preference samples for Creation, Math, and Code, respectively. The Creative dataset involves tasks on text creation such as writing poetry or crafting headlines. The Math dataset mainly includes middle and high school math problem-solvine tasks. The Code dataset comprises code-writing tasks in various programming languages and includes code-related problems. These datasets cover diverse sources, ranging from data generated by a commercial ChatBot, data generated by ChatGPT, and
https://gttuu.com// .aaafa/SFFT function learning
5

---

Published as a conference paper at ICLR 2025
data from open-source datasets like HH-rlhf (Bai al. 2022) and Infinity- -nntity-IIstrucel The prefer- ence annotation for these datasets is gathered from human annotators, with each sample annotated by one annotator and subsequently verified by another. In addition to the self-built datasets, train an open-source version of Con-J on a publicly available dataset
wewoor-Reward-reffeeee public benchmarks including "hinity-Preference""" UltraFeed- back (Cui et al., 2023) (select its test set according to HuggingFaccHH4 PKU-SafeRLHI' (Ji et al,, 2024). and Reward-Bench (Lambert et al. 2024). We ensure that no identical prompts appear in both the training and test sets by filtering them out of the training set.
80K-v0.1 andd test its performance
Model. We select Qwen2-7B Bren2-7B-Instruct (Yang et al., 2024a) as the base model train both the scalar model (named SM) and the proposed model (named Con-J). SM includes both pairwise and pointwise variants. Additionally, we included the original pre-trained Qwen2-7B-Instruee as an untrained variant of Con-J. We also compare Con-J with a range of I.L-s-as-Juudeee including GPT- 4c and two LLMssas-Judgee Auto-J (Li al. 2023) and Prometheus (Kim et al. 2024) trained by SFT, Llama series (Llama3. 1-8B, and Llama3. 1-0BB,, and Qwen series (Qwen2-7B. Qwen2.5- 72B). The details about hyperparameters, sampling, and sampling strategy Con-J provided in Section
4.2
MAIN RESULTS
Table presents the results of SM and Con-J on the self-built commercial datasets across three ver- tical domains. It can be observed that (i) there no significant difference between the pointwise and pairwise variants of SM. Although existing search suggests that concatenating the list of re- sponses improves performance for scoring the re- sponses (Jiang et al., 2023), we do not observe this effect our datasets. (ii) Both Con-J and SM out- perform off-the-shelf GPT-4o, indicating that small models trained on domain-specifie data effec- tively predict domain-related preferences. (iii) On the same preference datasets, Con-J consistently out- performs SM across all tasks with a significant gap on the Text Creation task. This indicates that Con-J is more effective at acquiring accurate judgment abilities than SM.
Table 1: Judgment accuracy of GPT-4o, SM, and Con-J. * indicates the performance dif- ference between Con-J is significant at p < 0.05 using a pair-wise t-test.
Model
Creation
Math
Code
GPT-4o
55.6*
74.8"
68.1*
SM (point-wise)
69.4"
84.8
69.4
SM (pair-wise)
69.2"
84.6
69.6
Con-J
72.4
85.0
70.1
We carry out an ablation study to investigate the vari- ants of Con-J. Given that current methodologies of- ten employ Supervised Fine-Tuning Tuning (SFT) to train LLMs-as-Judeee (Li et al. 2023 Kim et al. 2024). we developed an SFT variant Con-J, trained ex- clusively positive judgments using SFT loss. As illustrated in Table 2 Con-J trained with our proposed framework outperforms its variant with- out DPO loss across all datasets. This observation demonstrates the effectiveness of training from con- trastive judgments. In addition, Con-J outperforms its variant without hint-driven sampling, which re- lies solely repeated sampling and may be infeasi- ble construct contrastive judgment pairs for some prompts. Similar findings have been observed when using self-taught techniques to improve LLMs
Table 2: Judgment accuracy of Con-J and its variants. indicates the performance differ- ence between Con-J is significant at p < 0.05 using pair-wise t-test.
Model
Creation
Math
Code
Con-J untrained
33.6*
63.4"
61.7"
Con-J w/o Hint
61.3*
77.4*
68.2
Con-J w/o DPO
54.6*
64.2*
63.5*
Con-J
72.4
85.0
70.1
Zelikman et al. 2022). More variants of Con-J were tested and detailed Section A.2
http://hwgggggeec.oooo////ss///s Bere--aeeaaa--Peeeeee----------- http::/hhwgggggaaccc.oooouaaaaaa Preference
binarized

---

Published as a conference paper at ICLR 2025
Table 3: Accuracy of generative judges on the sets of four benchmarks: aafinity-Prefercce UltraFeddback, PKU-SafeRLHE, and Neward-Bench Results in bold are the best among all models and results with underline are the second-best.
Infinity- Preference
Ultra- Feedback
PKU- SafeRLHF
Reward-Bench Safety
Model
Chat
Chat-H
Reasoning
Llama3.1-8
59.0
62.9
66.4
80.7
49.8
64.0
68.1
Llama3.1-70 1-70B
64.0
71.4
67.6
97.2
70.2
82.8
86.0
Qwen2-7B
59.0
64.5
67.2
91.3
44.8
73.6
69.0
Qwen2.5-72B
70.0
66.0
58.7
86.6
61.4
74.5
90.7
Auto-J
69.0
63.9
66.9
93.0
40.0
65.5
50.5
Prometheus 2
68.0
63.3
63.0
85.5
49.1
77.1
76.5
GPT-4o
75.0
72.2
69.6
95.3
74.3
87.6
86.9
Con-J (ours) 81.0
73.0
68.4
91.3
79.6
88.0
87.1
For fair comparison with other LLMs-as-Judges,, we tested the open-source version of Con-J on public benchmarks, as shown Table 3 Con-J outperforms existing open-source LLMs-as-Judees in the vast majority of benchmarks, including commercial instruction-tuned models such as Llama series, Qwen series, and a series LLMM-as-Judgee trained specially for preference judgments such Auto-J and Prometheus 2, except in the Chat and Reasoning sub-task of Reward-Bent tionally, Con-J achieves comparable performance with the closed-source model GPT-4o across all benchmarks. Specifically, it surpasses GPT-4o in the Infinit--Preferenn and UltraFeedback bench- marks and also outperforms it in the average performance of the Reward-Bence
Addi-
4.3 PREFERENCE LEARNING YIELDS MEANINGFUL AND USEFUL RATIONALES.
We investigate the quality of rationales in the Math dataset because the evaluation of math problem- solving tasks is relatively more objective. We select 6 checkpoints trained on different numbers of contrastive judgment pairs, i.c., 2k, 4k, 8k, 16k, 32k, and 64k, with 64k being the final checkpoint. Then we prompt GPT-4o score the rationale's correctness (1 to 5) and its consistency with the predicted preference (1 to 3) (see the prompt in Table 99. Additionally, GPT-4o is tasked as meta-judge to judge whether Con-J makes correct preference predictions. If the judgments of GPT- 4o conflict with the dataset's true preference annotations, we exclude these questions for further analysis, as these questions may exceed GPT-4o's capabilities.
Experimental results are presented in Figure From Figure Baa, we observe that the correctness of the rationales improves when Con-J is trained with more data and achieves increased pref- erence prediction accuracy. However, we find that the consistency between Con-J's preference prediction and its rationales decreases with the increase in judgment accuracy, as shown Fig- ure Bj... These observations indicate that Con-J's abilities to make binary preference predictions and generate correct rationales both improve with training. However, the increase in inconsistency indicates that these improvements may not be balanced. We suspect that supervision from prefer- ence datasets, focused solely on binary preference prediction, enhances Con-J's binary prediction ability more than its reasoning ability, leading inconsistencies.
We further test whether the improved rationales can be used to help a weak judge make more accu- rate preference predictions. We use the untrained Con-J as the weak judge and prompt it with the rationales generated by the stronger model. As shown in Figure B(c), the weak judge yields more accurate preference prediction with the rationales provided by stronger model. Additionally, we find that rationales generated by GPT-4o can also improve the weak judge, with performance com- parable to the rationales from the strongest checkpoint of Con-J. This indicates that Con-J not only surpasses GPT-4o in preference prediction performance but also generates rationales with similar effectiveness.
Motivated by the above observation and the fact that the training of Con-J is only supervised by preference labels, we further investigated whether controlling the quality of the rationales can help Con-J make better preference prediction, which is elaborated in Section A.4 Experimental sults indicate that by controlling the quality of Con-J's rationale, we can not only improve the quality of the rationale but also further enhance the accuracy of the preference prediction.

---

Published as a conference paper at ICLR 2025
preference Prediction Agc
Rabionale Correciness
4.35
Rationale Consistency
0.80
Viece
act
2.9
0.75
0.70
8.20
0.65
4.15
0.60
0.60
0.65
0.70
0.75
0.80
0.65
0.7
0.75
0.8
Preference Prediction ACC
Preference Prediction Acc
Checkpoint
(a)
(b)
(c)
*) trained with different number of con- trastive judgment pairs 16K,3 (a-b) We prompt GPT-4o to evaluate the correctness of the rationales and their consistency with the predicted preferences. (c) We use the rationale from strong model as input to help weak model in its preference prediction.
Figure 3: We analyzed 6 checkpoints
0.5
Can-
Accurnty
Con J (wlo retionato)
0.0
0.3-
Accurro
0.7-
0.2-
0.3-
0.
0.50
1:00
0.50
1:00
0.50 0.67
1-Y
1-y
1-y
1-y
(a) Format (Adversarial)
(b) Format (General)
(c) Verbosity (Adversarial)
(d) Verbosity (General)
Figure 4: The performance of Con-J, Scalar model, and Con-J (without rationale). trained with varying degrees of training data bias in format and verbosity, where - 7 1 indicates bias.
The performance was evaluated on two test sets: an adversarial set, which injected inversed bias treatments with the training data, and a general set, which the original test
4.4 RATIONALES HELP CON-J BE MORE ROBUST TO DATASET BIAS.
Scalar models are known be susceptible biases in datasets, resulting in preference judgments that reflect biases in the pattern of data rather than true human values. To investigate the suscepti- bility of Con-J, we conduct a synthetic experiment that injects artificial bias into the data. We define the degree of bias in training set Prrain as 7. This indicates that the y proportion of the data contains unexpected biases that don't necessarily reflect human value, while the rest - y of the data are randomly sampled from the original training set.
We consider two widely studied biases: format bias and verbosity bias (Park et al., 2024; Singhal et al., 2023 Shen et al., 2023) (see details in Section A. All experiments are conducted on filtered Math dataset, where the filtering ensures that the solution in the preferred answer is correct, while incorrect in the less preferred answer. In this scenario, the correctness of the solution is prioritized over the format and verbosity of the answers. We investigate the performance of Con- J, SM, and Con-J trained and evaluated without (w/o) generating the rationales (see the prompt examples Table 7 under the injections of the addressed bias. We trained them on a biased dataset and tested their performance on an adversarial test set Dadversarial and a general test set general Dadversarial is constructed by injecting contrary bias the test set. For format bias, the contrary bias means rewriting the preferred answer with a more formal format and the less preferred answer with a more verbal format. For verbosity bias, the less preferred answer is rewritten to be more verbose using the same procedure applied to the preferred answer the training set. Deennerl is the original test set without any modification. All models achieved accuracy of over 99% on the biased training dataset D Prain when 7 = 1, indicating the models are thoroughly trained and are strongly influenced by the bias.
We present the experimental results in Figure 4 a-d) with the following observations:
(i) Con-J outperforms SM when trained biased data. Intuitively, Con-J does not train value head from scratch; instead, it leverages the text generation capabilities of pre-trained LLMs. We further investigate this performance gain in the context of format bias (adversarial), with 0.33. We
8

---

Published as a conference paper at ICLR 2025
Table 4: Example of rationales and binary preference generated by Con-J. We prompt GPT-4o to annotate the correctness (ranging from to 5) of the rationales and the consistency between the predicted preference and the rationale (ranging from to 3). Refer to more examples in Table 11
Prompt input
Rationale
Preferred
Question: What is the range of the oumeric output of sigmoid node in
-1 in the first answer is incorrect be- cause the outpat range of the sigmoid function to I. not including -L. Correctness: 5, Consistency:
Math (1)
neural network?
Answer
Answer The output of sigmoid node bounded berween and The output of a sigmeid node is bounded between D and 1.
Answer 2:
(correct)
Question: Soldiers built 2000-meter linc. They averaged 350 metors per day for the lirst 3 days. What is the required daily average for the
The first answer is straightforward and accurate. taking into account the total distance that needs be up and the known information: the second an- swer offers slightly simplified
remaiming days?
Answer
Math (2)
Answerl: (2000- - 350 x 3) $ 2 1400 + = 700 meters, Anewer:
average of 700 meters needs to
up each day for the last two days.
(correct)
Answer 2: (2000 - 350 x 3) + = (2000 - 1050) 2 = 950 =
pression and the resalt is correct. Correcmess: 2, Consistency:
475 meters. Answer: 475 meters for last two days.
observe that when Con-J outperforms SM, the untrained model achieves 85.6% accuracy compared to 49.8% when Con-J underperforms SM. This suggests that Con-J is likely to generate rationales aligning with preferences from its pre-trained data. However, this does not mean that Con-J's ability to resist bias compared to SM is entirely due to the initial model. As shown in Figure 4(b), Con-J's performance noticeably better than the initial model when 7 <= 0.67. This indicates that Con- J can not only retain knowledge from pretraining but also extract useful judgment standards even from biased data. However, this phenomenon also raises potential challenges when we need to align LLMs to preferences with fundamentally different values principles, which we left as future work.
(ii) When tested the adversarial datasets, Con-J w/o rationales significantly underperforms Con-J when y > 0.2 and y > 0.33 for the format bias and the verbosity bias, respectively. This indicates that Con-J becomes more robust at learning from biased data through training with rationales.
(iii) When tested on general test sets, Con-J w/o rationales demonstrates comparable performance to Con-J with rationales. We conduct further analysis under the verbosity bias with y=1.0 which categorizes the data samples into groups where the chosen answer is longer or shorter than the re- jected answer. We find that Con-J w/o rationale achieves an accuracy of 47.14% when the chosen answer is shorter than the rejected answer, whereas Con-J achieves an accuracy of 58.7 71% % (see Ta- ble p2p. This implies that, although the average performance of Con-J and Con-J w/o rationale is comparable, Con-J w/o rationale is more significantly impacted by bias.
(iv) Some existing research suggests that rationales or critics generated ahead of preference judg- ments can facilitate Chain-of-Thouglle (CoT)) reasoning and help the L.M make better judgments
(Lee al. 2024; Annknr al., 2024] Ye et al., 2024 Zhang et al. 2024p. A possible explanation for the lack of a similar CoT effect in our dataset is that the CoT process is often already embedded
in the responses, making the CoT procedure for judgment potentially unnecessary, which we further discussed in Section A.1
4.5 HUMAN EVALUATION & CASE ANALYSIS
We further conduct a human evaluation on the Math dataset to investigate the quality of the ratio- nales (see Section A.3 for details about annotator recruitment and analysis). The human evaluation involves two steps: (1) a quantitative annotation to assess the rationale's 's correctness and consistency using Likert scales, resembling the annotation performed by GPT-4o in Section 4.3 and (2) a qual- itative analysis where annotators openly discuss the main issues that exist in the rationales. For the quantitative annotation, Krippendorff's a values between GPT-4o and human annotators are 0.4427 for correctness and 0.6495 for consistency, indicating moderate and substantial agreement, respec- tively. This suggests a consistent trend between evaluations made by humans and GPT-40, despite some remaining disagreements. Such differences may arise from the limitations of GPT- 4o. For example, we observe that the a for correctness annotation is pretty low (0.1301) in cases where GPT-4 fails to make correct preference predictions.
In the qualitative analysis, annotators agreed that most of the rationales were correct and showed high consistency with the preference prediction, indicating that these rationales can support Con-

---

Published as a conference paper at ICLR 2025
J's preference prediction (e.g., Math (1) in Table 4). In addition to the high-quality rationales, especially focused on the cases where the rationale was assigned a low consistency (consistency or 2) or low correctness (correctness not higher than 4). The annotators deemed that in these cases, the rationales provided by Con-J may fail to identify key differences between the two answers, and therefore express explicit attitude toward either answer (e.g.. Math (2) in Table 4 Indeed, observed that the average accuracy of the preference prediction for low consistency cases is 59.9%, which significantly lower than the average performance (85.0%) but higher than chance. This indicates that Con-J sometimes struggles to provide a high-quality rationale, even when it makes a correct preference prediction.
5 RELATED WORK
LLM alignment. The initial approach developed for aligning LLMs with human values was rein- forcement learning from human feedback (RLHF) (Christiano et al. 2017 Liu et al.) (020). This technique involves training scalar reward model (RM) and then using reinforcement learning (RL) to optimize a policy according to the RM. Recently, a series of direct alignment from prefer- ence (DAP) works, such as DPO (Rafailov al.] 2024). SiLC (Zhao al.] 2023), and IPO Azar et al. (2024), have gained popularity. Unlike RLHF, DAP methods directly update the LLM using preference data, which is simpler and more stable. To scale the preference datasets, it is common to train an external machine model from existing preference datasets (Hou et al. (2024); : Wu et al.
(2024)) This online and scalable construction process enables DPO to be deployed in an itera- tive setting (Xiong et al. (2024); Xu et al. (2023)) or online setting (Guo et al. (2024)). Therefore, developing an accurate external model for preference judgment is a significant problem.
LLM-as---juddee Instead of training scalar model for preference judgment, employing LLMs as a generative judge has been a promising alternative (Zheng et al. 2023 Ye et al.; 2023). Efforts have been made to train language models specialized in evaluations. For example, Li ct al. (023))
Kim et al. (2024) construct instruction-tuning datasets by prompting GPT-4 and using supervised fine-tuning (SFT) train pre-trained LLM generative judge. Mahan et al. (2024) utilizes
similar seff-bootstrapiig idea and investigates the feasibility of utilizing such generative judge into
RLHF pipeline. Our contribution to the existing research is that Con-J uses self-sampled contrastive judgments under the supervision of preference data, with experiments showing its robustness and interoperability.
6 DISCUSSIONS AND CONCLUSIONS
We introduced Con-J, novel approach that trains an LLM-as-a-Judge by self-bootstrapped learning from preference data. Con-J addresses the limitations of scalar reward models, including lack of interpretabilityy and susceptibility to dataset bias. Our experiments commercial datasets across Text Creation, Math, and Code domains, as well as publicly available benchmarks, demonstrate the effectiveness of Con-J. Moreover, show that the correctness of the rationales generated by Con-J improves during learning from preference data. This enables Con-J not only to make accurate judgments but also to provide reasonable explanations, potentially facilitating nouaa--n--the-loge supervision of LLM alignment. Finally, we found that Con-J is less susceptible to biases in datasets compared to its variants without rationales and the scalar models.
As Al systems become more powerful, many suggest that they will reach the point at which human are unable to easily and reliably assess the quality of their outputs (Casper al. 2023). To address this issue, using another AI to supervise itself is viable solution; however, researchers suggest that these methods may fail without human involvement (Shumailov et al., 2024). This paper contributes to addressing this issue in two ways. On the one hand, Con-J can be used to supervise LLMs by acting as a judge. At the same time, Con-J produces an explanation of its output that is legible to humans or another trusted system. This indicates that we can spot errors made by Con-J. On the other hand, the training and construction of Con-J rely solely on preference data, which is easier to acquire from human annotators than high-quality instruction tunning data. In many cases humans often find it difficult to provide verbal reasons for their preference, the training of Con-J could be integrated with human preference annotations, thereby reducing human effort for LLM alignment.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D Chang, and Prithviraj Ammanabrolu. Critique-out-loud reward models. arXiv preprint arXiv:2408. 1791, 2024.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447 4455. PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint FxXx:2200400882. 2022.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307. 15217, 2023.
Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang. Dale Schu- urmans, Yuejie Chi, and Bo Dai. Value-incentivized preference optimization: A unified approach to online and offline rlhf. arXiv preprint arXiv:2405. 19320, 2024.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv: 16041660444444 74, 2016.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16359 2022.
Dumitru Erhan, Pierre-Antoine Manzagol, Yoshua Bengio, Samy Bengio, and Pascal Vincent. The difficulty of training deep architectures and the effect of unsupervised pre- pre-training. In Artificial intelligence and statistics, pp. 160- PMLR, 2009.
Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. Robust preference optimization through reward model distillation. arXiv preprini arXiv:2405.1933 2024.
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuchao Zhai Chengjin Xu, Wei Li, Ying- han Shen, Shengjic Ma, Honghao Liu, et al. survey on lmm-as-a-judge. arXiv preprint arXiv:2411.15594 2024.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, and Mathieu Blondel. Direct lan- guage model alignment from online ai feedback, 2024. URL 04792]
org./brgaa.. abs/
Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, and Yuri Burda. Prover-verifier games improve legibility of ml outputs. arXiv e-prints, pp. arXiv-2407, 2024.
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.0769) 2(4):5, 2024.
Zhenyu Hou, Yii Niu, Zhengxiao Du, Xiaohan Zhang, Xiao Liu, Aohan Zeng, Qinkai Zheng, Minlie Huang, Hongning Wang, Jie Tang, et al. Chatglm-rlhf: Practices of aligning large language models with human feedback. arXiv preprint 24Xv:24444000934 2024.

---

Published as a conference paper at ICLR 2025
Jian Hu, Xibin Wu, Weixun Wang, Dehao Zhang, Yu Cao, al. Openrlhf: An easy-to-use, scalable and high- perfpprrfomaceee rlhf framework. arXiv preprint arXiv:2405. 1114,, 2024.
Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D Lee, Wen Sun, Akshay Krishnamurthy, and
Dylan J Foster. Correcting the mythos of kl-regularization: Direct alignment without overparam- eterization via chi-squared preference optimization. arXiv e-prints, pp. arXiv-2407, 2024a.
Hui Huang, Yingqi Qu, Jing Liu, Muyun Yang, and Tiejun Zhao. An empirical study of Ilm-as- a-judge for IIm evaluation: Fine-tuned judge models are task-specific classifiers. arXiv preprint Seti:22003.22855 2024b.
Jiaming Ji. Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Qiu, Boxun Li, and Yaodong Yang. Pku-saferlhf: A safety alignment preference dataset for llama family models. arXiv preprint arXiv:2406. 15513, 2024.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.0256) 2023.
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. arXiv preprint arXiv:2405.01553. 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling, arXiv preprint arXiv:2403. 13787, 2024.
Gyeong-Gevt Lee, Ehsan Latif, Xuansheng Wu, Ninghao Liu, and Xiaoming Zhai. Applying large language models and chain-of-thouge for automatic scoring. Computers and Education: Artifi- cial Intelligence, 6:100213, 2024.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00 00267,, 2023.
Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:241 16594, 2024a.
Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou Qingyao Ai, Ziyi Ye, and Yiqun Liu. Lims-as-judges: A comprehensive survey lim-based evaluation methods, arXiv preprint arXiv:2412.05579 2024b.
Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating aligmment. arXiv preprint arX.v:2310.05470, 2023.
Fei Liu al. Learning to summarize from human feedback. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in rlhf: Your sft loss implicitly an adver- sarial regularizer. arXiv preprint arXiv:2405. 2024.
Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprint arXiv:2410.1 12832, 2024.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402. 13228, 2024.
12

---

Published as a conference paper at ICLR 2025
Junsoo Park, Seungyeon Jwa, Meiying Ren, Dacyoung Kim, and Sanghyuk Choi. Offsetbias: Lever- aging debiased data for tuning evaluators. arXiv preprint arXiv:2407.065511 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Perfor- mance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti-
mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp. -0006-306,, 2020.
Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. arXiv preprint arXiv:2310.00 10.05199, 2023.
Tu Shiwen, Zhao Liang, Chris Yuhao Liu, Liang Zeng, and Yang Liu. Skywork critic model series. https: httgs:://huggingffcee co co/Skyworkk September 2024. URL https:// hwwgingface.cc///o0 Skywork
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal.
Ai models collapse when trained on recursively generated data. Nature, 631(8022)755-755,,
2024.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023.
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with llm--s-a-metta-jddee arXiv preprint arXiv::407. 20t.::2400..9944,4 2024.
Wei Xiong, Hanze Dong. Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang- Iterative preference learning from human feedback: Bridging theory and practice for rihf under kl-constraint. In Forty-first International Conference on Machine Learning, 2024.
Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682. 2023.
An Yang. Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024a.
Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. Regularizing hidden states enables learning generalizable reward model for lmm. arXiv preprint arXiv:2406.1022 2024b.
Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thore, Juho Kim, and Minjoon Seo. Flask: Fine-grained language model evaluation based alignment skill sets. arXiv preprint arXiv:2307.10926 2023.
Zihuiwen Ye, Fraser Greenlee-Scotte Max Bartolo, Phil Blunsom, Jon Ander Campos, and Matthias Gallé. Improving reward models with synthetic critiques. arXiv preprint arXiv:2405.20850 2024.
Eric Zelikman, Yuhuai Wu, and Noah D Goodman. 0rXiv:2203.144633 2022.
Star: Self-taught reasoner. arXiv preprint
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408. 15240, 2024.
13

---

Published as a conference paper at ICLR 2025
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 EXPERIMENTAL SETUP
Hyper parameters. We train SM and Con-J from with the DeepSpeed library Rasley et al. 2020))
Zero Redundancy Optimizer (7eRO) Stage 3 Rajbhandari et al. (2020), gradient-checkpoiniig Chen et al. (2016) and FlashAttention Dao et al. (2022). We use bíloat16 (BF16) and tfloat32 (TF32) mix computation precision. We use cosine scheduler for the learning rate and 3% warmup. The maximum sequence length is set as 4,096. The batch sizes for SM and Con-J are set to 128 and 24, respectively, while their peak learning rates are set 9 10-6 and 5 x 10-?, respectively, in accordance with existing practices for 7B models.
For Con-J, we linearly combine the SFT loss and the DPO loss with = 1e-6. Here select the weight of the SFT component according to the reward curve of the chosen response (i.e., positive judgment in our paper). We observed the changes in the chosen reward during training and selected the smallest alpha that would not cause significant changes in chosen reward. This selection results in relatively small weight than existing work. In previous research of DPO training, the prompts, and chosen responses had diverse formats, whereas in our training, the chosen responses follow a fixed json-style template with key "rationale" and a key "better answer". So if the coefficient for the SFT component is too large, it can easily cause the chosen reward to increase rapidly, which could potentially damage the model's performance.
Sampling and inference strategy for Con-J. We use VLLM K Kwon et al. 2023) for the inference for Con-J. During repeated sampling and hint-driven sampling, we employ greedy sampling with top-p set at 0.9 and top-k at 20, with maximum output length set 512, a temperature of 1.0, and a repetition penalty 1.2. During the evaluation, we set top-p at 1.0 and a temperature of 0.0.
Note that the maximum output length is set as 512. In our experiments, the initial model needs to produce a lengthy judgment, it might get cut off, leading to an "unknown negative judgment is actually beneficial, when failing
observed that when
judgment". Rendering such judgment to do so, we empirically found that the Con-J's outputs will become increasingly lengthy during preference learning. However, simply setting fixed output length for any input may affect perfor- mance on tasks requiring longer outputs to achieve chain-of-thoupg empirical findings indicate that rationales have an insignificant CoT effect on Con-J, future work could explore the output length of an LLM beyond the scenarios studied in our research.
(CoT) effect. Although our
The design of bias injection For format bias, we use two different prompts rewrite the pair of answers (see prompt instructions in Table 88. Specifically, we rewrite the preferred answer with a more verbal format and the less preferred answer with a more formal format with GPT-4o. For verbosity bias, we follow existing practice (Zheng et al., (202)) by asking GPT-4o to rephrase the preferred answer without adding any new information and insert the rephrased text at the end of the original answer.
A.2 VARIANTS OF CON-J
We test several variants of Con-J, the variants are listed as follows: (i) Con-J untrained: the original pre-trained Qwen2-7B-Instre (ii) Con-J w/o Hint: (iii) Con-J w/o DPO: an SFT variant of Con-J, trained exclusively on positive judgments using SFT loss. (iv) Con-J w/o SFT: DPO variant of
Con-J, trained exclusively with DPO loss, i.e., the combination parameter = O. (v) Con-J w/o rationale: a variant using the prompt template Table requiring the LLM output only the binary judgment without any rationales. (vi) Con-J w/o reorder: average the performance with and without reordering answer and answer 2 in the input prompt.
Experimental results are presented in Table 10, Con-J outperforms its variant without hint-driven sampling. demonstrating the importance of ensuring all preference data can be used to construct contrastive judgments. Con-J also outperforms its variant without DPO loss. This observation demonstrates the effectiveness of training from contrastive judgments.
However, Con-J shows comparable performance with Con-J w/o SFT in Creation and Code, while outperforms Con-J w/o SET in Math. We further analyze the averaged reward of positive and neg- ative judgments (chosen and rejected) in the test set. For Con-J without SFT, we observe that the
15

---

Published as a conference paper at ICLR 2025
chosen rewards are relatively small in the Creation and Code datasets, with values of -0.73 and -0.46, respectively. However, in the Math dataset, the chosen rewards decrease to -2.37. The observation in the Math dataset aligns with previous research indicating that if SFT loss is helpful for stabilizing DPO training by preventing the decrease of the log probability of chosen response.
In addition, find that the performance difference between Con-J and Con-J w/o rationale is not significant. This indicates that Con-J can effectively learn only the preference prediction without rationales. Although some existing research suggests that rationales might have CoT effect boost preference prediction performance, this is not the case with our dataset. A possible explanation for the lack of similar CoT effect in our data is that the CoT process often already embedded in the responses, making the CoT procedure for judgment potentially unnecessary.
Finally, we find that reversing the order of answers has no significant impact on our datasets. Ac- tually, we performed random shuffle on the pair of answers when constructing the prompts. We found that Con-J's preference predictions on the test set were roughly 1:1 for the first and the second answers.
A.3 HUMAN EVALUATION
We conduct a human evaluation to evaluate the rationales. The human evaluation consists of two steps: a quantitative analysis evaluate the rationale's correctness and consistency with the final preference prediction, and a qualitative analysis where annotators openly discuss the annotation experience and the usefulness and main issues of the rationales, such as the underlying reasons for data samples with low consistency.
A.3.1 THE QUANTITATIVE ANALYSIS
In the quantitative annotation task, the annotators are required to annotate the correctness with a 5-point Likert scale and the consistency with 3-point Likert scale, which follows the instructions for GPT-40 (see Table 9). To kick off this task, the author team initially annotated 10 judgment examples generated by Con-J and conducted discussion reach an agreement the annotation criterion. Then, we recruited 6 annotators from the college students. All the annotators are graduate students (masters or PhD candidates) with mathematics skills at the level of university engineering students. Firstly, the annotators are required to read the instructions including the definitions of correctness and consistency (see Table Then, they will review the annotations for the 10 judgment examples to ensure that they have understood the annotation criterion. Then, we construct 100 data samples randomly sampled from the test set of the Math dataset, which contains a question, pair of answers, judgment made by Con-J, and judgment made by an untrained version of Con-J. Each annotator will annotate 50 out of the 100 data samples, ensuring that each sample annotated by three different annotators. To avoid position bias, we randomized the order of the data samples and the sequence of judgments (belonging to Con-J or Con-J untrained). On average, each annotation took about 3 minutes and each participant spent about 2.5 hours accomplish the annotation task. They were paid 40 dollars for their effort.
The average correctness score and consistency score evaluated by human annotators are 3.22 and 2.89, respectively. We calculated the inter-annotator agreement among human annotators and the agreement between the human majority vote and the machine annotations in terms of Krippen- dorff's a. The a values for human annotators are 0.7942 for correctness and 0,8508 for consistency. These imply that there is strong agreement among human annotators on this annotation task. Fur- ther, we calculated the agreement between the annotations from GPT-4o and the human's 's majority voting (select the median value when the three annotations are all different). The values are 0.4427 for correctness and 0.6495 for consistency, indicating moderate and substantial agreement, respec- tively. This implies that there is consistent trend between human annotators and GPT-4o, although some disagreements remain.
We further analyzed these disagreements to investigate their potential causes. First, the quality of the correctness annotations depends the ability of humans and machines to make correct preference predictions towards the pair of answers. Therefore, we categorized the data samples into two groups: those where GPT-4o, when prompted as the judge, made correct judgments, and those where it made incorrect judgments. We observed that in data samples where GPT-40 made correct judgments, its agreement with human annotators was higher than average (a =0.6201). However, in cases where
16

---

Published as a conference paper at ICLR 2025
GPT-4 fails to make correct judgments, its agreement with human annotators is low (a=0.1301) This observation suggests that we should not rely on GPT-4's 's annotation when the task may exceed its capabilities. Therefore, we excluded these samples from our experiments in Section 4.3 Sec- ond, We manually analyzed the disagreed consistency annotations made by humans and GPT-4o. We found in all of these data samples GPT-4o assigns higher consistency score than the human annotator. For example, GPT-4o may overscore some rationales that do not specify which responses are being evaluated (e.g., "The answer provided clear calculation process and gave an accurate answer, with more standard mathematical expressions and clear logical order"). This means that GPT-40 sometimes treats the rationales as consistent even if they lack of direct supporting factor for the preference prediction. We believe this issue could be solved through techniques such as few-shot prompting or reflection, which we leave as future work.
Next, we investigate and compare the annotation scores for Con-J and Con-J untrained. We found that the average rationale score generated by Con-J (3.47) was significantly higher than that of Con- J untrained (2.95), while the consistency of the rationale generated by Con-J (2.87) was slightly lower than that of Con-J untrained (2.91). The results are consistent with those evaluated using GPT-40, indicating that through preference learning, the rationales generated by Con-J improve correctness but may slightly reduce consistency. However, we also observed differences between the observations obtained from human evaluation and those from GPT-44 evaluation. For example, Con-J shows a greater improvement in correctness with preference learning under human evaluation compared GPT-4o. This difference indicates that humans and GPT-4o may have some different criteria in their evaluations, which we further discuss in the following section.
A.3.2 THE QUALITATIVE ANALYSIS
After the quantitative annotation, we further conduct qualitative analysis in which annotators openly discuss the annotation experience and the usefulness and main issues of the rationales. out of the 6 annotators who participated in the quantitative task agreed to join this discussion. In this process, annotators are presented with the annotations made by GPT-4o and other annotators. Firstly, the annotators discussed cases in which they disagreed on annotations, Most of the cases mainly arose from minor differences in the estimation of rationales. For example, there were cases where some participants gave a score of 4 (Mostly correct) while others gave a score of 5 (Com- pletely correct) for the correctness of the rationales. This occurred because participants might have had differing understandings of whether the rationale addressed the most important aspects for mak- ing the preference prediction. Further, the annotators discussed cases in which they disagreed with GPT-4o's s annotation. They generally agreed that these disagreements arise due to the mistake made by GPT-4o, particularly in overestimating rationales that have the correct attitude toward one of the answers but are based on incorrect reasoning, and in overestimating inconsistent responses as highly consistent.
Second, we analyzed the data samples with poor consistency (consistency scores and 2). We find that in these cases, Con-J often chooses to refrain from clearly expressing its preference, e.g., fuses to indicate which answer to refer to: "The answer provided a clear calculation process and gave an accurate answer, with more standard mathematical expressions and clear logical order.". and presents the weakness of both answers: "Answer 1 did not accurately understand the original mathematical problem; Answer 2 overly complicated the solution form, neglecting a more straight- forward calculation method." Furthermore, we examined the accuracy of Con-J's final preference prediction in cases where consistency scores assigned by GPT-4o were or 2. We found that the average accuracy for these cases is 59.9%, which is significantly lower than the average performance (85.0%). This suggests that low consistency generally occurs when Con-J lacks sufficient judgment ability, resulting in the generated rationale lacking an explicit attitude towards either answer and supporting the final preference prediction.
Finally, we analyzed cases where Con-J made correct preference predictions but the correctness of the rationales is not greater than 4. We found that for cases with a correctness score 4, the rationale usually has correct attitude towards either answer but might lack some important aspects. Here we provide an example case Math (4) in Table IT in which the rationale indicates the correctness of answer 1 but does not explicitly illustrate the mistake made by answer 2.
Additionally, for the vast majority of cases with a correctness score of 3 or fewer, we observe that the rationale does not clearly indicate an explicit attitude towards either answer. In this case, the con-
17

---

Published as a conference paper at ICLR 2025
sistency of these cases is also relatively low, e.g., the case presented in Table 4 Math (2). Actually, we also observe a positive correlation (Pearson's 's r=0.34) between the correctness and consistency of the rationales when Con-J made an inaccurate preference prediction. This indicates that Con-J is struggling in making the preference prediction for the task, resulting in the generated rationale showing bad quality.
A.4 CON-J IMPROVES WITH CONTROLLED RATIONALES
To test whether controlling the quality of the rationale can further improve Con-J's ability, we con- duct a pilot experiment where the quality of the rationales with the annotation consistency and correctness from GPT-4o (see Section 4.3) The control step is adopted on the data samples con- structed from the judgment sampling process (step in Figure 2. Considering the high cost of using GPT-4o for annotation, the Math task. The subset consists of 12,000 data samples in total. Each data sample consists of question, pair of answers, positive judgment, and negative judgment. We adopt GPT-4o to annotate the positive judgment and preserve samples in which GPT-4o agreed that the positive judg- ment is correct. This filtering step preserved a total of 9,050 samples. We then categorized these samples into two groups: high-quality group with the rationales' correctness score of and a con- sistency score of 3, and low-quality group with the remaining data samples. This step constructs 5869 data samples for the high- high-quality group and 3181 for the low-quality group. We then randomly select and retain 3181 data samples from the high-quality group to ensure that both groups have the same size. Then we use the proposed contrastive training method to train Con-J with high-quality group (denoted as Con-J-h) and low-quality group (denoted as Con-J-l), respectively.
randomly selected subset from the constructed data samples in
Experimental results show that the accuracy of preference prediction is 0.6759 for Con-J-h and 0.6427 for Con-J-I. This implies that improving the quality of the rationales can also help the binary preference prediction. We then test the quality of the rationales generated by Con-J-h and Con-J-l, respectively. We observe that the average correctness of the rationales generated by Con-J-h (4.25) is higher than Con-J-l (4.15) and Con-J untrained (4.145). This observation opens new avenues and benefits for human involvement in understanding preference models and enhances their ability not only make accurate preference judgments but also to base those judgments on correct reasoning. However, we observe that the consistency score of Con-j-h (2.85) is not higher than Con-J-I (2.87) and Con-J untrained (2.90). According to the human evaluation, Con-J may avoid expressing clear preferences in the rationale part when it lacks the ability to make judgments. Actually, the low con- sistency may not necessarily be bad, as it potentially indicates that the model not convincing in its judgment and may need more human involvement for these cases. In future work, more ef- fective data construction process may address this problem by designing more appropriate expected rationales for these situations, such as having the model output "I find it difficult to judge, but can provide some analysis..."
B LIMITATIONS
Several limitations of this work guide future directions including: (i) We demonstrate that preference learning can enhance the model's ability generate correct rationales. Another unresolved and intriguing question is whether enhancing the quality of rationales could also improve the model's preference prediction abilities. an important problem enhance the model's ability not only to make accurate preference predictions but also to base those judgments on correct reasoning. (ii) We demonstrated that Con-J can more effectively resist bias than SM in an adversarial experiment. However, further analysis is needed to understand why Con-J outperforms SM on complex, realistic datasets, and whether this is also related to bias. (iii) We suggest that Con-j can potentially facilitate human collaboration through interpretable preference judgments for LLM training. The design of such a pipeline is another interesting and valuable direction.
C ETHICS STATEMENT
With the advancement of LLM capabilities and alignment techniques, it is important to discuss the ethics issues in this paper. Here are some potential issues to consider:
18

---

Published as a conference paper at ICLR 2025
1. LLM Bias: LLM is prone to be biased, which raises significant ethical concerns that de- mand careful consideration to ensure its responsible usage. In addition to training methods and model architecture, the origin of bias is significantly related to the training data. The societal biases in the training data may affect the functioning of L.Mss, resulting in outputs that accentuate stereotypes paper investigates the bias in preference learning in a synthetic dataset and makes one hy- pothesis that the post-training preference dataset may contain serious bias. However, important to note that biases, particularly those related to majority and minority groups, are also prevalent during the pre-training. Therefore, we suspect that future work requires more rigorous dataset construction to reduce potential bias and feedback mechanisms to detect and address bias.
involve unjust discrimination against certain groups. This
2. Al Transparency: Nowadays AI is widely applied in decision-making planning. and var- ious related tasks. Ensuring the transparency and trustworthiness of AI in this process is important. The proposed Con-J enhances the interpretability of Al-based preference anno- tation by generating natural language rationales to support its preference prediction. This transparency allows humans to understand the reasoning behind AI decisions. However, we have also observed that the consistency between the Con-J's judgments and its ratio- nales decreases with preference learning. This poses challenge to building transparent Al systems.
3. LLM misinformation: Large language models (LLMs) are prone to generating misinforma- tion without factual basis. This misinformation may also be low in checkability, making it difficult for humans to verify (Hendrik Kirchner et al. 2024). Therefore, we have a significant concern that the rationales supporting the L.MM's preference predictions might be faulty and difficult to detect. Future efforts to analyze the rationales from the LLMs, and devising methods incorporating a fact-checking process help mitigate the risk of spreading misinformation are important.
D REPRODUCIBILITY
Open-source model and code are available at: wttph/uu..oom//eyyl199999 The code used in this paper is available under the Apache 2.0 license. All the experiments this paper are car- ried out based open-source frameworks, including Open-RLHF (Hu et al. 2024). Pytorch, and Transformers
19

---

Published as a conference paper at ICLR 2025
Table 5:
An example of prompt fed the LLM generate preference reward judgments, con- sisting of preamble (introduction and instructions describing the task), a question, and a pair of candidate answers. The preamble neutral and does not explicitly indicate which answer better.
Preamble
As an evaluation expert, given a question and its two possible answers, please choose which answer best aligns with coherence, accuracy, coverage, and over- all quality. Output your judgment in JSON format, where "rationale" is your explanation, and answerr is an integer type of or 2, for example, ("rationale": "your explanation", anttwer. 1}. Below the question and its candidate answers:
Question
The volume of rectangular prism is 96 cubic decimeters, and the base area is 16 square decimeters. What is its height in decimeters?
Answer
We can use the following formula to calculate the height h of the rectangular prism: V = S times h. Substitute the known values into the formula: 96 = 16 times h; = 6. Therefore, the height the rectangular prism is decimeters.
Answer
5. Height = Volume Base Area = 96 = 5 decimeters.
Prompt
As an evaluation expert, given a question and its two possible answers, please choose which answer best aligns with coherence, accuracy, coverage, and overall quality. Outpul your judgment in JSON format, where "rationale" is your explanation, and "better_unswer" is an integer type of or 2, for exam- ple, {"rationale": "your explanation", "better-answer": 1}. Below are the question and its candidate answers:
Question: The volume of rectangular prism is 96 cubic decimeters, and the base area is 16 square decimeters. What is its height decimeters?
Answer I: We
the following formula to calculate the height of the rectangular prism: V = S times h. Substitute the known values into the for- mula: 96 16 tiime h; 6. Therefore, the height of the rectangular prism is 6 decimeters.
Answer 2: 5. Height Volume Base Area 96 / 16 decimeters.
20

---

Published as a conference paper at ICLR 2025
Table 6: Prompt template with preamble using correct or incorrect hints, where a, are the ID of correct and incorrect answers, respectively, fa, }} (1, {[Question)), {fAnswer 1}}, [Answer 2 are the text content of the question, answer 1, and answer 2. respectively. When the LLM does output valid JSON format as expected (e.g.. "rationale". "your explanation", "better_anse _answer": a)), we use an alternative prompt (rows 3-4) to prompt again and insert its output as the rationale into the template.
Prompt with pream- ble correct
As an evaluation expert, given a question and its two possible answers, please choose which answer best aligns with coherence. accuracy, coverage, and overall quality, Below are the question and its candidate answers: Question: (Question]
Answer 1: [Answer
Answer 2: (Answer 2}}
Given that answer is better than answer 3, please provide the rationale. Output your judgment JSON format, where "rationale" your explanation, and
is integer rype of a.r example, {"rationale": "your explanation", "better answer": a.).
Prompt with pream- ble incorrect
As an evaluation expert, given question and its two possible answers, please choose which answer best aligns with coherence, accuracy, coverage, and overall quality. Below are the question and its candidate answers:
Question: { [[Question] }}
Answer 1: Answer
Answer 2: (Answr 2}}
Given that answer B is better than answer a, please provide the rationale. Output your judgment JSON format, where "rationale" is your explanation, and "better answer" is integer rype of for example, {"rationale"; "your explanation", "better answer": 3).
Prompt with pream- ble_correct (al- ternative)
As an evaluation expert, given question and its two possible answers, com- pare the answers according to their coherence, accuracy, coverage, and overall quality. Below are the question and its candidate answers:
Question: { [Quessiio]]
Answer 1: Annser 1
Answer 2: (Annseer 2}}
Given that answer is better than answer 3, please provide the rationale:
Prompt
As an evaluation expert, given question and ity possible answers, com- pare the answers according to their coherence, accuracy, coverage, and overall quality. Below are the question and its candidate answers: [Questiio)]
with pream- ble blc.incorrect (al- ternative)
Question:
Answer 1: [Answer
Answer 2: [[Answer 2
Given that answer better than answer a,, please provide the rationale:
21

---

Published as a conference paper at ICLR 2025
Table Prompt template for asking the generative LLM outputs only the binary judgment without any rationales. {[Question)), ), {[Answer 1)}, {[Answer 2}} } are the text content of the question, answer 1, and answer 2, respectively.
Prompt
As evaluation expert, given question and its possible answers, please choose which answer best aligns with coherence, accuracy, coverage, and overall quality Ouiput your judgment in JSON format in which "bet- ter ter-answer" is an integer type of I or 2. for example. { ["better_answer answer": l}. Do not include any additional explanations. Below are the question and its candidate answers:
Question: (Question)
Answer 1: [Anseer
Answer 2: (Answer 2}}
22

---

Published as a conference paper at ICLR 2025
Table 8: Prompt template for transforming the answers into different formats.
Prompt for rewriting the answer into more verbal format
You someone who works on popularizing mathematical knowledge. Please restate the following content in simpler, more accessible language without changing the original meaning, affecting its length, or adding extra informa- tion. Below is the input: {[Answer]]
Prompt for rewriting the answer into more formal format
You are a researcher the field of mathematics. Please restate the following content using precise mathematical language without changing the original meaning, affecting its length, or adding extra information. Below the input: { [Answer]]
Prompt for rewriting the answer to be more verbose
Please summarize the input by listing the key points in numbered format. Below is the imput: [[Answer]]
23

---

Published as a conference paper at ICLR 2025
Table 9: Prompt template for scoring the rationales and judgments of Con-J. (Juudgment] is the judgment generated by Con-J. the judgment is JSON style with a key named "rationale" that presents Con-J's reasoning and explanation for the task, and another key named "better answer" indicates Con-J's preference prediction.
#BBackgrundd
Given below is question and two corresponding answers:
Question: {[Questio]) }
Answer 1: (Answr 1
Answer 2: [Answer 2}}
A judge has assessed these two answers and judged which one is better. Here are its preference prediction (value of "better answerr) and the corresponding rationales (value of "rationale" "): (Judgment)
## #orkkflww and Scoring
Please analyze whether the judgment is correct and evaluate the rationale by scoring them on (i) correctness and (ii) consistency with the binary preference prediction. Here, correctness is defined by whether the rationales accurately understand the content of the question and the answers, provide a reasonable and correct analysis process, and are free of factual errors. Consistency is defined by whether the logical connection between the rationales and the final preference prediction is coherent, and whether both the rationales and the fi- nal preference prediction show the same sentiment or attitude towards the two answers.
Correctness:
I: Completely incorrect.
2: Mostly incorrect.
3: Partially correct.
4: Mostly correct.
5: Completely correct.
Consistency:
The sentiment or attitude of the rationales towards either answer (Answer
1 or Answer 2) is inconsistent with the final preference prediction.
2: The sentiment or attitude of the rationales towards either answer (Answer
1 or Answer 2) is not clear.
3: The sentiment or autitude of the rationales towards either answer (Answer
1 or Answer 2) is consistent with the final preference prediction. #ormatting
Please return the results in JSON format, for example: "Preference Predic- tion Correctness": "Correct", "Rationale Correctness": 4, "Rationale Con- sistency": 3, where "Preference Prediction Correctness" can be either "Cor- rect" or "Incorrect".
24

---

Published as a conference paper at ICLR 2025
Table 10: Judgment accuracy of Con-J and its variants. indicates the performance difference
between Con-J significant at < 0.05 using pair-wise t-test.
Model
Creation
Math
Code
Con-J untrained
53.6*
63.4*
61.7*
Con-J w/o Hint
61.3*
77.4"
68.2
Con-J w/o DPO
54.6*
64.2*
63.5*
Con-J w/o SFT
72.2
81.3*
70.3
Con-J w/o rationale
72.6
84.1
71.0
Con-J reorder
72.3
85.2
70.2
Con-J
72.4
85.0
70.1
25

---

Published as a conference paper at ICLR 2025
Table 11: Example of rationales and binary preference generated by Con-J. We prompt GPT-4o to annotate the correctness (ranging from to 5) of the rationales and the consistency between the predicted preference and the rationale (ranging from to 3).
Prompt input
Rationale
Preforred
Question: On the theme of comprehensive rumal revitalization, propose a paper title.
The second answer provides a bronder applicability and depth of description, with smoother and more natural con- first answer.
Creation (1)
Answer
Answer 1: Research on Strategic Pathways for Rural Revitalization from Collahorative Perspective.
(correct)
Answer 2: Comprehensive Rural Revitalization: Doop Integration of Strategies and Practices.
Correctness: 5, Consistency:
Question: Reverse the elements in an aray and make each character in elements lowercase, planetarr |'Earth', 'Saturn', 'Mar''l: I:
The first answer provides a clearer step-by-step solution the problem by achieving army reversal and lower- case conversion through independent functions, which makes the code more modular. The second answer is more concise nccurately meets ques- tion's 'sss requirements by lirst reversing the armay then converting it low- ercase.
Answer I:
def' reverse aray(arii:
return arri:-
I}:
revercase_array(arry
return [owlower()
Answer
Code (1)
x in arl:: planct.art
printfreersee array(planet art)xpeentoowweeese arraypl arr)):
('Earth'. 'Satum' 'Mars'3 I:
(incor- rect)
Answer planct.arr= ['Earth', 'Satum". 'Mars'):
planet.rrt = (p.lower)) for p in planet.ari]: peintpplanet.ar) This code will reverse order
elements the "planet arr arra,, und then lowercase each character in each element. The resalting array will [' l'mars', 'saturn', 'earth') []
Correcmess: 3, Consistency:
Question: Today is August 12, 2021. What was the date 30 days ago? Answer According the given information, today August 12, 2021, so 30 days ago was July 13, 2021. Therefore, 30 days ago was July 13,
The second answer does not pravide detailed explanation. whereas the first answer offers a detailed reasoning process and uses correct date subtrac- tion logic.
Math (3)
Answer
2021.
(incor-
Answer 2: July 12, 2021.
rect)
Correctness: 2, Consisteney:
Question: Given the four digits 2, 4, 6, and 7, a digit can be used more once in integer. how many different positive two-digit numbers can he formed?
Answer I dircctly combincs the ques- requiremntts provide rect solution along with detailed ex- planation.
Math (4)
Answer
combination (choosing one digit the first position and another for the second) is treated unique number, even if they are muthematically equivalent (like "22" and "44"). There total of C(4, C14, l)= such combinations (where denotes combinations). This is the final answer: 16 unique two-digit numbers can formed.
Answer
means
(correct)
Correctess:: 4, Consistency:
If we take order into account (e.g. "24" and "42" are different nombers), we to the concept permutaations P(n, r) (the number of ways arrange elements from n different elements), In this scenario, we use the formula P(4, 2) m1 (4-2)) = 4. 3 12 unique arrangements where "!" denotes factorial...
Answer 2:
26

---

Published as a conference paper at ICLR 2025
Table 12: Average performance of Con-J and Con-J w/o rationale on the general test set under the verbosity bias with ς 1. The data samples are grouped according to whether the chosen answer is longer than the rejected answer.
Chosen Length Rejected Length
Chosen Length Rejected Length
Con-J
0.6795
0.5871
Con-J w/o rationale
0.8301
0.4714
27

---

Published as a conference paper at ICLR 2025
Algorithm Constructing contrastive judgment pairs for Con-J
1: Input: n: pre-trained LLM; a preference dataset D {(u,a aa)))
2: Output: E: a set of constrastive judgment pairs.
3: ==
4: for (g,a".a"
D do p format(preamble (q,a (q,a=,at):)
5:
> Prompt construction with preamble
6:
Get (p)) with repeated sampling
D> Judgment generation with repeated sampling (p) (p) + filter (M (p)), filter (lteerincorrec)(M(())))) -Selection with ground
7:
truth preference
8:
E E (.p.Jn))a Jb) E (()p*, Jn E (p)
9:
Pr. Pr. format(preambll correct, (z, "e,,a"aa)i) );). format(preamble incorrect, (x, aaaaa=aa))))) b Using preamble with correct or incorrect hint to construct prompt
10:
Get (Pp).M with hint-driven sampling
11:
E E' U Ix)li E M Jn E (p.))
end for
13: Return E
28


Reviews:
[{"review_id": "qrrUVcUsqM", "reviewer": "Reviewer_eNC9", "summary": "The paper proposes a generative approach to build reward models for preference learning. They generate judgements with explanations for both positive and negative samples in preference pairs and then use DPO to align a generative reward model. They find that this contrastive approach leads to improved performance on internal and public benchmarks when compared against scalar reward models and other generative methods.", "questions": "Please see weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Very interesting and relevant approach that uses synthetic data to effectively propose a good empirical solution.\n* Generating rationales would be very useful to verbalize and detect biases in the preference data and can help provide better data quality.\n* I like that the authors train a model on publicly available datasets and report numbers on widely used benchmarks instead of just relying on the internal test sets.", "weaknesses": "* The approach feels like it would add a lot of overhead in alignment training - RLHF is already extremely computationally intensive and waiting for models to generate rationales before providing the judgement feels like it would not be a feasible approach to use for cutting-edge model alignment.\n* I would like to see some qualitative analysis of the rationales - there seems to be minimal information on how good they are and what they might be useful for. Has there been any human evaluation of the rationales when inconsistencies have been detected?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vrMGQPXqIq", "reviewer": "Reviewer_21Wd", "summary": "This paper introduces Con-J, a new way of learning preferences by augmenting human pairwise preferences with LM-generated rationales. First, a LM is prompted to generate rationale pairs ($j^+$, $j^-$) for either element of the preference pair ($a^+$, $a^-$). The base LM is then trained with DPO to optimize $j^+$ over $j^-$ (vs. DPO with scalar rewards, which is simply trained to optimize $a^+$ over $a^-$). Empirical studies are performed on three domains (Creation, Math, Code), which find that Con-J outperforms scalar reward models at providing the correct judgment, that it outperforms other generative judges (Auto-J and Prometheus 2), and and that it is less susceptible to bias and spurious correlations in the reward function.", "questions": "1. How exactly does Con-J without rationale differ from the scalar model (Figure 4)?\n2. One can also imagine a debiasing effect from using pre-trained LMs as rationale generators -- they are likely to generate rationales that align with preferences from their pre-trained data. While this may be ok if the preferences are generally in line with the pre-trained data, there may be issues with aligning to preferences with fundamentally different values or principles. Is there any investigation of whether the LM likes to generate the same set of values in its rationales?\n3. Are there overall statistics from section 4.5 about how consistent / correct the rationales tend to be? Either automated, or human annotated (perhaps on a subset).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The experimental results demonstrate that Con-J is an effective method, compared to scalar reward models. The results section is quite comprehensive, with baselines across multiple models and datasets. The analysis section is informative with lots of useful takeaways: including studies of how Con-J can be robust in the presence of bias, and where LMs can generate rationales that fail to support the answer judgment. Overall, the claims are well-supported.", "weaknesses": "1. The empirical results (Table 10, Figure 4(b)(d)) seem to indicate that the main contribution to Con-J's good results are not the rationales, but (presumably) the regularization in eq. 9 instead (It's unclear to me what exact parts Con-J without rational removes...). Experimentally, what happens when this regularization is ablated? Furthermore, the claims in the abstract/introduction seem to suggest that rationales themselves are the reason why Con-J succeeds. While experimentally it's demonstrated that the rationales do improve robustness to bias, it seems misleading to claim this when the main results actually suggest that the rationales do not improve judgment accuracy at all. It may be good to do a deeper analysis of what is actually inducing this improvement.\n2. Furthermore, the introduction highlights interpretability as a benefit of rationales (and puts it on the same priority level as robustness to bias). While intuitively rationales are indeed more interpretable, the paper never demonstrates the utility of this interpretability. What happens if you do have humans (or even other LMs) check the rationales and throw out inconsistent ones? (As far as I understand, the filtering step simply checks the correctness of the final answer, but not the rationale itself.) Is this interpretability actually useful for constructing better / more useful reward models? Will it be better to align downstream models with rationales?\n3. The method has many moving parts. While the figure is useful, parts of the text and figure could be clearer. Having a concrete example with a LM-generated rationale in the main body or the figure could be useful for visualizing the method. Furthermore, it may be good to list out all the moving parts, esp for the ablations it is unclear what parts are preserved and what parts are not (e.g. how exactly does Con-J without rationales differ from the scalar model).\n4. Note -- quite a few generative reward model methods have come out at about the same time this work (see https://arxiv.org/abs/2408.15240, https://arxiv.org/abs/2410.12832). While these can be considered concurrent work, perhaps a discussion of the tradeoffs of this method compared to other methods could be useful somewhere in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GN3kX8VO1I", "reviewer": "Reviewer_sJjd", "summary": "The paper introduces Con-J, an approach for training a generative judge using contrastive judgments for preference learning in order to address the problem of (i) lack of interpretability and (ii) susceptibility to biases in scalar reward models. Con-J involves creating contrastive judgments from a set of query-answer pairs sampled from a pre-trained LLM. These contrastive pairs are then used for preference finetuning, for example, direct preference optimization. Finally, they were able to show that Con-J performs better on several benchmarks such as RewardBench.", "questions": "Questions\n- [Q1] Can you clarify (or point to a URL) how you were able to obtain a test split for Ultrafeedback? The HuggingFace version (openbmb/Ultrafeedback) only contains a train split.\n\nComments/Nits:\n- [C1] Line 259-261. The “Code” vertical domain is not mentioned\n- [C2] Some LaTeX citation commands were used incorrectly. For example, \\citet is often used in place of \\citep, breaking the flow of writing. I highly recommend the authors to revisit these parts and update accordingly.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- [S1] The novelty of the paper’s contribution is clear. I appreciate how Con-J’s adjustments of the typical generative RM set-up was motivated.\n- [S2] The ablations are also reasonable, especially on the different variants of the contrastive pairs (e.g., positive judgments only, etc.).", "weaknesses": "Overall, I lean towards accepting the paper on the merit of its contribution's novelty (new way of framing the generative RM setup), but there are some sections--particularly in the analysis--that are not fully convincing and needs revisiting (see [W1]). \n\n[W1] I’m not entirely convinced with the analyses of Consistency and Correctness in Sections 4.3 and 4.5. I think it’s helpful if the paper first defines what “consistency” and “correctness” measure. In addition, it will be helpful to show that the GPT-4o scorer is reliable. Suggestion to the authors: define and motivate the aspects of consistency and correctness in the beginning of the paper.\n- The prompt template in Table 9 speaks about the emotional tone, which I think measures a different aspect than consistency. Perhaps that’s also the reason why the results  in Figure 3b are surprising (I would expect the consistency of the rationales would also improve) and aren’t consistent with the cited literature on chain-of-thought (CoT).\n- The scoring for Correctness is also confusing: the rubric for 3 can be confused for 2 and 4.  It’s also important to describe what correctness means for a subjective task such as Creation. I wonder what the findings would be if the rubric is more distinct.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j5xCBXHsLO", "reviewer": "Reviewer_KgUv", "summary": "The paper introduces Con-J, a generative judge trained via self-bootstrapped learning from preference data to overcome limitations of scalar reward models, such as lack of interpretability and dataset bias. This model is trained with a combination of direct preference optimization (DPO) loss for contrastive judgement learning with an additional SFT term over positive judgements. Experiments across various domains (Text Creation, Math, Code) and benchmarks show that Con-J performs effectively, improving rationale correctness during training. This enables accurate judgments and meaningful explanations, supporting human-in-the-loop LLM alignment. Con-J also demonstrates reduced susceptibility to dataset bias compared to scalar and rationale-free models.", "questions": "1. The paper seeks to address a crucial issue in preference (or reward learning), where the reward model may be prone to bias and is not easily understood from the human interpretability angle. However, the learning algorithm (objective) is not very novel and recent work [1,3-5] has already leveraged such DPO + SFT type losses have leveraged such as contrastive learning formulation. However, I could not find these works being cited unless I am missing something. \n\n2. While the idea is simple, my primary concern is that the generated rationales can be biased or be of low-quality if the chosen base model is not strong enough. Unless I am missing something, the authors do not conduct any human evaluation of the rationales being sampled.  How does one make sure that the rationales are at least surpass some *baseline* reasoning quality, regardless of the preference relation without a manual  verification component?\n\n \n\n3. How do you control for the length of the judgements being sampled? From my perspective, if the intention in the Con-J methodology is better/explainable reward learning with compute-efficiency, then would you not have to control for judgement length pre-inference? In that case, for questions that would naturally require a more thorough reasoning process (CoT), wouldn’t the preference judgement be partial or biased in such cases? If judgement-length is controlled, how can shortened or partial/biased judgements’ affect on preference learning be mitigated in light of the inconsistency of rationale vs binary decisions observed in section 4.3?\n\n\n4 .How was the alpha value chosen? [1] suggests that a larger $\\alpha$ (compared to this work) of 0.01 for text-regularization is optimal for generalizable reward learning, including testing on benchmarks like rewardbench. Also, [2,3] provides empirical evidence that DPO loss can lead to reduction in likelihoods of the winning responses. It’s reasonable to expect the SFT component helps in ensuring $\\pi_ (j+ / p)$ stays relatively high during optimization, but authors here use an $\\alpha$ of $1e^{-6}$ which is significantly lower than the optimal $\\alpha$ in [1]. Clearly, this affects SFT-based regularization, but was this $\\alpha$ value empirically determined or were there any particular motivations for choosing this value??", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is clearly-written and the motivation is clear that it attempts to bridge preference modeling with a generative component to reduce bias as well as to improve interpretability. Section 3 is pretty clear as in the math formulation and what the crucial components are in eq.4 and 5.  The empirical evaluation also seems reasonable with a variety of LLMs chosen and data domains being selected. I also find the illustrative examples and prompt-structures to be quite helpful for reproducibility and cross-comparisons.", "weaknesses": "I find this work to be a commendable engineering effort but with moderate novelty. In particular, is it not quite evident why do we need DPO’s formulation since the argument about similarity with pretraining objective and DPO’s policy being an optimal solution to the KL-constrained optimization problem is not directly addressed. Furthermore, SFT+ DPO combination for reward model training has been quite popular but I find the motivation for adding the SFT component to be less clear, especially the authors downweight this component with a relatively small weight compared to previous works. The authors argue that adding SFT avoids the reward overoptimization issue but I don't see an empirical evaluation of how this affects the log-likelihoods of pos and neg judgements in their Con-J training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)