Generative Reward Modeling via Synthetic Criteria Preference Learning
Authors: xiaobo liang, Haoke Zhang, Juntao Li, Kehai Chen, Qiaoming Zhu, Min Zhang
Accepted at: acl25

Generative Reward Modeling via Synthetic Criteria Preference Learning

Xiaobo Liang*,**, Haoke Zhang*,**, Juntao Li*,**, Kehai Chen*, Harbin Institute of Technology, Shenzhen, China
* Key Laboratory of Data Intelligence and Advanced Computing, Soochow University
{xbliang, ljt, qmzhu, minzhang}@suda.edu.cn
hkzhangnlp@stu.suda.edu.cn chenkehai@hit.edu.cn

Abstract

Generative Reward Models (GenRMs) leverage synthesized Chains of Thought (CoT) to reduce the need for massive labeled data, but this approach introduces risks of overoptimization due to the inability to guarantee the correctness of the CoTs. Identifying and optimizing unexpected behaviors within these synthesized CoT remains a challenge, as it heavily depends on precise annotations of intermediate behavior, similar to process supervision. In this work, we introduce a criteria-based preference tree for reward modeling, where each path in the tree represents a reasoning trajectory based on synthesized criteria. Crucially, each reasoning trajectory can be independently optimized through RL algorithm. These fine-grained process reward signals are derived from the inference-time computations and predefined rules, eliminating the need for human supervision. In experiments, SyncPL\textsuperscript{1} showed significant improvements over baselines on multiple human preference benchmarks. We further demonstrate that synthesized data can be learned using a long CoT format, analogous to an o1-like model, further enhancing performance while keeping stability and efficiency during training.

1 Introduction

With Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017; Schulman et al., 2017; Ouyang et al., 2022), large language models (LLMs) can effectively follow user instructions and generate responses that align with human values. Traditionally, discriminative reward models (e.g., Bradley-Terry models (Bradley and Terry, 1952)) have been used as proxies to evaluate human preferences. These reward models lack explicit reasoning, leading to significant overoptimization issues (Skalse et al., 2022; Chen et al., 2024) and often deteriorating on out-of-distribution data (Zhang et al., 2024). The reasoning models like OpenAI o1 and DeepSeek R1 (Guo et al., 2025) demonstrates that powerful CoT reasoning capability is key to performance improvement and scalability. As a potential alternative, GenRMs (Mahan et al., 2024; Ye et al., 2024) leverage the capabilities of LLMs, such as CoT reasoning and inference-time scaling, offering an interpretable evaluation process. However, this approach does not guarantee that the model generates reasonable CoT trajectory during inference. Especially in open-domain scenarios, the model often needs to consider multiple evaluation criteria (Liu et al., 2023; Wang et al., 2024b), such as accuracy, fluency, and consistency.

Recent studies (Kim et al., 2023; Saha et al., 2023; Liang et al., 2024) have used LLMs to synthesize criteria-based CoT for responses evaluation. In Section 3, we demonstrate how these methods can address challenges in reward modeling, such as implicit modeling, thus improving the accuracy of preference judgments. Additionally, GenRM ensures the effective parallel generation of multiple criteria to achieve more accurate judgment results. However, the compounding error of synthetic

![Illustration of SyncPL.](page_1012_368_388_388.png)

Figure 1: Illustration of SyncPL.

\footnotetext{* Corresponding Author}
\footnotetext{1 SyncPL means Synthetic Criteria Preference Learning.}
CoT hinders the model’s improvement through self-improvement, as the model struggles to verify the correctness of the CoT trajectories. To address this issue, we predefined two rules as meta criteria to filter high-quality trajectories.

As shown in Figure 1, the CoT trajectories in our proposed method, SyncPL, are composed of three essential elements: criteria, rubrics, and judgments. The judgments includes scores assigned to different responses (e.g., A and B), such as \( (1, 3) \). Furthermore, we will utilize the Monte Carlo rollout policy to expand CoT trajectories into a preference tree, which provides a manner to estimate the expected reward of different trajectories. To identify and optimize critical criteria, our method contains two stages: In the ranking phase, we first compute the score differences between responses A and B across various reasoning trajectories. By aggregating these differences, we determine the importance of each criterion. Specifically, the Ranking Rule identifies the relative importance of the criteria, while the Consistency Rule filters out trajectories with low confidence and inconsistent judgments. In the learning phase, we utilize these ranked trajectories to optimize LLMs through Rejection Sampling Fine-tuning (RSFT) and Direct Preference Optimization (Rafailov et al., 2024) (DPO). SyncPL achieves substantial performance gains across multiple preference benchmarks both on in-distribution and out-of-distribution (OOD) scenarios.

Benefiting from the rigorous data filtering, we attempted to combine different criteria-based judgments into a long CoT format and conducted RL training. We demonstrate that the SyncPL-o1-8 works effectively, surpassing the performance of the LLaMA3.1-70B. In a nutshell, our contributions are as follows:

• We are the first to introduce tree-based preference learning into generative reward modeling to optimize synthetic criteria trajectories.

• SyncPL-DPO prioritizes key and high-quality criteria, effectively reducing inference costs.

• We are the first to demonstrate and implement the o1-like long CoT format\(^2\) for GenRM through criteria-based process supervision.

\(^2\)It denotes multi-step reasoning across different criteria rather than merely a long response length.

![The difference between our proposed SyncPL and existing reward modeling approaches.](page_1016_153_393_312.png)

Figure 2: The difference between our proposed SyncPL and existing reward modeling approaches.

2 Related Works

Reward modeling plays a crucial role in aligning LLMs with human values and behaviors (Ouyang et al., 2022; Bai et al., 2022), particularly in ensuring their safe and effective deployment. To reduce the cost of annotating preference data, reward models are often used as proxies to provide reward signals for the policy model (Rafailov et al., 2024; Azar et al., 2024). Given the user query and the candidate responses A and B, the two main approaches are discriminative RM and generative RM.

Discriminative RM. The Bradley-Terry (Bradley and Terry, 1952) model improves pairwise preference selection through Luce-Shephard Choice Rule (Luce, 1959; Shepard, 1957). However, this method is prone to reward hacking (Yang et al., 2024; Coste et al., 2023; Pace et al., 2024; Shen et al., 2024), also known as overoptimization, as it easily learns spurious correlations between data and preference label. For example, Chen et al. (2024) demonstrate that verbose but less helpful responses often receive higher reward scores. To address this issue, Shen et al. (2023) introduce consistency training to mitigate spurious reward when handling lexically similar instructions. Despite it, discriminative RM still requires massive data to address the challenges posed by their inherent sparsity modeling (Sun et al., 2024).

Generative RM. GenRM can improves LLMs by synthesizing interpretable critiques. In practice, the LLM-as-a-Judge paradigm (Dubois et al., 2024; Wang et al., 2023; Zheng et al., 2024) demonstrates high consistency between LLMs and humans in evaluation tasks, further highlighting the potential of generative modeling. The self-reward modeling (Yuan et al., 2024b; Wu et al., 2024; Ye et al., 2024) approach showcases the optimization of syn-
thesized critiques through self-improvement. Furthermore, some researchers (Ankner et al., 2024; Yu et al., 2024) try to combine the strengths of both by jointly generating critiques and scalar reward scores. In contrast, as illustrated in Figure 2, our approach emphasizes criteria-based reward modeling by introducing a synthetic preference tree (Yuan et al., 2024a; Lehnert et al., 2024; Tian et al., 2024) to synthesize criteria and incorporating process supervision to reduce error accumulation.

3 Preliminaries

Implicit modeling in reward model training reduce data dependency but pose risks like reward hacking\footnote{For example, imbalanced label distributions may cause high-frequency labels to dominate predictions, misleading the model’s decisions.}. This section explores key challenges (see § 3.1) and suggests potential improvement strategies based on the findings of trials (see § 3.2).

3.1 Barriers to Progress

Implicit Modeling and Inference. In reward modeling, the number of required comparisons often far exceeds the available preference pairs (Sun et al., 2024). While sparse comparisons help reduce human annotation costs, they introduce the risk of failing to capture true order relationships, such as non-transitive preferences (Wang et al., 2024a). The human preference (\( y_a \) or \( y_b \)) can be considered as a weighted combination of multiple \( \mathcal{R} \) based on different *criteria* \( \mathcal{C} \):

\[
\mathcal{R}(x, y_a, y_b) = \sum_i^{|\mathcal{C}|} \alpha_i \hat{\mathcal{R}}(x, y_a, y_b, c_i),
\]
\[
\hat{\mathcal{R}}(x, y_a, y_b, c_i) \sim \mathcal{P}(y_r|c_i, x, y_a, y_b),
\]
\[
c_i = \mathcal{P}(c|x, y_a, y_b).
\]

However, this weighting coefficient \( \alpha \) is largely determined heuristically, making it impossible to obtain the value. Similarly, BT models and non-CoT GenRMs also rely on the implicit learning of human preferences rather than providing explicit reward signals, making it difficult for them to model complex preference relationships. Considering the overall reward \( \mathcal{R} \) is derived from a combination of criteria-based sub-rewards \( \hat{\mathcal{R}} \), can we enable the LLMs to generate these sub-rewards, thereby facilitating explicit generative reward modeling?

![Experimental Results of generative reward models (LLaMA3.1-8B) on RewardBench.](page_1016_172_393_355.png)

Figure 3: Experimental Results of generative reward models (LLaMA3.1-8B) on RewardBench.

3.2 Trial Experiment Design

To evaluate the effectiveness of criteria-based judgment, we adapt LLaMA3.1-8B-Instruct as a generative reward model. We utilize the Chat and Reasoning datasets from the human preference benchmark RewardBench. First, we guide the model to generate criteria-based CoT judgments using carefully designed prompts and then aggregate all judgments to obtain the final result. As illustrated in Figure 3, single-step reasoning represents the output of a traditional single-step GenRM, while multi-step reasoning corresponds to results derived from 1 to 9 different criteria. Furthermore, we employ rejection sampling to synthesize CoT trajectories on the UltraFeedback dataset for fine-tuning the LLMs, in order to evaluate whether self-improvement (using STaR-like methods (Zelikman et al., 2022)) can further enhance performance.

**Observation 1.** In the experiment, the curve represents the changes in cumulative judgment results based on an increasing number of criteria, demonstrating that multiple criteria-based preference judgment markedly improves model performance. Additionally, incorporating more criteria in the evaluation leads to enhanced accuracy (e.g., achieving an 11.63% improvement in reasoning tasks), surpassing the performance of using a single criterion.

**Observation 2.** Although we aim to optimize the model’s capabilities through the synthesis of CoT
and self-improvement, compounding errors4 result in a performance decline after fine-tuning.

Conclusion . It is evident that aligned LLMs inherently serve as criteria-based reward models, yet this capability still requires further guidance. The primary cause of “compounding errors” is the absence of human supervision in each CoT trajectory. Therefore, our main objective is to improve the quality of criteria-based CoT through explicit training and optimization. Additionally, multi-step reasoning introduces a significant inference cost, which poses challenges for practical deployment.

3.3 Motivation
A natural question is that Can LLMs optimize and prioritize the most critical criteria? Based on this idea, we reformulated the objective of reward modeling as a further optimization of Equation (1):

\[
\mathbb{I}\{\mathcal{R}(x, y_a, y_b)\} = \mathbb{I}\{\hat{\mathcal{R}}(x, y_a, y_b, c^*)\},
\]
\[
c^* = \arg\max_{c_i} \hat{\mathcal{R}}(x, y_a, y_b, c_i),
\]

where \( c^* \) represents the most critical criterion, \( \mathbb{I} \) represents the indicator function, which takes the value 1 if \( y_a \) is better, and 0 if \( y_b \) is better. This objective requires the explicit optimization of sub-rewards \( \hat{\mathcal{R}} \), but we lack explicit reward signals in practice. To address this limitation, outcome-based reward learning must be transformed into process-based reward learning. Specifically, in the training data, we only have \( \mathcal{R} \) rather than \( \hat{\mathcal{R}} \), which must be identified through searching and learning.

Remark: It is important to note that \( c^* \) is not a global optimal solution to the current preference problem, but rather a local optimal solution within the limited criteria candidate space in our work.

4 Methods
To achieve the abovementioned goals, we decompose it into two components: (1) obtaining estimated values for the sub-rewards \( \hat{\mathcal{R}} \), and (2) optimizing the LLMs to output the most critical criteria \( c^* \). Thus, we introduce a method SyncPL, as shown in Figure 4. This approach first ranks the CoT judgments according to synthesized criteria, generating sub-reward estimates for various trajectories. Then, we fine-tune the LLMs to emphasize critical criteria, ensuring that they are prominently reflected in the model’s predictions.

4.1 Ranking for Criteria Preference Tree
SyncPL first introduce an additional rubric node between the criterion node and the judgment node, as defined by the following:

\[
\mathcal{P}(r) = \mathcal{P}(r \mid x, c_i),
\]
\[
\mathcal{P}(y_r) = \mathcal{P}(y_r \mid x, y_a, y_b, c_i, r_i).
\]

Generative reward models need to generate the sequences of criteria, rubric, and judgment for each trajectory. The judgment is a text sequence generated by the model, including explicit evaluation scores (\( \hat{\mathcal{R}}_a, \hat{\mathcal{R}}_b \)) for the paired responses (\( y_a, y_b \)). As a result, we can obtain the final preference choice: If \( \hat{\mathcal{R}}_a > \hat{\mathcal{R}}_b \), then \( y_a \) is the better; if \( \hat{\mathcal{R}}_a < \hat{\mathcal{R}}_b \), then \( y_b \) is the better. The distributions \( \mathcal{P} \) in Equation 3 are modeled using LLMs.

Given the datasets \( \mathcal{D} = \{(x, y_a, y_b, l)_i\}_{i=1}^N \), we construct a preference tree using the Monte Carlo rollout policy, where each parent node generates multiple child nodes (two in practice). The preference label \( l \) is annotated by human as ground truth. In the implementation, the sub-tree for each criterion is represented as a complete binary tree, containing 4 distinct evaluation scores. Each path from the root node to a leaf node represents a complete evaluation trajectory \( \mathcal{T} = \{x, y_a, y_b, c_i, r_i, y_r\} \). It is important to note that we adopt two distinct rollout strategies: For criteria, we use LLMs to directly generate text sequence \( \{c_1, \cdots, c_{|C|}\} \) relevant to the user query, similar to the brainstorming. For rubric and judgment, we employ a sampling strategy based on the probabilities of the output.

To optimize the LLMs policy under different criteria, we leverage the oracle label \( l \) and predefined rules to perform credit assignment for various criteria. Unlike traditional Monte Carlo rollout that estimate confidence scores, we do not rely on preference scores for each criterion. Instead, these sub-reward scores are estimated based on the oracle preference and predefined rules:

• Ranking Rule: Criteria with larger score margins indicate higher confidence in evaluation by LLMs. Consequently, the evaluation trajectory of the current criterion should be generated with higher priority.

• Consistency Rule: In each sub-tree with the criterion as the root node, when conflicting judgments arise, the majority voting result can be regarded as final judgments. This suggests that judgments which deviate from the majority are probable to be erroneous.

4LLMs experience compounding errors, where initial mistakes in generation propagate as the sequence length increases.
Figure 4: SyncPL primarily consists: The ranking process focuses on synthesizing preference data, while the learning process aims to enhance model through inference-time computations. The retrain process can further enforce the reasoning ability by introducing a special prompt: [Keep going] indicates that the previous criteria-based judgment is correct, while [Reflection] signifies an inconsistency between reasoning and the oracle label.

Based on these rules, we can collect pairwise preference training data from different tree node:

Criteria. We calculate the score margins (\( |\mathcal{R}_a - \mathcal{R}_b| \)) for all criteria and reorder the output sequence \( \{c_1, \cdots, c_{|C|}\} \) based on the ranking results (\(|C|\) is set to 10). The reordered sequence is denoted as \( c^+ \), while the original sequence serves as \( c^- \).

Rubric & Judgment. For each rubric and judgment synthesized based on criterion, we apply the consistency rule to filter out sub-trees: inconsistent results are treated as rejected samples (\( r^- \) and \( y_r^- \)), while consistent results with large score margins are selected as chosen samples (\( r^+ \) and \( y_r^+ \)).

4.2 Criteria Preference Learning

In generative reward modeling, high-quality preference pairs (\( c^+, c^- \)), (\( r^+, r^- \)), and (\( y_r^+, y_r^- \)) are utilized for training both RSFT and DPO. Specifically, RSFT employs chosen samples (\( c^+, r^+, y_r^+ \)) as training data, while DPO leverages paired preferences for its training. The reward model can optimize the synthesis process\footnote{In practice, we use \( \mathcal{P}_\theta(c|x) \) instead of \( \mathcal{P}_\theta(c|x, y_a, y_b) \) to reduce computational cost.} through preference learning, such as the loss function for *criteria* is defined as follows:

\[
\mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{ref}) = - \mathbb{E}_{(x, c^+, c^-) \sim \mathcal{D}}
\]
\[
\left[ \log \sigma \left( \beta \log \frac{\pi_\theta(c^+|x)}{\pi_{ref}(c^+|x)} - \beta \log \frac{\pi_\theta(c^-|x)}{\pi_{ref}(c^-|x)} \right) \right],
\]

where \( \pi_{ref} \) is typically the RSFT model, \( \beta \) is the hyperparameter that controls the proximity of the policy \( \pi_\theta \) to the original policy \( \pi_{ref} \).

4.3 Retrain with Long CoT Format

OpenAI’s o1-series models introduce inference-time scaling through a long CoT reasoning process. Inspired by this, we also explore utilizing the long CoT for GenRM, leveraging high-quality criteria-based judgments generated by SyncPL-SFT. Specifically, we reorganize the training data by alternating between chosen and rejected trajectories, aiming to stimulate the model’s reflection ability. Subsequently, we retrain SyncPL-o1 based on LLaMA3.1-8B using DPO as before.

4.4 Implementation

Rejection sampling is designed to eliminate low-quality synthetic data by applying a consistency rule to filter out invalid trajectories. This process ensures that only evaluation results consistent with both majority voting and oracle labels are retained. If each sample includes 10 distinct criteria, the number of trajectories required for synthetic data is \( ((2 + 4) \times 10) \). Clearly, this process involves a significant amount of inefficient sampling.

Efficiency improvement . To enhance the efficiency of data synthesis, we generate evaluation trajectories for only **one** criterion per dialogue during the RSFT synthesis. For DPO, we synthesize trajectories for all **10** criteria simultaneously to en-
sure comprehensive preference learning. This will results in nearly a \(10 \times\) speedup. During DPO, we select only the inconsistent samples between the backbone and RSFT models, resulting in the removal of approximately 70% of the total dataset.

5 Experiments

5.1 Evaluation Datasets and Details
In experiments, we employ three distinct human preference benchmarks to evaluate the reward models: RewardBench (Lambert et al., 2024), Auto-J (Li et al., 2023), and MT-Bench (Zheng et al., 2024). Specifically, RewardBench comprises four distinct subsets: chat, chat-hard, safety, and reasoning, containing 358, 456, 740, and 1,431 samples, respectively. Each subset is designed to evaluate model performance in helpfulness, instruction-following, safety, and reasoning. Additionally, Auto-J and MT-bench include 1,392 and 3,355 dialogues across diverse domains. We exclude samples labeled as *tie* from the Auto-J. For evaluation, we adopt consistency metric between model predictions and human annotations.

For baseline models, we employ the state-of-the-art closed-source generative models GPT-4, GPT-4o, and DeepSeek V3. For the discriminative RM, we select LLaMA3-8B RM with ultrafeedback using the BT objective. For the generative RM, we select Prometheus2, which trained by GPT-4 synthesized data. Additionally, CLoud leverages LLaMA3-405B to generate CoT reasoning paths using a hybrid methods for model training.

5.2 Out-of-Distribution Reward Modeling
We utilize Qwen2.5-32B-Instruct-GPTQ-Int8 to construct CoT data and train the LLaMA3.1-8B Distill model as our backbone model. SyncPL leverages dialogue data from UltraFeedback and OffsetBias as training set to construct a fine-grained reward model. It is notable that the data distribution of these training datasets differs significantly from that of the evaluation benchmarks, meaning the results in Table 1 can be regarded as out-of-distribution (OOD) evaluations. With RSFT, SyncPL achieves superior alignment with human preferences, outperforming the LLaMA3.1-8B Distill model by 2.8 points. Moreover, SyncPL-DPO further enhances reward modeling capabilities, yielding a 3.8 point improvement.

SyncPL-o1 is trained on synthetic data generated by SyncPL-SFT, as it can produce rejected data to better guide the model’s reflection ability compared to SyncPL-DPO. Through reinforcement learning, SyncPL-o1 achieves superior performance, surpassing LLaMA3.1-70B (84.2 v.s. 84.0). While it obtains optimal performance with a single inference step, this primarily depends on SyncPL’s ability to optimize process-level data (criteria-based judgment). The introduction of process supervision signals ensures the correctness of the reasoning chain in long CoT, fully unleashing its potential.

5.3 In-Distribution Reward Modeling
To evaluate in-distribution judge performance, we fine-tune Mistral-7B-Instruct using the Fennec (Liang et al., 2024) training data, which includes 3K dialogue samples from Auto-J (Li et al., 2023), along with CoT trajectory data annotated by GPT-4. For a fair comparison, we adopt Agreement (AGR) and Consistency (CNS) as performance metrics, following Auto-J. Notably, CNS measures only positional consistency without considering correctness. As observed in Table 2, the RSFT and DPO models demonstrate significant performance improvements over most baseline methods on both Auto-J and Fennec, achieving the highest agreement score of 58.41.

Dialogue Correction. The core capability of GenRM is to identify and rectify flaws in dialogues, thereby enhancing overall quality. To evaluate this, we test our model’s ability to assess and refine dialogues generated by Alpaca-13B (Taori et al., 2023) and LLaMA2-7B Chat (Touvron et al., 2023) in MT-Bench. Unlike previous pairwise evaluations, MT-Bench is a multi-turn dialogue evaluation benchmark that utilizes GPT-4 to score AI responses on a scale from 1 to 10, subsequently establishing ranking relationships based on these scores. Specifically, we generate multiple judgments for the responses and refine those with scores below 3. As shown in Table 3, after refinement, both Alpaca-13B and LLaMA2-7B Chat achieve higher scores. Moreover, the DPO model’s corrections outperform those of the RSFT model, demonstrating that improved evaluation feedback can lead to significant enhancements in evaluation quality.

6 Analysis

6.1 Scalability of SyncPL’s Branching
To evaluate whether different criteria contribute to performance scalability, we compare model performance across different training stages using
<table>
  <tr>
    <th>Evaluator</th>
    <th>Backbone</th>
    <th colspan="2">Reward Bench</th>
    <th colspan="3">Safety</th>
    <th colspan="2">Reasoning</th>
    <th colspan="2">Avg.</th>
    <th>Auto-J</th>
    <th>MT-Bench</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>Chat</th>
    <th>Chat H.</th>
    <th>Safety</th>
    <th>Reasoning</th>
    <th>Avg.</th>
    <th>Auto-J</th>
    <th>MT-Bench</th>
  </tr>
  <tr>
    <td>GPT4-o</td>
    <td></td>
    <td>96.1</td>
    <td>76.1</td>
    <td>86.6</td>
    <td>88.1</td>
    <td>86.7</td>
    <td>74.6</td>
    <td>80.7</td>
  </tr>
  <tr>
    <td>GPT4</td>
    <td></td>
    <td>95.3</td>
    <td>74.3</td>
    <td>87.6</td>
    <td>86.9</td>
    <td>86.0</td>
    <td>73.1</td>
    <td>79.9</td>
  </tr>
  <tr>
    <td>DeepSeek-V3</td>
    <td></td>
    <td>96.9</td>
    <td>79.8</td>
    <td>87.0</td>
    <td>84.3</td>
    <td>87.0</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Prometheus2-7B</td>
    <td></td>
    <td>85.5</td>
    <td>49.1</td>
    <td>78.7</td>
    <td>76.5</td>
    <td>72.4</td>
    <td>75.1</td>
    <td>74.6</td>
  </tr>
  <tr>
    <td>Prometheus2-8x7B</td>
    <td></td>
    <td>93.0</td>
    <td>47.1</td>
    <td>83.5</td>
    <td>77.4</td>
    <td>75.3</td>
    <td>80.0</td>
    <td>73.6</td>
  </tr>
  <tr>
    <td>LLaMA3-8B RM</td>
    <td></td>
    <td>94.7</td>
    <td>44.9</td>
    <td>66.2</td>
    <td>83.6</td>
    <td>72.4</td>
    <td>79.3</td>
    <td>78.1</td>
  </tr>
  <tr>
    <td>CLoud</td>
    <td></td>
    <td>94.3</td>
    <td>46.5</td>
    <td>78.6</td>
    <td>84.6</td>
    <td>76.0</td>
    <td>81.2</td>
    <td>79.9</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B</td>
    <td></td>
    <td>80.7</td>
    <td>49.8</td>
    <td>64.0</td>
    <td>68.1</td>
    <td>65.7</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LLaMA3.1-70B</td>
    <td></td>
    <td><b>97.2</b></td>
    <td>70.2</td>
    <td>82.8</td>
    <td><b>86.0</b></td>
    <td>84.0</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B Distill†</td>
    <td></td>
    <td>92.9</td>
    <td>61.8</td>
    <td>82.5</td>
    <td>79.9</td>
    <td>79.3</td>
    <td>76.5</td>
    <td>73.2</td>
  </tr>
  <tr>
    <td>SyncPL-RSFT-8B</td>
    <td></td>
    <td>94.1<sub>0.16</sub></td>
    <td>66.4<sub>0.25</sub></td>
    <td>84.9<sub>0.20</sub></td>
    <td>83.0<sub>0.73</sub></td>
    <td>82.1</td>
    <td>78.0</td>
    <td>74.8</td>
  </tr>
  <tr>
    <td>SyncPL-DPO-8B</td>
    <td></td>
    <td>93.3<sub>0.15</sub></td>
    <td>71.3<sub>0.33</sub></td>
    <td><b>86.5<sub>0.15</sub></b></td>
    <td>81.2<sub>0.53</sub></td>
    <td>83.1</td>
    <td>77.1</td>
    <td>74.1</td>
  </tr>
  <tr>
    <td>SyncPL-o1-8B</td>
    <td></td>
    <td>93.9<sub>0.19</sub></td>
    <td><b>73.2<sub>0.93</sub></b></td>
    <td>85.8<sub>0.77</sub></td>
    <td>83.7<sub>1.12</sub></td>
    <td><b>84.2</b></td>
    <td>77.8</td>
    <td>73.5</td>
  </tr>
</table>

Table 1: The experimental results on RewardBench, Auto-J, and MT–Bench, where bold numbers indicate the best performance among open-source, small-size LLMs. † represent our reimplementations.

![Line plots showing judgment accuracy for different criteria and models](page_101_670_1242_312.png)

Figure 5: The results of judgments ensemble based on different criteria. SyncPL-o1 utilizes long CoT for preference judgments and inherently incorporates multiple criteria. One key advantage of an o1-like model is that it requires only a single inference step, whereas other models must perform multiple steps depending on the number of criteria.

<table>
  <tr>
    <th rowspan="2">Evaluator</th>
    <th rowspan="2">C</th>
    <th colspan="2">w/ Tie</th>
    <th colspan="2">w/o Tie</th>
  </tr>
  <tr>
    <th>AGR ↑</th>
    <th>CNS ↑</th>
    <th>AGR ↑</th>
    <th>CNS ↑</th>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>1</td>
    <td>62.28</td>
    <td>86.28</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>GPT-3.5</td>
    <td>1</td>
    <td>44.41</td>
    <td>72.39</td>
    <td>59.86</td>
    <td>73.57</td>
  </tr>
  <tr>
    <td>Auto-J †</td>
    <td>1</td>
    <td>55.13</td>
    <td>82.44</td>
    <td>74.13</td>
    <td>87.26</td>
  </tr>
  <tr>
    <td>Fennec</td>
    <td>1</td>
    <td>55.36</td>
    <td>83.80</td>
    <td>68.63</td>
    <td>86.33</td>
  </tr>
  <tr>
    <td></td>
    <td>5</td>
    <td>55.80</td>
    <td>85.52</td>
    <td>74.14</td>
    <td>89.19</td>
  </tr>
  <tr>
    <td>SyncPL-RSFT</td>
    <td>1</td>
    <td>56.87</td>
    <td>87.92</td>
    <td>70.92</td>
    <td>88.99</td>
  </tr>
  <tr>
    <td></td>
    <td>5</td>
    <td>56.32</td>
    <td>87.86</td>
    <td>75.17</td>
    <td>89.79</td>
  </tr>
  <tr>
    <td>SyncPL-DPO</td>
    <td>1</td>
    <td>57.54</td>
    <td>90.66</td>
    <td>78.60</td>
    <td>93.32</td>
  </tr>
  <tr>
    <td></td>
    <td>5</td>
    <td><b>58.41</b></td>
    <td><b>91.52</b></td>
    <td><b>79.69</b></td>
    <td><b>93.72</b></td>
  </tr>
</table>

Table 2: For a fair comparison on the Auto-J benchmark, we use only five criteria (C). Additionally, methods marked with † represent our reimplementations.

<table>
  <tr>
    <th>Models</th>
    <th>Rating</th>
    <th>Refine Rate</th>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>8.96</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LLaMA2-70B Chat</td>
    <td>6.99</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LLaMA2-7B Chat</td>
    <td>6.26</td>
    <td>-</td>
  </tr>
  <tr>
    <td>w/ SyncPL-RSFT Corr</td>
    <td>6.85</td>
    <td>87.5%</td>
  </tr>
  <tr>
    <td>w/ SyncPL-DPO Corr</td>
    <td><b>7.08</b></td>
    <td>72.5%</td>
  </tr>
  <tr>
    <td>Alpaca-13B</td>
    <td>4.97</td>
    <td>-</td>
  </tr>
  <tr>
    <td>w/ SyncPL-RSFT Corr</td>
    <td>6.61</td>
    <td>95.0%</td>
  </tr>
  <tr>
    <td>w/ SyncPL-DPO Corr</td>
    <td><b>6.85</b></td>
    <td><b>98.8%</b></td>
  </tr>
</table>

Table 3: Results of dialogue correction.

RewardBench. As shown in Figure 5, both the Distill model and SyncPL-SFT exhibit a significant performance improvement as the number of criteria increases. This effect is particularly notable in reasoning tasks, where different criteria significantly enhance the model’s judgment accuracy, with maximum improvements of +18.2% and +12.6%, respectively. Furthermore, the SyncPL-DPO model optimizes the ability to select key criteria more effectively, allowing it to make accurate judgments with only single criterion. This demonstrates that our preference optimization method effectively substantiates the claims presented in the motivation.
Figure 6: The results demonstrate the effectiveness of self-synthesized CoT data in combination with SyncPL.

<table>
  <tr>
    <th>Models</th>
    <th>Chat</th>
    <th>Chat H.</th>
    <th>Safety</th>
    <th>Reasoning</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td><b>SyncPL-RSFT</b></td>
    <td>94.1</td>
    <td>66.4</td>
    <td>84.9</td>
    <td>83.0</td>
    <td>82.1</td>
  </tr>
  <tr>
    <td>w/ UltraF.</td>
    <td>94.7</td>
    <td>52.2</td>
    <td>81.0</td>
    <td>83.4</td>
    <td>77.9</td>
  </tr>
  <tr>
    <td>w/ OffsetB.</td>
    <td>89.9</td>
    <td>67.7</td>
    <td>85.4</td>
    <td>82.1</td>
    <td>80.7</td>
  </tr>
  <tr>
    <td><b>SyncPL-DPO</b></td>
    <td>93.3<sub>±0.8</sub></td>
    <td>71.3<sub>±1.9</sub></td>
    <td>86.5<sub>±1.6</sub></td>
    <td>81.2<sub>±1.8</sub></td>
    <td>83.1</td>
  </tr>
  <tr>
    <td>w/ UltraF.</td>
    <td>94.9<sub>±0.2</sub></td>
    <td>54.4<sub>±2.2</sub></td>
    <td>83.1<sub>±2.1</sub></td>
    <td>82.6<sub>±0.8</sub></td>
    <td>78.8</td>
  </tr>
  <tr>
    <td>w/ OffsetB.</td>
    <td>85.5<sub>±4.1</sub></td>
    <td>71.2<sub>±3.5</sub></td>
    <td>85.3<sub>±0.1</sub></td>
    <td>77.1<sub>±5.0</sub></td>
    <td>79.8</td>
  </tr>
</table>

Table 4: Results on different dialogue datasets (OffsetBias and UltraFeedback) based on LLaMA3.1-8B.

Additionally, the SyncPL-o1 model employs a long CoT format, leveraging the key criteria extracted by SyncPL-SFT. This method proves to be more optimization-friendly in experiments while ensuring stable performance improvements. It can be observed that SyncPL-o1 consistently achieves superior results compared to others across almost all sub-datasets.

6.2 Ablations of Different Dialogue Datasets

As shown in Figure 4, we find that OffsetBias significantly improves evaluation performance on Chat Hard but does not outperform UltraFeedback in Chat evaluations. The primary reason for this is the distribution shift between Chat Hard and other datasets. Using OffsetBias alone as the training set leads to reward hacking, causing performance degradation across all datasets except Chat Hard. To address this issue, we combined UltraFeedback and OffsetBias for training in our previous experiments. Nevertheless, the distribution gap affects the DPO training process, emphasizing the need for more suitable dialogue datasets in future research.

6.3 SyncPL with Self-Improvement

In this subsection, we further examine the necessity of leveraging data distillation from larger models to warmup the backbone model. To investigate this, we retrain the SyncPL-self series models using only self-synthesized CoT data generated by LLaMA3.1-8B. As shown in Figure 6, the results reveal that small size LLMs struggle to enhance certain skills through test-time scaling, as also observed by Guo et al. (2025). Moreover, our findings indicate a substantial performance gap between SyncPL-self and the distilled model, particularly in Chat and Safety tasks, with a degradation of up to 25.9 points. A major factor is the presence of bias in Chat Hard, which makes small size models prone to misjudgment. These results highlight the critical importance of high-quality CoT data in improving the reasoning of small size LLMs.

Additionally, we observe that even with self-improvement, SyncPL-self achieves consistent performance gains on Rewardbench (except for safety, likely due to “alignment tax”). Notably, it surpasses SyncPL-o1 in chat and reasoning sub-tasks.

6.4 Error Analysis

We conduct an analysis of error types in chat datasets, revealing that 60% of errors stem from the preference for shorter responses, 13% are attributed to intent misinterpretation and inherent model limitations, and 27% involve high ambiguity. For more details, please refer to the Appendix A.2.

7 Conclusion

In this work, we introduce SyncPL, which reformulates the generative reward modeling process into a criteria-based preference tree, enabling LLMs to more effectively identify key criteria. Our experiments not only validate the efficacy of multi-criteria training but also demonstrate that SyncPL-o1 achieves the most substantial gains in the long-CoT format, outperforming even the LLaMA3.1-70B model. Beyond improving performance, our approach introduces a effective mechanism for process supervision in generative modeling, offering a
promising solution to mitigate compounding errors stemming from model limitations.

8 Limitations

The Candidate Space of Criteria . A intuitive limitation lies in the candidate space of criteria that LLMs consider during judgments, which directly affects whether the model can select the optimal criterion for making judgments. In our work, we rely on LLMs to autonomously generate criteria for evaluation, making the effectiveness of this approach partially dependent on the model’s inherent capabilities. Possible ways to expand the candidate space include, but are not limited to: (1) Using more strong LLMs for data synthesis. (2) Constructing a candidate pool list and retrieving relevant criteria as needed. Since this study primarily aims to validate the feasibility of criteria-based preference learning, we do not extensively explore these potential extensions.

Pure RL v.s. Process Supervision . The key difference from DeepSeek R-zero lies in the learning approach. R-zero relies solely on reinforcement learning, optimizing through rule-based rewards. In contrast, our method first leverages process supervision to generate effective intermediate trajectories before applying reinforcement learning for further improvement. Notably, the R-zero study demonstrates that their approach is ineffective on small-scale LLMs. In contrast, our method is specifically designed to enhance the performance of these models. Moreover, since reinforcement learning necessitates a rigorously defined candidate set for effective optimization, the training process can quickly collapse if high-quality solutions are rarely sampled. Our method addresses this challenge by filtering candidates based on multiple criteria, thereby ensuring a search domain that is highly conducive to optimization.

Acknowledgments

We want to thank all the anonymous reviewers for their valuable comments. This work was supported by the National Science Foundation of China (NSFC No. 62206194), the Natural Science Foundation of Jiangsu Province, China (Grant No. BK20220488), the Young Elite Scientists Sponsorship Program by CAST (2023QNRC001), and the Priority Academic Program Development of Jiangsu Higher Education Institutions. We also acknowledge MetaStone Tech. Co. for providing us with the software, optimisation on high performance computing and computational resources required by this work.

References

Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D Chang, and Prithviraj Ammanabrolu. 2024. Critique-out-loud reward models. arXiv preprint arXiv:2408.11791.

Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862.

Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345.

Lichang Chen, Chen Zhu, Davit Soselia, Jiahui Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Odin: Disentangled reward mitigates hacking in rlhf. arXiv preprint arXiv:2402.07319.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30.

Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. 2023. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.02743.

Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691.

Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. 2024. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. 2023. Prometheus: Inducing fine-grained evaluation capability in language models. arXiv preprint arXiv:2310.08491.

Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. 2024. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787.

Lucas Lehnert, Sainbayar Sukhbaatar, Paul Mcvay, Michael Rabbat, and Yuandong Tian. 2024. Beyond a*: Better planning with transformers via search dynamics bootstrapping. arXiv preprint arXiv:2402.14083.

Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. 2023. Generative judge for evaluating alignment. arXiv preprint arXiv:2310.05470.

Xiaobo Liang, Haoke Zhang, Helan hu, Juntao Li, Jun Xu, and Min Zhang. 2024. Fennec: Fine-grained language model evaluation and correction extended through branching and bridging. Preprint, arXiv:2405.12163.

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522.

R Duncan Luce. 1959. Individual choice behavior, volume 4. Wiley New York.

Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. 2024. Generative reward models. arXiv preprint arXiv:2410.12832.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.

Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. 2024. West-of-n: Synthetic preference generation for improved reward modeling. arXiv preprint arXiv:2401.12086.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE.

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506.

Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. 2023. Branch-solve-merge improves large language model evaluation and generation. arXiv preprint arXiv:2310.15123.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.

Jiaming Shen, Ran Xu, Yennie Jun, Zhen Qin, Tianqi Liu, Carl Yang, Yi Liang, Simon Baumgartner, and Michael Bendersky. 2024. Boosting reward model with preference-conditional multi-aspect synthetic data generation. arXiv preprint arXiv:2407.16008.

Lingfeng Shen, Sihao Chen, Linfeng Song, Lifeng Jin, Baolin Peng, Haitao Mi, Daniel Khashabi, and Dong Yu. 2023. The trickle-down impact of reward (in-) consistency on rlhf. arXiv preprint arXiv:2309.16155.

Roger N Shepard. 1957. Stimulus and response generalization: A stochastic model relating generalization to distance in psychological space. Psychometrika, 22(4):325–345.

Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. 2022. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460–9471.

Hao Sun, Yunyi Shen, and Jean-Francois Ton. 2024. Rethinking bradley-terry models in preference-based reward modeling: Foundations, theory, and alternatives. arXiv preprint arXiv:2411.04991.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca.

Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu. 2024. Toward self-improvement of llms via imagination, searching, and criticizing. arXiv preprint arXiv:2404.12253.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti
Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Mingzhi Wang, Chengdong Ma, Qizhi Chen, Linjian Meng, Yang Han, Jiancong Xiao, Zhaowei Zhang, Jing Huo, Weijie J Su, and Yaodong Yang. 2024a. Magnetic preference optimization: Achieving last-iterate convergence for language models alignment. arXiv preprint arXiv:2410.16714.

Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. 2023. Pandaml: An automatic evaluation benchmark for llm instruction tuning optimization. arXiv preprint arXiv:2306.05087.

Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. 2024b. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXiv:2406.08673.

Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. 2024. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge. arXiv preprint arXiv:2407.19594.

Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. 2024. Regularizing hidden states enables learning generalizable reward model for llms. arXiv preprint arXiv:2406.10216.

Zhiwen Ye, Fraser Greenlee-Scott, Max Bartolo, Phil Blunsom, Jon Ander Campos, and Matthias Gallé. 2024. Improving reward models with synthetic critiques. arXiv preprint arXiv:2405.20850.

Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, et al. 2024. Self-generated critiques boost reward modeling for language models. arXiv preprint arXiv:2411.16646.

Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. 2024a. Advancing llm reasoning generalists with preference trees. arXiv preprint arXiv:2404.02078.

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024b. Self-rewarding language models. arXiv preprint arXiv:2401.10020.

Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488.

Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. 2024. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.15240.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36.
A Appendix

A.1 Training details
We utilize the DeepSpeed (Rasley et al., 2020) library, Zero Redundancy Optimizer (ZeRO) (Rajbhandari et al., 2020) Stage 3, and FlashAttention (Dao, 2023), along with a mixed precision computation approach using bfloat16 (BF16) and tf32 (TF32), across 8 NVIDIA A100 GPUs. For optimization, we set the number of gradient accumulation steps to 32. The initial and SFT models are trained with a learning rate of 5e-6, while the DPO model is trained with a learning rate of 5e-7. Each training stage consists of one epoch, and we set \( \beta = 0.1 \) for training the DPO model. The prompt templates used in our evaluation are provided in Table 6.

A.2 Case Study

Case 1: Length Bias

Query: Who created the Superman cartoon character?
Response A: The Superman cartoon character was created by Jerry Siegel and Joe Shuster in 1938.
Response B : (ground truth) Superman, the iconic comic book superhero, was created by writer Jerry Siegel and artist Joe Shuster. Superman first appeared in Action Comics 1, which was published by Detective Comics, Inc. (later DC Comics) in June 1938. The character’s immense popularity established him as one of the most enduring and recognizable figures in the superhero genre.
Judge : Accuracy of Information: This criterion ensures that the response correctly identifies the creators of the Superman ...
Response A directly answers the user query by naming Jerry Siegel and Joe Shuster as the creators of the Superman cartoon character. ... The response is clear and to the point, making it easy for the user to understand the answer. ... Response B also correctly identifies Jerry Siegel and Joe Shuster as the creators of Superman. However, it provides additional information about the character’s first appearance and the publisher, which, while interesting, is not directly asked for in the user query. ....

Case 2: Length Bias

Query: what language does argentina people speak?
Response A: The official language of Argentina is Spanish.
Response B : (ground truth) The primary language spoken in Argentina is Spanish. Specifically, the variety of Spanish spoken there is known as Rioplatense Spanish, which is characterized by its unique accent and the use of the pronoun "vos" instead of "tú" for the second person singular. Additionally, there are various indigenous languages spoken in different regions of the country, as well as immigrant languages such as Italian, German, and English, but these are much less widespread than Spanish.
Judge : Accuracy of Information: This criterion ensures that the response correctly identifies the primary language spoken in Argentina. It is crucial because providing incorrect information can lead to misinformation and dissatisfaction with the response. ...
Response A directly answers the user query by stating that the official language of Argentina is Spanish. This response is concise and to the point, which aligns well with the user’s query. It does not provide additional information that might not be relevant to the user’s query, which can sometimes be seen as a positive aspect of brevity. ... Response B provides more detailed information about the language spoken in Argentina, including the specific variety of Spanish and the use of indigenous and immigrant languages. While this information might be interesting and informative, it goes beyond the scope of the user’s query, which was simply to identify the primary language spoken in Argentina. This makes the response less focused and potentially less useful for the user’s immediate needs. Comparison: Response A is more aligned with the user’s query, providing a direct and concise answer to the question. Response B, while informative, includes additional details that were not requested by the user, which can be seen as unnecessary and potentially confusing. ...
Case 3: Length Bias

Query: You are given a paper citation, convert it to the requested citation style.
Chicago: Vaswani, Ashish, Shazeer, Noam, Parmar, Niki, Uszkoreit, Jakob, Jones, Llion, Gomez, Aidan N., Kaiser, Lukasz, and Illia Polosukhin. "Attention Is All You Need." arXiv, (2017). https://doi.org/10.48550/arXiv.1706.03762.
MLA:
Response A: Vaswani, Ashish, et al. "Attention Is All You Need." ArXiv, 17 June 2017, https://doi.org/10.48550/arXiv.1706.03762.
Response B : (ground truth) Vaswani, Ashish, et al. "Attention Is All You Need." arXiv, 2017, doi:10.48550/arXiv.1706.03762. In the MLA citation style, the citation would be formatted as follows: Author(s) of the source: Vaswani, Ashish, et al. Title of the source: "Attention Is All You Need." Title of the container (if the source is part of a larger work): arXiv Publication date: 2017 Location (such as a URL or page range): doi:10.48550/arXiv.1706.03762
Note that in MLA style, the authors are listed with the last name followed by the first name, and the names are separated by a comma. ...
Judge : Accuracy of Citation Style: This criterion evaluates whether the response correctly converts the citation to the MLA style as requested. ... Response A provides a direct conversion of the Chicago citation style to MLA style, adhering closely to the MLA format guidelines. It includes the author’s last name, first name, title of the source, title of the container, publication date, and DOI. The response is concise and directly answers the user query without unnecessary elaboration. ... Response B attempts to explain the MLA citation style but does not provide a direct conversion of the citation as requested. Instead, it offers a general explanation of how to format a citation in MLA style, which is not what the user query asked for. The response is informative but does not fulfill the specific request for a citation in MLA style. ...

Case 4: Model Capability Limitation

Query: You should capitalize the sentence according to the guide. Guide: Every other letter alternates between lower case and upper case. Sentence: A giant spider blocks your path.
Response A: (ground truth) Sure! Here’s the sentence with alternating capitalization: A glaNt SpIdEr BlOcKs YoUr PaTh.
Response B : A GiANT spider BLOCKS your path.
Judge : Consistency with the Guide: The response must adhere strictly to the guide provided, ensuring that every other letter is alternately in lower case and upper case. This criterion is crucial as it directly measures the AI’s ability to follow instructions accurately. Judge: Response A Score: 1 Analysis of Response A: Response A does not follow the guide provided in the user query. The guide specifies that every other letter should alternate between lower case and upper case, but Response A does not maintain this pattern. Instead, it seems to randomly capitalize letters, which does not meet the criteria set by the user. Response B Score: 5 Analysis of Response B: Response B correctly follows the guide provided in the user query. It alternates between lower case and upper case letters as specified, maintaining the pattern throughout the sentence. This response accurately capitalizes the sentence according to the guide, making it the superior response. Comparison: Response A fails to adhere to the guide’s instructions, leading to a score of 1. In contrast, Response B successfully alternates between lower case and upper case letters as instructed, earning a score of 5. Response B is superior as it accurately follows the guide and provides the correct capitalization of the sentence. ... Relevance to the Query: ... Accuracy of Capitalization: ... Response A does not maintain this pattern. Instead, it randomly capitalizes letters, which does not meet the criteria set by the user. ... It alternates between lower case and upper case letters as specified, maintaining the pattern throughout the sentence. ...
So, the final decision is Response B
Case 5: High Ambiguity

Query: I want you to act as a Tiktok Ads Keywords generator. Your task is to come up with keywords that can be used in Tiktok advertising campaigns. The keywords should be relevant to the target audience and the product or service being advertised. Your responses should not include more than 5 keywords per request.

Response A: (ground truth) Sure, I’d be happy to help! Can you please provide me with some information about the product or service being advertised, as well as the target audience? This will help me generate keywords that are relevant and effective for your TikTok advertising campaign.

Response B : TikTok is a popular short-form video sharing app that ... Here are a few TikTok ads keywords generator ideas: 1. Trends and Hashtags: TikTok users are always on the lookout for the latest trends and hashtags. By incorporating these into your ads, you can increase the chances of your ads being seen by the right audience. ... 5. Comedy: TikTok users love to laugh, so if your ad is funny, you can use comedy-related keywords to target TikTok users. 6. Gaming: TikTok users also love gaming, so if your ad is about a new game, you can use gaming-related keywords to target. ...

Judge : Relevance to the Query: ... Response A is a good start as it acknowledges the need for more information about the product or service and the target audience. However, it does not provide any specific keywords or a list of ideas as requested in the user query. It is more of a request for additional information rather than a direct response to the query. ... Response B provides a comprehensive overview of the types of keywords that can be used in TikTok advertising campaigns. It includes a variety of categories such as trends, music, dancing, fashion, comedy, gaming, celebrities, animals, contests, and quotes. This response is more aligned with the user query’s request for a list of keywords and ideas. However, it could have been more concise and directly provided the requested 5 keywords per request as specified in the user query. ...

As illustrated above, we identify several types of errors observed during inference. The majority of these errors (from Case 1 to Case 3) arise from a preference for shorter responses. However, these shorter responses are not necessarily incorrect but may simply reflect a misalignment with human preferences at the time, which favored longer answers. Additionally, the error in Case 4 stems from the model’s inability to distinguish variations in letter casing, leading to misjudgments in specific cases. Finally, we present a scenario where both responses have flaws in Case 5. However, since a choice must be made, the ground truth is set to A. In fact, Response B also provides a concrete answer and could be considered more helpful than A in certain contexts. This issue primarily stems from the ambiguity in criteria selection, as different judges may prioritize different key criteria, leading to variations in judgment.

<table>
  <tr>
    <th rowspan="2">Evaluator</th>
    <th colspan="4">Reward Bench</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>Chat</th>
    <th>Chat H.</th>
    <th>Safety</th>
    <th>Reasoning</th>
  </tr>
  <tr>
    <td>GPT3.5-turbo</td>
    <td>92.2</td>
    <td>44.5</td>
    <td>65.5</td>
    <td>59.1</td>
    <td>65.3</td>
  </tr>
  <tr>
    <td>LLaMA3-8B</td>
    <td>85.5</td>
    <td>41.6</td>
    <td>68.0</td>
    <td>64.8</td>
    <td>65.0</td>
  </tr>
  <tr>
    <td>SyncPL-e1-16B</td>
    <td>83.8</td>
    <td>49.8</td>
    <td>67.4</td>
    <td>67.9</td>
    <td><b>67.3</b></td>
  </tr>
</table>

Table 5: The experimental results on RewardBench.

A.3 Results of MoE models

We train an MoE model and evaluate it on RewardBench based on DeepSeek-V2-Lite, which consists of 16B total parameters with 2.4B active parameters. As shown in Table 5, despite relying on only 2.4B active parameters, it still outperforms strong models such as LLaMA3-8B, demonstrating the generalizability of our approach.
<table>
  <tr>
    <th>Step</th>
    <th>Content</th>
  </tr>
  <tr>
    <td>Criteria</td>
    <td>For evaluating human satisfaction with responses from an AI assistant based on a [User Query], we need to brainstorm and establish ten [Evaluation Criteria] directly linked to the user’s query. These criteria play a crucial role in objectively assessing response content, with higher priority and greater evaluation weight.<br>***<br>As an illustration:<br>1. Relevance: Evaluate whether the response is directly related to the user’s query.<br>2. Criterion: Assess the correctness of the information provided in the response. etc.<br>***<br>[User Query]:<br>{query}<br>***<br>Please return ten [Evaluation Criteria]:</td>
  </tr>
  <tr>
    <td>Rubrics</td>
    <td>Consider a [User Query] and [Evaluation Criteria] for evaluating response satisfaction. Reflect on these criteria and offer a comprehensive [Scoring Guideline] on a scale of 1-5 (1 represents 'Not at all satisfactory' and 5 represents 'Extremely satisfactory'). Ensure that these guidelines are closely tied to both the user query and the assessment criteria, allowing for a precise evaluation of possible responses to the user query. Conduct a detailed comparison of the [Scoring Guideline] to ease adherence and assist individuals in assigning reasonable scores.<br>***<br>[User Query]:<br>{query}<br>***<br>[Evaluation Criteria]:<br>{criteria}<br>***<br>Please return detailed [Scoring Guideline]:</td>
  </tr>
  <tr>
    <td>Judgments</td>
    <td>Given a [Dialogue Context] and a [User Query], please score the responses (A and B) from two AI assistants according to the [Evaluation Criteria] and [Scoring Guideline]. Ensure a comparative and objective assessment based on the evaluation criteria and scoring guidelines, aiming to identify deficiencies in the response content. Provide a final score of 1-5 along with relevant explanations.<br>***<br>[Dialogue Context]:<br>{context}<br>***<br>[User Query]:<br>{query}<br>***<br>[Evaluation Criteria]:<br>{criteria}<br>***<br>[Scoring Guideline]:<br>{scoring guidelines}<br>***<br>[The Start of Response A]:{response1}[The End of Response A]<br>***<br>[The Start of Response B]:{response2}[The End of Response B]<br>***<br>Please return [Judge Result] as follows:<br>Response A Score: 3 Analysis of Response A: Explanation of the score for the Response A.<br>Response B Score: 3 Analysis of Response B: Explanation of the score for the Response B. Comparison: Discuss the comparative strengths and weaknesses of Response A and Response B [Judge Result]:</td>
  </tr>
</table>

Table 6: Prompts for multi-step evaluation. The criteria in Scoring Guidelines and Pairwise-eval is regularly extracted from the output of the first step. The scoring guidelines in Pairwise-eval are the output of the second step.

Reviewer paper summaries: The paper introduces SyncPL (Synthetic Criteria Preference Learning), a generative reward modeling approach that leverages synthesized criteria-based Chains of Thought (CoT) trajectories to optimize reward models without heavy reliance on human-annotated preference data. SyncPL builds a criteria-based preference tree where each path represents a distinct reasoning trajectory, independently optimized using Reinforcement Learning (RL). By introducing process-level supervision via predefined meta-criteria, SyncPL mitigates the risk of compounding errors common in generative reward models. Experiments demonstrate significant performance gains across multiple human preference benchmarks (e.g., RewardBench, Auto-J, MT-Bench), showing improved alignment with human preferences, particularly for small-scale language models.

This paper introduces SyncPL (Synthetic Criteria Preference Learning), a new method for training AI models to better judge the quality of text responses. The core problem is that while using AI-generated reasoning (CoT) to train these judges is cheaper than using massive human-labeled datasets, this AI-generated reasoning can be flawed, leading the judge model astray.

SyncPL tackles this by structuring the judgment process like a tree based on different evaluation criteria (like clarity, accuracy, etc.). For a given user query and two potential AI responses (A and B), the system first brainstorms relevant criteria. Then, for each criterion, it generates scoring rules (rubrics) and makes a judgment, scoring both A and B.

The clever part is how SyncPL automatically figures out which AI-generated reasoning steps are trustworthy without needing humans to check each step. It uses two main rules: a 'Ranking Rule' that gives more weight to criteria where the scores for A and B are very different (suggesting that criterion was important for the decision), and a 'Consistency Rule' that filters out judgments that disagree with the majority opinion for a specific criterion.

Using these rules, SyncPL filters the AI-generated reasoning, keeping the good parts and discarding the noisy ones. This filtered data is then used to train the judgment model (GenRM) using techniques like RSFT and DPO.  Experiments show SyncPL models perform well, often better than standard approaches and even larger models, on several benchmark tests.
Summary of strengths: Innovative Methodology: SyncPL introduces a novel criteria-based preference tree framework for generative reward modeling, effectively synthesizing fine-grained process supervision data.

Robust Empirical Results: The approach achieves state-of-the-art performance improvements, particularly notable in out-of-distribution scenarios, outperforming large baselines (e.g., surpassing LLaMA3.1-70B model).

Process Supervision Approach: Explicit criteria and rule-based filtering significantly reduce compounding errors, improving the reliability and correctness of generative CoT trajectories.

Efficiency and Scalability: SyncPL demonstrates practical efficiency through rapid synthetic data generation and selective optimization of critical criteria, significantly reducing computational overhead.

- SyncPL tackles a real and important issue in training reward models: how to reliably use synthetic data (like CoT) without the model learning incorrect reasoning patterns due to errors in that data.

- The use of the Ranking and Consistency rules to automatically supervise the process of reasoning, without needing humans to label every intermediate thought, is a nice way to improve data quality while keeping costs down. This is a practical advantage.

- The method explicitly deals with the fact that judging response quality often involves balancing multiple factors. The tree structure and ranking rule offer a reasonable way to manage this complexity.

- The paper backs up its claims with solid experiments on standard benchmarks. SyncPL models generally outperform other generative reward models and even some larger standard models, which suggests the approach is effective.
Summary of weaknesses: Although the paper introduces the innovative SyncPL framework for generative reward modeling, it still raises certain concerns and questions that could potentially limit its effectiveness and general applicability:
Unclear Criteria Selection:
The paper lacks detail on how the synthetic evaluation criteria generated by the LLMs are validated or selected, raising concerns about completeness and effectiveness.

Potential Bias from Synthetic Criteria:
The paper doesn't sufficiently address how automatically generated criteria can avoid inherent biases or ensure improved generalizability. Additional evidence or discussion is needed.

- The Ranking and Consistency rules are defined by the researchers. While they seem to work well here, they are essentially educated guesses ('heuristics'). It's unclear how well they would work in all situations or if they are truly the best possible rules. Performance might be sensitive to how these rules are set up.

- It would be helpful to add a short discussion about how sensitive the results might be to the exact design of the Ranking and Consistency rules. For instance, what happens if the threshold for 'consistency' changes?


- The whole process kicks off with the AI generating evaluation criteria. If the AI fails to come up with good, relevant criteria initially, the rest of the process might be less effective. The paper acknowledges this limitation ("Candidate Space of Criteria") but doesn't fully explore the impact of weak starting points.

- Generating and evaluating reasoning paths for multiple criteria  could become computationally expensive, especially if more criteria or deeper reasoning steps are needed. The practical limits on how complex this tree can get need consideration.

- In Section 4.4, the description of how data is selected for DPO training (mentioning "inconsistent samples") is not clear enough. It needs a clearer explanation of exactly which samples are chosen and why, as this is crucial for understanding the training process.

- The comparison to 'o1-models' is interesting, but lacks detail. What specific aspects (beyond the format) are truly analogous? Does SyncPL-o1 scale its compute cost with reasoning steps in the same way? More specifics would strengthen this connection.
Comments, suggestions and typos: Please check the weakness

see the weakness section above.
Overall assessment scores: 3.5

3
Metareview score: 3.5
Metareview: This paper presents a new Generative RM that leverages multiple self generated criteria-based judges for modeling preferences.
Summary of reasons to publish: The idea of utilizing criteria-based preference tree for reward modeling is interesting. The empirical results demonstrate the effectiveness of proposed approach. Plus, the direction of scaling self-generated criteria and rubrics is new/interesting.