Venue: NAACL 2025

# Self-Generated Critiques Boost Reward Modeling for Language Models

 Yue Yu1, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu,

Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan,

Chao Zhang2 , Melanie Kambadur, Dhruv Mahajan, Rui Hou3

Meta GenAI

Work done during an internship at Meta Llama Team.With Georgia Institute of Technology.Corresponding Authors: Yue Yu cyuevy@meta.com> and Rui Hou <rayhou@meta.come.

###### Abstract

Reward modeling is crucial for aligning large language models (LLMs) with human preferences, especially in reinforcement learning from human feedback (RLHF). However, current reward models mainly produce scalar scores and struggle to incorporate critiques in natural language format. We hypothesize that predicting both critiques and the scalar reward would improve reward modeling ability. Motivated by this, we propose Critic-RM, a framework that improves reward models using self-generated critiques without extra supervision. Critic-RM employs a two-stage process: generating and filtering high-quality critiques, followed by joint fine-tuning on reward prediction and critique generation. Experiments across benchmarks show that Critic-RM improves reward modeling accuracy by 3.7%-7.3% compared to standard reward models and LLM judges, demonstrating strong performance and data efficiency. Additional studies further validate the effectiveness of generated critiques in rectifying flawed reasoning steps with 2.5%-3.2% gains in improving reasoning accuracy.

## 1 Introduction

Reinforcement Learning from Human Feedback (RLHF) has been widely adopted to align large language models (LLMs) with human preferences Ouyang et al. (2022); Touvron et al. (2023); Dubey et al. (2024). Central to the RLHF process is the reward model (RM), which is trained to assign scores that quantify how well the model's outputs align with human judgments. The reward model defines optimization direction during training (e.g., reward signal in PPO), encouraging a policy LLM to generate more helpful, honest, and harmless responses ultimately enhancing the model's generation quality in real-world applications.

Standard reward models are typically trained using preference pairs and optimized to produce a single scalar score for each response. However, outputting a scalar score not only is hard to interpret but also fails to fully leverage the inherent language modeling capability that LLMs obtain from pretraining and post-training Zhang et al. (2024). Consequently, these reward models tend to be less data-efficient and prone to robustness issues, such as reward hacking Skalse et al. (2022); Chen et al. (2024b). Such limitations hinder the quality of feedback signals in RLHF and lead to suboptimal policy updates. On the other hand, the LLM-as-a-jadge paradigm offers an alternative, where the LLM first generates a critique and then optionally provides a discrete score as a quality proxy for a response Zheng et al. (2023). Combining the strengths of both paradigms -- integrating the interpretability and structured critique of LLM-as-the-judge with the scalar optimization framework of reward models -- has the great potential to address the limitations of each method and yield more robust and effective reward signals.

Despite its great premise, incorporating critiques into reward modeling presents several challenges.

(1) _Conflicting objectives_: Critique generation requires language modeling, while reward models provide scalar outputs, complicating its integration into language modeling. (2) _Evaluator limitations_: Off-the-shelf LMs are often not good evaluators, while additional fine-tuning requires costly humanized or annotated critiques. Recent work Ye et al. (2024) directly incorporates critiques generated from off-the-shelf LLMs for reward modeling, while Ankner et al. (2024) and Zhang et al. (2024) design a joint training approach for learning to generate the critique as well as rewards simultaneously via knowledge distillation. These methods typically rely on a strong teacher LLM to generate high-quality critiques, which can be costly and inefficient to obtain at scale in practice. Moreover, they

cannot be used to improve frontier models when a stronger teacher model does not exist.

In this work, we introduce **Critic-RM**, a new framework that enhances reward models using synthetic critiques, without relying on strong LLM teachers. Our approach draws inspiration from recent advances in self-improving language models Yuan et al. (2024); Wu et al. (2024), where models are iteratively refined using data generated by themselves. To apply a similar LLM self-improving paradigm in reward modeling, we hypothesize that it is crucial to inject LLM's critique generation ability into this process. Specifically, Critic-RM leverages an instruction-finetuned LLM as the backbone, which generates multiple candidate critiques, each with a discrete score (as explained below, for filtering critiques; not our final reward) for individual responses. However, these critiques can vary in quality, and poor-quality critiques often result in flawed quality predictions. To tackle this issue, we first apply a consistency-guided filtering technique, retaining only critiques whose scores align with human-annotated preference labels1. To further enhance the quality of these synthetic critiques, we additionally propose two strategies, _summarization_ and _ranking_, to refine the critiques used in training the reward model.

Footnote 1: This discrete score is only used for filtering critiques and being different from the final reward score of Critic-RM. Our Critic-RM eventually produces a continuous score, as explained in Section 3.3.

Once critiques are generated for each response, the main challenge lies in designing an effective training strategy to combine critique modeling and scalar reward prediction objectives. While LLMs benefit from learning through diverse critiques for each response Ho et al. (2023), reward modeling is prone to overfitting Dubey et al. (2024); Zhu et al. (2024); such a contradiction makes it nontrivial to determine the optimal learning steps. To address this issue, we introduce a simple weighting balancing strategy, where the model initially focuses on critique modeling loss, then gradually transitions to predicting rewards based on both the response and the critique. This approach balances the two learning objectives, allowing the model to excel at both _high-quality critique generation_ and _accurate reward prediction_.

The contributions of our work can be summarized as follows: (i) We propose Critic-RM, a framework to allow LLMs to take advantage of self-generated critiques for reward modeling. Critic-RM does not rely on supervision from additional teacher models compared to standard reward models. (ii) We propose a self-refinement technique to select high-quality critiques, and design a simple yet effective weight scheduling strategy to balance the learning objectives between critique generation and reward modeling. These techniques collaboratively equip the model with the dual capabilities of _high-quality critique generation_ and _accurate reward prediction_. (iii) We conduct experiments on three benchmarks covering over ten tasks on various domains, demonstrating the effectiveness of Critic-RM in precise reward modeling across diverse scenarios. Additional studies confirm the utility of generated critiques for Critic-RM in identifying and correcting mistakes made by LLMs.

## 2 Related Work

**Reward Models.** Building an accurate and robust reward model is a critical step for RLHF pipelines. Earlier work trains reward models with the ranking loss between chosen and rejected responses with the Bradley-Terry model Bradley and Terry (1952). To further improve upon this pipeline, Wang et al. (2024e,d,a) design fine-grained attributes to predict rewards toward different aspects, Chen et al. (2024b); Shen et al. (2024); Liu et al. (2025); Rame et al. (2024) promote the robustness of reward modeling via improved training techniques or model ensembling. More related to us, several very recent works (concurrent to us) also study generative reward modeling. Ye et al. (2024) directly augment

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Baselines** & **Input Format** & **Output Format** & \begin{tabular}{c} **Critique** \\ **Generation** \\ \end{tabular} & \begin{tabular}{c} **Require** \\ **Training** \\ \end{tabular} & 
\begin{tabular}{c} **Additional** \\ **Teacher Models** \\ \end{tabular} \\ \hline Standard RM Bradley and Terry & Single Response & Continuous Score & ✗ & ✓ & ✗ \\ BLAIF Lee et al. & Single Response & Continuous Score & ✗ & ✓ & ✓ \\ LLM-as-a-jadge Zheng et al. & Response Pairs & Discrete Score & ✓ & ✗ & ✗ \\ SynRM Ye et al. & Single Response + Critique & Continuous Score & ✗ & ✓ & ✓ \\ CLoud Anzher et al. & Single Response & Critique + Continuous Score & ✓ & ✓ & ✓ \\ GenRM Zhang et al. & Single Response & Critique + Reward Token & ✓ & ✓ & ✓ \\
**Critic-RM (Ours)** & Single Response & Critique + Continuous Score & ✓ & ✓ & ✗ \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison of our proposed method Critic-RM and closest baselines.

 

the response with critiques from a teacher model for reward modeling without training the RM for critique generation, and some studies (Zhang et al., 2024; Ankner et al., 2024; Mahan et al., 2024) attempted to learn reward models with additional critiques objective, with similar focus of our study. However, these methods rely on high-quality critiques from stronger teacher models for training, which can be costly to obtain in practice. They also don't provide a solution to reward modeling based on frontier LLMs where a teacher model doesn't exist. Also, they lack a unified approach to improve the quality of the critiques. Besides, Zhang et al. (2024) is specific to verifying math problem correctness, and being hard to map to subjective domains where there are no ground-truth answers.

**LLM-as-a-judge.** Recently, large language models (LLMs) have been proposed as cost-effective alternatives to human evaluation, and act as proxies for assessing text quality. Such methods often first provide explanations for judgments of the response, then output a discrete score or preference label as the prediction (Zheng et al., 2023; Li et al., 2023; Yan et al., 2024). CriticGPT (McAleese et al., 2024) has also extended this line of work into coding tasks, where the LLM critic models is fine-tuned to pinpoint problems in code from real-world assistant tasks. However, using off-the-shelf LLMs for evaluation introduces the risk of bias (Bavaresco et al., 2024), and they can be easily misled (Zeng et al., 2024). To address these challenges, recent studies (Wu et al., 2024; Wang et al., 2024b; Kim et al., 2024) have focused on collecting high-quality response pairs to train more accurate and reliable LLM-based evaluators.

**Self-alignment Techniques.** Aligning LLMs with human preferences often requires massive human annotations. To alleviate this reliance on human efforts, self-alignment leverages the model's own capabilities to refine its responses and align them with desired behaviors. Saunders et al. (2022); Madaan et al. (2023) use LLM itself to refine the original response at the inference time. Li et al. (2024) generate instruction prompts for web documents and subsequently select high-quality examples for instruction fine-tuning. Lee et al. (2024); Sun et al. (2024) leverage LLMs to create preference labels efficiently, Yuan et al. (2024) employ LLM itself to rank different responses to provide its own rewards during training, Pang et al. (2024); Gulcehre et al. (2023) improve LLM reasoning abilities through self-generated reasoning steps. A recent study (Wang et al., 2024) also employs self-improving techniques to train text evaluators, but it focuses on pairwise evaluation and generating synthetic preference pairs. In contrast, we combine self-generated _critiques_ with human-annotated preference pairs to enhance reward modeling performance.

## 3 Methodology

### Preliminaries

**Reward Modeling.** Let \(\mathcal{X}\) and \(\mathcal{Y}\) denote the space of prompts and responses, respectively. In the RLHF pipeline, human feedback is typically collected in the form of pairwise preferences between two responses \((y^{+},y^{-})\in\mathcal{Y}^{2}\) to a given prompt \(x\in\mathcal{X}\). Then, the preference dataset can be written as \(\mathcal{D}=\left\{(x_{i},y_{i}^{+},y_{i}^{-})\right\}_{i=1}^{|\mathcal{D}|}\), where the preference for \(y^{+}\) over \(y^{-}\) is denoted as \(y^{+}\succ y^{-}\). To model the pairwise preferences, the learning objective is to maximize the probability with Bradley-Terry model (Bradley and Terry, 1952) as

\[p\left(y^{+}\succ y^{-}\mid x\right)=\frac{\exp\left(r\left(x,y^{+}\right) \right)}{\exp\left(r\left(x,y^{+}\right)\right)+\exp\left(r\left(x,y^{-} \right)\right)}. \tag{1}\]

In practice, the reward model \(r_{\psi}\) is trained to minimize the following empirical negative log-likelihood loss (Stiennon et al., 2020):

\[\ell_{\text{f}}(\psi)=-\mathbb{E}_{\left(x,y^{+},y^{-}\right)\sim p}\log\left( \sigma\left(r_{\psi}\left(x,y^{+}\right)-r_{\psi}\left(x,y^{-}\right)\right)\right) \tag{2}\]

where \(\sigma\) denotes the sigmoid function.

**Problem Setup.** In this work, we investigate the usage of off-the-shelf instruction-finetuned LLM \(\mathcal{M}_{\theta}\) as the backbone for both the _critique generation model_ and _reward model_. Specifically, we denote

Figure 1: The framework of Critic-RM. For each preference pair, the LLM generates candidate critiques with discrete scores. Instance-level filtering reduces conflicts with preference labels, followed by quality-aware refinement to improve reward model training.

 

the critic generation model as \(g_{\phi}=h_{\texttt{g}}\circ\mathcal{M}_{\theta}\) and the reward model as \(r_{\psi}=h_{t}\circ\mathcal{M}_{\theta}\), where \(h_{\texttt{g}}\) and \(h_{t}\) stand for the language modeling head (inherited from the original \(\mathcal{M}_{\theta}\)) and reward modeling head (randomly initialized).

**Overview of Critic-RM.** The framework of Critic-RM is shown in Figure 1. Critic-RM generates candidate critiques for each prompt-response pair, filters noisy rationales to enhance preference pairs, and applies joint training to improve both critique generation and reward modeling. Section 3.2 provides more details for each step.

### Critique-augmented RM Training

**Overview.** To integrate the critiques into the reward modeling step, we view critiques as "latent variables", which serve as an intermediate variable between the response and the final reward. Specifically, we denote \(z^{+},z^{-}\) as critiques for chosen and rejected responses \(y^{+},y^{-}\) with prompt \(x\), respectively. Then, the overall learning objective \(p\left(y^{+}\succ y^{-}\mid x\right)\) can be recast as

\[p(y^{+}\succ y^{-}\mid x) =\sum_{z^{+},z^{-}}p(y^{+}\succ y^{-},z^{+},z^{-}\mid x)\] \[=\sum_{z^{+},z^{-}}p(y^{+}\succ y^{-}\mid z^{+},z^{-},x)\] \[\quad\times p^{*}(z^{+}\mid y^{+},x)\times p^{*}(z^{-}\mid y^{-} ,x). \tag{3}\]

Since \(p^{*}(\cdot\mid y,x)\) stands for the oracle distribution for critiques and is often not intractable, we aim to leverage the critic generation model \(g_{\phi}\) to generate the approximate distribution \(q_{\phi}(z\mid y,x)\). By adopting the Jensen's inequality (Details in Appendix A), the training objective can be written as the combination of

\[\mathcal{L}=-\log p(y^{+}\succ y^{-}\mid x)=\ell_{\text{r}}+\ell_{\text{c}}. \tag{4}\]

To interpret Eq. 4, it decomposes the reward model learning objective into two parts: (1) _Preference Modeling Loss with Critiques \(\ell_{r}\)_: the reward model \(r_{\theta}\) learn to predict the reward for each response conditioned on critiques; (2) _Critique Generation Loss \(\ell_{\text{c}}\)_: the LLM generation \(g_{\theta}\) is trained to generate critiques to approximate the oracle distribution \(p^{*}(\cdot\mid y,x)\). We will discuss how to train the reward model \(r_{\theta}\) and critique generation model \(g_{\theta}\) in the following.

#### 3.2.1 Critique-augmented Reward Prediction

To enable the reward model \(r_{\theta}\) to learn the preference with critiques (i.e. \(\ell_{\text{p}}\)) can be straightforward, as we only need to modify the input by augmenting response with critiques as

\[\begin{split}&\ell_{t}(x,y^{+},y^{-},z^{+},z^{-})=-\log p\left(y ^{+}\succ y^{-},z^{+},z^{-}\mid x\right)\\ &=-\log p\left(r_{\psi}(x,[y^{+};z^{+}])>r_{\psi}(x,[y^{-};z^{-} ])\right).\end{split} \tag{5}\]

In this way, for each prompt, the reward model will learn to generate the reward based on both responses and critiques. In practice, we put the critiques after the response and add a special token at the end of the critique for calculating the reward.

#### 3.2.2 Rationale Generation & Filtering

For critique generation loss, approximating \(p^{*}(\cdot\mid y,x)\) can be nontrivial as the primary challenge lies in the lack of high-quality critique annotations. To ensure the quality of the critiques, our key hypothesis is that _good critiques for responses should align well with human preference labels_. With this in mind, we design a generate-then-filter framework to create high-quality supervision signals for critique model training.

**Critique Generation.** To generate critiques without relying on stronger LLMs, we first prompt the LLM \(\mathcal{M}_{\theta}\) (with the same backbone as the reward model) and sample a set of \(N\) candidate critiques for each input prompt and response \((x,y)\) by following the procedure of the LLM-as-a-judge pipeline as \((\widehat{z}_{i},s_{i})_{i=1}^{N}\sim g_{\phi}(x,y)\), where \(\widehat{z}\) is the generated critique and \(s\) is a discrete score ranging from 1 to 10, indicating the quality of the response.

**Instance-level Critique Filtering.** To reduce the potential noisy critiques and encourage the consistency between critiques and preference labels, we propose to first retain instances guided by the score generated by the judge in the previous score as \(\mathcal{D}_{\textrm{sub}}=\{(x,y^{+},y^{-})\mid\bar{s}(x,y^{+})>\bar{s}(x,y^{- })\}\), where \(\bar{s}(x,y^{+})=\sum_{i=1}^{N}s_{i}^{+}/N\) and \(\bar{s}(x,y^{-})=\sum_{i=1}^{N}s_{i}^{-}/N\) stand for the average score for chosen and rejected responses, respectively. By applying this filtering process, we enhance the consistency of critiques with human preferences and minimize the impact of noisy instances.

**Quality-aware Critique Refinement.** The previous step mainly focuses on instance-level denoising, while for each (prompt, response) pair, the quality of different critiques also varies. To further improve the quality of critiques, we design a Metajudge-based technique Wu et al. (2024) to leverage LLM \(\mathcal{M}_{\theta}\) again to further refine the critiques in \(\mathcal{D}_{\textrm{sub}}\), with two possible variants:

* **Summarization-based Refinement**: We adopt the LLM as a summarizer to write 'meta 

critiques' given different critiques so that the LLM can identify the most common, reasonable feedback while mitigating the impact of the potential incorrect feedback. The final critique can be written as \({\mathcal{Z}_{\text{sumn}}=(z_{i})^{K}_{i=1}\sim g_{\phi}(x,y,\Pi_{j=1}^{N} \widehat{z}_{j})}\), where \(\Pi_{j=1}^{N}\widehat{z}_{j}\) is a permutation of \(N\) initial critiques. By sampling over different permutations of critiques, we can generate more diverse critiques for model training.
* **Ranking-based Refinement**: We use the LLM as a meta-juudge to create evaluation scores for critiques. Specifically, for each critique \(\widehat{z}_{i}\), we prompt the LLM to generate a discrete score from 1 to 10 as \(m_{i}\sim g_{\phi}(x,y,\widehat{z}_{i})\), which serves as a proxy for critique quality estimation. Then, we only retain top-\(K\) ranked Critiques as \({\mathcal{Z}_{\text{rank}}=(z_{i})^{K}_{i=1}=\mathrm{Top}\text{-}\mathrm{K}( \{\widehat{z}_{i}\}_{i=1}^{N})}\). In this way, we can preserve the critiques with the highest quality identified by the model itself.

Final Loss for Critique GenerationFrom the previous step, we augment the training set \({\mathcal{D}_{\text{sub}}}\) with self-identified high-quality critiques, denoted as \({\mathcal{D}_{\text{sub}}}=\{(x,y^{+},y^{-},{\mathcal{Z}^{+}},{\mathcal{Z}^{ -}})\}\). With the self-generated high-quality critiques \({\mathcal{Z}}\), we aim to use them to approximate the distribution of oracle distribution as \(p^{*}(z~{}|~{}y,x)=\mathbb{I}(z\in{\mathcal{Z}})/K\). Directly using this distribution in backward KL loss in Eq. 4 may lead to policy and entropy collapses Sessa et al. (2024); Agarwal et al. (2024). As a result, we use _forward KL_ loss to approximate this learning objective. Then using the empirical distribution, the KL divergence becomes:

\[\begin{split}\ell_{\text{c}}({\mathcal{Z}};x,y)&= {\mathcal{D}_{\text{KL}}(p^{*}(z~{}|~{}y_{i},x_{i})||q_{\phi}(z~{}|~{}y_{i},x_{ i}))}\\ &=\operatorname{\mathbb{E}}_{z\sim p^{*}(\cdot|y_{i},x_{i})}[ \log p^{*}(z~{}|~{}y_{i},x_{i})\\ &\qquad\qquad\quad-\log q_{\phi}(z~{}|~{}y_{i},x_{i})]\\ &=-\frac{1}{K}\sum_{z\in{\mathcal{Z}}}\log q_{\phi}(z~{}|~{}y, x)+\text{const.}\end{split} \tag{6}\]

Then, the overall loss for critique generation can be written as \(\ell_{\text{c}}(x,y^{+},y^{-},{\mathcal{Z}^{+}},{\mathcal{Z}^{-}})=\ell_{ \text{c}}({\mathcal{Z}^{+}};x,y^{+})+\ell_{\text{c}}({\mathcal{Z}^{-}};x,y^{- })\).

#### 3.2.3 Joint Learning of Critique Generation and Reward Modeling

To combine the reward modeling loss (Eq. 5) and critique generation loss (Eq. 6), one challenge lies in the different learning objectives for these two terms: for _critique generation_, the model \(g_{\phi}\) will benefit more from fine-tuning with diverse critiques from \({\mathcal{Z}}\). On the contrary, the reward model \(r_{\psi}\) is often observed with overfitting issues when finetuning with more than one round. To resolve this issue, we design a dynamic weight schedule approach, where we assign an additional weight \(\lambda(t)\) on Eq. 4, which is relevant to the training step \(t\), to balance between these two objectives as

\[\begin{split}{\mathcal{L}}(\phi,\psi)=& \operatorname{\mathbb{E}}_{(x,y^{+},y^{-},{\mathcal{Z}^{+}},{\mathcal{ Z}^{-}})\in{\mathcal{D}_{\text{sub}}}}\\ &\left[\lambda(t)\cdot\ell_{\text{c}}(\phi)+(1-\lambda(t))\cdot \ell_{\text{f}}(\psi)\right],\end{split} \tag{7}\]

where \(\lambda(t)\) is defined as

\[\lambda(t)=\begin{cases}1,&0<t<(K-1)T\\ 1-\beta\times\frac{t-(K-1)T}{T}.&(K-1)T<t<KT\end{cases} \tag{8}\]

Here, \(T\) represents the total number of training steps in one epoch. This approach allows the model to focus on critique generation during the initial phase of training and shifts to reward learning in the final round, mitigating the overfitting issue in the reward model.

### Critic-RM Inference

Compared to standard reward models, Critic-RM involves an additional step for each (prompt, response) pair during inference. Specifically, given the (prompt, response) pair \((x,y)\), the model will first generate a critique \(z\sim q_{\phi}(\cdot|x,y)\), then predict the reward for the response as \(r=r_{\psi}(x,[y,z])\).

Inference-time ScalingFollowing recent studies Ankner et al. (2024); Zhang et al. (2024), we also conduct inference-time scaling Wang et al. (2023) to improve performance. Specifically, we generate a set of \(m\) critiques as \({\mathcal{Z}}=\{z_{i}\}_{i=1}^{m}\sim q_{\phi}(\cdot|x,y)\) with non-zero temperatures, then predict the reward for the response as the average of reward over different critiques as \(r=r_{\psi}(x,[y,z_{i}])/m\).

## 4 Experiments

### Experiment Setup: Data Generation

To ensure the preference pairs are representative, we use both public and synthetic datasets for reward model training.

Public Preference DatasetsWe choose a set of datasets for reward model training with human-generated preference labels mainly from public, open-sourced datasets Ivison et al. (2024); Wang et al. (2024). We include the following datasets: (1) _General Chat Domain_: We include datasets from ChatArena Zheng et al. (2023) and AlpacaFarm-Human-Pref Dubois et al. (2023). (2) _Helpfultness Data_: We leverage Help 

Steer2 Wang et al. (2024)2 to create preference data. (3) _Reasoning_: We use Evol-instruct Xu et al. (2023) which contains preference pairs for complex instruction following, coding-related tasks. (4) _Safety_: We employ PKU-SafeRLHF Dai et al. (2024), which includes safety-related prompts paired with both safe and unsafe responses to form preference pairs.

Footnote 2: We follow the same approach as Wang et al. (2024a) to aggregate the score from different dimensions only only keep chosen/rejected pairs with score difference > 0.1.

**Synthetic Preference Datasets**: To incorporate additional preference supervision from different domains, we further include synthetic data using Llama-3.1 models.3 Specifically, for the math domain, we consider questions in GSM8K Cobbe et al. (2021) and the MATH dataset Hendrycks et al. (2021). For each math question, we use Llama-3.1-8b-instruct, and Llama-3.1-70b-instruct to generate candidate solutions with the prompt "_Given the following problem, reason step-by-step and give a final answer to the problem._", and generate multiple candidate solutions for a given prompt. We use those responses that lead to correct solutions as the chosen response while considering those responses with incorrect solutions as the rejected response. In the safety domain, we generate synthetic prompts following the safety principles outlined in SafeRLHF Dai et al. (2024) (e.g., Hate Speech, Offensive Language, Discrimination, Violence). To ensure balance, we also include scenarios where the model should not refuse to respond (e.g., Figurative Language, Safe Targets testing for ambiguous meanings) to avoid skewing the data toward over-conservatism.

Footnote 3: These synthetic data are used for both Critic-RM and our direct baselines.

### Experiment Setup: Evaluation Datasets

**Evaluation Benchmarks for Reward Models.** In our experiments, we mainly evaluate on _RewardBench_Lambert et al. (2024), which contains a collection of prompt-chosen-rejected triplets across chat, reasoning, and safety domains. Beyond RewardBench, we also aim to test the out-of-distribution generalization ability of reward models including _CrossEval_Zhong et al. (2024), _QA Feedback_Wu et al. (2023) and _SHP_Ethayarajh et al. (2022). For all tasks, we use the standard evaluation protocol and use _accuracy_ as the main metric. More details are in Appendix B.1.

**Evaluation Benchmarks for Critic Models.** We employ _CriticBench_Lin et al. (2024), a benchmark to evaluate LLMs' ability to critique and improve their reasoning across various tasks. We consider two dimensions for evaluation: (1) _Critique Accuracy_: where F1 Score is used to evaluate the correctness of critiques; (2) _Correction Accuracy_: where Accuracy is used to evaluate whether the model can generate correct answers based on critique feedback. More details are in Appendix B.2.

### Baselines

We consider baselines from three different groups: (1) _LLM-as-a-judge Baselines_: We consider Llama-3.1-70B/405B Dubey et al. (2024), GPT-4 and GPT-4o Achiam et al. (2023), Gemini-1.5-pro Reid et al. (2024) and recently proposed self-taught evaluator Wang et al. (2024b) based on Llama-3-70B for comparison. (2) _Standard Reward Models_: This line of models only outputs a scalar score for each (prompt, response) pair. We compare with standard RM Stiennon et al. (2020), Coherence-0514, SteerLM-RM Wang et al. (2024c), Nemotron-RM Adler et al. (2024). (3) _Reward Model with Critiques_: These studies are mostly relevant to us as they also leverage critiques to improve reward models. We compare with SymRM Ye et al. (2024) which directly augments responses with critiques for reward modeling, and CLoud Ankner et al. (2024) which jointly learn to generate critiques and predict rewards. It is worth noting that for most relevant baselines (e.g. RM, SynRM, CLoud), we reimplement those baselines with the same training data and backbone to ensure the _comparison is fair and meaningful_. We do not consider some reward model training techniques Wang et al. (2024c,a) as they focus on designing better learning objectives for standard reward models, which are orthogonal to our focus.

### Implemenation Details

We use Llama3.1-70B-Instruct Dubey et al. (2024) as the backbone in our main experiments. For critique generation, we set the temperature \(\tau=0.9\) and sample \(N=10\) candidate critiques for each response. For the critique filtering, we set \(K=2\) to select top-2 responses. For model fine-tuning, we use the Adam optimizer Kingma and Ba (2014) with the learning rate 2e-6, weight decay 0.1 and dropout 0.1. We set the global batch size to 64, \(\beta\) in Eq. 7 to 0.9 and train the model with \(2\) epochs. We observe that there exist several examples in AlpacaEval and ChatArena that share similar prompts 

with the target evaluation tasks, and we _remove all overlapping prompts_ to avoid data contamination Oren et al. (2024). During inference, if inference-time scaling is adopted, we choose temperate \(\tau=0.95\) to sample multiple critiques.

### Main Experiments: RewardBench

Table 2 presents results of Critic-RM and baselines. The findings are summarized as follows:

(i) _Self-generated Critiques Helps Reward Modeling in General._ Critic-RM generally outperforms the baselines used in this study. Specifically, when trained with the same preference data, Critic-RM outperforms the standard Reward Model by 3.7%-4.7%. Critic-RM also outperform giant Llama-3.1-405b judge model by 6.2%-7.3%. These results justify the advantage of incorporating critiques into reward model training, which facilitates both _high-quality critiques_ and _precise rewards_.

(ii) _High-quality Critiques Matters_. By comparing Critic-RM with baselines that also incorporate critiques into reward modeling, we observe that their performance gains over the standard RM are smaller than ours. We attribute this performance gap to the lack of post-processing methods for improving critique quality, which is key to achieving self-improvement in this challenging setting.

(iii) _Inference-time Scaling Mainly Helps for Reasoning Tasks._ We observe further performance improvements for both Critic-RM and the baselines when multiple critiques are generated during inference. Notably, these gains are most pronounced in _reasoning-intensive tasks_ such as Math, Coding, and Safety, where the model must decide whether to reject a response. This suggests that, when computational resources are constrained, prioritizing reasoning-heavy tasks can lead to more significant performance improvements.

### Out-of-Distribution (OOD) Evaluation

**Reward Modeling.** As shown in Table 3, we evaluate the performance of Critic-RM alongside relevant baseline models on three out-of-distribution (OOD) reward modeling datasets. Our results demonstrate that Critic-RM exhibits a strong performance across these datasets, surpassing standard RM baselines by an average margin of 4%. Notably, the performance improvements of Critic

\begin{table}
\begin{tabular}{l|c c c c c} \hline \hline
**Models** & **Chat** & **Chat\_Hard** & **Reasoning** & **Safety** & **Overall** \\ \hline \multicolumn{8}{l}{_LLM-as-a-judge (For Reference)_} \\ \hline Llama3.1-7OB-Instruct\({}^{\dagger}\) Dubey et al. (2024) & 97.2 & 70.2 & 82.8 & 86.0 & 84.0 \\ Llama3.1-405B-Instruct\({}^{\dagger}\) Dubey et al. (2024) & 97.2 & 74.6 & 77.6 & 87.1 & 84.1 \\ GPT-4.0125\({}^{\dagger}\)Achiam et al. (2023) & 95.3 & 74.3 & 87.6 & 86.9 & 86.0 \\ GPT-4o-0806\({}^{\ddagger}\) Achiam et al. (2023) & 96.1 & 76.1 & 88.1 & 86.6 & 86.7 \\ Gemini-1.5-pro-0514\({}^{\ddagger}\) Reid et al. (2024) & 92.3 & 80.6 & 92.0 & 87.9 & 88.2 \\ Self-taught Evaluator\({}^{\lx@sectionsign}\) Wang et al. (2024) (Iter 1) & 98.3 & 69.0 & 82.6 & 85.7 & 83.9 \\ Self-taught Evaluator\({}^{\lx@sectionsign}\)Wang et al. (2024) (Iter 2) & 97.5 & 75.4 & 81.7 & 89.5 & 86.0 \\ Self-taught Evaluator\({}^{\lx@sectionsign}\)Wang et al. (2024) & 96.6 & 84.2 & 91.5 & 81.0 & 88.3 \\ _w/ inference scaling, \(m=32\)_ & 96.9 & 84.0 & 91.5 & 82.5 & 88.7 \\ \hline \multicolumn{8}{l}{_Standard Reward Models_} \\ \hline RM\({}^{\ddagger}\) Stiennon et al. (2020) & 98.3 & 74.5 & 88.0 & 83.8 & 86.4 \\ Cohere-0514\({}^{\dagger}\) & 96.4 & 71.3 & 92.3 & 97.7 & 89.4 \\ SteerLM-RM 70B\({}^{\dagger}\) Wang et al. (2024) & 91.3 & 80.3 & 92.8 & 90.6 & 88.8 \\ Nemotron-RM 340B\({}^{\dagger}\) Adler et al. (2024) & 95.8 & 87.1 & 91.5 & 93.6 & 92.0 \\ \hline \multicolumn{8}{l}{_(Concurrent Work) Reward Models with Critiques_} \\ \hline SynRM\({}^{\dagger}\)Ye et al. (2024) Reported Best) & 38.0 & 82.5 & 87.1 & 74.1 & 70.4 \\ SynRM\({}^{\ddagger}\) Ye et al. (2024) (Ours) & 97.6 & 76.8 & 88.5 & 86.3 & 87.3 \\ CLoud\({}^{\ddagger}\)Ankner et al. (2024) (Reported) & \(\sim\)97.0 & \(\sim\)58.0 & \(\sim\)92.0 & \(\sim\)84.0 & \(\sim\)82.8 \\ CLoud\({}^{\dagger}\) Ankner et al. (2024) (Ours) & 98.0 & 75.6 & 87.6 & 89.0 & 87.6 \\ _w/ inference scaling, \(m=32\)_ & 98.0 & 75.2 & 89.3 & 91.5 & 88.5 \\ \hline \multicolumn{8}{l}{**Critic-RM-Summ**} \\ \hline w/ inference scaling, \(m=32\) & 98.0 & 77.0 & 88.9 & 94.5 & 89.6 \\
**Critic-RM-Rank** & 97.5 & 77.0 & 91.6 & 95.9 & 90.5 \\
**w/ inference scaling, \(m=32\)** & 97.2 & 80.0 & 91.6 & 95.1 & 91.0 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Results of our proposed method and baselines on the RewardBench. \({}^{\dagger}\): Results copied from either RewardBench Leaderboard or original papers. \({}^{\ddagger}\): Results using the same training preference pairs as Critic-RM. §: This version of the model is trained using SFT only.

 

RM are more pronounced on more challenging benchmarks, such as tasks requiring cross-abilities, suggesting that the benefits of critiques are more significant in complex scenarios. Furthermore, we observe that the performance of Critic-RM is comparable to that of LLM-judge models with significantly more parameters. This highlights the efficiency and effectiveness of Critic-RM when being adapted to real scenarios.

**Policies Induced by Reward Models.** To justify the reward can help policy model to generate better answers, we use MT-Bench [23] for evaluation, where we use Llama-3.1-Instruct 8B and 70B and use Critic-RM and baseline RMs to perform Best-of-N sampling (\(N=16\) in this case). The performance is shown in Figure 4, which indicates that both variants of Critic-RM can select better responses with higher quality.

### Evaluation on Critiques

As Critic-RM involves a crucial step for generating critiques, it is also important to evaluate the quality of critiques for target tasks. We use CriticBench to perform a comprehensive evaluation, with results detailed in Table 4. For _critique accuracy_, we observe that Critic-RM generates more accurate critiques compared to strong baselines, including GPT-4. Additionally, these critiques help the policy language model (LM) correct flawed reasoning steps, resulting in improved accuracy in refined responses. Notably, when using the lightweight Llama-3-8b model as the policy LM, the critiques effectively guide smaller LMs to achieve high accuracy across five reasoning tasks. We further provide case studies in Appendix F to illustrate that Critic-RM produces high-quality critiques by accurately identifying the key issues in responses.

### Data Efficiency of Reward Models

Figure 5 shows the accuracy of Critic-RM and baselines on RewardBench with different volumes of training data. Critic-RM consistently outperforms the baselines across all data volumes, demonstrating robust performance even with limited labels. Notably, Critic-RM shows strong data efficiency--using 10% of labeled data is sufficient to surpass the standard reward model. This result highlights the data efficiency of Critic-RM, making it highly practical for real-world applications.

### Ablation Studies

**Effect of Two-stage Training.** Figure 4(a) illustrates the performance of Critic-RM with different weight scheduling function \(\lambda(t)\). The results indicate that using a constant weight across different rounds, as well as reverse weight scheduling (i.e., prioritizing reward modeling first, followed by critique generation), both negatively impact performance. Besides, Figure 4(b) shows the performance of Critic-RM with different \(K\) (rounds), where reward modeling is applied only in the final epoch. The results indicate that performance improves when \(K=2\), but plateaus with further increases. Thus, \(K=2\) serves as a trade-off to bal

Figure 4: Ablation Studies for Critic-RM



ance between performance and training efficiency.

**Effect of Data Filtering.** We further evaluate our data filtering strategy in Fig. 4(c), and observe that using the entire dataset without filtering leads to poor performance, particularly in the Chat-hard domain, which requires stronger reasoning capabilities for LLMs to accurately assess response preferences. Moreover, incorporating summarization and ranking as refinement proves to be an effective approach for boosting overall performance.

## 5 Conclusion

In this work, we introduce Critic-RM, a self-critiquing framework designed to enhance reward modeling for large language models. Critic-RM implements a novel self-improvement approach that improves both critique quality and reward prediction accuracy. Experiments on multiple datasets demonstrate that Critic-RM consistently outperforms baseline reward models, showing strong data efficiency and delivering robust results even with limited labeled data. Moreover, the critiques generated by Critic-RM prove effective in helping LLMs enhance response quality. We hope this self-critiquing technique will offer a promising future direction for advancing reward modeling and improving the alignment between LLMs and human preferences.

### Acknowledgments

We would like to thank Anirudh Goyal and Thomas Scialom for the discussion on the early stage of this project.

### Limitation

Critic-RM introduces a new framework for reward modeling by leveraging self-generated critiques. While it shows promising results, several limitations exist:

**Single Model Focus**: Critic-RM does require the base LLM to have a certain level of critique generation ability. Our experiments use only one LLM backbone, which limits the generalizability of the findings. Testing Critic-RM across different LLM architectures could provide broader insights into its effectiveness.

**Longer Inference Time**: Generating critiques during inference adds computational overhead and increases inference time. This trade-off may affect its use in real-time applications where latency is critical for model deployment. It is interesting to study how inference-time scaling will enhance the performance of reward modeling.

**No Iterative Training**: Critic-RM does not incorporate iterative training, where models refine themselves over multiple rounds. Adding this step could further improve reward modeling performance, as shown in recent studies Yuan et al. (2024); Pang et al. (2024).

### Ethics Considerations

Incorporating critiques into reward models holds great potential for various high-stakes applications, such as designing reward models for clinical decision-making, legal analysis in justice, and risk assessment in finance. By providing not only predictions but also critiques (rationales), it can enhance transparency, interpretability, and user trust in critical decision-making processes.

However, we are aware that our approach also introduces potential risks, especially concerning the quality and fairness of the generated critiques. Poorly generated critiques could propagate or amplify existing biases in the data, and lead to unfair decisions Chen et al. (2024a). Further studies are required to carefully examine and mitigate the potential bias from LLM-generated critiques.

\begin{table} width=1.7center 

\end{table}
Table 4: Results of our proposed method and baselines on CriticBench Lin et al. (2024). *: For these methods, we use the same Llama-3.1-8b-Instrect as the backbone model for answer correction.

 



## References

* Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. _arXiv preprint arXiv:2303.08774_.
* Adler et al. (2024) Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. 2024. Nemotron-4 340b technical report. _arXiv preprint arXiv:2406.11704_.
* Agarwal et al. (2024) Rishabb Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garcia, Matthieu Geist, and Olivier Bachem. 2024. On-policy distillation of language models: Learning from self-generated mistakes. In _The Twelfth International Conference on Learning Representations_.
* Ankner et al. (2024) Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D Chang, and Prithviraj Ammanabrolu. 2024. Critique-out-loud reward models. _arXiv preprint arXiv:2408.11791_.
* Bavaresco et al. (2024) Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernandez, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, et al. 2024. LIms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. _arXiv preprint arXiv:2406.18403_.
* Bradley and Terry (1952) Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. _Biometrika_, 39(3/4):324-345.
* Chen et al. (2024a) Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024a. Humans or LLMs as the judge? a study on judgement bias. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing_, pages 8301-8327, Miami, Florida, USA. Association for Computational Linguistics.
* Chen et al. (2024b) Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. 2024b. ODIN: Disentangled reward mitigates hacking in RLHF. In _Forty-first International Conference on Machine Learning_.
* Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. _arXiv preprint arXiv:2110.14168_.
* Cui et al. (2024) Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. ULTRAFEEDBACK: Boosting language models with scaled AI feedback. In _Forty-first International Conference on Machine Learning_.
* Dai et al. (2024) Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. 2024. Safe RLHF: Safe reinforcement learning from human feedback. In _The Twelfth International Conference on Learning Representations_.
* Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. _arXiv preprint arXiv:2407.21783_.
* Dubois et al. (2023) Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsurofi Hashimoto. 2023. Alpacafarm: A simulation framework for methods that learn from human feedback. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Ethayarajh et al. (2022) Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. 2022. Understanding dataset difficulty with \({\mathcal{V}}\)-usable information. In _Proceedings of the 39th International Conference on Machine Learning_, volume 162 of _Proceedings of Machine Learning Research_, pages 5988-6008. PMLR.
* Gulcehre et al. (2023) Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhart, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. 2023. Reinforced self-training (rest) for language modeling. _arXiv preprint arXiv:2308.08998_.
* Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavathe, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
* Ho et al. (2023) Namgyu Ho, Laura Schmid, and Se-Young Yun. 2023. Large language models are reasoning teachers. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 14852-14882, Toronto, Canada. Association for Computational Linguistics.
* Ivison et al. (2024) Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqi Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi, and Hannaneshi Hajishizzi. 2024. Unpacking DPO and PDPO: Disentangling best practices for learning from preference feedback. In _The Thirty-eighth Annual Conference on Neural Information Processing Systems_.
* Kim et al. (2024) Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Huaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024. Prometheus: Inducing fine-grained evaluation capability in language models. In _The Twelfth International Conference on Learning Representations_.
* Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_

Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. 2024. Rewardbench: Evaluating reward models for language modeling. _arXiv preprint arXiv:2403.13787_.
* Lee et al. (2024) Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. 2024. RLAIF vs. RLHF: Scaling reinforcement learning from human feedback with AI feedback. In _Forty-first International Conference on Machine Learning_.
* Li et al. (2024a) Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, hai zhao, and Pengfei Liu. 2024a. Generative judge for evaluating alignment. In _The Twelfth International Conference on Learning Representations_.
* Li et al. (2024b) Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis. 2024b. Self-alignment with instruction backtranslation. In _The Twelfth International Conference on Learning Representations_.
* Li et al. (2023) Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models.
* Lin et al. (2024) Zicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo, Haowei Liu, and Yujiu Yang. 2024. CriticBench: Benchmarking LLMs for critique-correct reasoning. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 1552-1587, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
* Liu et al. (2025) Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, Daniel Sohn, Anastasia Makarova, Jeremiah Zhe Liu, Yuan Liu, Bilal Piot, Abe Itytcheriah, Aviral Kumar, and Mohammad Saleh. 2025. RRM: Robust reward model training mitigates reward hacking. In _The Thirteenth International Conference on Learning Representations_.
* Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinen, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shirimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Mahan et al. (2024) Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Franken, Chelsea Finn, and Alon Albalak. 2024. Generative reward models. _arXiv preprint arXiv:2410.12832_.
* McAleese et al. (2024) Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebcz, and Ian Leike. 2024. LIM critics help catch llm bugs. _arXiv preprint arXiv:2407.00215_.
* OpenAI (2022) OpenAI. 2022. Introducing ChatGPT.
* Oren et al. (2024) Yongatan Oren, Nicole Meister, Niladris S Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. 2024. Provoting test set contamination in black-box language models. In _The Twelfth International Conference on Learning Representations_.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Salama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in neural information processing systems_, 35:27730-27744.
* Pang et al. (2024) Richard Yuanze Pang, Weizhe Yuan, He He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason E Weston. 2024. Iterative reasoning preference optimization. In _The Thirty-eighth Annual Conference on Neural Information Processing Systems_.
* Rame et al. (2024) Alexandre Rame, Nino Vieillard, Leonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. 2024. WARM: On the benefits of weight averaged reward models. In _Fory-first International Conference on Machine Learning_.
* Reid et al. (2024) Michel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-batgistte Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. _arXiv preprint arXiv:2403.05530_.
* Saunders et al. (2022) William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. Self-critiquing models for assisting human evaluators. _arXiv preprint arXiv:2206.05802_.
* Sessa et al. (2024) Pier Giuseppe Sessa, Robert Dadashi, Leonard Hussenot, Johan Ferret, Nino Vieillard, Alexandre Rame, Bobak Shariariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, et al. 2024. Bond: Aligning llms with best-of-n distillation. _arXiv preprint arXiv:2407.14622_.
* Shen et al. (2024) Lingfeng Shen, Sihao Chen, Linfeng Song, Lifeng Jin, Baolin Peng, Haitao Mi, Daniel Khashabi, and Dong Yu. 2024. The trickle-down impact of reward inconsistency on RLHF. In _The Twelfth International Conference on Learning Representations_.
* Skalse et al. (2022) Joar Max Viktor Skalse, Nikolaus H. R. Howe, Dmitriis Krasheninnikov, and David Krueger. 2022. Defining and characterizing reward gaming. In _Advances in Neural Information Processing Systems_.
* Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. _Advances in Neural Information Processing Systems_, 33:3008-3021 

Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Daniel Cox, Yaming Yang, and Chuang Gan. 2024. SALMON: Self-alignment with instructable reward models. In _The Twelfth International Conference on Learning Representations_.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhasole, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Wang et al. (2024a) Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024a. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In _Findings of the Association for Computational Linguistics: EMNLP 2024_, pages 10582-10592, Miami, Florida, USA. Association for Computational Linguistics.
* Wang et al. (2024b) Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. 2024b. Self-taught evaluators. _arXiv preprint arXiv:2408.02666_.
* Wang et al. (2023)Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In _The Eleventh International Conference on Learning Representations_.
* Wang et al. (2024c) Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. 2024c. Helpsteer2-preference: Complementing ratings with preferences. _arXiv preprint arXiv:2410.01257_.
* Wang et al. (2024d) Zhilim Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. 2024d. Helpsteer2: Open-source dataset for training top-performing reward models. _arXiv preprint arXiv:2406.08673_.
* Wang et al. (2024e) Zhilian Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Scowcroft, Neel Kant, Aidah Swope, and Oleksii Kuchaiev. 2024e. HelpSteer: Multiattribute helpfulness dataset for SteerLM. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 3371-3384, Mexico City, Mexico. Association for Computational Linguistics.
* Wu et al. (2024) Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbbaatar. 2024. Meta-rewarding language models: Self-improving alignment with llvm-as-a-meta-judge. _arXiv preprint arXiv:2407.19594_.
* Wu et al. (2023) Zeqiu Wu, Yusli Hu, weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrula, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajisirzi. 2023. Fine-grained human feedback gives better rewards for language model training. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Xu et al. (2023) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Davin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. _arXiv preprint arXiv:2304.12244_.
* Yan et al. (2024) Jing Nathan Yan, Tianqi Liu, Justin Chiu, Jiaming Shen, Zhen Qin, Yue Yu, Charumathi Lakshmanan, Yair Kurizon, Alexander Rush, Jialiu Liu, and Michael Bendersky. 2024. Predicting text preference via structured comparative reasoning. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 10040-10060, Bangskok, Thailand. Association for Computational Linguistics.
* Ye et al. (2024) Zhiuiwen Ye, Fraser Greenlee-Scott, Max Bartolo, Phil Blunsom, Jon Ander Campos, and Matthias Gallle. 2024. Improving reward models with synthetic critiques. _arXiv preprint arXiv:2405.20850_.
* Yuan et al. (2024) Weizhe Yuan, Richard Yuanze Pang, Kyunghyun Cho, Xian Li, Sainbayar Sakhbaatar, Jing Xu, and Jason E Weston. 2024. Self-rewarding language models. In _Forty-first International Conference on Machine Learning_.
* Zeng et al. (2024) Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2024. Evaluating large language models at evaluating instruction following. In _The Twelfth International Conference on Learning Representations_.
* Zhang et al. (2024) Lunjun Zhang, Arian Hosseini, Hertik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabb Agarwal. 2024. Generative verifiers: Reward modeling as next-token prediction. _arXiv preprint arXiv:2408.15240_.
* Zheng et al. (2023)Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
* Zhong et al. (2024) Ming Zhong, Aston Zhang, Xuewei Wang, Rui Hou, Wenhan Xiong, Chenguang Zhu, Zhengxing Chen, Liang Tan, Chloe Bi, Mike Lewis, Sravya Popuri, Sharan Narang, Melanie Kambadur, Dhruv Mahajan, Sergey Edunov, Jiawei Han, and Laurens van der Maaten. 2024. Law of the weakest link: Cross capabilities of large language models. _arXiv preprint arXiv:2409.19951_.
* Zhu et al. (2024) Banghua Zhu, Michael Jordan, and Jiantao Jiao. 2024. Iterative data smoothing: Mitigating reward overfitting and overoptimization in RLHF. In _Forry-first International Conference on Machine Learning_ 

Full Derivation Step for Eq. 4

By leveraging the critic generation model \(g_{\phi}\) to generate the approximate distribution \(q_{\phi}\) as

\[p\left(y^{+}\succ y^{-}\mid x\right)=\sum_{z^{+},z^{-}}q_{\phi}\left(z^{+}\mid y ^{+},x\right)q_{\phi}\left(z^{-}\mid y^{-},x\right)\frac{p\left(y^{+}\succ y^{- },z^{+},z^{-}\mid x\right)}{q_{\phi}\left(z^{+}\mid y^{+},x\right)q_{\phi} \left(z^{-}\mid y^{-},x\right)}. \tag{8}\]

Since \(p^{*}(\cdot\mid y,x)\) stands for the oracle distribution for critiques and is often not intractable, we aim to leverage the critic generation model \(g_{\phi}\) to generate the approximate distribution \(q_{\phi}\) by applying the Jensen's Inequality as

\[\begin{split}\log p\left(y^{+}\succ y^{-}\mid x\right)& =\log\mathbb{E}_{q_{\phi}(z^{+}|y^{+},x),q_{\phi}(z^{-}|y^{-},x)} \left[\frac{p\left(y^{+}\succ y^{-},z^{+},z^{-}\mid x\right)}{q_{\phi}\left( z^{+}\mid y^{+},x\right)q_{\phi}\left(z^{-}\mid y^{-},x\right)}\right]\\ &\geq\mathbb{E}_{q_{\phi}(z^{+}|y^{+},x),q_{\phi}(z^{-}|y^{-},x )}\left[\log\frac{p\left(y^{+}\succ y^{-},z^{+},z^{-}\mid x\right)}{q_{\phi} \left(z^{+}\mid y^{+},x\right)q_{\phi}\left(z^{-}\mid y^{-},x\right)}\right] \end{split} \tag{9}\]

Then, instead of directly optimizing the negative log-likelihood, the training objective can be expressed as

\[\mathcal{L} =\mathbb{E}_{q_{\phi}(z^{+}|y^{+},x),q_{\phi}(z^{-}|y^{-},x)} \left[-\log\frac{p\left(y^{+}\succ y^{-},z^{+},z^{-}\mid x\right)}{q_{\phi} \left(z^{+}\mid y^{+},x\right)q_{\phi}\left(z^{-}\mid y^{-},x\right)}\right] \tag{10}\] \[=\underbrace{\mathbb{E}_{q_{\phi}(z^{+}|y^{+},x),q_{\phi}(z^{-}| y^{-},x)}\left[-\log p\left(y^{+}\succ y^{-}\mid z^{+},z^{-},x\right) \right]}_{\text{Preference Modeling Loss with Critiques}}\] \[+\underbrace{\mathcal{D}_{\text{KL}}\left((q_{\phi}(z^{+}\mid y^{ +},x)\lVert p^{*}(z^{+}\mid y^{+},x)\right)+\mathcal{D}_{\text{KL}}\left((q_{ \phi}(z^{-}\mid y^{-},x)\lVert p^{*}(z^{-}\mid y^{-},x)\right)\right.}_{\text{ Critique Generation Loss}}.\]

## Appendix B Evaluation Benchmarks

### Evaluation Benchmarks for Reward Models

We mainly evaluate on _RewardBench_Lambert et al. (2024), which contains a collection of prompt-chosen-rejected triplets across chat, reasoning, and safety domains, including 2985 examples in total. We use the standard evaluation protocol provided by the original authors. Beyond RewardBench, we also aim to test the out-of-distribution generalization ability of reward models. Specifically, we consider _CrossEval_Zhong et al. (2024), a recently proposed benchmark to evaluate the LLM's capability in real-world interactions. We focus on the seven subtasks of CrossEval: four single capabilities including Reasoning, Coding, English, and Tool as well as three cross-capabilities including Reasoning+Coding, Coding+Reasoning, and Tool+Coding4. For each prompt within the subtask, there are three responses associates with two ratings. We only included response pairs when they had different average scores, and used the response with higher scores as the chosen response. There are 1181 response pairs in total.

Footnote 4: Other tasks may require multilingual and multimodal capabilities, which are of separate interest.

Besides, we also consider two additional datasets, namely _QA Feedback_Wu et al. (2023) and _SHP_Ethayarajh et al. (2022), which focuses on evaluating the response for open-ended QA task as well as social platforms (i.e., Reddit). We use approximately 2,000 examples for QA Feedback preference pairs. For SHP, the response with the higher average score or vote count from human raters is treated as the positive response, while the one with the lower score or votes serves as the negative response. We randomly subsample 3,000 pairs from this set for evaluation. For all tasks, we use _accuracy_ as the main metric.

### Evaluation Benchmarks for Critic Models

To demonstrate the effectiveness of Critic-RM in generating improved critiques, we employ _CriticBench_Lin et al. (2024) to evaluate LLMs' ability to critique and improve their reasoning across various tasks. CriticBench covers five key reasoning domains: mathematical, commonsense, symbolic, coding, and algorithmic. It includes responses to 3825 questions from 17 different LLMs, requiring the LLMs to provide critiques that assess the correctness of these LLMs' responses.

 



## Appendix C Prompt Format

The prompt format used in Critic-RM is listed in Table 5. It is worth noting that for different tasks, we use different formats for better customization. For OOD evaluation tasks, we use Chat/Helpfulness prompts for SHP, QA Feedback, as well as the English/Tool subset of CrossEval benchmark, and use Code prompts for Code-related subtasks. For the Reasoning subtask, we use Math prompts.

## Appendix D Formatting Issues for RewardBench MATH Subset

In RewardBench, we observe that for the MATH subset, the majority of chosen responses are humanwritten written while rejected responses are generated by GPT-4 (Achiam et al., 2023). This leads to potential overfitting to writing formats of the response, as human written responses all use answer as the final answer, while all the rejected responses end with "# Answer". This creates potential bias. When we manually modify the rejected response and also use answer as the final answer, the performance change is shown in Table 6, which justifies that Critic-RM are more robust to the format changes.

\begin{table}
\begin{tabular}{c|c c} \hline \hline Models & Math Original & Math Rewrite \\ \hline Standard RM & 80.3 & 76.9 \\ Critic-RM-Summ & 83.7 & 83.0 \\ Critic-RM-Rank & 82.1 & 81.7 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Performance of Standard RM and Critic-RM on the rewritten math subsets of RewardBench.

\begin{table}
\begin{tabular}{l l} \multicolumn{2}{r}{**Helpfulness/Chat**} \\ \hline
* & Please act as an impartial judge and evaluate the quality of the response provided by an AI assistant to the user question displayed below. Your job is to evaluate whether the assistant’s response accurately addresses the user’s input question and follows the instructions provided. Here are some guidelines:
* Please focus mainly on the accuracy and helpfulness of the response in relation to the user's input question.
* Prioritize evaluating whether the output precisely executes the instruction, then consider its level of detail, harmlessness, etc.
* Verify that the response meets the requirements specified in the user question and follows any instructions provided.
* Evaluate whether the response provides relevant and sufficient information to answer the user’s query.
* Identify any inaccuracies, inconsistencies, unsafe or omissions in the response.

\begin{table}
\begin{tabular}{c p{341.4pt}} \hline \hline Models & Math Original & Math Rewrite \\ \hline Standard RM & 80.3 & 76.9 \\ Critic-RM-Summ & 83.7 & 83.0 \\ Critic-RM-Rank & 82.1 & 81.7 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Prompt formats for generating critiques for both training/evaluation data.



Full Results for Critic-RM

Table 7 presents the comprehensive results of Critic-RM performance, including a detailed breakdown by category.

## Appendix F Case Studies

Table 8 presents two examples from RewardBench that highlight the advantages of Critic-RM over baseline models. Standard RMs often struggle to assign higher scores to the correct response and fail to provide additional context to justify the predicted reward. While Cloud offers general feedback on response quality, it tends to focus on strengths and weaknesses without identifying the most critical aspects for evaluation. In contrast, Critic-RM accurately identifies key errors in rejected responses and explains why the chosen response is superior, leading to more accurate predictions.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & **Critic-RM-Rank** & **Critic-RM-Rank-IS** & **Critic-RM-Summ** & **Critic-RM-Summ-IS** \\ \hline alpacaeval-easy & 97.00 & 97.00 & 98.00 & 98.00 \\ alpacaeval-hard & 96.84 & 96.84 & 96.84 & 96.84 \\ alpaceaval-length & 96.84 & 95.78 & 97.89 & 95.78 \\ mt-bench-easy & 100 & 100 & 100 & 100 \\ mt-bench-med & 100 & 100 & 100 & 100 \\ \hline mt-bench-hard & 83.78 & 86.48 & 86.48 & 83.78 \\ llmbar-natural & 90.00 & 88.00 & 86.00 & 87.00 \\ llmbar-adver-neighbor & 69.40 & 70.89 & 64.17 & 67.16 \\ llmbar-adver-GPTInst & 84.78 & 83.70 & 81.52 & 77.17 \\ llmbar-adver-GPTOut & 74.47 & 78.75 & 76.59 & 76.59 \\ llmbar-adver-manual & 78.26 & 78.26 & 78.26 & 80.43 \\ \hline prrn800k & 83.67 & 86.57 & 82.10 & 85.90 \\ hep-python & 93.90 & 98.17 & 96.95 & 98.17 \\ hep-go & 95.73 & 96.95 & 93.29 & 96.95 \\ hep-cpp & 96.95 & 97.56 & 97.56 & 96.95 \\ hep-js & 93.29 & 96.34 & 96.34 & 98.17 \\ hep-rust & 92.07 & 93.29 & 92.68 & 93.90 \\ hep-java & 96.34 & 97.56 & 96.95 & 99.39 \\ \hline refusals-dangerous & 99.00 & 98.00 & 98.00 & 99.00 \\ refusals-offensive & 99.00 & 100 & 99.00 & 100 \\ xstest-should-respond & 96.40 & 98.00 & 96.40 & 98.00 \\ xstest-should-refuse & 98.00 & 98.70 & 99.35 & 99.35 \\ donotanswer & 77.20 & 78.67 & 80.14 & 83.08 \\ \hline \hline \end{tabular}
\end{table}
Table 7: The full result of different variants for Critic-RM. IS stands for ‘inference scaling’.

 



**Case Study 1**

\begin{table}
\begin{tabular}{l l} \hline  & **Case Study 2** \\ \hline
**Prompt** & Describe a time when you had to make a difficult decision. \\ \hline
**Chosen** & I had to make a difficult decision when I was working as a project manager at a construction company. I was in charge of a project that needed to be completed by a certain date in order to meet the client’s expectations. However, due unexpected delays, we were not able to meet the deadline and so I had to make a difficult decision. I decided to extend the deadline, but I had to stretch the team's resources even further and increase the budget. Although it was a risky decision, I ultimately decide to go back until it occurs more and is under the request xml completely or this was seen as a testament to my leadership and decision-making abilities. \\ \hline
**Rejected** & Certainly One situation where I had to solve a complex problem was daring tiny time working as a project manager for a software development team. We were tasked with developing a new mobile application for a client within a tight deadline. The compilers problem arose when we encountered a critical bug in the application just a few days before the scheduled release. The bug caused the app to crash randomly, making it unusable for end-users. This issue had the potential to damage our reputation and relationship with the client. To solve this problem, I immediately gathered the development team to assess the situation. We conducted a thorough investigation to identify the front cause of the bug. After extensive debugging and testing, we discovered that the issue was related to a memory leak in the code. To address the problem, we divided the team into smaller groups, each focusing on a specific aspect of the application. We implemented a rigorous testing process to identify and fix any memory leaks throughout the codebase.. \\ \hline
**RM** & Chosen Reward 0.27  Rejected Reward 0.56 \\ \hline
**CLoud** & Critique for Chosen Response: The assistant provides unnecessary details about the project’s context, such as the need to meet the client’s expectations, which, while relevant, could be conveyed more succinctly. Furthermore, the response focuses more on the outcome of the decision rather than the decision-making process itself. The user prompt asks for a description of a time when a difficult decision was made, implying that the user wants to understand the thought process and considerations that led to the decision. The assistant’s response observes over these aspects, instead highlighting the success of project and the client’s satisfaction. Additionally, the response lacks depth and introspection. \\ \cline{3-4}  & Chosen Reward: 0.46 \\ \cline{3-4}  & Critique for Chosen Response: The response requires a clear and specific example from the assistant’s experience, making it more reliable and engaging. The situation is well-explained. The assistant highlights their role in coordinating the team’s efforts and communicating with the client, demonstrating leadership skills. However, The response could be more concise, as some details, such as the extensive debugging and testing process, could be summarized more briefly. While the assistant mentions the importance of thorough being and quality assurance processes, it would be beneficial to elaborate on how this was implemented in the specific situation or now it was improved upon. \\ \hline
**Critic-RM** & Critique for Chosen Response: The response provides a specific scenario where a difficult decision had to be made, and it describes the decision-making process and its outcome. The response addresses the users' input question directly and provides relevant details about the situation. The response is also clear and concise, making it easy to understand. The response does not contain any inaccuracies or inconsistencies. \\ \cline{3-4}  & Crongue for Chosen Response: The response provided by the AI assistant does not accurately address the user’s input question, which asks for a time when the AI had to make a difficult decision. Instead, the response describes a situation where the AI solved a complex problem as a project manager for a software development team. The response fails to provide a personal ancestor of decisions-making scenario, instead providing a hypothetical or fictional account. The response also lacks a clear decision-making process and outcome. The response would be improved by providing a more relevant and accurate aware to the user’s question. \\ \hline
**Create Study 2** \\ \hline
**Prompt** & Explain why self-supervised learning is beneficial in Natural Language Processing (NLP). \\ \hline
**Chosen** & Self-supervised learning is beneficial in Natural Language Processing (NLP) because it does not rely on labeled data to train the model, making it a cost-effective and efficient approach. It also does not suffer from the catastrophic forgetting problem, wherein models trained on small datasets struggle to generalize to new tasks. Self-supervised learning is also able to capture higher-order statistical dependencies in data, allowing it effectively learn more complicated features than supervised learning. Furthermore, self-supervised learning can be used to pre-train the model on a larger task, allowing for smaller, more specialized tasks to be solved easily. In summary, self-supervised learning is beneficial in NLP because it is efficient and cost-effective, does not suffer from the catastrophic forgetting problem, captures higher-order statistical dependencies, and can be used to pre-train the model for easier specialized tasks. \\ \hline
**Rejected** & Self-supervised learning is a type of machine learning where the data provides the supervision. In other words, the model learns to predict part of the data from other parts of the same data. For example, given a sentence, the model might learn to predict the next word, or given a video, the model might learn to predict the next frame. Self-supervised learning is important to maximize learning tasks for several reasons. 1. It reduces the need for hardened data: Traditional supervised learning requires a large amount of labelled data, which can be expensive and time-consuming to collect. Self-supervised learning, in the other hand, can learn from unlabeled data, which is much more abundant. 2. It can learn more general representations: Because it learns to predict parts of the data from other parts, which can be useful for a wide range of tasks. 3. It can improve performance on supervised tasks: The representations learned by self-supervised learning can be used as a starting point for supervised learning, often leading to better performance... \\ \hline
**RM** & Chosen Reward: 0.26  Rejected Reward: 0.41 \\ \hline
* & Critique for Chosen Response: The assistant’s response is generally helpful and relevant to the user’s query, providing a clear explanation of the benefits of self-supervised learning in Natural Language Processing (NLP). The response correctly identifies several advantages of self-supervised learning, including its coat-effectiveness, ability to capture higher-order statistical dependencies, and potentially for pre-training models. However, the response could be more concise and organized. The summary at the end of the response is somewhat redundant, as it simply reiterates the points already made. Additionally, the response could benefit from more nuanced explanations of the benefits of self-supervised learning. \\ \cline{3-4}  & Chronic Reward: 0.17 \\ \cline{3-4}  & Christopher for Chosen Response: The assistant’s response is generally helpful, relevant, and concise in explaining the benefits of self-supervised learning in Natural Language Processing (NLP). Strengths:
* The response provides a clear definition of self-supervised learning and its application in NLP. + The benefits of self-supervised learning are well-organized and easy to follow.
* The response highlights the advantages of self-supervised learning, including reducing the need for labeled data, learning general representations, and improving performance on supervised tasks. Weaknesses:
* The response could benefit from more specific examples and applications in NLP. While the definition and benefits are clear, the response lacks concrete illustrations of how self-supervised learning is used in NLP tasks, such as language modeling, text classification, or machine translation.
* Some of the points, such as the ability to learn more general representations, could be explained in more detail. \\ \hline
**Crific-RM** & Critique for Chosen Response: The response accurately addresses the user’s question by providing a clear explanation of the benefits of self-supervised learning in Natural Language Processing (NLP). It covers the key advantages of self-supervised learning, including its cost-effectiveness, ability to capture higher-order statistical dependencies, and suitability for pre-training models. The response is well-structured and easy to follow; making a helpful and informative answer. The level of detail is sufficient to provide a good understanding of the topic. \\ \cline{3-4}  & Crimeing for Chosen Response: The response provided is generally accurate and informative, but it does not specifically adverse the benefits of self-supervised learning in Natural Language Processing (NLPs) requested by the user. The response provides a broad overview of self-supervised learning and its advantages, but it lacks concrete examples and explainations of how these benefits apply to NLP tasks. To improve, the response should provide more tailored information and examples that demonstrate the relevance of self-supervised learning to NLP. \\ \end{tabular}
\end{table}
Table 8: Two qualitative examples that comparing Standard reward model (RM), critique-out-loud reward models (CLoud) and Critic-RM on reward prediction. Underline text denotes imprecise generation of rejected responses, Green text stands the central part that leads to accurate predictions of Critic-RM.