Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge

## Abstract

Abstract Large Language Models (LLMs) are rapidly surpassing human knowledge in many domains. While improving these models traditionally relies on costly human data, recent self-rewarding mechanisms (Yuan et al., 2024c ) have shown that LLMs can improve by judging their own responses instead of relying on human labelers.
However, existing methods have primarily focused on improving model responses rather than judgment capabilities, resulting in rapid saturation during iterative training. To address this issue, we introduce a novel Meta-Rewarding step to the self-improvement process, where the model judges its own judgements and uses that feedback to refine its judgment skills.
Surprisingly, this unsupervised approach improves the model’s ability to judge and follow instructions,
as demonstrated by a win rate improvement of Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2, and 20.6% to 29.1% on Arena-Hard.
These results strongly suggest the potential for self-improving models without human supervision.

## 1 Introduction

Large Language Models (LLMs) are advancing significantly in their ability to follow instructions
and respond to user queries
(OpenAI, ; Touvron et al., ). An important phase in training these models is instruction tuning (Ouyang et al., ), which typically involves training LLMs on datasets curated by humans, either via supervised finetuning or preference optimization. Nevertheless, the acquisition of human-generated data is both costly and time-consuming. Furthermore, the quality of such data is inherently constrained by the limitations of human capabilities.
The so-called ‘Super Alignment’ challenge (Burns et al., )
aims to find a solution to steering or controlling potentially super-intelligent AIs when their actions are inherently beyond human abilities to judge.

Among the potential solutions to this challenge, self-judging by the AI emerges as a particularly promising approach. Yuan et al. () introduces an iterative Self-Rewarding mechanism that enables an LLM to improve autonomously.
The process involves a single model that takes on two distinct roles, as an actor and as a judge. As an actor, the model produces responses that are aimed to fulfill specific instructions. As a judge (a special kind of acting), the model evaluates these responses via LLM-as-a-Judge prompting (Zheng et al., ) and assigns rewards. The objective of the actor during this self-play is to maximize its reward, thereby improving its ability to follow instructions.

We hypothesize that a major limitation of this previous work is that its learning objective enhances the model’s ability as an actor to generate better responses, while overlooking improving the model’s ability as a judge. If the ability to judge does not improve then training the actor over iterations can quickly saturate – or worse could overfit the reward signal, a.k.a. reward hacking.
Consequently, it is imperative to also improve the model’s capabilities as a judge in addition to its ability to act.

In this paper, we propose a novel method called *Meta-Rewarding* which assigns rewards to its own judgements to train the model’s ability to judge.
The key idea is to introduce a third role of *meta-judge*, whose task is to evaluate the model’s own judgements. While the judge evaluates the actor’s responses, the meta-judge evaluates the judge’s judgments (including rewards that it assigns) using a mechanism similar to LLM-as-a-Judge, which we term LLM-as-a-Meta-Judge.
The meta-judge enables us to build
training data containing preference pairs of judgements, in addition to the standard preferences
between actor responses derived from the standard judge. Our
Meta-Rewarding method thus aims
to explicitly improve both the acting and judging skills of a model – whereby these combined skills
should help to enhance its instruction following ability as an actor. It is important to note that all
three roles - actor, judge, and meta-judge - are performed by the same model, thereby maintaining a self-improving nature that requires no extra human data.

In addition to enhancing the judging ability through Meta-Rewarding, we also address the length-bias issue in the judging process (Singhal et al., ).
Like other reward models, the judge tends to favor long responses,
which can make response length grow during iterative DPO (Yuan et al., ).
To counteract this, we combine the judge score with length information to determine the winning response, ensuring that a shorter response is chosen when scores are close.

In our experiments we start from Llama-3-8B-Instruct and perform multiple iterations of our Meta-Rewarding training.
When evaluated on AlpacaEval 2 (Dubois et al., ), we see a substantial improvement in the length-controlled (LC) win rate (from 22.9% to 39.4%), even outperforming GPT-4-0314(^1^11[https://tatsu-lab.github.io/alpaca_eval/](https://tatsu-lab.github.io/alpaca_eval/)).
We also observe that our method outperforms standard Self-Rewarding training even if it is enhanced with our length-bias improvements (35.5% vs 39.4%), highlighting the importance of the meta-judge.
We also see similar improvement on Arena-Hard benchmark (Li et al., ), which is a benchmark targeting models’ ability to answer complex and hard questions.

## 2 Meta-Rewarding

Figure: Figure 1: Meta-Rewarding iterative training scheme. The language model at step $t$ behaves as an actor to generate responses to instructions, as a judge to assign rewards to those responses, and as a meta-judge to evaluate its own judgments. The judgments are used to create preference pairs to improve its ability to act, and the meta-judgments are used to create preference pairs to improve its ability to judge. Both preference pair sets are used together to train the model for the next iteration.
Refer to caption: x1.png

In our method, we assume a setup where we only have an initial seed model, an instruction-tuned LLM, and no further human supervised training data.
The idea is to generate training data from the model itself through an iterative self-play process.
In this process, the model assumes three main roles: as an actor, it generates responses to given prompts; as a judge, it evaluates and scores its own responses; and as a meta-judge, it compares the quality of its own judgments.

While training the actor to generate better responses to user queries is the final objective, this training’s efficacy relies on the accuracy of the judge.
As the judge’s accuracy increases, it will provide higher quality feedback for training the actor, ultimately leading to a better actor.
Therefore, the goal of Meta-Rewarding is to improve the model’s capability both as actor and judge during training.
The role of the meta-judge is to provide feedback necessary for training the judge.

At a high level, as depicted in Figure 1, our method is an iterative training scheme that starts from a given seed LLM, which assumes all three roles.
An iteration starts with the actor generating multiple response variations
for each prompt.
This is followed by the judge evaluating each response
using an LLM-as-a-Judge prompt and generating a judgement
that contains a score.
This score then allows us to build preference pairs of responses for training the actor.
For training the judge, we pick a single response
and let the meta-judge compare two of its judgement variations
generated by the judge to determine which one is better using an LLM-as-a-Meta-Judge prompt, see Figure 2.
This step enables us to create preference pairs of judgements that can be used for training the judge.

Once we have the preference data both for the actor and the judge, then we apply preference optimization on the dataset via DPO (Rafailov et al., ).
Note that while other RLHF methods can be employed, we chose to use DPO because of its simplicity and stability.
After the training, we end up with an improved model that will be then used for the next iteration, both for generating training data and as an initial model for the optimization.
Next, we will describe each preference data creation process in detail.

Figure: Figure 2: Prompt used by the meta-judge to compare given two judgements.

### 2.1 Actor Preference Dataset Creation

Our approach to create the actor preference dataset on a given iteration is built upon the pipeline introduced by Yuan et al. (), with a crucial modification to incorporate a length-control mechanism.
As we see later in Section 3.5, this change proves to be essential in preventing the responses from lengthening and improving the length-controlled win rate.
The dataset creation process consists of three main steps:

Sample Responses from Actor. We assume we have a given set of prompts. For each prompt $x$, we generate $K$ different responses $\{y_{1},\ldots,y_{K}\}$ by sampling from the current model $M_{t}$ at iteration $t$.

Aggregate Multiple Judgments. For each response $y_{k}$, we generate $N$ different judgments $\{j_{k}^{1},\ldots j_{k}^{N}\}$ from $M_{t}$ using an LLM-as-a-Judge prompt (shown in Section A.1).
The prompt instructs the model to evaluate the given response $y_{k}$ for prompt $x$ according to a fixed rubric and output its chain-of-thought reasoning and a final score out of 5.
We use regular expressions to parse the scores, discarding any judgments with parsing errors or those not adhering to the 5-point scale. The final reward score for each response is then calculated by averaging all valid judgment scores.

Preference Data Selection with Length-Control.
The previous work simply selects the highest $S_{\text{max}}$ and lowest $S_{\text{min}}$ scored responses as the chosen $y_{c}$ and rejected $y_{r}$ as a preference pair for each prompt. However, this leads to length explosion where responses get longer with each iteration.
This is due to the length-bias of the judge, a well-know issue in reward models (Dubois et al., ; Park et al., ; Yuan et al., ).
To mitigate this, we introduce a simple length-control mechanism.
We define a quality tier parameter $\rho\in[0,1]$ to control the trade-off between score-based selection and length consideration. Responses with scores in the top tier, specifically within the range $[(1-\rho)S_{\text{max}}+\rho S_{\text{min}},S_{\text{max}}]$, are considered to have similar quality. For selecting the chosen response $y_{c}$, we opt for the shortest response within this top tier. This approach helps to counteract the tendency of judges to favor longer responses, which can lead to biased training data. Conversely, for the rejected response $y_{r}$, we select the longest response with a score in the range $[S_{\text{min}},(1-\rho)S_{\text{min}}+\rho S_{\text{max}}]$.
Setting $\rho$ to 0 effectively disables the length-control, reverting to a purely score-based selection.

### 2.2 Judge Preference Dataset Creation

Unlike the judge that provides score-based judgements, we design the meta-judge to operate in a pairwise mode by comparing two given judgements.
Thereby, we adopt the following three steps for generating and selecting chosen and rejected pairs, while carefully controlling for positional bias:

Response Selection:
To prepare effective training data for the judge, we focus on responses where the judge is the least certain, as measured by the variance of the scores it has given. To be more specific, we first compute the score variance given by the $N$ different judgments for every response $y_{k}$. We then pick the response ${y}$ with the highest score variance for each prompt $x$ to be used in the judge training. If multiple responses have the same variance, we break ties randomly.

Pairwise Meta-Judge Evaluations: For each selected response ${y}$, we have up to $N$ corresponding judgments, denoted as $\{{j}^{1},\ldots,{j}^{N}\}$. We then evaluate each pair of different judgments $({j}^{m},{j}^{n})$ using a meta-judge prompt shown in Figure 2.
This *LLM-as-a-Meta-Judge* prompt includes the original prompt $x$, response $y$, and its two judgements $({j}^{m},{j}^{n})$ as well as the rubric used by the judge.
Then the model is asked to generate chain-of-thought reasoning followed by its choice of the better judgement.
Again this uses the same LLM model, but acting as a meta-judge this time.

To mitigate positional bias (where the meta-judge might e.g. tend to prefer the judgment that appears first), we prompt the model twice by changing the ordering of the two judgements.
In addition, we also introduce weighted scoring for winning in the first vs second positions. We define ${win}_{\text{1st}}$ and $win_{\text{2nd}}$ as the total wins in the first and second positions respectively, and calculate the weights as:

$$ $\omega_{1}=\frac{win_{\text{2nd}}}{win_{\text{1st}}+win_{\text{2nd}}},\quad\quad\omega_{2}=\frac{win_{\text{1st}}}{win_{\text{1st}}+win_{\text{2nd}}}.$ $$

The result of a single battle between judgments $({j}^{m},{j}^{n})$ is defined as:

$$ $\displaystyle r^{mn}=\begin{cases}1&\text{If the meta-judge prefers }m\ \text{wins}\\ -1&\text{If the meta-judge prefers }n\ \text{wins}\\ 0&\text{If tie or parse error}.\end{cases}$ $$

We then construct a battle matrix $B$ as the weighted sum of the battle results:

$$ $B_{mn}=\omega_{1}{\bm{1}}[r^{mn}=1]+\omega_{2}{\bm{1}}[r^{nm}=-1]$ $$

Elo Score and Pairs Selection:
The next step is to convert the battle matrix into rewards (meta-rewards) corresponding to each judgement.
Inspired by Zheng et al. (), we determine the Elo score $\varepsilon_{m}$ for each judgment ${j}^{m}$ by solving the following maximum likelihood estimation problem:

$$ $\arg\max_{\varepsilon}\sum_{m,n}B_{mn}\log\left(\frac{e^{\varepsilon_{m}-\varepsilon_{n}}}{1+e^{\varepsilon_{m}-\varepsilon_{n}}}\right).$ $$

This approach allows us to compute scores that account for the positional bias in the meta-judge evaluations, providing a more accurate reward signal representing the judgment quality.
When creating the preference pairs, we select the chosen $j^{c}$ and rejected $j^{r}$ as the judgment with the highest and lowest Elo score respectively, breaking ties randomly.

However, we find that the meta-judge can also exhibit length-bias similar to the judge, preferring verbosity when evaluating judgments. This bias results in chosen judgments being, on average, longer than rejected ones. If left unchecked, this tendency could lead to increasingly verbose model outputs after training.
To overcome this verbosity issue, we implement an additional filtering step to filter out preference pairs where the chosen judgment exceeds a certain length threshold. This process effectively penalizes excessively long generations, helping to maintain a balance between quality and conciseness in the judge’s outputs.

## 3 Experiments

### 3.1 Experimental Setup

We use instruction-finetuned Llama-3-8B-Instruct as a seed model, and
otherwise closely follow the experimental setup of Yuan et al. ().
Before our Meta-Rewarding training, we first perform supervised finetuning (SFT) of the seed model on the Evaluation Fine-Tuning (EFT) dataset from Yuan et al. ().
This dataset is built from Open Assistant (Köpf et al., ) and provides initial LLM-as-a-Judge training data of ranked human responses, thus aiding the model to act as a judge.
Since the seed model is already instruction finetuned, we skip training directly on human responses for the actor.
We refer to this model as *SFT on EFT*, or simply SFT for short.

Figure: Figure 3: AlpacaEval 2. Length-controlled (LC) win rate increases with Meta-Rewarding iterations, even approaching Claude-Opus level. The Self-Rewarding w/LC baseline lags behind in later iterations due to its lack of judge training.
Refer to caption: x2.png

For Meta-Rewarding iterations,
we utilize 20,000 prompts from Yuan et al. () that were generated by Llama-2-70B-Chat using an 8-shot prompt.
We provide a visualization of their distribution in Appendix Figure 6. For each iteration, we sample 5,000 prompts from this seed set and conduct four iterations in total. The iterative process is formally defined as follows:

- Iter 1
Obtain $M_{1}$ by training using DPO (initialized from the SFT model) on both actor and judge preference pairs generated by the SFT model.
- Iter 2
Obtain $M_{2}$ by training $M_{1}$ using DPO on actor and judge preference pairs generated by $M_{1}$.
- Iter 3
Obtain $M_{3}$ by training $M_{2}$ using DPO exclusively on actor preference pairs generated by $M_{2}$.
- Iter 4
Obtain $M_{4}$ by training $M_{3}$ using DPO exclusively on actor preference pairs generated by $M_{3}$.

We provide a detailed recipe for training in Section A.3. In each iteration, we generate $K=7$ response variations per prompt using temperature 0.8 and top_p 0.95. This results in a total of 35,000 responses per iteration. We then filter out identical responses, typically removing no more than 50 duplicates.
Next, we generate $N=11$(^2^22We chose this value based on our early experiments showing optimal performance at this number, with further increases yielding similar or worse correlation with human judgments.) different judgments for each response using the same sampling parameters.

### 3.2 Evaluation Methods

As Meta-Rewarding aims to improve the model both as an actor and a judge, we evaluate its performance in both of these roles.
In addition, we also compare it against a Self-Rewarding baseline (Yuan et al., ) in the same setup, equipped with the same length-control mechanism.
This allows us to measure the gains brought by the judge training data generated via meta-rewarding.

Actor’s Instruction Following We make use of three well-established auto-evaluation benchmarks based on GPT4-as-a-Judge: AlpacaEval 2 (Dubois et al., ), Arena-Hard (Li et al., ) and MT-Bench (Zheng et al., ). These benchmarks focus on different aspects of the model. For instance, AlpacaEval mainly focuses on chat scenarios, where the prompt sets cover a diverse range of daily questions.
In comparison, Arena-Hard consist of more complex or challenging questions, where they satisfy more criteria in the predefined 7 aspects (creativity, complexity, problem-solving, etc). Notably, Arena-Hard has the highest correlation with Chatbot-Arena among popular open-ended LLM benchmarks (Li et al., ). MT-Bench has 8 different question categories and evaluates the multi-turn conversation ability of the model.

Judge’s Reward Modeling To evaluate the reward modeling capability of the judge, we measure the correlation of our judge scores with human preferences, as well as a strong AI judge when human labeling is not available.
We quantitatively calculate the Spearman correlation and agreement between the model-generated ranking with the human-labeled preferences provided in the Open Assistant dataset.
We use a held-out split of 190 samples, with each sample consisting of a prompt and several human ranked responses, totalling 580 different responses. Additionally, we also measure the judge’s performance on ranking responses generated by the seed model, which is considered to be more in-distribution compared to human or other model generated responses.
This is because the judge is mainly trained and applied on samples that are self-generated. However, in this case, we do not have ground-truth human preference labels, so we adopt the strong judge gpt-4-1106-preview as a proxy.

### 3.3 Instruction Following Evaluation

**Table 1: AlpacaEval 2: The evaluation on AlpacaEval shows significant improvement with Meta-Rewarding training. While the seed model Llama-3-8B-Instruct only achieves 22.92% length-controlled (LC) win rate against GPT4-Turbo, our 4-th iteration achieves 39.44%.**
| Model | LC win rate | Win rate | Length |
| --- | --- | --- | --- |
| Llama-3-8B-Instruct (Seed)(^3^33Our evaluation shows slightly higher numbers, with the LC Winrate 24.57%, Winrate 24.89% and Length 1936. This is likely due to a different inference template.) | 22.92% | 22.57% | 1899 |
| SFT on EFT | 25.47% | 25.10% | 1943 |
| Self-Rewarding LLM (Yuan et al., ) + LC |  |  |  |
| Iteration 1 | 26.93% | 27.12% | 1983 |
| Iteration 2 | 30.38% | 29.77% | 1940 |
| Iteration 3 | 34.87% | 34.59% | 1967 |
| Iteration 4 | 35.49% | 35.37% | 2005 |
| Meta-Rewarding LLM (Ours) |  |  |  |
| Iteration 1 | 27.85% | 27.62% | 1949 |
| Iteration 2 | 32.66% | 33.29% | 2001 |
| Iteration 3 | 35.45% | 37.24% | 2064 |
| Iteration 4 | 39.44% | 39.45% | 2003 |

Figure: Figure 4: Fine-grained AlpacaEval LC Winrate Analysis. We classify all 805 AlpacaEval test prompts into 20 categories, while discarding 2 categories that have less than 10 questions. Meta-Rewarding improves upon Llama-3-8B-Instruct for 17 out of 18 categories.
Refer to caption: x3.png

Meta-Rewarding iterations significantly improves the win rate.
In Figure 3, we show the length-controlled (LC) win rate of our method over its training iterations on the AlpacaEval benchmark.
Overall, we see a substantial increase from 22.9% to 39.4%, outperforming GPT-4 and approaching close to the Claude Opus model.
This is a remarkable result considering our model has only 8B parameters and our training did not utilize any extra human data beyond the seed model (except the EFT dataset used in the SFT stage).
In addition, our method surpasses the strong baseline of SPPO (Wu et al., ), which has a similar iterative training setup using Llama-3-8B-Instruct, but uses a reward model that was trained on a large set of human and GPT-4 data.
Despite its reliance on a strong external reward model as a judge, SPPO achieves 38.77% LC win rate, which is slightly lower than our method.

The meta-judge and length-control mechanism are important.
The Self-Rewarding baseline with our length-control (LC), which lacks the meta-judge for training the judge, also brings improvement, but to a lesser degree, especially in later iterations.
This signifies the importance of training the judge and the effectiveness of the meta-judge in achieving this.
As shown in Table 1,
the average response length (measured in characters) does not grow substantially over training iterations, proving the effectiveness of our length-control mechanisms (see ablations in Section 3.5).

Meta-Rewarding improves nearly all instruction categories.
We perform a fine-grained analysis by breaking down the 805 questions in AlpacaEval into 18 categories(^4^44We dropped 2 categories that had less than 10 samples.) given in Yuan et al. ().
Notably, we find significant improvements in most of the categories as shown in Figure 4, including categories that require a considerable amount of knowledge and reasoning, e.g. science, gaming, literature, etc. However, there are also categories like Travel or Mathematics, where the model only has slight improvement compared with the seed model Llama-3-8B-Instruct.

Meta-Rewarding improves answering of complex and hard questions. We further evaluate our method’s performance on answering complex and challenging prompts using Arena-Hard. The evaluation results in Table 2 show that Meta-Rewarding is able to improve the score in all 4 iterations, showing a substantial improvement (+8.5%) compared with the seed model (20.6%).
This further validate the effectiveness of our method.

Meta-Rewarding does not sacrifice multi-turn ability despite training only on single-turn.
We perform MT-Bench evaluation to examine the loss in multi-turn conversation ability since we trained only on single-turn data. The result (detailed in Appendix Table 6) shows that Meta-Rewarding significantly improves the Turn 1 Score from 8.319 to 8.738 in the last iteration, while sacrificing no more than 0.1 in Turn 2 Score. This is a large improvement on Self-Rewarding + LC, as it typically sacrifices more than 0.2 in Turn 2 score while not improving the Turn 1 score.

**Table 2: Arena-Hard: Although our prompt set mainly consists of Open Assistant-like prompts, which are far from the distribution of Arena-Hard (which is selected from the highest quality clusters from the Chatbot Arena dataset), we observe a substantial improvement. Four iterations of Meta-Rewarding brings +8.5% increase over the seed model.**
| Model | Score | 95% CI | Length |
| --- | --- | --- | --- |
| Llama-3-8B-Instruct (Seed) | 20.6% | (-2.0, 1.8) | 2485 |
| SFT on EFT | 24.2% | (-2.0, 1.8) | 2444 |
| Self-Rewarding LLM (Yuan et al., ) + LC |  |  |  |
| Iteration 1 | 23.2% | (-1.7, 1.9) | 2438 |
| Iteration 2 | 26.3% | (-2.1, 2.3) | 2427 |
| Iteration 3 | 28.2% | (-2.0, 1.9) | 2413 |
| Iteration 4 | 27.3% | (-2.0, 2.2) | 2448 |
| Meta-Rewarding LLM (Ours) |  |  |  |
| Iteration 1 | 25.1% | (-1.9, 1.8) | 2395 |
| Iteration 2 | 27.4% | (-2.0, 2.0) | 2416 |
| Iteration 3 | 27.6% | (-2.3, 2.6) | 2501 |
| Iteration 4 | 29.1% | (-2.3, 2.1) | 2422 |

### 3.4 Reward Modeling Evaluation

We evaluate the judging accuracy of our models on responses generated by the seed model Llama-3-8B-Instruct. In the absence of human labeling, we measure the correlation between our model and the currently strongest judge model, gpt-4-1106-preview. Our analysis employs two slightly different settings, primarily differing in how they handle ties given by the judge models. We begin with a fixed set of Open Assistant prompts that do not overlap with our training prompts.

For the *GPT-4 Chosen Pairs* setting, we generate two responses using the seed model for each prompt. We then generate preference labels with GPT-4 judge using a prompt adopted from AlpacaEval (see Section A.1). To mitigate positional bias, we make two judgements by switching the positions of the compared responses. We retain the data only where the two judgments agree, discarding the rest. This process yields a total of 170 pairs with preference judge labels.
Subsequently, we use the model being evaluated to predict rankings on those pairs, employing the same procedure as before by generating 11 judgments and averaging their scores.
We calculate two metrics: agreement (counting ties as 0.5) and agreement without ties (removing all ties predicted by the weaker judge and assessing agreement on the remaining pairs).

For the *Self-Chosen Pairs* setting, we generate 7 responses from the seed model and rank them using the target model.
Again, we use the same procedure of averaging of 11 judgements.
We select the highest and lowest scoring responses as the predicted chosen and rejected pairs, respectively. We then perform the same judgment using the strong GPT-4 model and report the agreement and agreement without ties metrics.

The model improves in judging after performing judge training: Our analysis shown in Table 3 reveals significant improvements in the correlation between Meta-Rewarding and the strong GPT-4 judge compared to the Self-Rewarding baseline in both evaluation settings. The enhancement is most notable in the agreement without ties metric.
For Self-Chosen Pairs, the improvement reaches up to +12.34% (Iteration 2) when comparing the same iterations of both models, while in the GPT-4 Chosen Pairs setting, the increase exceeds +6%. These results demonstrate the effectiveness of the Meta-Rewarding methodology in refining the model’s judgment capabilities, bringing its evaluations substantially closer to those of more sophisticated language models like GPT-4.

Meta-Rewarding training improve judging correlation with Human. We examine the judge’s correlation with the human-ranked responses from the Open Assistant dataset.
We use the same average over 11 judgments to get the predicted ranking, and then measure the agreement as well as the average Spearman correlation (over prompts). As shown in Appendix Table 7, there is a notable increase in correlation with human judgement, especially in Meta-Rewarding LLMs.
However, this improvement is not sustained over later training iterations, likely due to a distribution shift in the model-generated responses compared to the human responses.

**Table 3: Judge agreement with GPT-4 on responses generated by the seed model: Evaluation of the judge’s correlation with GPT4 on the Open Assistant test set, with responses generated by Llama-3-8B-Instruct.**
|  | GPT-4 Chosen Pairs | Self-Chosen Pairs |  |  |
| --- | --- | --- | --- | --- |
| Model | Agreement | Agree wo Tie | Agreement | Agree wo Tie |
| Llama-3-8B-Instruct (Seed) | 55.95% | 56.49% | 55.80% | 61.03% |
| SFT on EFT | 51.48% | 51.79% | 61.66% | 73.51% |
| Self-Rewarding LLM (Yuan et al., ) + LC |  |  |  |  |
| Iteration 1 | 56.54% | 57.97% | 55.17% | 59.59% |
| Iteration 2 | 52.67% | 53.43% | 54.89% | 60.00% |
| Iteration 3 | 55.65% | 55.90% | 61.13% | 72.68% |
| Iteration 4 | 52.97% | 53.12% | 64.44% | 78.42% |
| Meta-Rewarding LLM (Ours) |  |  |  |  |
| Iteration 1 | 56.54% | 57.23% | 60.06% | 68.75% |
| Iteration 2 | 55.05% | 56.58% | 61.57% | 72.34% |
| Iteration 3 | 58.63% | 61.24% | 63.43% | 76.80% |
| Iteration 4 | 57.44% | 59.54% | 64.50% | 79.33% |

### 3.5 Ablations and Analysis

Length-Control Mechanism: Our length-control mechanism is essential in maintaining a balance between comprehensiveness and conciseness of the model responses. We compare the last training iteration with different length-control parameter choices $\rho$ and present the results in Table 4.
Using $\rho=0$ is equivalent to not performing any length-control in the preference data selection. As expected, training this way makes the model excessively verbose for both models, and negatively affects the LC win rate as shown for Self-Rewarding LLMs.

**Table 4: Effect of Length-Control Parameter $\rho$ on AlpacaEval: We find that the length-control parameter $\rho$ significantly impacts both the win rate and length-controlled (LC) win rate. Using a larger threshold decreases the model generation length, and vise versa. While turning off the length-control mechanism ($\rho=0$) increases the win rate, it hurts the LC win rate and makes the responses longer. Choosing a balanced length-control parameter provides a balanced final performance. We also compare our length-control with a naive filtering based on the response length (Filter $>2500$), but this hurts both win rates, demonstrating the effectiveness our length-control mechanism.**
| Model | LC win rate | Win rate | Length |
| --- | --- | --- | --- |
| Self-Rewarding LLM + LC |  |  |  |
| Iteration 3 (Base) | 34.87% | 34.59% | 1967 |
| Iteration 4 ($\rho=0$) | 34.68% | 36.11% | 2063 |
| Iteration 4 ($\rho=0.1$) | 35.49% | 35.37% | 2005 |
| Iteration 4 ($\rho=0.3$) | 35.83% | 31.95% | 1806 |
| Iteration 4 (Filter $>2500$) | 32.90% | 33.33% | 1982 |
| Meta-Rewarding LLM (Ours) |  |  |  |
| Iteration 3 (Base) | 35.45% | 37.24% | 2064 |
| Iteration 4 ($\rho=0$) | - | - | 2212 |
| Iteration 4 ($\rho=0.3$) | - | - | 2127 |
| Iteration 4 ($\rho=0.35$) | - | - | 2067 |
| Iteration 4 ($\rho=0.4$) | 39.44% | 39.45% | 2003 |

Training with an External Reward Model:
Meta-Rewarding employs an LLM-as-a-Judge prompt to judge its own responses.
Instead, we experiment with using a strong external reward model Starling-RM-34B (Zhu et al., ) to select actor preference pairs. However, we find that Starling-RM-34B failed to increase the LC win rate of AlpacaEval in the first iteration (24.63% vs 27.85%), perhaps due to its length bias.

**Table 5: Meta-Judge Statistics. We observe growing biases in the meta-judge towards preferring higher score judgements or those in the first position.**
|  | Score Bias | Positional Bias |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Meta-Judge | Higher Win | Lower Win | Same Score | Diff Score | All |
| Iteration 1 | 63.04% | 36.96% | 47.79% | 41.12% | 43.92% |
| Iteration 2 | 97.68% | 2.32% | 87.75% | 56.18% | 68.11% |

Figure: Figure 5: Change in Scoring Distribution: Training the judge using the meta-judge changes its score distribution significantly. Notably, the judge tends to concentrate more into giving a high score. As a result, the mean score is increased from 4.1 to 4.7+ after two iterations of training.
Refer to caption: x4.png

Meta-Judge Biases: After the first iteration of Meta-Rewarding training, the meta-judge becomes more likely to prefer a higher score judgment nearly all the time, as shown in Table 5. This score-bias, in turn, significantly shifts the scoring distribution of the judge towards the full score of 5.
For the positional bias, we also see an increasing trend of during the training, especially for comparing two judgments with the same score.

Judge Scoring Shift.
To investigate the judge score distribution change during Meta-Rewarding training iterations, we use the same validation prompts as used for reward modeling evaluation. We generate 7 responses on each prompt using Llama-3-8B-Instruct, then generate 11 judgments for each response. Figure 5 is a visualization of the scoring distribution, where the density is estimated using Gaussian kernel density estimation (Davis et al., ). Training the judge using the meta-judge further increases its likelihood of generating higher scores. However, we notice that the first 2 iterations of the judge training makes it prefer to assign scores 4.5, 4.75, 4.9 even though the scores should be integers according to the instruction.
Although these are high scores, they provide more granularity and distinguishing ability for separating different quality responses.

## 4 Related work

RLHF
Significant efforts have been made towards aligning LLMs with human values. These alignment strategies can be broadly classified into aligning with a reward model or aligning directly based on a preference dataset. Ziegler et al. (); Stiennon et al. (); Ouyang et al. (); Bai et al. () train a fixed reward model from human preference data, and then use the reward model to train via reinforcement learning (RL), e.g. via Proximal Policy Optimization (PPO) (Schulman et al., ). To further reduce engineering costs, P3O (Wu et al., ) derived the contrastive policy gradient, which has shown superior performance over PPO while removing the need for a value function. In contrast, methods such as Direct Preference Optimization (DPO) (Rafailov et al., ) avoid training the reward model entirely, and instead directly train the LLM using human preferences. Several other such competing methods exist as well (Xu et al., ; Zhao et al., ; Zheng et al., ; Yuan et al., ).
Iterative DPO (Xu et al., ) uses a reward model to build preference data from model responses for multiple rounds of DPO training, with improved results.

LLM-as-a-Judge
Using LLM-as-a-Judge for evaluation (Li et al., ; Dubois et al., ; ; Saha et al., ; Bai et al., ) and training reward models (Lee et al., ; Zhu et al., ; Chen et al., ; Li et al., ) has become a standard practice. Some works, such as Kim et al. (; ), have investigated how to construct datasets for training a LLM-as-a-Judge. However, these approaches typically use human data or data coming from a much stronger model. In contrast, our approach emphasizes self-improvement of judgment skills.

Super Alignment The idea of aligning a very capable model that even surpasses human level is called super alignment. Since current AI alignment methods mostly rely on either supervised fine-tuning (SFT) with human-provided demonstrations (Sanh et al., ; Wei et al., ; Chung et al., ) or reinforcement learning from human feedback (RLHF) (Ziegler et al., ; Stiennon et al., ; Ouyang et al., ), their capabilities would be inherently limited as humans cannot always provide helpful demonstrations or supervision on the hard tasks beyond their expertise (Sharma et al., ). Several promising directions toward super alignment exist, including using models to assist human supervision (Scalable oversight (Bowman et al., ; Saunders et al., ; Leike et al., ; Lightman et al., )), automatic search for problematic behaviors or internals (Interpretability (Perez et al., ; Bills et al., ; Templeton, )) and more. Perhaps the closest direction to our work is using AI to produce feedback for training AI, which is also known as RLAIF (Zhu et al., ; Lee et al., ). For example, Constitutional AI (Bai et al., ) uses an LLM to give feedback and refine responses, and uses this data to train a reward model, which is then used to train the language model via RL.
McAleese et al. () trained CriticGPT to write critiques that highlight inaccuracies in ChatGPT answers.
Self-Rewarding Yuan et al. (), the closest work to ours which we build upon, is an iterative training scheme where the model acts as a judge to evaluate its own responses and then that feedback is used in the preference optimization.
However, as far as we know, less work has focused on training the actor and the judge simultaneously during self-improvement.

## 5 Limitations

A deficiency in our experimental setup is the 5-point judging system that we chose, following Yuan et al. (). We discovered that this scoring method often results in ties due to minimal quality differences between responses, necessitating careful averaging of multiple judgments to differentiate between them. Moreover, as training progressed, responses increasingly approached the maximum score, making further improvements difficult to detect. A more nuanced scoring system that covers diverse aspects (Wang et al., ) or a comparison-based approach might address these issues.

Another significant limitation lies in the judge training process. Despite our efforts to mitigate positional bias of our meta-judge, this issue persists and hindered further improvements in Iteration 3. The judge also demonstrated a tendency to assign higher scores, which accelerated score saturation and reduced its ability to discriminate between responses. Furthermore, the judge showed limited improvement in evaluating non-self-generated responses in our evaluations. We believe there is substantial room for improvement if these issues can be effectively addressed, which could significantly boost the overall effectiveness of our approach.

## 6 Conclusion

In this work, we propose a novel mechanism for improving the judging skill of models by using a meta-judge that assigns meta-rewards
to select chosen and rejected judgments for preference optimization.
This addresses a major limitation of the Self-Rewarding framework (Yuan et al., ), specifically the lack of training the judge. To make Meta-Rewarding training work,
we additionally introduce a new length-control technique to mitigate the issue of length explosion when training with AI feedback.
The effectiveness of our method is demonstrated through auto-evaluation benchmarks AlpacaEval, Arena-Hard, and MT-Bench.
Remarkably, even without additional human feedback, our approach significantly improves upon Llama-3-8B-Instruct and surpasses both Self-Rewarding and SPPO (Wu et al., ), a strong baseline that relies heavily on human feedback. Furthermore, when we evaluate our model’s judging ability, it shows significant improvement in correlation with both human judges and strong AI judges like gpt-4-1106-preview.
Overall, our findings provide strong evidence that self-improving the model without any human feedback is a promising direction for achieving super alignment.

## Appendix A Appendix

### A.1 Judge Prompt

We adopt the same judge prompt as in Yuan et al. ().

### A.2 GPT4 Judge Prompt

We adopt this prompt from AlpacaEval, which is proved to have high correlation with human judges.

Figure: Figure 6: Distribution of Prompts: A t-SNE to visualization of three sources of prompts: training prompts, AlpacaEval prompts and Arena-Hard prompts. The embedding of the prompts are calculated by text-embedding-3-small. Our training prompts are closer in distribution to AlpacaEval prompts, while Arena-Hard is more concentrated into a subset of the distribution.
Refer to caption: figures/prompt_distribution.png

**Table 6: MT-Bench: Since our training mainly focus on the first-turn capability, we observe a significant improvement in the Turn 1 Score. While the Self-Rewarding baseline suffer from a large drop in Turn 2 score, our Meta-Rewarding only sacrifice slightly and even improving the Turn 2 score in Iteration 3 & 4.**
| Model | Score | Turn 1 | Turn 2 | Length |
| --- | --- | --- | --- | --- |
| Llama-3-8B-Instruct | 8.116 | 8.319 | 7.911 | 1568 |
| SFT on EFT | 7.943 | 8.138 | 7.747 | 1511 |
| Self-Rewarding LLM + LC |  |  |  |  |
| Iteration 1 | 7.909 | 8.144 | 7.671 | 1576 |
| Iteration 2 | 7.894 | 8.200 | 7.588 | 1570 |
| Iteration 3 | 7.984 | 8.231 | 7.734 | 1528 |
| Iteration 4 | 8.028 | 8.381 | 7.675 | 1539 |
| Meta-Rewarding LLM |  |  |  |  |
| Iteration 1 | 7.994 | 8.263 | 7.725 | 1555 |
| Iteration 2 | 8.198 | 8.794 | 7.595 | 1577 |
| Iteration 3 | 8.341 | 8.731 | 7.950 | 1596 |
| Iteration 4 | 8.288 | 8.738 | 7.838 | 1592 |

**Table 7: Judge’s Correlation with Human: We measure the judge’s agreement (with and without ties) with humans on the Open Assistant test set. Spearman correlation represent the ranking spearman correlation with the ground truth averaged over prompts.**
| Model | Agreement | Agree wo Tie | Spearman corr. |
| --- | --- | --- | --- |
| Llama-3-8B-Instruct | 62.81% | 64.18% | 0.315 |
| SFT on EFT | 63.20% | 64.59% | 0.321 |
| Self-Rewarding LLM + LC |  |  |  |
| Iteration 1 | 63.04% | 65.04% | 0.298 |
| Iteration 2 | 64.14% | 67.17% | 0.347 |
| Iteration 3 | 60.23% | 61.63% | 0.251 |
| Iteration 4 | 61.48% | 62.22% | 0.283 |
| Meta-Rewarding LLM |  |  |  |
| Iteration 1 | 57.73% | 61.98% | 0.210 |
| Iteration 2 | 66.64% | 68.33% | 0.382 |
| Iteration 3 | 63.35% | 65.24% | 0.329 |
| Iteration 4 | 62.96% | 64.82% | 0.326 |

### A.3 Training Details

For the SFT model, we train for a total of 10 epochs using a learning rate $5\times 10^{-8}$ and global batch size of 32. We employed cosine learning rate scheduling and saved a checkpoint after every epoch. We selected checkpoint from epoch 5 as the final model.

For all DPO training, we also trained for 10 epochs, with a learning rate of $5\times 10^{-6}$, $\beta=0.1$ and global batch size of 32. We adopted cosine learning rate scheduling.

For Self-Rewarding training, during Iteration 1 we set $\rho=0$ for actor data creation and applied a filter to exclude pairs where the chosen response length exceeded $2500$ characters. We selected the checkpoint from epoch 5 for this iteration. In both Iteration 2 & 3 we continue with $\rho=0$ and chose checkpoints from epoch 1 and epoch 2 respectively. For Iteration 4, we adjust $\rho$ to $0.1$ and selected the checkpoint from epoch 2.

For Meta-Rewarding training in Iteration 1 we set $\rho=0$ for actor data creation, and we filtered out pairs with chosen response length exceeding $2500$ characters. Additionally, for the judge data creation, we filtered out pairs if the chosen judgment length exceeded $1100$. We selected checkpoint from epoch 6 for this iteration. In Iteration 2, we increased $\rho$ to $0.32$ and set the threshold to $1000$ for judge data filtering, we selected the checkpoint from epoch 4. In Iteration 3 we maintain $\rho$ at $0.32$ and chose the checkpoint from epoch 2. Finally, in Iteration 4, we further increased $\rho$ to $0.4$ and again selected the checkpoint from epoch 2.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In the implementation of the judge model and meta-judge model, the judge model is a point-wise scorer while the meta-judge model is a pairwise evaluator. What is the underlying rationale for this differentiation?
2. What would be the implications of adopting a consistent approach across both models - either exclusively point-wise scoring or pairwise evaluation?"
3. Regarding the evaluation of reward modeling, it would be valuable to conduct a comparative analysis between the Meta-Rewarding LLM and several established reward models, including nvidia/Llama-3.1-Nemotron-70B-Reward[1], Skywork/Skywork-Reward-Llama-3.1-8B-v0.2[2], and RLHFlow/ArmoRM-Llama3-8B-v0.1[3]. This comparison could be performed across two types of benchmarks: those specifically developed in the paper and official reward model benchmarks such as RewardBench[4] and Preference Proxy Evaluations[5].
4. Recent self-improvement approaches, such as iterative DPO[6], SPPO[7], and SimPO[8], have demonstrated considerable performance gains in AlpacaEval 2.0 when utilizing external reward models (RM) for response scoring. However, this appears to contradict the experimental findings presented in Section 3.5, which suggest that the incorporation of external RMs diminishes the model's overall performance. Could the authors elaborate on this apparent discrepancy and provide more insights about these results?

[1] HelpSteer2-Preference: Complementing Ratings with Preferences

[2] Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs

[3] Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts

[4] RewardBench: Evaluating Reward Models for Language Modeling

[5] How to Evaluate Reward Models for RLHF

[6] https://snorkel.ai/blog/how-snorkel-topped-the-alpacaeval-leaderboard-and-why-we-re-not-there-anymore/

[7] Self-Play Preference Optimization for Language Model Alignment

[8] SimPO: Simple Preference Optimization with a Reference-Free Reward
soundness: 3
strengths: 1. The motivation of explicitly enhancing judging ability, which establishes the upper bound for self-improvement of LLMs, is compelling and well-founded.
2. The proposed Meta-Rewarding approach offers the advantage of improving the response generation and judging capabilities simultaneously.
3. The writing and presentation are clear and easy to read.
summary: This study presents an innovative Meta-Rewarding mechanism to enhance the existing self-rewarding framework. This new approach enables the model to evaluate its judgments and incorporate this meta-level feedback to improve both its acting and judging skills simultaneously. The implementation of this mechanism in the Llama-3-8B-Instruct has yielded significant improvements in both instruction-following and reward-modeling capabilities, highlighting the potential for autonomous self-improvement without direct human supervision.
weaknesses: 1. The primary contribution of this paper is utilizing Meta-Rewarding to address performance saturation during iterative training, however, the judge score shifting results in less training iterations of the judge model than the actor model. This limitation not only results in the diminished performance of the judging model in the final iteration (as evidenced by GPT-4 Chosen Pairs in Table 3) but also contradicts the method's intended purpose, thereby constraining the scalability of the proposed approach.
2. The paper lacks sufficient discussion of the rationale for employing the model as its meta-judge in supervising the judging model. A critical question that emerges is the underlying mechanism behind the effectiveness of this self-supervision approach.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: 1. The authors highlight the importance of improving the model’s judging ability through iterative self-evolve, which is conceptually sound.
2. They designed separate methods for Actor Data Creation and Judge Data Creation and trained the model using DPO.
3. A new length-control technique was proposed to mitigate the issue of length explosion when training with AI feedback.
summary: The authors propose the Meta Rewarding approach, where an LLM acts as an actor, judge, and meta-judge during iterative self-evolve to enhance its abilities as the actor and judge. Through experiments, they verify that the LLM's capabilities as both actor and judge improve over multiple iterations.
weaknesses: 1. A logical concern arises regarding the authors’ approach of using the LLM as a meta-judge to improve its judging ability. If so, wouldn’t the meta-judge also need enhancement? Would this necessitate a meta-meta-judge?
2. The authors compare the Meta Rewarding and Self Rewarding methods, showing performance improvements. However, I am curious if computational costs were aligned in the comparison. This includes the overhead for generating Actor Data and Judge Data and the amount of data used for training after filtering. Especially given that the Meta Rewarding method requires additional Judge Data for training, clarifying this aspect is crucial to alleviate concerns about performance gains resulting merely from increased data construction and training costs.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Given the failure of improvement of judgments due to biases in meta-judge, does it make more sense to have two different LLMs one that mainly focuses on improving the judge and another that focuses on only improving the actor? Perhaps training both tasks simultaneously into a single model leads to biases in Meta-Judgements.
- Even with the length conditioned training, the average length of responses keeps increasing up until iteration 3 in Tables 1 and 2. By virtue of the preference selection process, if the chosen responses are guaranteed to be smaller than rejected responses in DPO training, is there any intuition why the length still increases in Meta-Rewarding process?
- From Figure 3, it seems like the performance from Meta-Rewarding hasn't saturated on Alpaca Eval. I'm curious if authors tried more iterations of this process and if it helped.
soundness: 3
strengths: - The paper shows an interesting paradigm for language model tuning where a single model can simultaneously optimize for better response generation and also become a better judge.
- The proposed strategy to reduce length bias seems effective given their revaluation results (Table 1, 2, and 4)
- The authors provide numerous analyses and ablations related to judge alignment and fine-grained category analysis in Alpaca Eval. It is especially exciting to see that iteration 4 of their proposed Meta-Rewarding strategy improves stably in all 18 sub-categories of AlpacaEval.
summary: The authors aim to improve recent self-rewarding mechanisms (Yuan et al., 2024c). Self-rewarding LLM improves the LLM response generation by judging its own responses. Typically there are two roles of LLM in the Self-rewarding setup, actor and judge. The actor generates responses towards certain prompt distribution and the same model then acts as a judge and discerns the good and bad outputs. 
However, the authors recognize a key limitation of the Self-rewarding LLM approach, that it only focuses on improving the actor and not the judge. To mitigate this, they propose a 3rd role of meta-judge, which evaluates the judgments and creates preference pairs for judge training. They call their new step Meta-Rewarding i.e. labeling its own judgments. Uniquely, in their setup, the same LLM plays all three roles - actor, judge, and meta-judge. 

Another issue with the Self-rewarding setup is its tendency to increase length after each improvement iteration. To fix this, authors propose a new scheme for sampling responses, that prefer shorter responses over longer ones. In the judge role, LLM assigns a score out of 5 signifying the quality of the response for the prompt. For a given set of responses for a prompt, the authors define a region around the min and max score using an interpolation hyperparameter $\rho \ge 0$. All the responses within this boundary region are considered equally good and the shortest one is selected as the chosen preference whereas the longest one is selected as the rejected preference. 

Overall their training setup iteratively trains an LLM as an actor and judge such that both operations improve resulting in a process that doesn't require human labels. The actor training uses the labels from the judge role whereas the judge training uses the labels from the meta-judge role. Their evaluations on Arena-Hard and length-controlled Alpaca Evals show that the Meta-Rewarding process can show increased performance compared to Self-Rewarding while avoiding its length increase issue (i.e. responses from the final iteration of Meta-Rewarding are usually smaller than the responses from the final iteration of Self-Rewarding).
weaknesses: - The meta-judge becomes heavily biased with just one iteration of training, whereby it only picks the first position or the judgment with a higher score without actually evaluation the quality of the judgment.
   - Accordingly, out of the 4 iterations of Meta-Rewarding, the judgment training is stopped in iterations 3 and 4 while only the actor is being trained.
   - There is also a change in score distribution from the judgment phase (figure 5), indicating that subsequent iterations in Meta-Rewarding will suffer from too many ties and unclear labels for actor training.
   - These limitations of the judgment process, bring serious concerns regarding the scalability of the Meta-Rewarding approach.
- (follow-up from the previous point) In Table 3, the judgment agreement increases in iterations 3 and 4, even though only the actor model is trained in these iterations. What explains this improvement? It is unclear why actor training alone would improve judge agreement.
    - Also, given the small size of the eval set and small differences between agreement rates of Self-Rewarding and Meta-Rewarding models in Table 3, I'm concerned that those results are statistically significant.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - Last equation  before S3: $\epsilon_m$ and $\epsilon_n$ are two distinct dimensions of $\epsilon$? I don't see where you defined $\epsilon$. Make it clear. 


Missing citations: Here are several works that are not cited in your work: 
- Crystal: Introspective reasoners reinforced with self-feedback, 2023
- Generating sequences by learning to self-correct, 2023
- The self-Instruct paper, 2023
- The Reflexion paper, 2023


Suggestion: There is also literature that casts doubt on the abilities of LLMs to self-improve out of the box. It would be great to also expand on this literature: 
- The DeepMind paper: Large language models cannot self-correct reasoning yet, 2024 
- Self-[In]Correct: LLMs Struggle with Discriminating Self-Generated Responses, 2024
- The Google paper: LLMs cannot find reasoning errors, but can correct them given the error location, 2024
- ... (there are a few others that I might be missing)


Your work is also related to the growing literature on inference-scaling. As the prior work has shown, inference-scaling works. Naturally, one can extend these to data generation, as you do in your work. 


Finally, there is prior work on alignment of various sorts that also does length-controlled preference data selection (https://arxiv.org/pdf/2402.07319, https://arxiv.org/abs/2404.03862); highlighting for your attention. These may be hidden in experimental details since, typically, no one makes a big deal of their length-controlled data filtering/selection.
soundness: 2
strengths: - Overall, the idea is imaginative and interesting.  The method is simple and intuitive.
- The performance increase is large (22.9% → 39.4%) on AlpacaEval 2.0 LC.
- The paper is clearly written, with details on the data construction and synthesis, making the work reproducible. Overall, the writing is smooth, and I was able to follow most of the ideas (there are places which can be improved, as I discuss later on).
summary: The paper proposed Meta-rewarding, an additional layer on synthetic data generation that uses judgments on (model’s) prior judgments in order to improve the quality of synthetic data generation and hence improve the quality of the aligned model. This added verification layer sits on top of the model's generation ability to mitigate saturation in “self-rewarding mechanisms”. 
The paper uses  Llama-3-8B-Instruct for evaluation and conducts evaluation with length-normalized AlpachaEval 2. The evaluation shows that the proposed approach improves upon the prior work by Yuan et al.
weaknesses: - The proposed approach requires more computation (compared to their baseline Yuan et al). Hence, if the comparison is for a fixed compute budget, which is the best data augmentation method, one needs to redo the experiments (e.g., Fig3) to obtain compute-adjusted comparison results. I acknowledge that it's fair to control for "iteration" only if the goal is to solve the saturation problem in data augmentation, but that’s a different goal. 
- The entire setup of self-rewarding only assumes no human annotator available, but ignores the fact that you can take off-the-shelf reward models to rank your responses. It would be useful to compare self-rewarding with LLM-judge and iterative DPO with rejection sampling / Best-of-N sampling, which I believe is a more practical setting that is adopted by many works [1, 2]. If you can take off-the-shelf SFT models, and assume “no human annotator available”, why can’t you take off-the-shelf reward models under the same assumption.
- The setup assumes that the base LM is bad at judging (or otherwise, you wouldn’t need to train the judging ability) but good at meta-judging (since you take the meta-judge results as gold results to train the judge) - which might be true for LLama-3-8B-Instruct. This work only investigated ONE base model (llama3-8b-inst) on TWO different benchmarks (Arena Hard, Alpaca Eval 2.0), therefore it is unclear if the assumption still holds for other models / benchmarks. As the LLama3 Instruct model’s training data is unknown - it is unclear whether the improvements comes from the proposed pipeline or simply because LLama 3 instruct is trained on human annotated meta judging data - which contradicts the setup of “without human labelers”.
- It is unclear what is the upper limit of self-rewarding [3] and meta-rewarding (this work) as the former only reports performance up to 3 iterations and this work only reports performance up to 4 iterations. If, after these iterations, the performance stops increasing, then this should also be reported so that the community can learn about the upper bound of self-rewarding mechanisms. The authors claim that the proposed method solves the “rapid saturation” problem in [3], but extending from 3 iters to 4 can hardly be viewed as solving “rapid saturation”.

References 

- [1] The Llama3 Herd of Models (Dubey et al., 2024)
- [2] Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (Xiong et al., 2023)
- [3] Self-Rewarding Language Models (Yuan et al., 2024)

Meta-review: The paper proposes self improving with a meta judge.

Strengths:
Timely topic  
Clear writing  
Interesting approach and techniques, with benefits (e.g. improving generation and verification at once)

Weaknesses:  
There were concerns regarding the compute necessary, and the compute of this vs baseline methods.  
It is unclear whether results (such as about the quality of models as a judge and a meta judge) would replicate to other models and are general.
See more proposals for improvements in reviews.

Overall the paper has merit but had enough open issues to call for addressing them before publishing.

Decision: Reject