Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge

Authors: Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason E Weston, Sainbayar Sukhbaatar
Primary area: foundation or frontier models, including LLMs
Keywords: self-improving, self-rewarding, LLM, LLM-as-a-judge, instruction following, super alignment

Abstract:
Large Language Models (LLMs) are rapidly surpassing human knowledge in many domains. While improving these models traditionally relies on costly human data, recent self-rewarding mechanisms (Yuan et al., 2024) have shown that LLMs can improve by judging their own responses instead of relying on human labelers. However, existing methods have primarily focused on improving model responses rather than judgment capabilities, resulting in rapid saturation during iterative training. To address this issue, we introduce a novel Meta-Rewarding step to the self-improvement process, where the model judges its own judgements and uses that feedback to refine its judgment skills. Surprisingly, this unsupervised approach improves the model's ability to judge and follow instructions, as demonstrated by a win rate improvement of Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2, and 20.6% to 29.1% on Arena-Hard. These results strongly suggest the potential for self-improving models without human supervision.

Decision: reject
Reviewer scores: 6, 5, 6, 3

Reviews and discussion:
<thread>
creation date:
1729919064331

thread_type:
official_review

<thread_object>
creator: reviewer_kg4s

<content>
summary:
The paper proposed Meta-rewarding, an additional layer on synthetic data generation that uses judgments on (model’s) prior judgments in order to improve the quality of synthetic data generation and hence improve the quality of the aligned model. This added verification layer sits on top of the model's generation ability to mitigate saturation in “self-rewarding mechanisms”. 
The paper uses  Llama-3-8B-Instruct for evaluation and conducts evaluation with length-normalized AlpachaEval 2. The evaluation shows that the proposed approach improves upon the prior work by Yuan et al.

strengths:
- Overall, the idea is imaginative and interesting.  The method is simple and intuitive.
- The performance increase is large (22.9% → 39.4%) on AlpacaEval 2.0 LC.
- The paper is clearly written, with details on the data construction and synthesis, making the work reproducible. Overall, the writing is smooth, and I was able to follow most of the ideas (there are places which can be improved, as I discuss later on).

weaknesses:
- The proposed approach requires more computation (compared to their baseline Yuan et al). Hence, if the comparison is for a fixed compute budget, which is the best data augmentation method, one needs to redo the experiments (e.g., Fig3) to obtain compute-adjusted comparison results. I acknowledge that it's fair to control for "iteration" only if the goal is to solve the saturation problem in data augmentation, but that’s a different goal. 
- The entire setup of self-rewarding only assumes no human annotator available, but ignores the fact that you can take off-the-shelf reward models to rank your responses. It would be useful to compare self-rewarding with LLM-judge and iterative DPO with rejection sampling / Best-of-N sampling, which I believe is a more practical setting that is adopted by many works [1, 2]. If you can take off-the-shelf SFT models, and assume “no human annotator available”, why can’t you take off-the-shelf reward models under the same assumption.
- The setup assumes that the base LM is bad at judging (or otherwise, you wouldn’t need to train the judging ability) but good at meta-judging (since you take the meta-judge results as gold results to train the judge) - which might be true for LLama-3-8B-Instruct. This work only investigated ONE base model (llama3-8b-inst) on TWO different benchmarks (Arena Hard, Alpaca Eval 2.0), therefore it is unclear if the assumption still holds for other models / benchmarks. As the LLama3 Instruct model’s training data is unknown - it is unclear whether the improvements comes from the proposed pipeline or simply because LLama 3 instruct is trained on human annotated meta judging data - which contradicts the setup of “without human labelers”.
- It is unclear what is the upper limit of self-rewarding [3] and meta-rewarding (this work) as the former only reports performance up to 3 iterations and this work only reports performance up to 4 iterations. If, after these iterations, the performance stops increasing, then this should also be reported so that the community can learn about the upper bound of self-rewarding mechanisms. The authors claim that the proposed method solves the “rapid saturation” problem in [3], but extending from 3 iters to 4 can hardly be viewed as solving “rapid saturation”.

References 

- [1] The Llama3 Herd of Models (Dubey et al., 2024)
- [2] Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (Xiong et al., 2023)
- [3] Self-Rewarding Language Models (Yuan et al., 2024)

questions:
- Last equation  before S3: $\epsilon_m$ and $\epsilon_n$ are two distinct dimensions of $\epsilon$? I don't see where you defined $\epsilon$. Make it clear. 


Missing citations: Here are several works that are not cited in your work: 
- Crystal: Introspective reasoners reinforced with self-feedback, 2023
- Generating sequences by learning to self-correct, 2023
- The self-Instruct paper, 2023
- The Reflexion paper, 2023


Suggestion: There is also literature that casts doubt on the abilities of LLMs to self-improve out of the box. It would be great to also expand on this literature: 
- The DeepMind paper: Large language models cannot self-correct reasoning yet, 2024 
- Self-[In]Correct: LLMs Struggle with Discriminating Self-Generated Responses, 2024
- The Google paper: LLMs cannot find reasoning errors, but can correct them given the error location, 2024
- ... (there are a few others that I might be missing)


Your work is also related to the growing literature on inference-scaling. As the prior work has shown, inference-scaling works. Naturally, one can extend these to data generation, as you do in your work. 


Finally, there is prior work on alignment of various sorts that also does length-controlled preference data selection (https://arxiv.org/pdf/2402.07319, https://arxiv.org/abs/2404.03862); highlighting for your attention. These may be hidden in experimental details since, typically, no one makes a big deal of their length-controlled data filtering/selection.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693489032

thread_type:
official_review

<thread_object>
creator: reviewer_nekw

<content>
summary:
The authors aim to improve recent self-rewarding mechanisms (Yuan et al., 2024c). Self-rewarding LLM improves the LLM response generation by judging its own responses. Typically there are two roles of LLM in the Self-rewarding setup, actor and judge. The actor generates responses towards certain prompt distribution and the same model then acts as a judge and discerns the good and bad outputs. 
However, the authors recognize a key limitation of the Self-rewarding LLM approach, that it only focuses on improving the actor and not the judge. To mitigate this, they propose a 3rd role of meta-judge, which evaluates the judgments and creates preference pairs for judge training. They call their new step Meta-Rewarding i.e. labeling its own judgments. Uniquely, in their setup, the same LLM plays all three roles - actor, judge, and meta-judge. 

Another issue with the Self-rewarding setup is its tendency to increase length after each improvement iteration. To fix this, authors propose a new scheme for sampling responses, that prefer shorter responses over longer ones. In the judge role, LLM assigns a score out of 5 signifying the quality of the response for the prompt. For a given set of responses for a prompt, the authors define a region around the min and max score using an interpolation hyperparameter $\rho \ge 0$. All the responses within this boundary region are considered equally good and the shortest one is selected as the chosen preference whereas the longest one is selected as the rejected preference. 

Overall their training setup iteratively trains an LLM as an actor and judge such that both operations improve resulting in a process that doesn't require human labels. The actor training uses the labels from the judge role whereas the judge training uses the labels from the meta-judge role. Their evaluations on Arena-Hard and length-controlled Alpaca Evals show that the Meta-Rewarding process can show increased performance compared to Self-Rewarding while avoiding its length increase issue (i.e. responses from the final iteration of Meta-Rewarding are usually smaller than the responses from the final iteration of Self-Rewarding).

strengths:
- The paper shows an interesting paradigm for language model tuning where a single model can simultaneously optimize for better response generation and also become a better judge.
- The proposed strategy to reduce length bias seems effective given their revaluation results (Table 1, 2, and 4)
- The authors provide numerous analyses and ablations related to judge alignment and fine-grained category analysis in Alpaca Eval. It is especially exciting to see that iteration 4 of their proposed Meta-Rewarding strategy improves stably in all 18 sub-categories of AlpacaEval.

weaknesses:
- The meta-judge becomes heavily biased with just one iteration of training, whereby it only picks the first position or the judgment with a higher score without actually evaluation the quality of the judgment.
   - Accordingly, out of the 4 iterations of Meta-Rewarding, the judgment training is stopped in iterations 3 and 4 while only the actor is being trained.
   - There is also a change in score distribution from the judgment phase (figure 5), indicating that subsequent iterations in Meta-Rewarding will suffer from too many ties and unclear labels for actor training.
   - These limitations of the judgment process, bring serious concerns regarding the scalability of the Meta-Rewarding approach.
- (follow-up from the previous point) In Table 3, the judgment agreement increases in iterations 3 and 4, even though only the actor model is trained in these iterations. What explains this improvement? It is unclear why actor training alone would improve judge agreement.
    - Also, given the small size of the eval set and small differences between agreement rates of Self-Rewarding and Meta-Rewarding models in Table 3, I'm concerned that those results are statistically significant.

questions:
- Given the failure of improvement of judgments due to biases in meta-judge, does it make more sense to have two different LLMs one that mainly focuses on improving the judge and another that focuses on only improving the actor? Perhaps training both tasks simultaneously into a single model leads to biases in Meta-Judgements.
- Even with the length conditioned training, the average length of responses keeps increasing up until iteration 3 in Tables 1 and 2. By virtue of the preference selection process, if the chosen responses are guaranteed to be smaller than rejected responses in DPO training, is there any intuition why the length still increases in Meta-Rewarding process?
- From Figure 3, it seems like the performance from Meta-Rewarding hasn't saturated on Alpaca Eval. I'm curious if authors tried more iterations of this process and if it helped.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720826942

thread_type:
official_review

<thread_object>
creator: reviewer_zyxk

<content>
summary:
The authors propose the Meta Rewarding approach, where an LLM acts as an actor, judge, and meta-judge during iterative self-evolve to enhance its abilities as the actor and judge. Through experiments, they verify that the LLM's capabilities as both actor and judge improve over multiple iterations.

strengths:
1. The authors highlight the importance of improving the model’s judging ability through iterative self-evolve, which is conceptually sound.
2. They designed separate methods for Actor Data Creation and Judge Data Creation and trained the model using DPO.
3. A new length-control technique was proposed to mitigate the issue of length explosion when training with AI feedback.

weaknesses:
1. A logical concern arises regarding the authors’ approach of using the LLM as a meta-judge to improve its judging ability. If so, wouldn’t the meta-judge also need enhancement? Would this necessitate a meta-meta-judge?
2. The authors compare the Meta Rewarding and Self Rewarding methods, showing performance improvements. However, I am curious if computational costs were aligned in the comparison. This includes the overhead for generating Actor Data and Judge Data and the amount of data used for training after filtering. Especially given that the Meta Rewarding method requires additional Judge Data for training, clarifying this aspect is crucial to alleviate concerns about performance gains resulting merely from increased data construction and training costs.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730778501706

thread_type:
official_review

<thread_object>
creator: reviewer_p6cq

<content>
summary:
This study presents an innovative Meta-Rewarding mechanism to enhance the existing self-rewarding framework. This new approach enables the model to evaluate its judgments and incorporate this meta-level feedback to improve both its acting and judging skills simultaneously. The implementation of this mechanism in the Llama-3-8B-Instruct has yielded significant improvements in both instruction-following and reward-modeling capabilities, highlighting the potential for autonomous self-improvement without direct human supervision.

strengths:
1. The motivation of explicitly enhancing judging ability, which establishes the upper bound for self-improvement of LLMs, is compelling and well-founded.
2. The proposed Meta-Rewarding approach offers the advantage of improving the response generation and judging capabilities simultaneously.
3. The writing and presentation are clear and easy to read.

weaknesses:
1. The primary contribution of this paper is utilizing Meta-Rewarding to address performance saturation during iterative training, however, the judge score shifting results in less training iterations of the judge model than the actor model. This limitation not only results in the diminished performance of the judging model in the final iteration (as evidenced by GPT-4 Chosen Pairs in Table 3) but also contradicts the method's intended purpose, thereby constraining the scalability of the proposed approach.
2. The paper lacks sufficient discussion of the rationale for employing the model as its meta-judge in supervising the judging model. A critical question that emerges is the underlying mechanism behind the effectiveness of this self-supervision approach.

questions:
1. In the implementation of the judge model and meta-judge model, the judge model is a point-wise scorer while the meta-judge model is a pairwise evaluator. What is the underlying rationale for this differentiation?
2. What would be the implications of adopting a consistent approach across both models - either exclusively point-wise scoring or pairwise evaluation?"
3. Regarding the evaluation of reward modeling, it would be valuable to conduct a comparative analysis between the Meta-Rewarding LLM and several established reward models, including nvidia/Llama-3.1-Nemotron-70B-Reward[1], Skywork/Skywork-Reward-Llama-3.1-8B-v0.2[2], and RLHFlow/ArmoRM-Llama3-8B-v0.1[3]. This comparison could be performed across two types of benchmarks: those specifically developed in the paper and official reward model benchmarks such as RewardBench[4] and Preference Proxy Evaluations[5].
4. Recent self-improvement approaches, such as iterative DPO[6], SPPO[7], and SimPO[8], have demonstrated considerable performance gains in AlpacaEval 2.0 when utilizing external reward models (RM) for response scoring. However, this appears to contradict the experimental findings presented in Section 3.5, which suggest that the incorporation of external RMs diminishes the model's overall performance. Could the authors elaborate on this apparent discrepancy and provide more insights about these results?

[1] HelpSteer2-Preference: Complementing Ratings with Preferences

[2] Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs

[3] Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts

[4] RewardBench: Evaluating Reward Models for Language Modeling

[5] How to Evaluate Reward Models for RLHF

[6] https://snorkel.ai/blog/how-snorkel-topped-the-alpacaeval-leaderboard-and-why-we-re-not-there-anymore/

[7] Self-Play Preference Optimization for Language Model Alignment

[8] SimPO: Simple Preference Optimization with a Reference-Free Reward
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes self improving with a meta judge.

Strengths:
Timely topic  
Clear writing  
Interesting approach and techniques, with benefits (e.g. improving generation and verification at once)

Weaknesses:  
There were concerns regarding the compute necessary, and the compute of this vs baseline methods.  
It is unclear whether results (such as about the quality of models as a judge and a meta judge) would replicate to other models and are general.
See more proposals for improvements in reviews.

Overall the paper has merit but had enough open issues to call for addressing them before publishing.