Kati8/ICLR_2023_2025 huggingface dataset:
authors: Leitian Tao
authors: Yixuan Li
reviews: 
Comments: This work proposes to investigate whether "weak" language models could serve as a reward model/preference generator for alignment. Experiments demonstrate that they can to varying extents and match some of the benefits of HF/AIF.
Strengths: + ailgnment with synthetic feedback/preference is an important research question
+ the experiments are comprehensive
Weaknesses: - On the high-level framing, why is "weak LLM as feedback" positioned as the "middle ground" (line 42) between human feedback and AI/LM feedback? You could say that weak LLM feedback is still AI feedback, just with a smaller/weaker model, and not necessarily offers something strikingly new that make it "middle ground" in the "spectrum".

- By "weak LLM" the authors actually mean "small LLM": for example, "while a model like GPT-4 might have trillions of parameters,
a weak LLM might only have hundreds of millions or even fewer" (lines 44-45). The experiments focus on size too by employing OPT models, which are small compared to today's standards. However, "small LLM" for feedback is not necessarily "weak": if you take a small model and train it on large quantities of quality human preference data, it might become a moderate reward model. It would be great to disentangle "weak" and "small" in the narrative and also in the experiments.

- Section 2 is a bit trivial and takes up too much space. These math primers on DPO/PPO were not particularly helpful in later sections too. It would be great to assume readers of an "alignment" paper are already familiar with the basics, move them to the appendix, and use the space for analysis/results.

- I see that there is some investigation into the quality of preferences by the "weak" OPT model. Maybe something like RewardBench might offer a more rigorous quantitive evaluation.

- I would argue that things like self-rewarding language models [1] did things similar to "feedback from a weak model", albeit not as weak as OPT. Starting from a "weak-ish" models that are not good in both generating and evaluating, they show it is possible to jointly improve both. I wonder if the uniqueness of this paper's contribution might be weakened a bit by works like this.

- Ultimately, some might say that this work is a fine analysis paper on a niche technical design choice and does not offer something greater: taking RLAIF pipelines, changing the "AI" model part to a smaller model, and run some evaluation. Please highlight additional contribution if any.

[1] Yuan, Weizhe, et al. "Self-Rewarding Language Models." ICML 2024.
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper investigates the use of a weak supervised LLM to annotate preference data, as a middle ground between labor-intensive, costly human annotations and computationally expensive annotation based on high-capacity LLMs such as GPT-4. More specifically, the weak LLM (with approx. 100M parameters) is fine-tuned with DPO on a set of human labeled preference data and then its associated implicit reward model is applied to an unlabeled set to predict preferences. The resulting completed preference dataset is ultimately used to align a student LLM, whose performance enables measuring the quality of the annotations provided by the weak LLM.

The proposed weak LLM-based annotator is evaluated through multiple experiments where it is compared against both human annotators and stronger LLM annotators. It also includes experiments with weak LLMs and students models from different model families. Overall, the results show that using weak LLM-based annotations lead to similar or better performance compared to using human annotations or prompted high-capacity LLMs.
Strengths: - The paper is clear, polished and reads very well.
- The experiments are comprehensive and convincing.
- The proposed idea of using a weak supervised LLM as annotator is likely to have a significant impact on future LLM alignment practices.
Weaknesses: - The choice of using DPO fine-tuning to obtain a weak LLM-based annotator is not fully intuitive and lacks motivations, as only a reward model is needed for the preference annotation -- a model with generation capabilities is unnecessary for this (more details on this point in the "Questions" section).
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4
reviews: 
Comments: This paper introduces an alignment approach using weak large language models (LLMs) that balances cost and efficiency. The authors find that weak LLMs can generate feedback comparable to or even better than human feedback and stronger LLMs and propose the method to employ a semi-supervised framework where a weak LLM, trained on a small labeled dataset, provides preference feedback for a larger unlabeled dataset, reducing reliance on human annotation.
Strengths: 1. This paper proposes a cost-effective alignment framework that reduces computational costs and reliance on extensive human feedback,  using weak LLMs that have significantly fewer parameters than high-capacity models.

2. The study demonstrates that weak LLMs can perform as well as, and in some cases better than, human annotators in generating preference feedback for alignment through extensive experiments. For example, by demonstrating similar alignment success across different model families and sizes, the study provides a strong case for the generalizability of the weak LLM approach, suggesting that high alignment quality can be achieved without the latest or most powerful models.
Weaknesses: 1. The paper lacks a thorough analysis of feedback consistency over repeated evaluations. For example, when weak LLM and human preferences conflict, the study does not adequately explore the stability of weak LLM feedback. A quantitative analysis of consistency across multiple runs or various weak models would be helpful to assess whether the weak LLM’s feedback is repeatable and reliable. 

2. The evaluation metrics are only based on 2 models, the gold reward model and GPT-4. If the models are biased, the evaluation given by them would be unreliable and it could lead to skewed assessments that do not accurately reflect the quality of weak LLM alignment.
Recommendation: 6
Reviewer confidence: 3
Soundness/correctness: 4
Clarity: 3