reviews: 
Comments: The paper explores the potential of using weak large language models (LLMs) for AI alignment. It proposes a new framework that leverages weak LLMs to provide feedback, thereby reducing reliance on costly human labor and computational resources. Experimental results show that even weak LLMs with as few as 125 million parameters can offer feedback comparable to human feedback. The paper also analyzes the quality of the feedback from weak LLMs and finds that it can surpass human judgment in some cases, although there is uncertainty in fine differentiation. The study offers a new perspective for AI alignment and demonstrates that weak LLM feedback is a promising, scalable and sustainable solution.
Strengths: 1.The paper introduces a framework that leverages weak LLMs for alignment, which offers a promising alternative to existing methods that rely of either extensive human labor or expensive computational resources.

2.The paper conducts comprehensive evaluation of weak LLM feedback across various model scales and families. The result consistently demonstrate the effectiveness of this approach, highlighting its potential for practical application.

3.The paper is well-organized, with clear sections and logical flow. The authors effectively communicate the research objectives, methodology, and findings.

4.The research has significant practical implications for the development and deployment of aligned LLMs. Utilizing weak LLM feedback can reduce costs and improve efficiency, making alignment more scalable and accessible.
Weaknesses: 1.In this paper， the authors propose a framework that utilizes weak LLMs for alignment, striving to strike a balance between RLHF and RLAIF, it appears that the weak-to-strong approach and methodology in the paper are still quite similar to Burns et al. (2024). Burns et al. (2024)’s study focus on reward modeling and binary classification, where the outputs in this paper are either scalar or categorical labels. Your study is closely tied to alignment and involves a more complex output space. However, it seems that the two are consistent in their methodological approach. Therefore, I think that there are still some limitations in terms of novelty. I hope you can explain more about the difference between the two works.

2.The paper points out that even advanced LLMs like GPT-4 may show feedback inconsistency when the quality difference between two responses is subtle. In the case where it is uncertain whether gold RM is stronger than GPT-4, when using gold RM for evaluation, is it also possible to make incorrect judgments about two responses with similar quality, leading to bias in the evaluation？Additionally，When using GPT-4 for evaluation, although the prompt already mentions and ensures that the order of the responses should not affect its judgment, in actual experiments, it is difficult to avoid the model's preference caused by the order. A fairer approach would be to swap the positions of the two responses and conduct the experiment again. I believe that conducting the experiment twice with different positions would result in a more accurate evaluation.

3.While the paper demonstrates the effectiveness of weak LLM feedback across alignment task, it would be beneficial to investigate its applicability to more complex tasks.
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 4