Kati8/ICLR_2023_2025 huggingface dataset:
authors: Aviral Kumar
authors: Vincent Zhuang
authors: Rishabh Agarwal
authors: Yi Su
authors: Avi Singh
authors: Kate Baumli
authors: Shariq Iqbal
authors: Colton Bishop
authors: Rebecca Roelofs
authors: Lei M Zhang
authors: Kay McKinney
authors: Disha Shrivastava
authors: Cosmin Paduraru
authors: George Tucker
authors: Doina Precup
authors: Feryal Behbahani
authors: Aleksandra Faust
reviews: 
Comments: This paper introduces ScoRe, a multi-turn online reinforcement learning (RL) approach aimed at enabling self-correction in model responses. Through this method, the model learns to identify and correct its own responses, improving overall performance. The authors first analyze existing methods, identifying two main factors that currently limit self-correction capabilities. Based on this analysis, the proposed method employs a two-stage online RL training process with specific optimization objectives to reduce distribution shift and behavior collapse. As a result, the method outperforms other baselines.
Strengths: 1. The self-correction setting in this paper is more realistic, increasing its applicability to real-world scenarios.
2. The paper provides an insightful analysis of prior self-correction methods, identifying two key factors that limit effectiveness: distribution shift and behavior collapse. This analysis offers valuable insights that can inspire future research.
3. ScoRe demonstrates superior performance compared to baseline methods. Additionally, all experiments were conducted on Gemini, a robust baseline model, which further validates the approach.
Weaknesses: 1. In Stage 2, the authors use reward shaping to prevent the model from collapsing to a non-self-correcting solution. The chosen hyperparameter $\alpha$ , $\alpha >1$, could potentially result in the overall sign of $r(y1,y*)$ being negative, which may lead the model to introduce minor errors in the first step to leave room for self-correction in subsequent steps.
2. I believe the primary purpose of using self-correction is to achieve better performance. To support this goal, more baseline comparisons should be introduced. For instance, self-improvement methods like REST-EM, which involve only a single round of generation without self-correction during inference, rely solely on the model itself and should be comparable.
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4
reviews: 
Comments: The authors present a technique, ScoRe, for teaching LLMs to self-correct. The technique uses two RL training phases: in the first, the model is trained to give correct second-turn answers without giving different first-turn answers than the base model; in the second, the model is trained on a reward signal that incentivizes correct answers in both turns as well as improvement from the first turn to the second. SCoRe is motivated via a fine-grained analysis of the failures modes of prior SFT-based techniques for teaching self-correction.
Strengths: 1. The paper's analysis of the failure modes for prior SFT-based methods is very insightful, with the authors making use of edit-distance-based metric and an analysis of train-test differences to understand why prior methods fail to learn self-correction or fail to generalize out-of-distribution.
2. The results appear relatively strong, with SCoRe substantially outperforming prior methods on the evaluations presented. 
3. The presentation is overall relatively clear.
Weaknesses: 1. Certain choices in the technique don't appear to be "as simple as possible," and the text doesn't consistently do a good job of motivating these choices. (See questions.)
2. I would like to see these results compared to the very simple baseline of RL directly against the final answer, but with the self-correction prompt inserted after the first turn.
Recommendation: 8
Reviewer confidence: 2
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: The paper introduces SCoRe, a novel multi-turn RL method to enhance the self-correction ability of LLMs. The SCoRe improves LLMs' performance in correcting their mistakes without needing extra external supervision. Compared to supervised fine-tuning (SFT), which struggles with distribution shift and behavior collapse, SCoRe utilizes multi-turn RL with regularization strategies, achieving good accuracy gains on MATH and HumanEval benchmarks.
Strengths: 1.  The work identifies and studies two limitations of existing self-correction methods: distribution shift and behavior collapse.

2. The work proposes a novel and original multi-turn RL method.  The method's significance lies in its potential to address key limitations of existing approaches.

3. The quality of empirical analysis is good by showing improvements in self-correction metrics on established datasets and ablation studies on various components of the proposed method.

4. The work is presented with commendable clarity, including detailed explanations of the algorithm and experimental setup, making it accessible to readers.
Weaknesses: 1. No experiments are conducted with open-source models such as the Llama series.

2. The models are trained for only two attempts, leaving the scalability of the proposed method to additional attempts uncertain.
Recommendation: 8
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper presents SCoRe, a novel reinforcement learning (RL) approach to enhance self-correction in large language models (LLMs). Unlike previous methods that rely on multiple models or external feedback, SCoRe employs a multi-turn online RL mechanism using self-generated data. This two-stage process begins with training on a base model's correction traces to avoid behavior collapse, followed by multi-turn RL with reward shaping to promote effective self-correction. SCoRe is the first approach to attain positive self-correction result, surpassing traditional fine-tuning and prompting-based methods on math and code benchmarks.
Strengths: 1. First approach for making self-correction really work.
2. Very solid experiments and ablation studies along with in-depth analysis providing insights for achieving inference time scaling like OpenAI's o1 series.
Weaknesses: 1. This work conduct experiments on private Gemini series which is hard to reproduce, it would be beneficial to include experiments on open-source models (Llama 3).
2. This work only explore 3 datasets (HumanEval, MBPP and MATH) on code and math. It would be better to introduce more datasets of varying difficulty levels (e.g. AIME).
3. Also, it would be better to conduct experiments on a broader range of diverse subject (e.g. Physics, Chemistry).
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4