SAIL: Self-improving Efficient Online Alignment of Large Language Models

Authors: Mucong Ding, Souradip Chakraborty, Vibhu Agrawal, Zora Che, Chenghao Deng, Alec Koppel, Mengdi Wang, Dinesh Manocha, Amrit Singh Bedi, Furong Huang
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: RLHF, Alignment, Online Alignment, Self-Play

Abstract:
Reinforcement Learning from Human Feedback (RLHF) is a critical method for aligning large language models (LLMs) with human preferences. However, existing offline alignment approaches, such as DPO, IPO, and SLiC, rely heavily on static datasets of human preferences, often leading to suboptimal performance. Recent efforts in the literature have moved towards online RLHF methods, but they lack a unified framework and suffer from distribution shift issues. In this work, we formalize online LLM alignment as a bilevel optimization problem. By reducing this formulation to a more computationally efficient single-level first-order method, utilizing reward-policy equivalence, we propose SAIL (Self-improving Efficient Online Alignment).SAIL generates new samples and iteratively refines model alignment through online exploration and regulation of preference labels. This enables continuous, self-improving alignment and generalizes prior online RLHF methods as special cases. Compared to state-of-the-art RLHF methods, SAIL delivers significant performance gains, with up to 11.6\% improvement in win rate and a 3.6-point increase in evaluation rewards, while maintaining low computational overhead.

Decision: reject
Reviewer scores: 6, 6, 8, 3

Reviews and discussion:
<thread>
creation date:
1729772623071

thread_type:
official_review

<thread_object>
creator: reviewer_7i95

<content>
summary:
The paper addresses the limitations of traditional reinforcement learning from human feedback (RLHF) methods for aligning large language models (LLMs) with human preferences. The authors propose a unified framework for online RLHF formulated as a bilevel optimization problem, which they simplify to a single-level method for efficiency. This approach, called SAIL, allows for continuous model improvement through online exploration and iterative refinement of preference labels, mitigating issues related to distribution shifts and reducing reliance on static preference oracles. Experimental results demonstrate significant performance gains, with SAIL outperforming state-of-the-art RLHF methods.

strengths:
(1) The paper introduces a novel unified framework for online RLHF that effectively addresses the challenges of static datasets and distribution shifts.
(2) By reducing a bilevel optimization problem to a single-level method, SAIL maintains theoretical benefits while significantly lowering computational costs, making it more practical for real-world applications.
(3) The self-improving aspect of SAIL allows models to iteratively enhance alignment without extensive supervision, addressing the challenge of needing constant access to human preference data.
(4) Extensive experiments validate the effectiveness of SAIL, showing substantial improvements in performance metrics compared to existing methods, thus showcasing its applicability across various datasets.

I would consider rescoring if the authors can solve my concern.

weaknesses:
(1) The method does not improve much in the AlpacaEval 2.0 Score. The author should give a detailed explanation. And why not use metrics like length-controlled win rate?
(2) Authors should compare more advanced preference optimization algorithms like ORPO and SimPO. And current results are not impressive for the alignment community.
(3) Why did the author just include MMLU as the downstream task metric? They should incorporate more tasks (eg., arc-challenge) like the similar self-improvement work SPIN (ICML24) to better illustrate their contribution.
(4) In the alignment area, it's better to conduct experiments in the Arena-Hard benchmark since it's a common metric to evaluate the alignment ability.

questions:
See the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730490911716

thread_type:
official_review

<thread_object>
creator: reviewer_zous

<content>
summary:
The paper introduces SAIL (Self-improving Efficient Online Alignment), an approach for online reinforcement learning from human feedback (RLHF) that aims to align large language models (LLMs) with human preferences. SAIL addresses limitations in offline RLHF methods by framing online LLM alignment as a bilevel optimization problem, which it reduces to a single-level first-order optimization method to enhance computational efficiency. The approach allows for continuous model improvement by generating samples iteratively, regulating preferences, and exploring online feedback. SAIL's self-improvement mechanism enables it to reduce reliance on preference oracles, thus allowing for more scalable alignment. Empirical evaluations demonstrate significant performance improvements over standard RLHF baselines.

strengths:
1. **Innovative Formulation**: The paper provides a novel formulation of online RLHF through bilevel optimization, enhancing computational efficiency by reducing this problem to a single-level optimization, which is a significant advancement for practical LLM training.
2. **Effective Self-improvement Mechanism**: SAIL effectively addresses challenges related to reliance on preference oracles, making online alignment more feasible by leveraging the model's self-generated responses for iterative improvement.
3. **Comprehensive Evaluation**: The paper includes extensive experiments that demonstrate substantial improvements in evaluation reward, win rate, and efficiency over other methods like DPO, supporting SAIL's efficacy and computational advantage.
4. **Scalability and Adaptability**: SAIL’s approach to handling distribution shifts and reducing oracle reliance presents a promising method for more scalable RLHF applications, especially for emerging large-scale LLMs.
5. **Detailed Experiment Design and Baselines**: The experiment section is well-structured, covering a range of metrics (reward-margin, eval-reward, win rate) and configurations (SAIL-PR, SAIL-PP, SAIL-DP), providing insights into the trade-offs and performance across different setups.

weaknesses:
1. **Limited Exploration of Alternative Utility Functions**: The method relies on the Bradley-Terry preference model, which may not be optimal for all RLHF applications. Future work could benefit from exploring alternative utility models that account for more nuanced preference data.
2. **Scalability Concerns for Larger Models**: Although the paper demonstrates SAIL’s effectiveness on LLMs with up to 8B parameters, additional scaling experiments would strengthen the paper's claims about computational efficiency for significantly larger models.
3. **Dependency on Initial Offline Dataset**: While SAIL reduces oracle dependency, it still relies on an initial offline dataset to bootstrap alignment. Further discussion on managing this dependency, especially when starting with limited labeled data, could be beneficial.
4. **Potential Overfitting in SAIL-DP**: The paper mentions that SAIL-DP shows signs of overfitting on in-distribution responses, suggesting that the method may benefit from more refined regularization techniques to ensure robust generalization to out-of-distribution samples.

questions:
1. The paper demonstrates SAIL's efficiency with models up to 8B parameters. Could you share any considerations or expected challenges for scaling SAIL to significantly larger models, such as those with over 100B parameters?

2. SAIL currently relies on the Bradley-Terry preference model. Have you considered experimenting with other preference models, and do you anticipate any impact on alignment performance if different utility functions are used?

3. SAIL-DP seems to show some overfitting on in-distribution responses. Could you discuss any regularization techniques you considered or plans to mitigate this, particularly to enhance generalization to out-of-distribution data?

4. Given the dependence on an initial offline dataset, how does SAIL perform in situations with minimal or noisy initial data? Are there strategies within the current framework to mitigate issues arising from a limited initial dataset?

5. Could you provide more detail on the computational costs of SAIL, particularly in comparison with other RLHF approaches? How does the single-level optimization approach compare in terms of resource requirements, and what practical considerations should be kept in mind when implementing it?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730646081261

thread_type:
official_review

<thread_object>
creator: reviewer_urgr

<content>
summary:
The authors identify three significant challenges in online RLHF algorithms: Challenge 1: the interdependence between models and data in implicit reward learning; Challenge 2: the computational complexity of bi-level optimization; and Challenge 3: the reliance on preference oracles. They propose SAIL to address these challenges. 

The main contributions of the paper can be summarized as follows:

1. **Unified LLM Alignment Mathematical Framework**: The authors have designed a principled online RLHF framework that provides concrete guidance for generating new responses, assuming the existence of a preference oracle.

2. **Adaptive Direct Preference Optimization**: By introducing a DPO-style analysis, the authors present an efficient single-layer solution capable of effectively addressing distribution shifts and providing a scalable online preference optimization method.

3. **Introduction of a Self-Improvement Mechanism**: This mechanism reduces the reliance on preference oracles.

4. **Extensive Experimental Evaluation**: The experiments conducted demonstrate that SAIL significantly outperforms baseline methods.

strengths:
1. Introducing Bi-level Preference Optimization: The process of bi-level preference optimization is integrated into the modeling of online RLHF. By leveraging the unique correspondence between the reward function and the LLM policy, this approach innovatively transforms the process into an equivalent single-layer form that is easier to solve.

2. Extensive Experiments on SAIL: Comprehensive and rich experiments were conducted to address the three significant challenges in online RLHF and to demonstrate the relevant applications of SAIL.

weaknesses:
Regarding the three variants of the SAIL method, Table 3 shows that in the Eval-Reward and MT-bench columns, the SAIL method performs worse than the baseline DPO. Please clarify whether these experimental results undermine the assertion that the SAIL method is superior to the baseline DPO.

questions:
There is a large amount of blank space below Section 6.1. Is there any missing content in this part of the paper?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696090534

thread_type:
official_review

<thread_object>
creator: reviewer_rdtx

<content>
summary:
Compared to offline RLHF methods, online RLHF methods empirically show stronger performance, yet is computationally expensive, vulnerable to distribution shifts and lacks a unified framework. The authors ablate different online RLHF methods based on all possible combinations (namely, SAIL-PR, SAIL-PP, SAIL-DP) which could be useful for future work exploring online RLHF methods. Personally, it was surprising that SAIL-PP generally works on par or slightly better than SAIL-PR, which open up further research questions on what would be the optimal way to obtain preference dataset.

strengths:
* The authors test of two LLM-as-a-Judge benchmarks as well as on a well-established classification benchmark, and their results are consistent.
* The authors provide a theoretical explanation of why their method works effectively.
* Showing all possible combinations at Figure 2 helped understanding what kind of online RLHF methods one should consider
* The results are consistent across smaller models (0.5B) up to widely used scale models (8B).

weaknesses:
* As a practitioner, at least the presentation/writing wasn't clear enough to agree that SAIL provides a unified framework for those who might want to consider using online RLHF in future works. I would personally suggest adding a section explains about how one could use SAIL instead of iterative DPO methods, as well as a huge emphasis on how the provided code could be used.
* There is a huge emphasis on trying to improve reward models (on RewardBench) to mitigated reward model overoptimization & train better LMs. I am curious if given a fixed budget/time limit, whether one should try to employ online RLHF methods or try to enhance reward models in general.
* I would suggest adding an explanation of what is the limitation of online RLHF methods that the paper could not address. For example, it is still unclear on what is the best practice to "whether to discard instances from a preference dataset that have a subtle difference on the preference strength" or "would it be beneficial to employ more models when gathering responses when consisting a preference dataset".

questions:
* Reward margin and offline-reward evaluation is interesting by itself and could provide information of the effectiveness of the method, but I personally think is not as an important measurement as pairwise winrate. Could you elaborate on Section 6.1 why one should consider looking into it?

* Please check the questions in weaknesses as well!
</content>
</thread_object>

</thread>
<thread>
creation date:
1731715465328

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Status Update on Additional Experiments

comment:
Thank you for your detailed feedback and suggestions for additional experiments. We have carefully reviewed all comments and experimental requests, and are actively conducting the requested evaluations. We will provide a comprehensive response with results soon. We greatly appreciate your constructive feedback and patience as we work to strengthen our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces SAIL, a self-improving online RLHF approach for aligning large language models (LLMs). SAIL frames online alignment as a bilevel optimization problem, reducing it to a computationally efficient single-level method. The framework enables continuous improvement by iteratively generating samples, updating preference labels, and leveraging implicit reward functions. SAIL demonstrates performance gains on benchmarks like MT-Bench and RewardBench, reducing reliance on human preference oracles.

Key weaknesses include limited comparisons with recent methods (e.g., ORPO, SimPO), insufficient evaluation on diverse tasks, and unclear scalability and computational efficiency. The experimental setup lacks depth, with limited downstream tasks and inconsistent metrics. While the theoretical framing is novel, the practical contributions appear incremental.