SeRA: Self-Reviewing and Alignment of LLMs using Implicit Reward Margins

Authors: Jongwoo Ko, Saket Dingliwal, Bhavana Ganesh, Sailik Sengupta, Sravan Babu Bodapati, Aram Galstyan
Primary area: foundation or frontier models, including LLMs
Keywords: Preference Alignment, Large Language Models, Implicit Reward Margin, Sample Selection, Preference Bootstrapping

Abstract:
Direct alignment algorithms (DAAs), such as direct preference optimization (DPO), have become popular alternatives to Reinforcement Learning from Human Feedback (RLHF) due to their simplicity, efficiency, and stability. However, the preferences used by DAAs are usually collected before alignment training begins and remain unchanged (off-policy). This design leads to two problems where the policy model (1) picks up on spurious correlations in the dataset (as opposed to only learning alignment to human preferences), and (2) overfits to feedback on off-policy trajectories that have less likelihood of being generated by the updated policy model. To address these issues, we introduce Self-Reviewing and Alignment (SeRA), a cost-efficient and effective method that can be readily combined with existing DAAs. SeRA comprises of two components: (1) sample selection using implicit reward margin to alleviate over-optimization on such undesired features, and (2) preference bootstrapping using implicit rewards to augment preference data with updated policy models in a cost-efficient manner. Extensive experiments, including on instruction-following tasks, demonstrate the effectiveness and generality of SeRA in training LLMs with diverse offline preference datasets and and DAAs.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730268733173

thread_type:
official_review

<thread_object>
creator: reviewer_aqa6

<content>
summary:
Direct alignment algorithms (DAAs) like direct preference optimization (DPO) face challenges with static off-policy data, leading to spurious correlations and overfitting. In this paper, the authors propose a novel method called SeRA that addresses the mentioned issue by using sample selection and preference bootstrapping to improve alignment. The authors use extensive experiments to show SeRA enhances DAA performance on the Ultrafeedback dataset.

strengths:
This paper provides a clear motivation for the authors' proposed method from both theoretical and empirical perspectives. The experimental results show that their proposed method has a significant improvement on the alignment task given the Ultrafeedback dataset and the authors perform extensive empirical analysis to explain the mechanism behind the proposed method.

weaknesses:
1. The paper lacks more detailed discussions on the literature analyzing the overoptimization issue for LLM alignment and RLHF with self-generated data. For example, the authors could discuss the literature [1]-[5] in this paper.


2. Compared with other papers that study LLM alignments (e.g. [5]. [6]), this paper lacks empirical results on the academic benchmarks in coding, reasoning, and math. The authors should evaluate their proposed methods on these academic benchmarks.

3. It is a little confusing that the authors leave Remark 1 in Line 254 without an in-depth explanation of the derived bound. The derived bound seems to be adapted from the common analysis of empirical risk minimization. If there are novel analytical techniques for the current problem (analyze the performance of the proposed method), the authors should highlight the novel part and show that the vanilla DPO fails to have such a bound. 

4. Minor issues or suggestions:
(1) In the comment on Line 14 of Algorithm 1, you should better mention the 'implicit reward' instead of 'reward'.
(2) More descriptions of benchmarks should be given. (even in the appendix)


[1] Coste, Thomas, et al. "Reward Model Ensembles Help Mitigate Overoptimization." The Twelfth International Conference on Learning Representations.

[2] Liu, Zhihan, et al. "Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer." arXiv preprint arXiv:2405.16436 (2024).

[3] Eisenstein, Jacob, et al. "Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking." arXiv preprint arXiv:2312.09244 (2023).

[4] Liu, Aiwei, et al. "Direct large language model alignment through self-rewarding contrastive prompt distillation." arXiv preprint arXiv:2402.11907 (2024).

[5] Wu, Yue, et al. "Self-play preference optimization for language model alignment." arXiv preprint arXiv:2405.00675 (2024).

[6] Chen, Zixiang, et al. "Self-play fine-tuning converts weak language models to strong language models." arXiv preprint arXiv:2401.01335 (2024).

questions:
1. In Line 246, the authors claim that the inherent problem of DPO is that DPO treats all preference samples equally, even when the true preferences are not correct. In the BT preference model, the annotation is stochastic instead of deterministic, which means that this phenomenon should not be an issue from the theoretical perspective. Could the authors give a more detailed explanation?

2. The proposed method requires sampling multiple times for each prompt in the dataset. I wonder if the baseline methods and the proposed method have the same computation budget.

3. In this paper, the authors consider introducing a filtration on self-generated responses to get a preference pair for each prompt. However, I am curious about what would happen if the authors use all the self-generated responses (e.g. N responses) in pairs to create N(N-1)/2 preference pair. Such an idea is used in SPPO ([1]), hence a comparison between the proposed method in this paper and SPPO may be important.

4. Since the authors use the Ultrafeedback dataset for training, I am curious about the meaning of the 'Ultrafeedback' benchmark. Is this the test split of the Ultrafeedback? 


[1] Wu, Yue, et al. "Self-play preference optimization for language model alignment." arXiv preprint arXiv:2405.00675 (2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730576939357

thread_type:
official_review

<thread_object>
creator: reviewer_ppvp

<content>
summary:
The paper presents a new algorithm called Self-Reviewing and Alignment (SeRA) for direct aliment algorithms, which can be adapted to different preference optimization methods such as DPO, SLiC, and IPO. The two components of SeRA are sample selection and preference bootstrapping. Instead of using human feedback, AI judgment, or an external reward model to label the preference data, SeRA uses an implicit reward model for the sample selection and data bootstrapping. The paper provides a theoretical guarantee for DPO. Also, the paper presents extensive empirical studies across different datasets, model architectures, and alignment frameworks to show the effectiveness of SeRA.

strengths:
* The paper presents both theoretical and empirical validation, covering a variety of models, datasets, and SAA methods. Furthermore, the component analysis is thorough, evaluating most elements of the algorithm for a more nuanced understanding.
*  The IRM method enables training SAA iteratively without relying on external reward signals. This innovation simplifies the algorithm’s implementation and reduces the computational resources required.
* The paper is well-structured, making the concepts accessible and the methodology easy to follow.

weaknesses:
* The theoretical upper bound on the risk of the DPO is only weakly connected to the proposed SeRA. Additionally, the theoretical analysis does not account for the impact of multiple iterations on the algorithm's performance.

* The paper does not provide a thorough study into how the number of iterations affects SeRA's performance. While Figures 4 and 7 indicate that iter 3 consistently outperforms iter 2, it remains unclear whether further increasing the number of iterations would lead to additional performance improvements.

questions:
* Is there an estimation of the uniform covering number for various hypothesis classes, such as linear functions and neural networks?

* Can the theoretical results presented in the paper be extended to apply to other SAA methods?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701382903

thread_type:
official_review

<thread_object>
creator: reviewer_sfx5

<content>
summary:
This paper studied the drawbacks of direct alignment algorithms that overfit the dataset and optimize the model to focus on the spurious correlations. To address these limitations, the authors proposed Self-Reviewing and Alignment which selects preference samples with high implicit reward margins.

strengths:
1. The paper is well-written and easy to follow.
2. The experiments are extensive and well support the claims in the paper.

weaknesses:
1. Since the DPO implicit reward can serve as a good approximation of the preference score, e.g., some of the top reward models in RewardBench [1] are from DPO implicit rewards, relabeling the preference data with DPO reparameterized reward is a natural way, and very much like the standard LLM-as-judge and RM-as-judge methods used to rank the preference pairs. In fact, similar approaches have been explored in [2,3].
2. The authors may argue that their method uses the implicit reward gap as a data selection metric, instead of only using the implicit reward for relabeling. However, given that implicit rewards are a form of reward, it might be the case that the data selection process mitigates the issues of DPO such as unlearning the high-quality chosen responses when the reward gap is small [2,4], which makes the performance improve. If this is true, then we may use any reward models to filter out similar-quality pairs, or use all preference data with the reward-aware algorithms proposed in [2,4].
3. The proposed method is computationally expensive since the implicit rewards need to be calculated for each data point, while the iterative DPO does not. Besides, given that the base models (e.g. the SFT models) outperform GPT 3.5 by a large margin and UltraFeedback uses GPT 3.5 for preference ranking, it might be the case that the reason that SeRA outperforms DPO is that the implicit reward is more accurate, even not removing any data. Therefore, I would like to see how iterative DPO performs when using DPO implicit reward to rank the pairs.
4. Based on the above comments, the evidence that SeRA is better than DPO since it addresses the spurious correlations issue may be weak.
5. It is also unclear what Theorem 1 indicates and how it relates to the filtering process of SeRA.

[1] Lambert et al., RewardBench: Evaluating Reward Models for Language Modeling.\
[2] Zhang et al., Reward-Augmented Data Enhances Direct Preference Alignment of LLMs.\
[3] Zhong et al., DPO Meets PPO: Reinforced Token Optimization for RLHF.\
[4] Nvidia, Nemotron-4 340B Technical Report.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702960450

thread_type:
official_review

<thread_object>
creator: reviewer_feme

<content>
summary:
This work provide a method SeRA for Direct Alignment Algorithms (DAA). This method mainly uses the DAA policy as a reward model and provide implicit reward margins (IRM) for pairs. This IRM can be used for data selection and can also score the on-policy data to get more data. Many experiments are done to support the performance of this method.

strengths:
1. This work is well-written and easy to read.

2. The proposed method is simple and don't need further human or AI annotation.

3. This method is suitable for different DAA methods, which makes it a quite general method.

4. The experimental results show the effectiveness.

weaknesses:
1. As far as I can understand, I think this method mainly uses the DAA model as a reward model to generate on-policy signals. Therefore, I am quite curious about the comparison between SeRA and a direct DAA which uses on-policy preference data. For this direct DAA, the on-policy data might come from GPT-4 or a reward model trained from previous data. 

2. I hope more intuition about that SeRA can mitigate over-optimization can be provided. Fig 2 shows that SeRA selected data have less dependency on response length but I am curious why this dependency can be removed.

3. I wonder if there are some other designs for r_t in Equ.(7), Sec. 3.4?

questions:
See weakness part above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732083534297

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response by Authors

comment:
We sincerely thank all the reviewers (**@AqA6, @pPvp, @SFx5, @FEME**) for their careful review of our paper and for providing insightful and constructive feedback. We are pleased that the reviewers recognized the clear motivation behind our proposed method from both theoretical and empirical perspectives (**@AqA6, @pPvp**). We are glad to hear that (1) the reviewers found our paper well-written, well-structured, and easy to follow (**@pPvp, @SFx5, @FEME**) and (2) that the experimental results demonstrate significant improvements and effectively support the claims in our paper (**@AqA6, @pPvp, @SFx5, @FEME**). We have endeavored to address the concerns raised by the reviewers in our individual responses and have revised our manuscript accordingly based on their comments.

The following revisions have been made:

- Much clearer description of Remark 1 for @AqA6, @pPvp, @SFx5, @FEME (Section 3.2 & Appendix B)
- Additional literature review for @AqA6 (Appendix A)
- Clearer description of Algorithm 1 and benchmarking details for @AqA6 (Appendix C.1)
- Extension of Figure 2 across various selection ratios to further support our claims (Appendix D.4)
- Performance analysis for subcategories of the Evol-Instruct benchmark for @AqA6 (Appendix D.5)
- Discussion on the computational budget of SeRA for @SFx5 (Appendix D.6)
- Experimental results for additional iterations for @pPvp (Appendix D.7)

The revisions are marked in $\color{magenta}{\text{magenta}}$ in the updated paper. Additionally, we are happy to engage in further discussions during the rebuttal phase.

Thank you for your time and thoughtful feedback.

Sincerely,

The Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732383758870

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Reminder: Response Review for All Reviewers

comment:
Dear Reviewers,

Thank you for your thoughtful comments on our work. We have carefully addressed your initial feedback and provided corresponding responses, along with an updated manuscript.

We wanted to kindly follow up to see if there are any additional thoughts or clarifications you would like us to consider. Please let us know if there’s anything further we can do to clarify any points or address any concerns.

Thank you again for your time and efforts!

Sincerely,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposed a framework that iterates between selecting samples based on the margin of the implicit reward model and training the model using the selected preference data. The paper claims that this approach addresses the over-optimization issue from offline DPO. The paper empirically demonstrates that their approach indeed outperforms baselines under multiple models and datasets.