Title: KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexity
Venue: NeurIPS 2025
Accepted: yes
Keywords: Alignment

multiple reference model

RLHF

Sample complexity
Abstract: Recent methods for aligning large language models (LLMs) with human feedback predominantly rely on a single reference model, which limits diversity, model overfitting, and underutilizes the wide range of available pre-trained models. Incorporating multiple reference models has the potential to address these limitations by broadening perspectives, reducing bias, and leveraging the strengths of diverse open-source LLMs. However, integrating multiple reference models into reinforcement learning with human feedback (RLHF) frameworks poses significant theoretical challenges, where achieving exact solutions has remained an open problem. This paper presents the first \emph{exact solution} to the multiple reference model problem in reverse KL-regularized RLHF. We introduce a comprehensive theoretical framework that includes rigorous statistical analysis and provides sample complexity guarantees. Additionally, we extend our analysis to forward KL-regularized RLHF, offering new insights into sample complexity requirements in multiple reference scenarios. Our contributions lay the foundation for more advanced and adaptable LLM alignment techniques, enabling the effective use of multiple reference models. This work paves the way for developing alignment frameworks that are both theoretically sound and better suited to the challenges of modern AI ecosystems.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies RLHF with multiple reference models. Traditionally, RLHF is trained with a single reference model. But that is over restrictive because the model can inherits biases of that single pre-trained source. On the other hand, it is also inefficient because there are a lot of pre-trained models available. So far, there is a gap in understanding this new training paradigm because unlike RLHF with single reference model, there’s no exact solution yet. This paper pushes in this direction and provide solutions for RLHF with multiple reference models for both reserver and forward KL constraint.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengthes\n    - Expect for the experimental section, I think the paper is written very clearly with solid motivation and the theory sections are easy to understand.\n    - The research direction on RLHF with multiple reference models is an interesting direction and this paper provides some theoretical justifications in that direction.\n- Weaknesses\n    - The experimental section can be expanded a bit. I am confused about what algorithm is introduced based on the theoretical results.\n\nQUESTIONS:\n- Line 113-114: It’s also possible that two outputs are sampled from a distribution different from the reference policy? An example of which would be the Antropic HH dataset which is collected offline. And how are outputs sampled with multiple reference polices?\n- How realistic is assumption 4.2? Typically reward function are trained based on existing LLMs. For example, one could train the last linear layer of a pre-trained LLM. In that case, the VC dimension is finite but the reward function class is not finite.\n- I am always a bit confused about the practicality of coverage assumption. It seems like because outputs are sampled from reference policy, I don’t need lower bound of all \\pi_ref(y|x) but just the ones that can be realistically sampled? Although I do acknowledge the technical difficulties involved.\n- If the assumptions in 4.2 is not needed for 5.1, maybe move the placement of assumptions?\n- In the proof of theorem 5.1, you mentioned that “Clearly, the right side of (29) is maximized when the KL divergence is set to zero.” Is it possible that there exists a policy with non-zero KL divergence but high exponential value? (I'll update my score based on answer to this question since I might have missed something obvious)\n- Does theorem 5.1 only works with optimal reward?\n- I am confused about what the actual algorithm is? How does it rely on the exact solution of RLHF with multiple reference models? The experiment section is written with a lot of details missing.\n\nLIMITATIONS:\n- I have questions about the limitations of assumptions which are discussed in the question section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo address the significant theoretical challenges of integrating multiple reference models into reinforcement learning with human feedback (RLHF) frameworks, this paper provides the exact solution to the multiple reference model problem under reverse KL-regularized RLHF. A comprehensive theoretical framework is proposed, incorporating rigorous statistical analysis and offering theoretical guarantees on sample complexity. The analysis is further extended to forward KL-regularized RLHF, presenting new insights into sample complexity requirements in multiple reference scenarios. The main contributions are as follows:\n\n1. A comprehensive mathematical framework is proposed for reverse KL-regularized RLHF with multiple reference models, providing the exact solution to this problem and calculating the maximum objective value.\n\n2. Theoretical guarantees are established for the proposed multiple reference model setting under reverse KL-regularization.\n\n3. The multiple reference model scenario is also studied under forward KL-regularized RLHF, and its sample complexity is analyzed.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The theoretical analysis is novel, particularly in the authors' innovative and comprehensive investigation of the sub-optimal gap and optimal gap for both forward KL and backward KL regularizations, effectively filling a gap in the existing theoretical literature.\n2. The theoretical results are well supported by experimental validation. The authors designed dual-scenario experiments to verify that their closed-form solution outperforms existing approximate methods and single-reference models.\n\nWeaknesses:\n\n1. The theoretical results are only validated on two relatively small models and on specific tasks, without involving larger-scale models. This limitation may affect the generality and rigour of the conclusions.\n2. Although the theoretical proofs are clear and rigorous, the paper lacks intuitive explanations of the theoretical results. It would be valuable if the authors could provide more intuitive interpretations or insights into their theoretical findings.\n\nQUESTIONS:\nFor detailed comments, please refer to the Weakness section. If the authors could clarify the issues raised therein, I would be happy to consider raising the score.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper derives closed-form solutions and sample-complexity bounds for reverse and forward KL–regularized RLHF with multiple reference policies. The optimal policy is a normalized geometric mean (reverse KL) or weighted arithmetic mean (forward KL) of references, exponentiated by scaled rewards. Additionally, the paper adds GSM8K  and UltraFeedback as experiments.

STRENGTHS AND WEAKNESSES:
---
* Quality:  
    * Strengths: Provides closed-form multi-reference policy (Eq. (14)) with tight $O(1/n)$ and $O(1/\sqrt n)$ bounds under relaxed coverage.  
    * Weaknesses: Relies on bounded rewards and finite reference class; coverage constants $C_{\mathrm{rkl}},C_{\mathrm{fkl}}$ are never instantiated.
---

* Clarity:  
    * Strengths: Reverse and forward KL derivations are presented in parallel, and the core policy formula is highlighted.  
    * Weaknesses: Heavy notation (escort distributions, functional derivatives) and no running example or notation cheat sheet hamper readability.

---

* Significance:  
    * Strengths: Replaces global coverage with a local KL-ball condition, still achieving the best known rates; unifies reverse/forward KL analysis.  
    * Weaknesses: Practical impact on large LLMs is unclear—combining $K$ models per update may not scale and experiments are limited to small models.

--- 
* Originality:  
    * Strengths: First to derive exact multi-reference RLHF solutions without surrogate bounds, highlighting geometric vs. arithmetic blends.  
    * Weaknesses: Omits discussion of links to errors-in-variables or robust regression literatures that also perturb feature distributions.

QUESTIONS:
* Can one use approximation for reducing cost in the given analysis framework?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical analysis of KL-regularized RLHF using multiple reference models. The authors derive the first exact closed-form solutions for both reverse KL (RKL) and forward KL (FKL) regularized objectives in this multi-reference setting. They further provide sample complexity guarantees for the optimality and sub-optimality gaps for both frameworks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n*   **Novel Theoretical Contribution:** The core strength is the derivation of an exact closed-form solution for the multi-reference RKL-RLHF problem (Thm 5.1). This is a significant step beyond prior work (Le et al., 2024) which relied on a lower-bound approximation. The resulting optimal policy as a normalized geometric mean of reference policies (Eq. 15) is an elegant result.\n*   **Comprehensive Analysis:** The anlysis is thorough, covering both RKL and FKL regularization. For each, the paper derives optimality and sub-optimality gap bounds (Thms 5.2, 5.3, 6.2, 6.3), providing a complete theoretical picture. The comparison in Table 1 clearly situates the contributions within existing literature.\n*   **Strong Motivation:** The paper makes a compelling case for using multiple reference models to improve diversity, reduce bias, and leverage the growing ecosystem of open-source LLMs. Teh motivation is well-grounded and timely.\n\n**Weaknesses:**\n*   **Limited Empirical Validation:** This is the main weakness. While the experiments in Sec 8 support the theory, they are limited in scope. The use of relatively small models (0.5B) and only two datasets (GSM8K, UltraFeedback) makes it hard to gauge the practical significance for SOTA LLM alignment. The performance gains shown in Fig 1 over the single-reference baseline are modest.\n*   **Practicality of Assumptions and Implementation:** The paper is heavily theoretical. The practical implications of the Local KL-ball Coverage assumption (4.4, 4.5) are not fully explored. For instance, how does the constant $C_\\{ε_\\{rkl}}$ scale with the number of reference models? Also, the computation of the normalization term $F_\\alpha(x)$ in Eq. 15 seems computationally prohibitive (sum over all $y$ in $Y$); more discussion on how this is handled in the DPO-style experiment would be valuable in the main text.\n*   **Significance of FKL Results:** While the FKL analysis is complete, the experimental results (Fig 1a, 1b) show it consistently underperforming the RKL approach. This raises questions about teh practical utility of the multi-reference FKL methd, even if the theory is sound.\n\nQUESTIONS:\n1. The normalization constant $F_\\alpha(x)$ in Eq. 15 requires a sum over the entire vocabulary, which is often intractable. How was this handled in the offline DPO experiment? Does the corresponding DPO loss function for multiple references (presumably in App. F) circumvent this explicit calculation?\n2. The coverage constants $C_\\{ε_\\{rkl}}$ are central to the bounds but their dependency on the number (K) and diversity of reference models is unclear. Cud you provide some intuition on how these constants might scale with K? Does the bound become vacuous if the reference models are very different?\n3. The empirical gains from using multiple references ($\\alpha>0$) are positive but appear somewhat marginal in Fig 1. In what practical scenarios do you hypothesize the multi-reference approach would provide a much more substantial improvement over a strong single-reference baseline?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}