Title: Optimal Policy Minimum Bayesian Risk
Venue: NeurIPS 2025
Accepted: no
Keywords: Inference Scaling

Best-of-N

Minimum Bayesian Risk

Optimal Policy
Abstract: Inference scaling can help LLMs solve complex reasoning problems through extended runtime computation. On top of targeted supervision for long chain-of-thought (long-CoT) generation, purely inference-time techniques such as best-of-N (BoN) sampling, majority voting, or more generally, minimum Bayes risk decoding (MBRD), can further improve LLM accuracy by generating multiple candidate solutions and aggregating over them. These methods typically leverage additional signals in the form of reward models and risk/similarity functions that compare generated samples, e.g., exact match in some normalized space or standard similarity metrics such as Rouge. Here we present a novel method for incorporating reward and risk/similarity signals into MBRD. Based on the concept of optimal policy in KL-controlled reinforcement learning, our framework provides a simple and well-defined mechanism for leveraging such signals, offering several advantages over traditional inference-time methods: higher robustness, improved accuracy, and well-understood asymptotic behavior. In addition, it allows for the development of a sample-efficient variant of MBRD that can adjust the number of samples to generate according to the difficulty of the problem, without relying on majority vote counts. We empirically demonstrate the advantages of our approach on math (MATH-500) and coding (HumanEval) tasks using recent open-source models. We also present a comprehensive analysis of its accuracy-compute trade-offs.


Peer reviews:
comments: SUMMARY:
This paper proposes a Optimal Policy Mimimum Bayes Risk Decoding (MBRD) for LLM. MBRD aims to decode from a distribution maximize some utility, for example reward function. in expectation. For entropy-regularized RL, with relative entropy to some reference distribution. It's well-known that the optimal policy admits a closed-form. Due to the intractable sum over the string space $V^+$, it requires one to sample from an unnormalized energy model. This paper leverages techniques from previous to sample from unnormalized energy-based model to develop OP-MBRD and its rao-blackwellization version, for LLM decoding.

STRENGTHS AND WEAKNESSES:
## Strengthes

- Very clear and enjoyable writing

- Throughout discussions on related works

- Elegant solution with minimal implementation overhead

- Strong empirical performance on MATH and HumanEval

## Weaknesses

- Not a weakness, but I am not particularly familiar with the string of works mentioned in L171 (DPG, GDC++, RSO [Liu et al., 2024], BRAIn). So I am not sure if there is any similar works worthwhile noticing or comparing.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Optimal Policy Minimum Bayesian Risk Decoding (OP-MBRD), a decoding algorithm that generalizes standard minimum Bayes risk decoding (MBRD) using KL-regularized RL.The method defines an optimal policy distribution that interpolates between the model’s own generation distribution and a reward-weighted distribution. The authors derive an importance-sampled estimator for expected similarity between outputs under this optimal policy and show asymptotic consistency (Eq. 14).\n\nBuilding on this, the paper proposes a sample-efficient variant that adjusts the number of generated candidates based on an interpretable acceptance-weighted metric. Experiments on MATH-500 and HumanEval using open-source models (Qwen, Granite, Phi-4) show that OP-MBRD performs robustly across model and task settings, often matching or outperforming baselines such as best-of-N sampling and MBRD with reward scores. The method is particularly effective under lower-resource settings or for weaker models, is modular and can be readily integrated with existing decoders.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) Proposes a novel MBRD formulation based on KL-regularized optimal policy inference, integrating reward modeling and similarity-based decoding in one framework (Section 4.1, Eq. 6–11).\n\n2) Derives a self-normalized importance-sampling estimator with convergence guarantees, generalizing both majority voting and reward-weighted decoding (Eq. 11, Eq. 14).\n\n3) The efficient variant (OPE-MBRD) introduces a principled mechanism for adaptive sampling based on a soft estimate of example difficulty, improving compute efficiency when the reward model is well-calibrated (Figure 1 right).\n\n4) Experimental results show that OP-MBRD generally performs comparably or better than BoN and MBRD(EM*R), particularly for smaller models and in low-sample regimes (Figure 1, Figure 2).\n\nWeaknesses:\n1) The method’s performance is sensitive to PRM calibration. For example, using a math-trained PRM on coding tasks in HumanEval leads to poor adaptation by OPE-MBRD (Figure 2 right), reducing its reliability under domain mismatch.\n\n2) All experiments fix the reference model to be the generator, limiting the method’s ability to exploit stronger teacher models or mixture-of-agents style configurations. This undermines part of the theoretical generality of OP-MBRD.\n\n3) The paper does not compare against recently competitive inference-time techniques such as speculative decoding, stepwise verifier-guided reranking, or tree-of-thought search. This makes it difficult to assess how OP-MBRD compares against stronger baselines beyond BoN and standard MBRD.\n\n4) It seems like Section 4 could use some additional figures or an algorithmic pseudocode. The estimator derivation is difficult to parse without a strong RL background.\n\nQUESTIONS:\nHave you tried using pR ≠ p, e.g., by using a stronger teacher model or an ensemble as the reference? This would better demonstrate the flexibility of the OP-MBRD framework.\n\nIn cases where PRMs are misaligned (e.g., math PRM applied to code), could the model detect or correct for this dynamically? Would adaptive β tuning help?\n\n\nCan you provide any timing comparisons or latency overhead metrics for OP-MBRD vs BoN and MBRD(EM*R)? How does wall-clock cost scale?\n\n\nCould OP-MBRD be integrated with multi-step inference strategies (e.g., beam search or tree-of-thought reasoning)? Would the optimal policy still be applicable at intermediate steps?\n\n\nHave you explored the method’s robustness under noisy PRMs? Some form of regularization may be necessary for stable deployment in open-ended domains.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Optimal Policy Minimum Bayesian Risk Decoding (OP-MBRD), a novel decoding method for large language models that integrates minimum Bayes risk decoding with concepts from KL-controlled reinforcement learning. The proposed approach offers three primary contributions: (1) improved robustness and accuracy over standard MBRD and best-of-N sampling, especially under performance gaps between generators and reward models; (2) an asymptotic guarantee that OP-MBRP converges to MBRD over a reward-weighted distribution; (3) a sample-efficient variant (OPE-MBRD) that dynamically adjusts the number of generated samples based on problem difficulty using expected optimal policy samples. Experiments on MATH-500 and HumanEval benchmarks demonstrate consistent gains across diverse open-source models (Qwen, Granite, Phi-4), particularly for well-calibrated generator-reward model pairs. The method retains the simplicity of traditional MBRD while adding only one hyperparameter ($\\beta$).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- Clarity. The paper is well-written and logically structured. Technical concepts are clearly explained.\n- Theoretical contribution. The paper effectively bridges KL-regularized RL (optimal policy derivation) and MBRD, providing a principled reward-integration framework.\n- Significance. OPE-MBRD’s adaptive sampling reduces compute costs without accuracy loss, addressing key inference-scaling challenges.\n- Experiments. Thorough evaluation across various model sizes (1.5B–20B) and diverse tasks (MATH-500/HumanEval), demonstrating consistent gains.\n\n**Weaknesses**\n- Statistical robustness. The result in Fig 1/2 lacks variance quantification (e.g., error bars/confidence intervals), limiting reproducibility assessment.\n- Novelty. The proposed OP-MBRD primarily combines BoN and MBRD paradigms. The theoretical formalization is rigorous but it minimally extends beyond prior MBRD-with-rewards concepts.\n\nQUESTIONS:\n- Will you add error bars/standard deviations to results (e.g., Fig 1 pass@1 curves) to clarify performance stability?\n\n- Can OP-MBRD handle larger LLMs (e.g., 90B llama 3.2)? Would sample diversity effects differ at scale?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Optimal Policy Minimum Bayesian Risk Decoding (OP-MBRD), an inference-time method to enhance LLM reasoning by improving upon and unifying existing techniques like Best-of-N (BoN) and standard MBRD. The core contribution is a framework that performs MBRD over a theoretically-grounded ``optimal policy'' derived from KL-constrained policy optimization over a reward model (i.e., standard RLHF setting). As sampling from this optimal policy is intractable, the paper uses importance sampling to approximate its expectations using samples from a base generator. The paper also proposes a sample-aware variant, OPE-MBRD, that adaptively adjusts the number of samples based on problem difficulty. Through empirical validation on MATH and HumanEval benchmarks, the paper demonstrate that OP-MBRD achieves robust performance compared to baselines and that OPE-MBRD offers a favorable accuracy-compute trade-off.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\nThe paper's core idea is simple yet interesting. The approach of combining principles from reward model training with inference-time decoding is a promising direction. The writing is clear and easy to understand, and the set of experiments conducted on MATH and HumanEval is quite thorough, providing a solid demonstration of the proposed method's capabilities.\n\n## Weaknesses\n**Lack of Algorithmic Explanation and Theoretical Grounding:** Although the paper is easy to read conceptually, it is hard for the reader to piece together the algorithm that was finally implemented. Including a formal algorithm block would be very useful for understanding the final method and for reproducibility. Furthermore, while the theoretical results are cited from other papers, the submission would be much stronger if it provided some brief primers to contextualize the theory and explain why the assumptions hold for this specific application.\n\n**Experimental Limitation with $p_R = p$:** A second weakness concerns the scope of the experimental validation. The paper makes a crucial simplifying assumption by setting the reference policy $p_R$ to be the same as the generator policy $p$ in all experiments (line 259). This nullifies the $\\log \\frac{p_R}{p}$ term in the core formulation (Eq. 10), effectively reducing the method to a form of normalized, reward-weighted sampling. While acknowledged, this sidesteps a key question of how OP-MBRD performs when the generator and reference models differ, which is the standard setting in RLHF and where the framework's full potential lies, and the empirical validation seems to not be comparing against policies trained using RLHF which makes the empirical section incomplete.\n\nQUESTIONS:\n**Algorithmic Clarification:** Regarding the importance weights in Equation 9, how is the intractable $\\max_{y'} \\tilde R(y', x)$ term calculated in the general case where $p_R \\neq p$? The paper explains the simplification for $p_R = p$ (line 263), but it is unclear how this would be handled otherwise. Is the maximization simply performed over the N generated samples? \n\n**Empirical Setup Clarification:** This is a question about the baseline policies. The introduction discusses RL-trained models, but the experimental setup excludes them (lines 235-237). Could you please confirm if the generator models used for all methods (including baselines like BoN and MBRD) are standard instruction-tuned, or if they have undergone some training with RL for reasoning?\n\nLIMITATIONS:\nYes, limitations discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-21"}