Title: Preference-based Reinforcement Learning beyond Pairwise Comparisons: Benefits of Multiple Options
Venue: NeurIPS 2025
Accepted: yes
Keywords: Preference-based Reinforcement Learning

Ranking Feedback

Plackett–Luce Model

Reinforcement Learning from Human Feedback

Dueling Bandit
Abstract: We study online preference-based reinforcement learning (PbRL) with the goal of improving sample efficiency. While a growing body of theoretical work has emerged—motivated by PbRL’s recent empirical success, particularly in aligning large language models (LLMs)—most existing studies focus only on pairwise comparisons. A few recent works  (Zhu et al., 2023, Mukherjee et al., 2024, Thekumparampil et al., 2024)  have explored using multiple comparisons and ranking feedback, but their performance guarantees fail to improve—and can even deteriorate—as the feedback length increases, despite the richer information available. To address this gap, we adopt the Plackett–Luce (PL) model for ranking feedback over action subsets and propose **M-AUPO**, an algorithm that selects multiple actions by maximizing the average uncertainty within the offered subset. We prove that **M-AUPO** achieves a suboptimality gap of $\tilde{\mathcal{O}}\left( \frac{d}{T} \sqrt{ \sum_{t=1}^T \frac{1}{|S_t|}} \right)$, where $T$ is the total number of rounds, $d$ is the feature dimension, and $|S_t|$ is the size of the subset at round $t$. This result shows that larger subsets directly lead to improved performance and, notably, the bound avoids the exponential dependence on the unknown parameter’s norm, which was a fundamental limitation in most previous works. Moreover, we establish a near-matching lower bound of $\Omega \left( \frac{d}{K \sqrt{T}} \right)$, where $K$ is the maximum subset size. To the best of our knowledge, this is the first theoretical result in PbRL with ranking feedback that explicitly shows improved sample efficiency as a function of the subset size.


Peer reviews:
comments: SUMMARY:
The paper proposes M-AUPO, a novel algorithm for preference-based reinforcement learning (PbRL) that leverages ranking feedback over multiple actions (beyond pairwise comparisons) under the Plackett-Luce (PL) model.

STRENGTHS AND WEAKNESSES:
Strengths:

1. First work to rigorously show that larger ranking feedback subsets improve sample efficiency in PbRL, with matching lower bounds.
2. Addresses limitations of pairwise comparisons in RLHF and provides a principled way to exploit richer feedback.
3. Validates theory on both synthetic and real-world data, outperforming baselines like DopeWolfe.

Weaknesses:

1. The per-round cost scales with $K^3d^3$ for rank-breaking (RB) loss, which may limit scalability for very large $K$.
2. Relies on linear reward models and the PL model for rankings; robustness to misspecification is unclear.

QUESTIONS:
1. The RB loss has tighter bounds but higher computational cost. In practice, how do you decide between PL and RB losses?
2. Given the prominence of pairwise preference-based RLHF (e.g., PPO + BT model in [35] or DPO [38]), did you consider comparing M-AUPO against such methods?
3. See the weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose M-AUPO, an algorithm that selects action subsets by maximizing average uncertainty, to enhance sample efficiency for online PbRL. The authors theoretically prove a suboptimality gap and lower bound for M-AUPO, avoiding the exponential dependence on parameter norms. Experiments on synthetic and real-world datasets validate its effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The theoretical derivation is sufficient.\n2. The paper is well organized and easy to follow.\n\nWeaknesses:\n1. In RLHF, it's no surprise that increasing the number of options for the same context leads to performance improvements. For example, for RB methods, this approach means more amount of data, so the reward model will be more accurate. What I think more interesting is the discussion of whether to increase the number of questions or the number of options per question for a fixed amount of total data. For example, the first set of experiments have N questions with M options per question; the second set of experiments had N/2 questions with 2M options per question; which one is better?\n2. The experimental part is too simple. 1. Missing important baselines, e.g., other algorithms that can handle pairwise comparisons in the RB setting, such as DPO; 2. Missing dopewolfe vs. m-aupo comparisons with **varying K** (the last subfigure in Figure 1 and 2).\n3. Lack of hyperparametric analysis. The paper does not show the values of regularization parameter $\\lambda$ and step size $\\eta$ hyperparameters in the experiments and lacks tuning or ablation studies on hyperparameters.\n\nQUESTIONS:\n1. See weaknesses.\n2. The denominator of Eq. (1) should be $\\sigma_k$ and not $\\sigma_j$, right?\n\nLIMITATIONS:\nSuggest adding a limitation paragraph about the effect of inherent limitations of PL or BT assumptions on RLHF.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to design an algorithm for learning from multiple-option feedback in an online PbRL setting with stronger theoretical guarantees. The paper proposes M-AUPO which uses online mirror descent on the comparison loss and selects an assortment of actions by maximizing the feature uncertainty relative to a reference. The paper provides a sub optimality gap and performs experiments on synthetic data and real-world data with fixed features.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper considers a novel setting of multi-option PbRL where contexts are drawn from an unknown distribution and the assortment is selected. \n- Background on PbRL and the algorithm are clearly presented.\n- The paper proposes a new algorithm based on selecting a diverse set of actions and using an online mirror descent algorithm with theoretical guarantees. \n- The paper performs numerical experiments following the theoretical setting and with real-world data. \n\nWeaknesses:\n- Full proofs are not provided. \n- It seems it could be difficult to generalize M-AUPO to more practical settings beyond a linear model. \n- The full bound consists of the term consider the leading term and an additional term with an $e^B$ dependency but decays faster with $T$. The second term is considered negligible for large enough $T$, but if $T$ large enough for this term to be neglected is being considered, under the same setting, the bounds from existing work with $e^B$ dependency would actually be better than the bound presented especially since the provided bound has a factor $e^{4B}$. Based on this, the claims can be misleading and it is unclear whether the sub optimality of M-AUPO is better in practice. \n- Experiments only compare to DopeWolfe. \n- The synthetic experiments work with $B=1$ and the decrease in sub-optimality seems relatively small with $K$ and based on the bound, the effect of $K$ could be smaller with larger $B$. It would be useful to see how this generalizes with varying $B$. \n- The real world setting uses fixed features which differs from most practical settings which often train a full model or train at least multiple layers of a model. As a result it seems difficult to be able to make claims about the effect of multi-option feedback for tasks such as training a language model based on the analysis.\n\nQUESTIONS:\n- Can you provide synthetic experiments with varying B and also provide more details such as what the norms of the features look like in the real-world setting?\n- Can you clarify why the provided bound is an improvement? It seems the second term alone for larger $B$ is weaker than existing bounds. \n- Can you provide comparison to other methods beyond DopeWolfe? In particular, it would be good to see comparisons of sub-optimality gap to the methods presented in Table 1. \n- Would it be possible to extend the experiments/method beyond a linear model?\n- Can you provide the proofs for all results?\n\nHaving the full proofs and clarifications about the provided bound in particular would increase the score.\n\nLIMITATIONS:\nNo, limitations around the method and analysis being restricted to linear models could be more explicitly presented and limitations of the bound under different conditions (larger B, smaller T) should be addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes selecting multiple actions in preference-based learning to improve sampling efficiency. The authors provide theoretical guarantees for the proposed algorithm, showing that the suboptimality gap is $O(\\frac{d}{T}\\sqrt{\\sum_{t=1}^T\\frac{1}{|S_t|}})$, where $|S_t|$ is the size of the action subset offered at round $t$.  This result eliminates the exponential dependence on parameter norm bound  $e^B$. The authors also establish a theoretical lower bound that matches the upper bound.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths: \n 1. This work is the first  theoretical guarantee that larger subsets directly improve sample efficiency in PbRL.   \n 2.  This result eliminates the exponential dependence on parameter norm bound  $e^B$. And previous related works (eliminates $e^B$) only considered two options.  \n   \n\n### Weakness\n1. The parameter $B$ is the upper bound on the norm of $\\theta$ in the linear reward model, i.e., $\\Theta:= \\{ \\theta:\\mid \\|\\theta\\|_2 \\leq B\\}$. Generally, it is a small constant. Therefore, the theoretical improvement does not appear to be particularly significant.\n2. Each iteration requires solving the optimization problem (9), and the authors report a time complexity of $N^2K$. When the number of arms is large, the algorithm incurs a high computational cost.\n3. The paper assumes a finite number of arms ($N$). Under the same setting (linear rewards with a finite number of arms), previous works [1] and [2] have provided regret upper bounds of $O(\\sqrt{dT})$ (which correspond to a suboptimality gap of $O(\\sqrt{\\frac{d}{T}})$), which are better than the $O(\\frac{d}{\\sqrt{T}})$ bound presented in this paper.\n\n[1] Saha, A. (2021). Optimal algorithms for stochastic contextual preference bandits. Advances in\nNeural Information Processing Systems 34 30050–30062.\n\n[2] Bengs, V., Saha, A. and H¨ullermeier, E. (2022). Stochastic contextual dueling bandits under\nlinear stochastic transitivity models. In International Conference on Machine Learning. PMLR.\n\nQUESTIONS:\n1. In the field of contextual dueling bandits, theoretical guarantees are also often provided in terms of regret. The authors use the weaker notion of suboptimality gap. Is there an inherent difficulty in this work to provide a regret analysis?\n2. Saha, A. (2021) also provides a regret lower bound (matching the upper bound) of $O(\\sqrt{dT})$. In contrast, this paper gives a suboptimality gap lower bound of $O(\\frac{d}{\\sqrt{T}})$. How can this difference be explained?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}