Title: Efficient Preference-Based Reinforcement Learning: Randomized Exploration meets Experimental Design
Venue: NeurIPS 2025
Accepted: yes
Keywords: RLHF

Preference learning

RL

Optimal design
Abstract: We study reinforcement learning from human feedback in general Markov decision processes, where agents learn from trajectory-level preference comparisons. A central challenge in this setting is to design algorithms that select informative preference queries to identify the underlying reward while ensuring theoretical guarantees. We propose a meta-algorithm based on randomized exploration, which avoids the computational challenges associated with optimistic approaches and remains tractable. We establish both regret and last-iterate guarantees under mild reinforcement learning oracle assumptions. To improve query complexity, we introduce and analyze an improved algorithm that collects batches of trajectory pairs and applies optimal experimental design to select informative comparison queries. The batch structure also enables parallelization of preference queries, which is relevant in practical deployment as feedback can be gathered concurrently. Empirical evaluation confirms that the proposed method is competitive with reward-based reinforcement learning while requiring a small number of preference queries.


Peer reviews:
comments: SUMMARY:
This paper studies preference-based reinforcement learning. They consider a MDP with linear reward function, while one can only query preference feed-back. They study two learning criterions: regret and sub-optimality, and propose a meta algorithm with theoretical guarantees. After that, they propose a more practical algorithm with lazy updates. Numerical experiments results are also provided.

STRENGTHS AND WEAKNESSES:
### Strengths

The preference-based learning setting is reasonable. The structure of this paper is clear and easy to follow. Numerical experiments are provided.

### Weaknesses

1. Although there are only a few previous work studying preference-based setting, I did not find much technical contribution in this paper. It seems to me most of the results are standard and non-surprising. 

2. Another part makes me confused is that how important is the efficiency in learning the transition function. 

    I think this paper studies the setting where both transition and reward functions are unknown. However, it seems to me the main theorems about regret and sub-optimality only considered the (sample) efficiency in learning reward function. 

    But if this is the case, it seems to me it is unnecessary to assume a PAC-RL oracle, and just assume the transition is known in advance or it has been estimated accurately enough, given that we do not care about the cost in this stage. Or as mentioned by authors in the conclusion, it is enough to just assume a reward-free algorithm as an RL oracle.

QUESTIONS:
The optimal dependence on $d$ in regret should be $O(\sqrt{d})$ (and $O(d)$ in the sample complexity guarantees), at least in linear MDP/bandits setting. Is there any hope to improve the results in this paper?

LIMITATIONS:
N.A.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of designing computationally tractable and query-efficient algorithms for RLHF. The authors propose a meta-algorithm, Randomized Preference Optimization (RPO), which uses randomized exploration (inspired by Thompson sampling) to learn a linear reward model from trajectory preferences. The paper provides both regret and last-iterate (PAC-style) guarantees for RPO under the assumption of access to a PAC-RL oracle. To address practical issues like high query frequency and latency, the authors introduce an improved variant, LRPO-OD, which selects informative query batches, enabling parallelization. The efficacy of these methods is demonstrated theoretically and empirically on a simulated robotics task.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper provides a clear framework for RLHF with strong theoretical guarantees (regret and PAC bounds)\n2. The LRPO-OD variant introduces practical variantions (lazy updates and D-optimal design) to enable parallelization, directly addressing a key bottleneck in real-world RLHF.\n\nWeaknesses:\n1. Empirical validation is limited to a single, simple simulated environment, making it difficult to assess real-world performance and scalability.\n2. The framework is confined to a linear reward model and does not address the challenge of generalizing its posterior sampling approach to non-linear settings. It is unclear if the cartpole problem uses neural networks. Clarification on this would be useful.\n 3. The paper doesn't cite recent, very relevant work on active exploration for preference learning (e.g., Dwaracherla et al., 2024: https://arxiv.org/pdf/2402.00396), miscontextualizing its contribution.\n\nQUESTIONS:\n1. What was the specific feature mapping used in the Cartpole experiments?\n\n2. What is the computational complexity of the greedy D-optimal design step, and how does it scale with the batch size?\n\n3. Can you discuss the relationship between your work and recent active exploration literature like Dwaracherla et al. (2024)?\n\nLIMITATIONS:\nThe authors discuss several limitations, but the discussion would be stronger if it also acknowledged the challenges of generalizing the proposed sampling method beyond the linear reward model assumption.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes efficient RLHF algorithms for general MDPs when learning from preference feedback. First, it proposes a randomized exploration meta‐algorithm that achieves regret guarantees under RL‐oracle assumptions. Second, it proposes another algorithm which proposes a batch of queries at a time to enhance the query efficiency. Experiments show the effectiveness of their algorithms.\n\nSTRENGTHS AND WEAKNESSES:\nI found the clarity of this paper to be great. It is easy to follow, and all algorithms and theoretical results are well explained. Careful comparisons to existing work are also provided, and thus I appreciate the time the authors spent on writing this.\n\nThe paper proposes two algorithms with theoretical guarantees. Although i didn't checked the proofs in detail, the results look sound. One weakness is that the paper could benefit from a clearer “highlight” of its main novelty. From my understanding, Algorithm 1 extends standard MLE-based RL algorithms to the RLHF setting – Im not trying to diminish this contribution; getting those ideas to work under RLHF is a cool contribution; but its main components (randomization, maintaining the \\$V\\$ matrix, and MLE) all appear in related work (such as those mentioned at the end of page 5), so its technical novelty is somewhat limited, that is, it seems hard to find something super insightful from Algorithm 1. Similarly, while Algorithm 2 is an algorithm of different style, it adapts the determinant‐trick from standard RL to RLHF. However, I acknowledge that such contribution is important and clearly nontrivial, thus I’m leaning toward acceptance.\n\nQUESTIONS:\nIn what situation will we set $\\alpha=0$ in algorithm 1? It looks like $\\alpha=1$ is strictly better.\n\nLIMITATIONS:\nThe authors discussed the limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of preference optimization in reinforcement learning, introducing a meta-algorithm where collecting preferences is an integral part of the learning process. The proposed method, Randomized Preference Optimization (RPO), involves sampling trajectories, collecting preferences over these trajectories, and then sampling reward parameters from a confidence set derived from the gathered preferences. The authors provide regret bounds for the meta-algorithm, assuming access to a PAC-RL algorithm capable of producing approximately optimal policies.\n\nTo reduce annotator burden and latency associated with labeling each trajectory, the paper proposes an extension: Lazy RPO with Optimal Design (LRPO-OD). This variant collects trajectories in batches and consults the oracle only when the information gain (measured by the determinant of the design matrix) increases by a constant factor. Within each batch, a greedy D-optimal design procedure is used to select the most informative trajectory comparisons to query. Theoretical analysis shows that LRPO-OD retains the original regret guarantees (up to constant factors) while significantly reducing the number of required preference queries---only $O(T / \\log T) $ in expectation over $ T $ rounds.\n\nEmpirical results on the Isaac-Cartpole environment demonstrate that both RPO and its lazy variants perform competitively with reward-based RL methods, with LRPO-OD achieving the highest returns while issuing the fewest preference queries.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n**Idea and Motivation**:  \nThe paper explores a compelling and timely idea, particularly relevant given the increasing focus on preference-based learning. One of the major bottlenecks in this field is the annotator burden and latency associated with collecting human preferences. The proposed approach directly addresses this challenge, making the contribution both relevant and potentially impactful.\n\n**Writing and Presentation**:  \nThe writing and structure of the paper are strong. Concepts are introduced in a logical sequence, and the overall exposition is clear and accessible---even for readers without a strong theoretical background.\n\n**Theory**:  \nWhile I do not have the theoretical background to verify the correctness or novelty of the formal results, the theorems are well-motivated and clearly explained. The theoretical development appears sound and well-integrated into the broader argument of the paper.\n\n## Weaknesses\n\n**Empirical Experiments**:  \n- The learning curves exhibit large overlapping confidence intervals. Although the intent seems to be showing that preference-based methods can match reward-based RL performance, conducting more trials would increase confidence in the empirical claims.\n- While the experiments support the effectiveness of RPO and LRPO-OD, they do not clearly isolate the benefit of reward parameter randomization. Including a baseline without randomization would help establish the specific advantage of RPO.\n- It is unclear what feature function was used in the experiments---i.e., was $\\phi(s, a)$ based on raw features or learned representations? Clarifying this would\n\nQUESTIONS:\n**Relation to existing RLHF work with reward randomization**:  \nSome related work on reward parameter randomization in the context of RLHF is discussed in the weaknesses section. It would be helpful to clarify how this work relates to or differs from prior approaches.\n\n**Experimental details**:  \nAdditional experimental concerns are raised in the weaknesses section.\n\n**Clarifications requested**:\n- What is the motivation for including the case with $\\alpha = 0$ in RPO?\n- From the regret bounds, it appears that both $\\alpha = 0$ and $\\alpha = 1$ result in the same cumulative regret---summing over the sub-optimality term in Theorem 3.5 yields a bound equivalent to Theorem 3.4. Could the authors clarify if there is any practical or theoretical distinction between the two settings?\n- In Algorithm 2, what role does $W_t$ play in tracking $V_t$? An explanation of how $W_t$ helps manage or estimate $V_t$ would be appreciated.\n\nLIMITATIONS:\nYes, discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}