Title: value: Provable Offline Preference-Based Reinforcement Learning
Authors: value: Wenhao Zhan
Masatoshi Uehara
Nathan Kallus
Jason D. Lee
Wen Sun
Venue: ICLR 2024
Keywords: value: reinforcement learning theory
offline reinforcement learning

Abstract: value: In this paper, we investigate the problem of offline Preference-based Reinforcement Learning (PbRL) with human feedback where feedback is available in the form of preference between trajectory pairs rather than explicit rewards. Our proposed algorithm consists of two main steps: (1) estimate the implicit reward using Maximum Likelihood Estimation (MLE) with general function approximation from offline data and (2) solve a distributionally robust planning problem over a confidence set around the MLE. We consider the general reward setting where the reward can be defined over the whole trajectory and provide a novel guarantee that allows us to learn any target policy with a polynomial number of samples, as long as the target policy is covered by the offline data. This guarantee is the first of its kind with general function approximation. To measure the coverage of the target policy, we introduce a new single-policy concentrability coefficient, which can be upper bounded by the per-trajectory concentrability coefficient. We also establish lower bounds that highlight the necessity of such concentrability and the difference from standard RL, where state-action-wise rewards are directly observed. We further extend and analyze our algorithm when the feedback is given over action pairs.


Review discussion:
Most work in PbRL focuses on empirical evaluations, therefore additional work considering the theoretical background is important. This work is especially relevant, as it is not restricted to a specific class of function approximation and considers return-based utility. Reward-based utility is usually easier to handle, but can induce abnormalities in the preference setting, therefore the return-generalization is also highly interesting. Adapting the concentrability coefficient to this setting is interesting as it allows to separate (offline) optimization of the policy from the issues of exploration/exploitation.
The most substantial weakness of the paper is, that the related work focuses on other, theoretical work, but not practical. E.g. the idea of return-based utility approximation is not novel (e.g. Preference-Based Policy Learning, Akrour, 2011 - see Wirth 2017 for more examples). This is relevant because of two reasons.
First, it allows the reader to determine which algorithms fit under the proposed framework and second, it clarifies the differences between the proposed algorithm and other works. Considering the algorithm is mentioned as one of the main contributions, but likelihood-based estimation of the return utility is already known, the novelty of the algorithm is a bit limited.
Besides that, there are several possibilities to improve the presentation:
- Reward is defined wrt. trajectories, but this is commonly denoted return in the PbRL literature
- Some relevant information is left to the appendix - mostly the feasible implementation and the comparison to Zu 2023. Especially some details concerning the feasible implementation are relevant for practioners.
- Remark 3 is not explained
Furthermore, to the reviewer, one of the statements is not obvious:
- cannot be relaxed to the per-step concentrability without additional assumptions, such as on the reward structure - Why does it follow from the Theorems, that state-action rewards are not enabling this?
As small remarks:
- Missing reference: "..per-step concentrability coefficient Cst commonly used in the general offline RL literature"
- Missing reference: "..It is well-known that when the reward is state-action-wise, the optimal policy π is both Markovian and deterministic."
- Definition 1 is a bit hard to understand, because g is never define.
- The slack parameter in Sec. 4.1 has the wrong symbol
- Typo in Alg 2: "Distributionally robust plnanning"
Please explain "cannot be relaxed to the per-step concentrability without additional assumptions, such as on the reward structure" (see above)
No ethics review needed.
Thank you for the valuable feedback! Here are our responses:
- **Comparison against Practical Work**
We mainly discuss theoretical works in the related works because our paper also focuses on the theory of offline RLHF.
That said, **our algorithm is still quite different from the practical algorithms presented in [1]**.
Although they also use MLE to estimate the reward model, they typically run RL algorithms **with respect to the learned MLE rewards** (i.e., $\hat{r}$ in Algorithm 1) without any pessimism. The lower bound in [Theorem 3.9, 2] has shown that this kind of algorithm can fail in the worst cases.
In contrast, we **use pessimism** to circumvent such failure. More specifically, we **construct a confidence set of the reward model and then evaluate each policy with the most pessimistic reward model inside the confidence set**. This is the key difference between our algorithm and existing commonly used algorithms in [1], and this modification is crucial to overcome their limit.
- **Reward vs Return**
Thanks for pointing it out! We will revise this in the final version.
- **Appendix**
Thanks for pointing it out! We will move the key points in Appendix to the main paper in the final version.
- **Remark 3**
We will clarify it as follows. In line 4 of Algorithm 1, we need to compute $\mathbb{E} _ {\tau \sim \mu _ {ref}}[r (\tau)]$, which might be computationally inefficient since the space of all trajectories is large. In this case, we can sample $N_0$ trajectories $(\tau ^ i) _ {i = 1} ^ {N _ 0}$ from $\mu _ {ref}$ and then simply use $ \frac{1}{N _ 0}\sum _ {i=1} ^ {N _ 0} r(\tau ^ i)$ as a surrogate of $\mathbb{E} _ {\tau \sim \mu _ {ref}}[r (\tau)]$. From Azuma-Hoeffding inequality, this will only incur an error scaling with $\frac{1}{N _ 0}$.
Therefore as long as we choose $N _ 0 = O(1 / \epsilon ^2)$, this error can be neglected while line 4 becomes more computationally efficient.
- **Cannot Relaxed to Per-step Concentrability**
Theorem 2 shows that there exists **a set of MDPs whose rewards are defined per step (i.e., $r(\tau) = \sum _ {h=1} ^ H r _ h (s _ h, a _ h)$)** and **an offline dataset which satisfies the per-step concentrability** such that  **no offline algorithm** is able to learn a near-optimal policy. This implies that with only per-step concentrability, we are not able to find a high-quality policy, even if we assume the rewards of the MDPs are defined per step.
In contrast, with our trajectory-wise concentrability coefficient in Definition 2, we are able to learn a near-optimal policy for any MDPs, no matter whether the reward is defined over trajectories or defined per step. We will add this explanation in the final version.
- **Missing Reference**
We are sorry about the missing references. We list the references here and will add them in the main paper:
Per-step concentrability coefficient: [3-7], Optimal policy: [8]
- **Definition of $g(\cdot|\tau _ 1,\tau _ 2)$**
We are sorry about the unclearness of the definition of $g$. $g$ is a function mapping from $(\mathcal{T} \times \mathcal{T})$ (recall that $\mathcal{T} = (\mathcal{S} \times \mathcal{A}) ^ H$ is the set of all possible trajectories) to $\mathbb{R}^2$ and thus $g(\cdot|\tau _ 1, \tau _ 2)$ is a two-dimensional vector. It doesn’t need to be a distribution, i.e., $g(\cdot|\tau _ 1, \tau _ 2)$ is not necessarily positive and $\Vert g(\cdot|\tau _ 1, \tau _ 2) \Vert _1$ doesn’t need to be 1. Similar functions are elaborated in the definitions in Appendix E. We will revise this in the final version.
- **Wrong Symbol and Typo**
Thanks for pointing them out! We will correct them in the final version.
[1] Wirth, C., Akrour, R., Neumann, G., Fürnkranz, J., et al. (2017). A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1–46.
[2] Zhu, B., Jiao, J., and Jordan, M. I. (2023). Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. arXiv:2301.11270.
[3] Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A. (2021). Bellman-consistent pessimism for offline reinforcement learning. arXiv:2106.06926
[4] Uehara, M. and Sun, W. (2021). Pessimistic model-based offline rl: Pac bounds and posterior sampling under partial coverage. In arXiv:2107.06226.
[5] Zhan, W., Huang, B., Huang, A., Jiang, N., and Lee, J. (2022a). Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory, pages 2730–2775. PMLR.
[6] Shi, L., Li, G., Wei, Y., Chen, Y., and Chi, Y. (2022). Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. arXiv:2202.13890.
[7] Li, G., Shi, L., Chen, Y., Chi, Y., andWei, Y. (2022b). Settling the sample complexity of model-based offline reinforcement learning. arXiv:2204.05275.
[8] Bertsekas, D. P. (2017). Dynamic programming and optimal control. Athena Scientific.
Thanks for the clarifications. I slightly increased my score, mostly due to the "Comparison against Practical Work". I expected this to also be included into the CR version. (At least as a condensed version)
Thank you so much for increasing the score! We will include that part in the final version.
* The algorithm and analysis are sound.
* Considering multiple cases, e.g., unknown transition, action-based comparison, and fitting them within similar algorithmic framework.
* Nice comparison with previous work in Sec.4.1, Remark 2, 4.
* This paper is fully focusing on theory. It would be nice to also have some empirical results.
* It seems to me that the link function in Eq.1 is known? Would it work if the link function is unknown?
* For unknown transition kernels, would the result extend to the case where the transition kernels' data are sampled separately from the preference data?
No ethics review needed.
Thank you very much for the positive feedback! Here are our responses:
- **Empirical Results**
We are currently considering adding an empirical case study in future work.
That said, **we indeed have some discussions about the practical implementation of our algorithm in Appendix B**.
In summary, we use the Lagrangian multiplier to convert our algorithm into a maximin optimization problem and demonstrate how to compute the gradient of the objective function so that we can use gradient ascent-descent to solve it. Given that gradient descent-ascent has shown superior performance in standard offline RL problems [1], we believe that the above approach can also work well in practice.
- **Link Function**
Yes, we assume the link function is known.
When the link function is unknown, we may consider a link function class $\mathcal{F}$.
Then we can generalize our analysis by implementing MLE over **the joint function class** $(\mathcal{F}, \mathcal{G} _ r)$, i.e., $(\hat {r}, \hat{\Phi}) = \arg\max _ {r \in \mathcal{G} _ r, \Phi\in\mathcal{F}} \sum _ {n=1}^N \log P _ {r,\Phi}(o=o^{n}\mid\tau^{n,1},\tau^{n,0})$ where $P_{r,\Phi}(o= 1\mid\tau^1, \tau ^ 0) := \Phi( r(\tau ^ 1) - r (\tau ^ 0))$. We leave its formal analysis to our future work.
- **Unknown Transition Kernel**
Yes, the analysis still holds. We reuse the dataset in order to reduce sample complexity, but you can definitely collect a different dataset for transition learning as long as the dataset has a bounded single-policy concentrability coefficient (Definition 3).
[1] Rigter, M., Lacerda, B., and Hawes, N. (2022). Rambo-rl: Robust adversarial model-based offline reinforcement learning. Neurips 22.
Thank you for your comment!
You are welcome!
(1) The problem of reinforcement learning with human feedback, specifically offline preference learning is highly relevant.
(2) Their algorithm provides a guarantee of being able to learn the target policy using polynomial number of samples.
(3) They extend their algorithm to settings with unknown transition kernel and state-action preferences.
(4) They introduce a new single-policy concentrability coefficient to measure the coverage of the target policy on the dataset.
(1) There are no implementation details about the algorithms. These algorithms are theoretically sound but there is no intuition for the reader on how to implement them or construct a practical algorithm.
(2) There are no experiments either.
So there is no way to check how this method empirically scales.
(3) Several terms in the paper are not well explained.
For example, $\epsilon$-bracketing: what do you mean by g(.|tau1, tau2)? Is g a probability measure, or a preference?
(4) Proposition 1: For epsilon being arbitrarily small, and the bounds B and R being arbitrarily large, log N_G_r can be arbitrarily large making the sample bounds weak.
(5) The sample bounds i.e. sample complexity N = O(..log (..1/N)). How can N be on both sides, something seems missing.
(1) In Section 4.1, it is mentioned the distance between r and r* is computed as total variation distance or l1 norm. Arent r and r* scalars?
(2) Does C_tr (per trajectory concentration coefficient) not depend on \mu_1? If so, why?
(3) How does C_r(G_r, \pi_tar, \u_1) reduce to sqrt(Ctr)?
(4) It looks like pi, mu and d are used interchangeably. I recommend correcting this.
No ethics review needed.
Thank you for the valuable feedback. Here are our responses:
- **Implementation Details and Empirical Experiments in Weaknesses**
We would like to clarify that this is mainly a **theoretical** work that strives to find sample efficient offline RLHF algorithms with **provable** guarantees.
That said, we would like to point out that we indeed **have discussions about the practical implementation of our algorithm in Remark 1 in the main text and Appendix B**.
In summary, we use the **Lagrangian multiplier** to convert our algorithm into a **maximin optimization problem** and demonstrate how to compute the gradient of the objective function so that we can use **gradient ascent-descent** to solve it. Given this type of gradient descent-ascent has shown superior performance in standard offline RL problems [1], we believe that the above approach can also work well in practice in our preference-based RL setting. We will emphasize this part more in the final version.
- **Response to the claim: There are no experiments either.
So there is no way to check how this method empirically scales.**
We would like to mention that **there are many practical and impactful works about offline RL in top machine learning conferences without any experiments**.
First, for example, in the literature on offline model-free RL, [2] proposed the first provably efficient algorithm with general function approximation.
While this paper does not have any experiments, it is later implemented in another follow-up paper [3]. It is reported that the proposed algorithm achieves SOTA and beats many offline RL baselines, such as CQL, COMBO ICL, and TD3+BC.
Similarly, in the literature on model-based RL, [4] proposed the first probably efficient algorithm with general function approximation.
While this paper does not have any experiments again, later, it is implemented in another follow-up paper [1], and it is reported that it achieves SOTA and beats many baselines, such as again MOReL, CQL, IQL, and TD3+BC. **In this way, even with no experiments, theoretically backed algorithms have been empirically impactful.**
- **Definition of $g(\cdot|\tau _ 1,\tau _ 2)$  in Weaknesses**
We are sorry about the unclearness of the definition of $g$. $g$ is a function mapping from $(\mathcal{T} \times \mathcal{T})$ (recall that $\mathcal{T} = (\mathcal{S} \times \mathcal{A}) ^ H$ is the set of all possible trajectories) to $\mathbb{R}^2$ and thus $g(\cdot|\tau _ 1, \tau _ 2)$ is a two-dimensional vector. It doesn’t need to be a distribution, i.e., $g(\cdot|\tau _ 1, \tau _ 2)$ is not necessarily positive and $\Vert g(\cdot|\tau _ 1, \tau _ 2) \Vert _1$ doesn’t need to be 1. Similar functions are elaborated in the definitions in Appendix E. We will revise this in the final version.
- **Proposition 1 in Weaknesses**
**We would like to state that our bounds are not weak compared to standard RL literature**.
First, as you notice, the log bracket number increases **logarithmically** as the accuracy $\epsilon$ decreases.
However, as we will later see in Theorem 1, we just set $\epsilon=1/N$, which results in an additional $\log(N)$ term. This term is negligible compared to the leading term $1/N$.
Second, the log dependence on $B$ and $R$ is very standard in the literature on statistical machine learning in general, as seen in Section 4,5 in [5] and Definition 8.1 in a standard RL textbook [6].
- **Sample Bounds in Weaknesses**
We are sorry for the misleading expression. Here, we take $N$ so that the right-hand side of (2) in Theorem 1 is less than $\epsilon$.
So, this expression means if $N$ satisfies (2), we can get an $\epsilon$-optimal policy.  To remove $N$ from the right-hand side, with some algebra, we can state that as long as $N=O(\log(1/\epsilon)/\epsilon^2)$ (when we just focus on the dependence on $\epsilon$), we can get an $\epsilon$-optimal policy. We will clarify it in the final version.
- **$r$ and $r ^ * $ in Questions**
No, $r$ and $r ^ * $ are reward **functions**. They map a trajectory $\tau$ to a scalar.
Thus, here $r$ ($r^*$) can be viewed as a vector whose $\tau$-th entry corresponds to the value of $r(\tau)$ ($r^*(\tau)$) for each $\tau \in \mathcal{T}$.
- **$C _ {tr}$  in Questions**
No, it doesn’t depend on $\mu _ 1$ because when we reduce  $C _ r$ to $C _ {tr}$, we take $\mu _ {ref} = \mu _ 1$ in $C _ r(\mathcal{G} _ r, \pi _ {tar}, \mu _ {ref})$ and then $\mu _ 1$ is canceled out  by $\mu _ {ref}$.
- **from  $C_r(\mathcal{G} _ r, \pi _ {tar}, \mu _ {ref})$ to $C _ {tr}$  in Questions**
Note that in this reduction we take $\mu _ {ref}$ to be $\mu _ 1$. From Cauchy-Schwartz inequality, we have
$$ \sqrt{  \mathbb{E} _ {\tau ^ 0 \sim \pi _ {tar}, \tau ^ 1 \sim \mu _ 1}\big[|r ^ *(\tau ^ 0) - r ^ *(\tau ^ 1) - r(\tau ^ 0) + r(\tau ^ 1)| ^ 2 \big] } \geq  \mathbb{E} _ {\tau ^ 0 \sim \pi _ {tar}, \tau ^ 1 \sim \mu _ 1}\big[|r ^ *(\tau ^ 0) - r ^ *(\tau ^ 1) - r(\tau ^ 0) + r(\tau ^ 1)|\big].$$
This implies that
$$C^2_r(\mathcal{G} _ r, \pi _ {tar}, \mu _ 1) \leq \sup _ {r} \frac{ \mathbb{E} _ {\tau ^ 0 \sim \pi _ {tar}, \tau ^ 1 \sim \mu _ 1}\big[|r ^ *(\tau ^ 0) - r ^ *(\tau ^ 1) - r(\tau ^ 0) + r(\tau ^ 1)| ^ 2\big]}{ \mathbb{E} _ {\tau ^ 0 \sim \mu _ 0,\tau ^ 1 \sim \mu _ 1}\big[|r ^ *(\tau ^ 0) - r ^ *(\tau ^ 1) - r(\tau ^ 0) + r(\tau ^ 1)| ^ 2\big]}$$
Since $\tau _ 0$ and $\tau _ 1$ are independent, we have
$$C ^ 2 _ r(\mathcal{G} _ r, \pi _ {tar}, \mu _ 1) \leq \sup _ {r} \max _ {\tau _ 0, \tau _ 1 }\frac{ d ^ {\pi _ {tar}}(\tau _ 0) \mu _ 1(\tau _ 1)}{ \mu _ 0(\tau _ 0) \mu _ 1(\tau _ 1)} =\max _ {\tau} \frac{d ^ {\pi _ {tar}}(\tau)}{ \mu _ 0(\tau)}.$$
- **$\pi, \mu$ and $d$ in Questions**
We are sorry for the unclearness. For $\pi$, we use $\tau \sim \pi$ to denote the distribution of $\tau$ when executing $\pi$ in the MDP.
When $\tau$ follows a more general distribution $\mu$, i.e., when the distribution may not be induced by a policy $\pi$, we use $ \tau \sim \mu$. For $d$, we only use $d ^ {\pi} (\tau)$ to denote the probability of $\tau$ under policy $\pi$. We will further clarify these in the final version.
[1] Rigter, M., Lacerda, B., and Hawes, N. (2022). Rambo-rl: Robust adversarial model-based offline reinforcement learning. Neurips 2022.
[2] Xie, T., Cheng, C. A., Jiang, N., Mineiro, P., & Agarwal, A. (2021). Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34, 6683-6694.
[3] Cheng, C. A., Xie, T., Jiang, N., & Agarwal, A. (2022, June). Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning (pp. 3852-3878). PMLR.
[4] Uehara, M. and Sun, W. (2021). Pessimistic model-based offline rl: Pac bounds and posterior sampling under partial coverage. ICLR 22
[5] Wainwright, M. J. (2019). High-dimensional statistics: A non-asymptotic viewpoint (Vol. 48). Cambridge university press.
[6] Agarwal, A., Jiang, N., Kakade, S. M., & Sun, W. (2019). Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 32
Dear reviewer,
There are only three days left for the discussion period. Do you have any other questions that we can help with?
Thanks a lot for your response. Thanks a lot for pointing out the discussion on the practical implementation.
I do understand that the paper focuses on theoretical results and introduces several theoretical insights that are valuable for the community.
On the other hand, I feel it is also important for works to introduce a practical version of their algorithms, and provide some empirical evaluations to support their theoretical claims.
For instance, if you provide an algorithm with better sample efficiency, you could have come up with a very simple experiment showing how the algorithm has better sample efficiency than previous methods.
Nevertheless, I believe that the theoretical insights introduced in the paper are relevant to the community and have increased my score.
Thank you very much for increasing the score! We appreciate your feedback and will try adding some experiments in the future.
The paper is clearly written.
The paper provides several strong and novel theoretically results on preference-based RL in offline setting. Some of these results also generalize the results of Zhu et al for linear function approximators to general function approximators.
Although the main contribution is theoretical, it would be nice to see an empirical case study or two  where the given algorithm works as expected.
No questions.
No ethics review needed.
Thank you very much for your positive feedback! We are currently considering adding an empirical case study in future work.
That said, **we indeed have some discussions about the practical implementation of our algorithm in Appendix B**.
In summary, we use the Lagrangian multiplier to convert our algorithm into a maximin optimization problem and demonstrate how to compute the gradient of the objective function so that we can use gradient ascent-descent to solve it. Given that gradient descent-ascent has shown superior performance in standard offline RL problems [1], we believe that the above approach can also work well in practice.
[1] Rigter, M., Lacerda, B., and Hawes, N. (2022). Rambo-rl: Robust adversarial model-based offline reinforcement learning. arXiv preprint arXiv:2204.12581.
Thank you for the comment!
You are welcome!

Meta-review:
role: AC
data: metareview: value: The authors address the problem of offline preference-based reinforcement learning (PbRL) with human feedback. Based on feedback in the form of preferences between trajectory pairs, they estimate the latent reward and then essentially formalise the learning task as a planning problem. For this setting, they provide interesting theoretical results and formal guarantees. 

Although the reviewers raise a number of critical points in their original reports, there is agreement that the paper has great potential and makes a significant contribution. The authors showed a high level of commitment during the rebuttal phase and did their best to respond to the comments and to improve the submission. This was appreciated and positively acknowledged by all. In the discussion between authors and reviewers, some critical points could be resolved and some questions clarified. Although practical results and experiments would have further strengthened the paper, there is agreement that the paper has enough merit to be accepted.
justification_for_why_not_higher_score: value: Strong paper, but there are even better ones.
justification_for_why_not_lower_score: value: Strong enough paper.
role: PC

Decision: Accept (spotlight)