Adversarial Policy Optimization for Offline Preference-based Reinforcement Learning

Authors: Hyungkyu Kang, Min-hwan Oh
Primary area: reinforcement learning
Keywords: preference-based reinforcement learning, reinforcement learning with human feedback, offline reinforcement learning

Abstract:
In this paper, we study offline preference-based reinforcement learning (PbRL), where learning is based on pre-collected preference feedback over pairs of trajectories. While offline PbRL has demonstrated remarkable empirical success, existing theoretical approaches face challenges in ensuring conservatism under uncertainty, requiring computationally intractable confidence set constructions. We address this limitation by proposing Adversarial Preference-based Policy Optimization (APPO), a computationally efficient algorithm for offline PbRL that guarantees sample complexity bounds without relying on explicit confidence sets. By framing PbRL as a two-player game between a policy and a model, our approach enforces conservatism in a tractable manner. Using standard assumptions on function approximation and bounded trajectory concentrability, we derive a sample complexity bound. To our knowledge, APPO is the first offline PbRL algorithm to offer both statistical efficiency and practical applicability. Experimental results on continuous control tasks demonstrate that APPO effectively learns from complex datasets, showing comparable performance with existing state-of-the-art methods.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 8

Reviews and discussion:
<thread>
creation date:
1729763074984

thread_type:
official_review

<thread_object>
creator: reviewer_bzb5

<content>
summary:
The authors propose a novel variant of the two-player formulation of PbRL, that allows the deviation of statistical bounds as well as a computational efficient implementation. The basic element of the proof is a novel sub-optimality decomposition. They also offer implementation details as well as an empirical comparison with other SoTA algorithms.

strengths:
Theoretically sound approaches to PbRL are an important tool for researchers, because they allow conclusions beyond the empirically tested scenarios. Together with the also available empirical evaluation, this work becomes a significant contribution. This is further strengthened by the fact, that the algorithm is competitive to the SoTA. This is not expected for an algorithm derived from an mostly theoretical work.
The primary contribution of the work is an original contribution, which is embedded in a established framework.
Quality and Clarity are also good, with only minor limitations. Especially the formalization/introduction and the embedding into the available, related work are excellent. However, an overview table of related work, showing the available bounds, given assumptions & constraints would strength this further. This would enable the reader to directly pinpoint the variant considered by the authors, wrt. other works.

weaknesses:
The most substantial weakness, are the repeated references to the appendix. The appendix should not be assumed to be available to every reader. In most cases, the explanations in the main paper are sufficient to follow the work without the appendix, but there are exceptions. E.g. the KL regularizer in alg 1&2 which is only requried for Alg.3, but this is not explained. In general, the authors should ensure that the paper is self contained. Stating a condensed version of what is written in the appendix (like for Theorem 4.1) is a good practice.

Furthermore, there are some strong assumptions, that should be discussed, like the fixed-length requirement or the Markovian reward. Both assumptions are correctly stated, but it is not clear if they are a substantial requirement, or just in place for enabling the formal proof.

Additionally, the experiments should be extended by additional ablation studies. As example, the impact of smaller/larger D_traj sets would be interesting (independent of the number of preferences). Another improvement would be adding a rank or critical distance plot for table 1, as a direct comparison with the SoTA is inconclusive. However, given that experiments are not the primary scope of the paper, these are only a smaller concerns.

One small typo: The results of baselines Oracle, PT, DPPO, and IPL are taken from Choi et al. (2024) - "Oracle" should likely by "MR"

questions:
Can you please point me to the description of the size of $D_{traj}$ used in the experiments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730049259686

thread_type:
official_review

<thread_object>
creator: reviewer_ftyb

<content>
summary:
This work explores the innovative realm of PbRL, addressing the challenges of ensuring conservatism under uncertainty. APPO is designed to ensure conservatism in PbRL without the need for explicit construction of intractable confidence sets. This is achieved by framing PbRL as a two-player game between a policy and a model, which allows for a tractable enforcement of conservatism. Experimental results show that APPO performs comparably to existing state-of-the-art algorithms in continuous control tasks．

strengths:
APPO is designed to optimize the learning process in preference-based reinforcement learning (PbRL) by effectively utilizing preference feedback, which allows for faster convergence and improved sample efficiency compared to traditional methods. The proposed method can be integrated with any standard unconstrained reinforcement learning algorithm, making it versatile and applicable across various domains. Additionally, the paper provides theoretical bounds on the sample complexity of the proposed method.

weaknesses:
The performance of APPO is sensitive to the choice of the conservatism regularizer coefficient (λ). While the algorithm can learn with a range of λ values, improper tuning can lead to suboptimal performance and stability issues, which may require additional effort in hyperparameter optimization. Moreover, the theoretical guarantees of APPO rely on standard assumptions regarding function approximation and bounded trajectory concentrability. If these assumptions do not hold in certain environments, the performance and reliability of the algorithm may be compromised.

questions:
1.What is the impact of the conservatism regularizer coefficient (λ) on the performance of APPO? How can an appropriate λ value be selected?

2.How does APPO compare to other IQL-based algorithms in terms of hyperparameter tuning advantages and disadvantages?

3.What happens to the performance of APPO if these assumptions do not hold in certain environments?

4.What are the scalability concerns of APPO in high-dimensional state or action spaces? Are there any issues related to computational complexity?

5.In practical applications, how does APPO cope with sparse or non-representative preference data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730671243143

thread_type:
official_review

<thread_object>
creator: reviewer_bkme

<content>
summary:
This paper studies offline PbRL. The authors propose APPO, which frames PbRL as a two-player game between the policy and an adversarial reward model. This algorithm enforces pessimism without explicit confidence sets, making the method more computationally tractable than the existing literature. The paper further provides theoretical guarantees, demonstrating that APPO achieves strong sample complexity bounds under standard assumptions on function approximation and trajectory concentrability. Experimental results on continuous control tasks (in Meta-world) show that APPO achieves performance comparable to state-of-the-art methods.

strengths:
1. Algorithmic novelty: The Q function estimation part in Algorithm 2 is interesting and novel.

2. Valid theoretical guarantee: The authors provide a rigorous sample complexity analysis under standard assumptions.

3. Qualified empirical performance: The experiments show that the practical implementation of APPO is a competitive empirical algorithm. This part is lacked in many existing works.

weaknesses:
1. Limited novelty: the essential idea is not very novel. In standard RL, there have been works to use two-player games to get rid of confidence sets (e.g., 'Adversarially Trained Actor Critic for Offline Reinforcement Learning' by Ching-An Cheng) and this work is very similar to this line of work. In addition, the authors should highlight new analysis techniques in the paper, if any.


2. Complicated algorithm: although Algorithm 2 doesn't need confidence sets, the algorithm is still not computationally efficient due to Line 4. Typically, most of the existing works think a method computationally efficient if it can be solved with least squares oracles and MLE oracles. However, I think Line 4 in Algorithm 2 cannot be solved with these oracles.

questions:
1. The authors claim that using Lagrangian multipliers to get rid of the confidence sets is not applicable here (Line 204-207). I am not quite convinced because such an approach is effective in standard RL and I hope the authors can elaborate more.


2. For Algorithm 2, can we just replace the true environment in Algorithm 1 with the estimated environment $\widehat{P}$ (adding some regularizers to quantify the uncertainty of $\widehat{P}$ at the same time)? Then the resulted algorithm could be less complicated.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691252386

thread_type:
official_review

<thread_object>
creator: reviewer_dhla

<content>
summary:
This paper introduces APPO, a novel algorithm for offline PbRL that utilizes a two-player game formulation to produce a robust policy less prone to model errors. By avoiding explicit confidence sets, APPO achieves computational and statistical efficiency. The paper derives a sample complexity bound for APPO under standard assumptions and presents experimental results demonstrating performance comparable to SOTA offline PbRL methods on continuous control tasks.

strengths:
This paper is well-motivated and organized. The proposed approach is theoretically sound and solid. The authors derive sample complexity bounds without relying on explicit confidence sets, offering statistical efficiency and practical applicability. The approach is evaluated on continuous control tasks, with results showing that it performs on par with or surpasses SOTA baselines.

weaknesses:
Please see Questions.

questions:
1. The paper only evaluates APPO on a medium-replay dataset from Metaworld. It would be beneficial to evaluate APPO on a wider range of datasets to assess its generalizability and robustness.

2. Previous work on offline PbRL, such as CPL [1], directly learns a policy without RL, as noted in the related work section. Using a supervised learning approach, CPL achieves comparable performance with SOTA baselines while significantly reducing computational complexity. How does APPO compare to CPL, and what are the advantages of using APPO over CPL?

3. Could the authors discuss the limitations of APPO?

[1] Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W. Bradley Knox, and Dorsa Sadigh. Contrastive preference learning: Learning from human feedback without reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper studied offline preference-based learning and proposed an algorithm that is more computationally tractable than prior works. Theoretically, the paper extends ideas from pessimistic offline RL literature to offline preference learning. From the empirical side, the proposed algorithm achieved good performance on the standard continuous control tasks.