Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper looks at the problem of offline RL from human preferences. Instead of first learning a reward model from human preferences and then training with offline RL, this paper learns a context embedding from trajectories, and then learns a policy that is conditioned on the context. Then, to optimize for the human preferences, it learns the optimal context, which can be fed into the policy. 

strength_and_weaknesses: 
Generally I like this research direction, and I think that learning context from offline data can be useful for other tasks as well, such as learning options
 Unfortunately, it doesn’t look like the method convincingly outperforms existing baselines. However, it does perform competitively, and I think that the fact that it doesn’t learn a reward model is a nice aspect.
I like the experiments showing that the learned optimal context aligns with preferences. 
A question: what is the main difference between learning a reward model and learning a context + the optimal context? It seems almost the same thing. Could they be thought of as the same thing in a broader unified perspective?  Why would we expect OPPO to perform better than learning a reward model? 


clarity,_quality,_novelty_and_reproducibility: The method seems like a clear extension of HIM, but seems novel enough. The algorithm 1 is confusing. It seems reproducible enough. 
summary_of_the_review: I think this is a nice idea, and it should be useful for the community.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies preference-based offline reinforcement learning where we only have access to the preferences over offline trajectories. Compared to the standard offline RL setting, PbRL doesn't assume the reward function is available, which is more realistic. To solve the problem, the authors propose an iterative algorithm, which optimizes two objectives iteratively and can directly output a policy, instead of first learning a reward function from the preferences and then training a policy on top of it. Empirically, the authors compare their algorithm with several baselines on the D4RL benchmark and demonstrate improved performance. 
strength_and_weaknesses: Strength:
- The problem setup in this paper is more realistic than most of the offline RL papers, where the reward is always available for each state-action pair. 

- The proposed algorithm is new and can greatly improve the performance over baseline methods on the D4RL benchmark.

Weakness:
- The motivation of OPPO is unclear to me. Especially, why optimizing two (heuristic) objectives iteratively is a better choice than solving two separated optimizations, i.e., first, learn a reward function based on the preference, and then fit a policy. Can you provide a more rigorous justification for it? 

- In addition, I also noticed that DT+$r_{\psi}$ performs quite close to (only slightly worse than) OPPO, but it performs slightly better than DT+r (ground-truth reward). This is quite surprising, can you explain why this is the case? Also, it seems the OPPO has little advantage over the two-stage algorithm, DT+$r_{\psi}$. 

- The paper is not easy to follow, it has too many vague, informal and unsupported arguments. E.g., 

> "OPPO learns the optimal (offline) policy (π(a|s, z∗)) directly and thus avoids the potential information bottleneck caused by separately learning a reward function"  what do you mean by information bottleneck?

> "A better estimate of the optimal embedding provides hints for the encoder to extract more valuable features, while a better hindsight information encoder, on the other hand, accelerates the search process for the optimal trajectory in the high-level embedding space":  what do you mean by "hints"? Can you provide evidence for supporting this argument?

> "It is worth mentioning that the posterior of the optimal embedding $z^\star$: why call it a posterior? Shouldn't $z^\star$ just be a point?

- What is $\ell$? I did not see the definition of it.

- Is $\pi(\cdot|s,z)$ a parameteric model?

- In general, offline RL relies on pessimism to overcome the distribution shift. It seems that OPPO doesn't need pessimism to perform well. If so, can you explain why it doesn't need?

- Can you provide a comparison on the *Random* datasets, e.g., Hopper-Random, Walker-Random, and Halfcheetah-Random? Also, why is the Ant dataset missing?
clarity,_quality,_novelty_and_reproducibility: The algorithm is new. Code was submitted, but I did not run it. The writing needs to be improved. There are too many informal arguments.
summary_of_the_review: Overall, I think the problem studied in this paper is very interesting and close to reality. However, I am not really convinced with the motivation of the proposed algorithm, and why we need to do so as in this paper. In addition, the writing of this paper needs to be improved. There are too many vague/ambiguous and informal arguments. The authors should make explainations more formally, and provide either theoretical justifications or empirical results to backup their arguments.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents an approach for learning to solve task from offline data using preference feedback and no reward function. Instead of learning a reward function, a latent embedding is learned that encode information about what it means to complete the target task. The latent embedding, z, is learned from preferences over the offline data used to train the policy. The latent embedding z and the policy are learned separately, where there during policy learning and inference, next actions are conditioned on both z and the current observation. Policy training is conducted following Hindsight Information Matching. Experiments are conducted in the Hopper, Walker, and Half cheetah locomotion tasks from D4RL with offline data coming from the replay buffer of a medium quality policy, rollouts from a medium quality policy, and rollouts from a medium-expert quality policy. To evaluate the quality of the learned latent embedding z*, a t-SNE plot is used to visualize how similar z* is a an ideal trajectory. Across conditions, z* is close to the ideal trajectory. The performance of policy rollouts using the "optimal" z embedding, a z embedding from a trajectory with large returns, and a z embedding with low returns are compared. Across conditions rollouts from the "optimal" z embedding out perform the z embedding from the trajectories with high and low returns. Policy rollout performance is also compared against other offline RL approaches (decision transformer with the ground-truth reward, decision transformer with a learned reward, CQL with the ground-truth reward, and behavior cloning). In 1/3 of the conditions, the proposed methods out perform the baselines. Finally, the impact of iterating between the Hindsight Information Matching objective and policy learning is evaluated via t-SNE projections and visualization. In all but the medium condition, the learned z* and the ideal trajectory are further apart than in the first experiment. 
strength_and_weaknesses: Strengths:
- The idea is really interesting. This is probably one of the first works I have seen that attempts to learn a policy from preferences in the absence of a reward function. Moving away from reward functions seems like it could have benefits and improve policy learning. 
- The work is well grounded in the literature and authors do a good job of contextualizing where the method fits and how it is different from what has been done.
- A good amount of detail has been given to specify how OPPO works meaning the method and the results should be reasonable to reproduce. 

Weakness:
- The proposed method does not consistently outperform the baselines. Clear benefits of the method are not well motivated, especially given the results. For example, the authors motivate their approach by stating that learning reward functions from preferences may result in a reward function that is not correctly calibrated to the preferences. The authors do not follow up with specific scenarios where or under which the reward functions are not well calibrated. It would have been nice to see clear cases where mis-calibration occurs when learning reward functions and then results demonstrating that OPPO is well calibrated in the identified scenarios. 
- The authors only present results using a synthetic preference labeller that gives perfect preference feedback with respect to the ground truth reward. However, humans are not likely to provide such high quality labels. As the quality of the offline dataset impacts rollout performance, it is likely that preference label quality will also impact rollout performance. It has been shown in the BPref (Lee et al. 2021) paper that labeller quality does impact policy performance.
- I did not see a discussion about the impact of the amount of preference feedback on rollout performance. In online PbRL methods where the reward function is learned, the amount of preference feedback greatly impacts policy performance. It would have been great to see results on this and for there to have been a discussion.
- The authors claim that their method is guaranteed to learn the optimal latent embedding z, however no proof that optimality is guaranteed is provided. The experiments (i.e. t-SNE plots) are not enough evidence to conclude optimality is guaranteed. 
- Prior online PbRL papers have included results from human experiments where actual humans provide the preference feedback. It would have been nice to see results with humans providing the preference feedback.
- The authors do not appear to have compared against Shin & Brown 2021, who they call out as a very similar method that learns a reward function from preferences over an offline dataset and then use offline RL methods to learn a policy. It may have been present in the baselines, but it wasn't clear to me.
clarity,_quality,_novelty_and_reproducibility: The idea is a creative combination of Hindsight Information Matching methods and preference-based reinforcement learning. It seems fairly novel, however it would have been nice to a discussion on the differences between the OPPO and Skill Preferences (https://proceedings.mlr.press/v164/wang22g/wang22g.pdf), both of which use preference feedback to learn latent embeddings used to condition a policy's behavior. In the case of Skill Preferences, the latent embeddings correspond to skills (sub-tasks) instead of complete tasks as in OPPO. Additionally in Skill Preferences, after the skill embeddings are learned, online PbRL is used to learn a policy to select a skill and used a preference-learned reward function. However, it would have been nice to see a discussion of the key differences in how the latent embeddings are learned. 

From a reproducibility perspective, enough detail seems to have been provided that it should be possible to reproduce the results. However, I did not attempt to reproduce the results and therefore cannot say for certain. It would be great for the authors to release the code and make OPPO more reproducible. The hyper-parameter values for the number of preferences given does not appear to be given, but I may be missing it. 

In terms of clarity and quality of writing, there are a number of places where the authors would benefit from another read through of the paper. There were many sentences with an awkward construction, e.g. "Sequence modeling enables to model behavior without....". While the construction did not make the paper incomprehensible, reducing the number of sub-clauses and parentheticals in each sentence would make the paper much easier to read and follow. 
summary_of_the_review: In summary, the paper presents an interesting idea that seems of benefit to pursue. However, given that OPPO did not consistently outperform the baselines, better motivation for why the method should be used instead needs to be provided. The benefits of OPPO over the baseline need to made more clear. Additionally, there are further experiments that would be helpful to see to fully understand the benefits and robustness of the method. My recommendation for the paper is therefore based upon the need for more experiments and the lack of consistent improvements in performance relative to baselines.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors introduce a novel approach for Preference-based Reinforcement Learning in an offline setting. In contrast to other methods, they do not learn an explicit reward function, but use information matching against a latent representation of an (approximate) optimal trajectory. The latent representation is learn via a transformer approach. The algorithm is evaluated on three benchmark domains, including analyzation of three specific aspects of the approach. Namely, an qualitative evaluation of the embedding space in terms of distance to an optimal trajectory, under a simplified learning scheme and effects of using different points in the latent space as conditioner.
strength_and_weaknesses: The paper is very well written and clear to understand, with one, small exception (see clarity). The approach itself mainly builds on existing ideas, but goes beyond a simple, straight-forward combination. Furthermore, it is nicely aligned with the current State-of-the-Art and represents a meaningful step forward. The relation to related work is clearly state. The evaluation is ok, but could be improved. Mostly, because only one of the 4 competitors (Tbl.2) is a true PbRL algorithm. Furthermore, 3 domains (with 3 variants each) is ok, but leaves room for substantial improvement. Especially, non-locomotion tasks should be considered. Additionally, the required number of preference queries should be evaluated, as this is usually a limiting factor in PbRL (due to costly, human involvement). The qualitative ablation studies are a good addition to the evaluation. 
clarity,_quality,_novelty_and_reproducibility: The approach is novel and relevant to the research area. The paper is of high quality and very good to follow, including the mathematical sections. However, to the reviewer it is not completely clear why $I_{\theta}$ is updated by (6) and (8) independently and not a joint objective. This way, it seems like, the magnitude of the updates may implicitly define a tradeoff between potentially conflicting targets?
BC in Tabl.2 is never introduced. Reproducibility is likely, as code is available.


summary_of_the_review: The paper is very good and the clarity issue are not relevant for acceptance, but could improve the paper further. Only the evaluation is a relevant concern, but is still extensive enough to warrant a recommendation.

In fact, the reviewer believes, that the method is even more capable than mentioned. In theory, the method should be applicable to preference signals, based on a non-Markovian expert evaluations. This is an important issue for real world PbRL, because real humans cannot be assumed to conform to the Markov property. Therefore a discussion of this assumption (or even evaluation), would be quite interesting.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper consider an offline preference-based reinforcement learning setting and propose a method for learning a behavioural policy in an end-to-end fashion avoiding a naive two-stage strategy in which one first infers a reward function and then performs a standard approach for offline reinforcement learning. The authors claim that this is beneficial "when encountering complex tasks" (e.g., with non-Markovian information in the preferences) and when there are noisy preference ("a two-step procedure is subject to noisy reward labels") because in such a case "the learned reward function is imperfect, [and] noise would be introduced to policy learning".

The reviewers agreed that these claims are sensible and an end-to-end approach for policy learning in general useful. However, the authors fail to show that these claims are indeed true. In particular, no experiments are presented which demonstrate the claimed advantages of their approach. The authors should present experiments with non-Markovian rewards and noise in the preference feedback to show that their approach really has the claimed advantages. Additionally understanding how the method behaves in terms of the amount of preference feedback provided would be important to study. Furthermore, the writing of the paper should be improved. Not all details of the proposed approach are easy to comprehend from the given writeup, e.g., the possible meaning of the losses, information bottleneck, etc. Some of these problems can be fixed by moving material from the appendix to the main paper, some others might require a more careful rephrasing of the authors' approach.

In summary: while the authors' approach is promising and sensible, the current evaluation of the approach is not convincing and should be improved for future submissions. Furthermore, the paper and proposed method would be more accessible if clarity of the paper was improved.
justification_for_why_not_higher_score: The reviewers rating are 6 / 3 / 3  / 8. The review of the reviewer giving a score of 6 is relatively superficial and the confidence of the reviewer is low, so I discounted this review. The remaining ratings show a relatively clear signal for rejection and also the most positive reviewer agreed in the discussion to the concerns of the negative reviewers.
justification_for_why_not_lower_score: n/a
Decision: Reject