Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a transformer-based neural architecture, Preference Transformer, to model human preferences in preference-based reinforcement learning. Specifically, this work adopts the assumption of non-Markovian rewards and utilizes transformer-based architecture in modeling sequential data to capture temporal dependencies in human decisions and infer critical events in a trajectory.
strength_and_weaknesses: Strengths:
1. This paper is well-written and easy to understand.
2. The proposed new preference predictor that depends exponentially on the weighted sum of non-Markovian rewards is reasonable and effective.
3. Conducted Experiments are well-designed. Preference Transformer are evaluated both on real human preferences and synthetic preferences. The authors demonstrate the relationship between them and show that existing benchmarks based on synthetic preferences may not be enough.
Weaknesses:
1. It would be interesting to see justifications for the architecture selection. Did you compare the GPT with other transformer architectures in experiments?
2. In Section 5.2, the authors demonstrate that capturing long-term information is the reason why PT is more competitive than other baseline models, but LSTM-based NMR could also capture temporal dependencies. There should be detailed explanations to clarify the demonstration.

clarity,_quality,_novelty_and_reproducibility: The writing is clear and easy to follow.
The proposed preference predictor based on non-Markovian assumption is novel and effective.
The released code seems easy to run and install.

summary_of_the_review: This paper introduces a novel transformer-based architecture to model human preferences under the non-Markovian assumption in preference-based reinforcement learning. The overall framework seems to be effective and performs well according to the experiment results. Though some of the claims have minor issues, the overall paper is well-written and easy to follow.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Given: just preferences over agent trajectories (and not rewards),
and the "underlying" reward being non-Markovian and weighted.

Approach: the authors propose the Preference Transformer model which learns the "latent reward" based on the whole past being fed into a Transformer to output current reward and weight. Then from this reward they can classically train an agent.

The contribution includes this architecture as well as experiments

strength_and_weaknesses: **Strengths.**

Except for details, overall the paper is well written.

The preference RL setting is interesting,
and,
if in deed the rewards are non-Markovian,
the idea of Transformers is well motivated.

So there is a conceptual as well as empirical contribution.


**Weaknesses and improvement points:**

I think the precise contribution over previous work needs to be clarified:
* From a first look at the paper, I had these questions: does a preference transformer already exist, but just for Markovian, unweighted rewards, and the contribution is just to extend it to the non-Markov, weighted setting? Or is the contribution, to introduce the preference transformer thing altogether, and directly in the general from (non-M. ...)?
* When looking into it in more detail, I understand it as follows: it seems (Early 2022) is closest, since they also address preference RL with non-Markov rewards, and with a similar relation between rewards and preferences, but the difference is that (Early 2022) (1) use LSTM instead of Transformers and (2) don't allow for weighing. This, in particular, should be made more clear.

Regarding experiments, there need to be some clarifications, and some results are counterintuitive:
* For the tasks that are studied, are the true reward functions actually Markovian or not? I don't find that info. Or is there no true reward for some tasks?
* Sec 5.4: if the reward is actually Markovian, then the much simpler MR baseline model (which assumes a Markovian reward via an MLP) actually has *no approximation error in that sense (i.e., up to functional approximation, the dependencies between the variables can in principle perfectly be captured by this model class)*, but is much simpler in terms of capacity (so, should be more data efficient). I'm really surprised that nonetheless PT outperforms it. Can you explain why PT outperforms it?

**Minor points:**

* Note that non-Markovian rewards can be turned into markovian my changing the state.
* How does it relate to V/Q value function? It's not necessary, but 1-2 sentences could be interesting.
clarity,_quality,_novelty_and_reproducibility: See above.
summary_of_the_review: Overall, it is an interesting idea to use Transformers for non-Markovian reward and weighting, as part of preference RL. The writing overall is OK-good, though some clarifications are needed. 

The main weaknesses for me are that the contribution is interesting but rather limited compared to (Early 2022) (replace LSTM by Transformer and add weighting). Additionally, I'm not sure how often actually non-Markovian rewards are needed, since they can be mitigated by good state representations. And, in the experiment, if the reward is Markovian, I didn't understand why nonetheless the non-Markovian method performs better.

When ignoring significance/size of contribution, I lean accept; when not ignoring it, I lean reject. In favor of the doubt, overall I lean accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper describes an approach for learning a reward function in preference-based reinforcement learning based on transformers.  This provides the advantages that the model allows credit assignment within the behavior trajectory to correctly weight significant state/actions, and that less feedback samples that has typically been required to learn the reward.
strength_and_weaknesses: +The paper is well written

+The transformer approach naturally lends itself to credit assignment, which is important for human preferences as human are significantly influenced by surprising events.

-Given an advantage of this approach is feedback-efficient, it would have been nice to have compared against PEBBLE given that is designed for efficiency.

clarity,_quality,_novelty_and_reproducibility: In Table 1 — is there an explanation for why NMR performs so poorly in hopper-medium-replay-v2?

You show the agreement between humans and scripted feedback, but I am wondering about the inter-annotator agreement amongst the authors.  Also, how does the agreement change as the model improves?

How was the feedback signal formed from the human annotators?  Was it majority vote?  What about ties?

summary_of_the_review: Reward learning for preference-based RL is receiving increasing interest, so this is timely.  Transformers are a natural way to accommodate the problem of credit assignment over a trajectory.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors introduce a novel method for Preference-based Reinforcement Learning, which targets the problem of non-Markovian rewards. Namely, that human trajectory preferences cannot be assumed to be based on a sum of Markovian rewards, as expected in common RL and PbRL settings. The method learns non-Markovian rewards, based on the assumptions, that trajectory segment returns can be defined as weighted sum over history dependent state, action rewards. The learning algorithm uses a transformer architecture to these ends. Besides the novel PbRL algorithm, the authors contribute an empirical evaluation, demonstrating the advantages on 5 different domains (with 2-4 variants each).
strength_and_weaknesses: The consideration of non-Markovian trajectory preferences is an important issue in PbRL, because the Markov property is usually not assumed to hold in this setting. Therefore, non-Markovian PbRL is important to move from synthethic evaluations to real world applications. The paper is mostly well written, but some assumptions are hidden and some clarifications are required. (See next Section) The evaluation covers a sufficiently divers set of domains and real, human evaluations are usually to be preferred over pure synthethic trials. However, the evaluation is lacking in several parts:

- Evaluation is only done in terms of reward. However, user preference may not be aligned with the given reward function. In fact, if it is, the domain does not require a non-Markovian approach. Therefore, evaluation should be performed in terms of preference agreement of the resulting policy. Furthermore, the number of queries is usually also deemed an important metric, because human evaluations are costly.
- The scripted evaluation uses a Markovian reward, but the NMR and PT approaches are still able to outperform the MR variant in several domains. This needs to be better explain and evaluated. In fact, a non Markovian reward should be used for the evaluation.
- The RL algorithm used to maximize the reward assumes a common MDP structure. This means, it will likely not correctly consider the non-Markovian property or the weighting factors. Considering it works, it may be that the learned reward "collapses" to an Markovian variant (also see next Section concerning this issue). To resolve this issue, it should be validated that the learned reward cannot be approximated with a Markovian definition and/or a PO-MDP or N-state MDP RL algorithm should be used.


clarity,_quality,_novelty_and_reproducibility: As mentioned, the authors need to clarify some issues and make some assumptions explicit:
- $r_t$ is only defined inline in the text, which makes it hard to determine how many states are available for the reward approximation. To my understanding, 1 to H states are available, depending on the position of the state, action pair in the sequence?
- Compared trajectory segments always have same length (H), but trajectories can be longer than H? This would be a deviation from most other PbRL algorithms and poses some more questions. Namely, usually PbRL usually assumes that compared trajectories are starting in the same state. Otherwise, comparability may not be given, because the frame of reference differs (two different "problems").
- Due to the same length, weighted sum and weighted average are equivalent (up to a multiplicative factor)

These clarifications are important, because if my understanding is correct, and important issue may arise: For several state, action rewards, no history information is available. This means, the algorithm can only learn a conventional, Markovian reward. In case, very high numbers of preferences are used, this will affect many states, potentially leading to a Markovian solution in general. Therefore, it is a bit in question if the reason for the observed improvements are really due to the non-Markovian modelling. To resolve this issue, all rewards should have access to the same number history states. Formally, this would allow an N-state history MDP formalization.

The presentation is good, only colors in Fig.4 should not be reused (scripted, human). The algorithm is novel and interesting, especially the combination with a transformer-based model. Reproducibility should be possible, given source code an hyperparameters are given.

summary_of_the_review: The paper is a nice read and the method is novel and interesting. However, the stated issues with a main property of the algorithm prevent an accept. Although, to be fair, the authors claims are not directly related to the property and only refer to "real human preferences" and are therefore mostly substantiated. Some issues concerning the evaluation metrics and clarity remain, but can likely be resolved without substantial time effort.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents Preference Transformer that uses a transformer architecture to model human preferences in preference-based reinforcement learning. 

After author rebuttal, it received scores of 5668. The reviewer who gave a score of 5 also clarified that he/she will not argue for rejection. All the reviewers agree that the proposed model for prefrence-based RL is novel, the empirical performance is good, and the presentation is clear. On the other hand, some more clarification and explanations on experimental results are needed, e.g., why PT and NMR outperform MR in scripted teacher settings.

Overall, the AC thinks that the proposed method in this paper is novel and interesting, therefore, recommends acceptance of the paper.
justification_for_why_not_higher_score: The idea presented in this paper is novel and interesting, however, some more clarification and explanations are needed to make the paper in a better shape, so the AC do not think it can be leveled up to a Spotlight paper. 
justification_for_why_not_lower_score: All the reviewers are positive about the paper. The method presented is novel, and one reviewer also commented that the contribution is timely, the reviewer who gave a score of 5 also agrees to converging to positive outcome, so the AC decides to recommend acceptance.
Decision: Accept: poster