Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper attempts to reformulate RL as a sequence modeling problem and hence exploit the recent advances in sequence modeling like Transformers to design an RL agent. Authors show that with little modification to existing sequence to sequence models and beam search techniques, they can perform well in imitation learning, goal-conditioned RL, and offline RL.
main_review: While this is an interesting research direction, I have several major concerns about this work.

1. I do not think that throwing away all the dynamic programming techniques and replacing them with Transformers will help us solve RL. While I disagree with this idea, my decision is not based on the fact that I do not like this approach.
2. I feel like authors do not highlight the limitations of this simple sequence to sequence approach well enough. Limitations mentioned in section 5 are not serious limitations! Would this approach work when there is stochasticity? Non-stationarity? More diverse tasks? This simple approach would suffer a lot in complex environments and this has to be made clear. While it is ok for having these limitations, it is crucial for this paper to have an explicit limitations section so that it does not mislead the community.
3. Why did the authors decide to go with the proposed discretization approach? It would be nice to see the performance of some baseline representations to understand the significance of this representation.
4. Line 133 -> The maximum tokens in 512. What are the values of N and M? How many transitions can you keep in the history if your maximum token length is 512? This is very useful information to know.
5. Again, Figure-1 experiments are very misleading. You are comparing Transformers models with PETS. PETS uses simple feedforward networks. I would like to see the comparison between Transformers and PlaNet (Hafner et al 2019).
6. Figure 3 - Are these the only two types of attention you have seen in the trained model? How much cherry-picking happened here? Please explain the selection process here.
7. How is this work different from [2]? It is not clear whether the performance trends that authors show here would be the same if they use proper recurrent models in their baselines as well.


Minor comments:

1. Figure 2 and Figure 4 - Please specify the number of runs.
2. Figure 4 - What is the y axis?

References:

[1] Learning Latent Dynamics for Planning from Pixels. Hafner et al 2019
[2] Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions. Schmidhuber 2019

limitations_and_societal_impact: Check the main review.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In a departure from traditional RL approaches, the paper applies transformer-based sequence modeling (with beam search) to a variety of RL tasks. Evaluations show the proposed Trajectory Transformer (TTO) to be competitive with baselines on three domains.
main_review: - Originality

The proposed approach is novel and well-motivated, given the successes of transformers in sequence modeling.


- Quality

The experimental results indicate that TTO can be usefully applied to at least a few, somewhat peripheral (non-online) tasks that fall within the broad collection of work associated with RL. However, the paper is worded in a way that is easy to interpret too broadly, starting with the catchy title, which could make many readers think of online RL, since that is the core of the field. Some readers may not even realize that online RL is excluded by the final sentence of the abstract, since imitation learning and offline RL are common pretraining steps performed prior to online RL training, and since goal conditioning often appears in the online RL setting. Expectations could be better managed by making the title less all-inclusive. For instance, “**Non-Online** Reinforcement Learning as One Big Sequence Modeling Problem”. (I’m sure the authors could think of something better than I just did.) And line 308 in the conclusion could also be more carefully qualified, as “**certain** reinforcement learning problems”.

Any reader who initially assumes that this work reduces online RL to sequence modeling, freeing them from the tyranny of estimating cumulative reward, will be disappointed by the fine print that follows. As explained in section 3.2, this paper’s imitation learning task is not of the usual kind, and the goal-conditioned task is outside the online setting. Line 86 says “Our model treats states, actions, and rewards interchangeably”, but this is not quite true for the offline RL task, where reward-to-go is used as the beam search heuristic. Line 193 argues that these reward estimates need not be particularly accurate, but why assume that online RL requires more accurate value estimates than beam search does?

I don’t believe that a bold new RL approach such as sequence modeling needs to achieve SOTA or even be applicable to the most common RL problems in order to merit publication. But I do believe the claims should be more explicitly scoped.


- Clarity

Apart from needing greater clarity in its claims, the paper is generally clear and well-written. I see a few other areas for improvement.

Line 107 says that continuous states and actions were discretized “for processing by a discrete-token architecture”, leaving readers to assume that the transformer architecture itself is being discussed. But transformers have already been successfully applied to online RL in a variety of ways, using both continuous and discrete observations and/or actions. So it would help to explain more directly that the discrete-token architecture chosen for this work was minGPT along with its discrete token embedding and decoding machinery.

This work replaces some of the traditional RL machinery with beam search, which is a tricky tool with its own practical limitations that deserve much greater attention for the benefit of potential users in the RL community. Algorithm 1 describes beam search at a certain level, but the algorithm box or accompanying text should include basic explanations of the individual symbols and how they are to be used in an RL task. In addition, the critical issues like beam size, branching factor, search depth, and heuristic pruning factors need to be given concrete numbers in the discussion of each task. Many of these details could be relegated to an appendix, but they are as important as any hyperparameter value. 

What is the interpretation of the vertical axis in Figure 4 and Figure 6?

The binary footprint of Figure 5 should be compressed. It dramatically slows down my browser every time I try to scroll past it.


- Significance

This is an interesting new approach that may deliver benefits in certain types of RL problems, and may even lead to useful online techniques.


- Conclusion

Given the shortcomings described above, I cannot support publication of the paper in its current form. But I believe the work makes important contributions which deserve a wide audience, so I hope the paper’s claims can be clarified so as not to mislead over-anxious readers.

- POST-RESPONSES UPDATE

The authors have addressed all of my reservations. With these revisions, the paper seems properly scoped and easier to understand. I have raised my score by 2 points.
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a sequence modeling approach for reinforcement learning (RL) problems. This proposed approaches enables to tackle diverse range of problem settings and unifies many of the standard components RL algorithms (e.g. as policies, models, and value functions) under a single sequence model. The authors present empirical results that illustrate the effectiveness of their model on imitation learning, goal-reaching, and offline reinforcement learning problems.
main_review: *Originality*: In my view, the usage of contemporary sequence modeling toolbox as a for sequential decision-making tasks is sufficiently novel. It provides new prospective on how to make use of highly successful transformer archtecture in various contexts and get good results. The related work adequately cited?

*Quality*: The submission is technically sound. Claims are supported by the large number of experimental results. The visualisations and qualitative examples are also assisting the reader.

*Clarity*: The manuscript is clear, well organized and easy to follow. 

*Significance*: In my view, these are interesting results that the community can learn from and build upon. 

### Cons

- The authors admit that predictions with Transformers is slower and more resource intensive. This is expected. However, I think the reader would benefit from knowing how much slower TTO is compared to baselines. Hence I ask the authors to add that information for their experiments (imitation learning, goal-conditioned RL, and offline RL).
- I'd be curious to see if goal-reaching is possible in more challenging domains compared to 4 rooms. Perhaps in procedurally generated mazes that change at every episode (e.g. MiniGrid).

# Post rebuttal update

I thanks the authors for their response and for running additional experiments using MiniGrid. I've also read the reviews by my fellow reviewers and the authors rebuttal to them (including many additional experiments). Therefore, I have increased my score by 1.
limitations_and_societal_impact: Yes. Several limitations of the work are described in the Discussion section. There are no ethical concerns, as far as I'm concerned.

See above for suggestions.
review_title: null

Meta-review: The reviews were thorough and there was good interaction with the authors. 

I'd like to suggest that the authors may have come off as overly aggressive in their pushing for additional reviewer interaction, especially when the interaction was already quite good. This aggressiveness may backfire sometime in the future. 

That said, the paper does seem to make a reasonable contribution and the delta between what is promised and the original submission is not overly large. 
Decision: Accept (Spotlight)