Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: * This paper introduces Decision Transfomer: a transformer based model that treats RL as a simple sequence modelling problem, and thus is able to eschew dynamic programming bootstrapping involved in contemporary RL algorithms.
* The experiments on Atari and gym environments show that by using an appropriate return as “prompt”, DTs can model the desired behavior, including perform comparably with state-of-the-art offline RL algorithms. 

main_review: This paper proposes a really compelling idea: Can we treat Reinforcement Learning as just another sequence modelling problem? Doing so allows us to re-use all the machinery already developed for modelling sequences in NLP (such as transformers), and removes the need to carefully design RL algorithms. 

The core idea is to let transformers model trajectories in an environment where each trajectory is a sequence of returns, states and actions. Interestingly, only predicting actions is found to be necessary (as opposed to predicting other tokens of states and returns as well). Modelling the full trajectory could be a nice direction for future work: something like predicting discretized embeddings of states using a VQ-VAE could work well.  Either way, the idea is indeed simple in hindsight but quite effective: a hallmark of influential papers. 

The writing is exceptionally clear throughout, and the authors have put in a good amount work to make the exposition as clear as possible. Figure 1 succinctly describes the idea, and Figure 2 along with the illustrative example does a great job of walking the reader through how the idea would work in a toy setting. 

The experiments on Atari are on only 4 games, it would be great if future versions of the paper can expand on this. Some aspects of the experiments do seem problematic:

* **Per-game hyper-parameter tuning**: the context length and desired return ratio are tuned individually for each environment. On Atari, this is not kosher, and the norm is to use a single set of hyper-parameters for all games. The goal of-course is not to build specialized agents for each Atari game, but rather a single algorithm that can master all games, and per-game hyperparam tuning goes against this ethos. This makes comparisons to the baselines unfair / uncontrolled, since I don’t think CQL / REM did per-game hyper-param tuning. 
* **Problematic model selection**: It’s not mentioned in the paper, but I imagine the way hyperparams were chosen was by doing *online* policy evaluation. This violates the offline learning assumption as assuming access to online evauluation is often not realistic. [1] and [2] point out these issues and recommend evaluation protocols, so I encourage the authors to show results with offline selection as well. 

Minor:
* The introduction should mention upside-down RL since its very similar in similar in spirit, instead of delegating it as another entry in the related work. DTs are a natural evolution of the initial idea proposed in upside-down RL, so it deserves a more central discussion.
* line 109 should say section 5.4 instead of section 5.3

The analysis section is quite thorough and well done, and answers a lot of questions the reader might have about DTs. The sections on modelling diverse returns and sparse rewards are quite informative. 

Overall, I think this paper is great fresh new perspective on RL and will undoubtedly be influential. There are some problems with the experimental evaluation in the current version, but nothing that can’t be fixed. I am therefore evaluating assuming these issues can be ironed out. 

[1] Gulcehre, C., Wang, Z., Novikov, A., Le Paine, T., Gomez Colmenarejo, S., Zolna, K., ... & de Freitas, N. (2020). Rl unplugged: Benchmarks for offline reinforcement learning. NeurIPS 2020

[2] Paine, Tom Le, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. "Hyperparameter selection for offline reinforcement learning." arXiv preprint arXiv:2007.09055 (2020).
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper investigates the power of Transformers in sequential decision-making tasks. The authors suggest to use trajectory modeling in contrast to using Transformers as a model architecture in conventional RL algorithms. The proposed algorithm is called decision transformer and outperforms model-free offline RL algorithms in tasks requiring long-term credit assignment. 
main_review: I enjoyed reading this paper. The paper is beautifully organized and very well written. Decision Transformer is well motivated and positioned. 

The only concern that I have which I think is extremely important is the following: 

The authors did not compare Decision Transformers to other related works where Transformers are used as a model architecture within an existing RL method. For instance, the authors did in fact mention (very briefly) the differences of their method compared to reference 43 of the paper in the related works section, but there is no experiment comparing the tradeoff between “Transformers as a model architecture in RL” vs “the paradigm shift proposed here”. Could you please comment? The paper can benefit significantly from an additional experiment here.

limitations_and_societal_impact: The authors discussed partially the limitations of their approach.
I do not see any potential negative societal impact from this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a sequence-model approach for offline RL, which does not need to solve cumbersome dynamic programming subproblems. It utilizes a transformer architecture to effectively model state-action sequence which is conditional to the return-to-go. The proposed method shows competitive or superior performance compared to the previous state-of-the-art in various offline RL datasets.
main_review: Pros
- The paper provides a novel and valuable direction to solve RL problems without TD learning which have been fundamental to conventional RL algorithms despite being hard to optimize.
- Transformers have shown remarkable performance across NLP, vision, and GNN. This work introduces a nice way to integrate this architecture to the RL domain.
- The proposed method performs exceptionally well on long-term credit assignments, which seems plausible considering the origin of the transformer architecture.

Cons
- Based on the comparisons with %BC, it seems the proposed method is just a better (or more sample-efficient) way to perform behavior cloning. Can the authors provide some evidence that DT is more than a BC? For example, can the method discover behaviors not directly included in the dataset?
- The authors argue that %BC is not a realistic approach as it needs online evaluations to choose the optimal subset size. However, this goes the same with DT as it needs to tune its hyperparameters ($K$, return-to-go). The appendix shows that the optimal hyperparameters vary between environments.
- The paper does not list the hyperparameter search procedure (search range, selection criteria) at all, which is especially important for offline RL as online evaluations are (conceptually) impossible.
- Sensitivity to the hyperparameters ($K$, return-to-go conditioning) is not given.
- Std values for %BC results are missing in Table 3.
- Exploration is a crucial part for RL algorithms. However, it is unclear how to expand this work to perform explorations with DT.

Other comments
- Does the method fail when it is given an extremely OOD return-to-go value (e.g., $\times 0$, $\times 5$, $\times 10$, ...) ?
- How does the method perform on other D4RL datasets such as *-random and *-full-replay?
- The algorithm is explained only in high-level phrases. Including a pseudo-code of the training and the inference procedure may help the readers understand the method concretely.

========== Update ==========

The author response addresses most of my concerns, and I think the updated manuscript will be valuable to the machine learning community. I am increasing my rating to 6.
limitations_and_societal_impact: Limitations
- Noted above

Societal Impact
- N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper uses the Transformer model as an auto-regressive model to predict the actions (decisions) givens the state inputs, past actions, and more importantly, the return-to-go. Training such a policy becomes a supervised-learning problem, similar to behavior cloning. The paper applies this method to offline RL problems and demonstrates that policy trained in this supervised manner achieves competitive results compared to SOTA model-free offline RL algorithms.
main_review: This paper is a combination of existing techniques (Transformer + Upside-down RL) and shows that this combination yields better or comparable performance compared to BC and some SOTA offline RL algorithms.

Strengths:
* The method is clearly explained. The paper proposed a simple method to improve behavior cloning performance by conditioning the policy on the return-to-go. Based on the paper writing, one can easily implement the method.

* I like the idea of conditioning the policy on the return-to-go value. In behavior cloning, typically the policy only depends on the states (sometimes also the previous actions). If the training dataset contains multiple state-action pairs on the same state but with different actions, then the policy learning can be hurt as it cannot capture multi-modality. The paper proposes to condition the policy with the return-to-go value as well. With the return value, now the model can have some capacity to fit multiple state-action pairs in the same state because it is very likely that these state-action pairs will have different return values even though the states are the same. This technique can be useful when there is multiple action supervision in the same state or when there is suboptimal action supervision. Note that as the authors also point out in the related work, there are prior works that use this idea such as the Upside-down RL paper.

* The paper does a thorough comparison with SOTA offline RL methods and provides experiments on several environments.


Weaknesses:

* I cannot get why %BC is not a realistic approach. A better explanation would be helpful. In Decision Transformer (DT), one would need to know the return-to-go value to deploy the policy as it's part of the policy input. Then BC can also use this value to filter out the good trajectories on which BC is performed. So it seems that both methods can have the same assumption. If the authors mean that it requires policy deployment to determine whether to use 10%, 25%, 40%, or 100% of the training data, then one can use a threshold return value to filter out the trajectories. So instead of showing the BC performance with %BC, one can show the BC performance trained with datasets that are filtered with different threshold return values.

* This paper seems to over-emphasize the role of the Transformer. From my understanding, conditioning the policy on the return-to-go value seems to be the key to why it can outperform BC. If the authors think that Transformer is the key, then it is necessary to show this with a baseline comparison on LSTM/GRU policies that have the same policy input as the ones used in the paper.

* I would like to see more analysis on why the method works. While there are many experiments showing the performance of DT, little analysis is actually trying to explain why DT works better than BC. Section 5.1 gives some analysis that BC trained with top $X$% of data has similar performance with DT. This could be that DT learns to do BC only on good trajectories and ignore the rest, or DT is able to fit all the trajectories by conditioning the policy on return-to-go value which can tell how good the action is. Also, Is it because DT has the capacity to better deal with multi-modality via the return-to-go policy input? 

* An ablation study on the value of the sequence length $K$ is necessary to understand how important this hyperparameter is and is it very sensitive to each game.

* If it is the case that policy in DT can fit all trajectories because the return-to-go value can help the policy distinguish different action supervision even if the state input is very similar, then I would like to see experiments on binary sparse-reward environments. In these environments, the return values for all the successful episodes will be the same (e.g., $+1$) and the return values for all failed episodes will also be the same (e.g., $0$). How does DT perform in these sparse environments?

* The details of the Key-to-Door environments are missing.

limitations_and_societal_impact: The paper did not state clearly the limitations of the proposed method. Possible limitations can be on the type of environments where the method can work, the episode length constraint due to the use of attention module, the sensitivity of the hyperparameters such as the training sequence length $K$, etc.
review_title: null

Meta-review: After an involved discussion on several topics regarding experimental practices and relation to prior art, the reviewers have settled on a range of positive scores. The cited strengths of the paper include the potential impact of introducing modern sequence modeling tools for RL problems, both the offline-RL domain in the paper's experiments and potentially more areas in the future. I personally found several areas where further technical detail and discussion would be desireable in the final version:
A) Expand your analysis of the method's limitations. Note the requests of reviewer dbvQ, but additionally, consider the balance and fairness in your text between identifying the "deadly triad" of deep TD compared to your speculations on the peformance of transformers for the task. The empirical comparison shows DT is strong but not overwhelming and therefore we may find several of the same, or new, problems specific to modeling the relations of reward/return/state/actions over time are present. Help the reader to understand the limits you expect for your method and the community to know when to use this and where to push new methods next. 
B) While the training procedure is well documented, some reviewers mentioned the lack of detail about using your model at test/inference time. Even in offline RL benchmarks, one must execute evaluation rollouts where novel states can be encountered and the future return is not available. I'd like to see a full description of the algorithm to perform these rollouts with your model (perhaps in picture form analagous to Fig 1).
 
Decision: Accept (Poster)