Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper investigates the properties of decision transformer models trained on multiple Atari games using off-line RL. The model is found to be superior to multi-game models using other approaches, for example online non-transformer models or transformer models trained with behavioral cloning. The performance gap between single-game trained models vs. multi-game trained models, observed in some earlier works, was found in this study as well. 
strengths_and_weaknesses: Strengths: 

The experiments here are interesting and insightful, likely to be of general interest to the ML community. The work also seems technically sound.   

Weaknesses:

Please see the limitations section below.

questions: Please see the limitations section below.
limitations: I understand that these are ultimately empirical results, but the authors could perhaps do a better job explaining why these observed results might make sense: for example, why online C51 doesn’t seem to do as well as the offline DT or why non-transformer models in general don’t seem to do as well as transformer models. The lack of any scaling with model size for CQL in Fig 5 (in fact inverse scaling with model size in Fig 5b!) also seems surprising a priori, for example. What is the hypothesized explanation for this?  

In Figure 5a, the decision transformer is claimed to exhibit power law scaling with model size, but this doesn’t seem correct. First of all, it's not really possible to conclude a power law or anything like that from three data points only with any degree of confidence. Secondly, the figure shows a roughly linear increase in performance for the decision transformer in a semilogx plot. This corresponds to a logarithmic improvement in performance with model size (in fact it seems like the scaling on a linear-linear plot will be worse than logarithmic because the trend seems slightly slower than linear in the semilogx plot in Fig 5a). A power law would be linear in a log-log plot.

Also relatedly, were separate hyperparameter searches performed for different model sizes in the scaling experiments in Figure 5 and was the amount of compute (FLOPs) controlled between different model sizes (need to train smaller models for a larger number of iterations)? If not why not? If not, wouldn't that possibly underestimate the performance of the smaller size models in these experiments, as demonstrated, for example, in the new scaling laws paper by Hoffmann et al. (2022)?

In section 4.6, the best model rollouts are compared against the best training data. But, I’m really not sure if this is a fair comparison. With enough samples, one is essentially guaranteed to surpass the top training data eventually. Shouldn’t the model be expected to do better on average than the expert rollouts in the training data ideally? The performance gap between single-game and multi-game models shown in Fig. 1 (lack of positive transfer between games on average), also observed in other works, suggests this is not really the case yet with multi-game models.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces the possibility of having a generalist RL agent that is based on the transformer. The paper demonstrate the effectiveness of a single RL agent, and that its scaling trend follows those of language and vision. The idea is validated on Atari games. 
strengths_and_weaknesses: Strengths:
The paper is easy to follow and the problem is well motivated, What really stands out the detailed description and the solid experiments performed. I think the results could have great impact on RL, and potentially making large transformer based offline RL training the new dominant paradigm for the field, especially with the scaling behavior.

Weakness
I guess the main weakness is the novelty since going from decision transformer to multitask decision transformer seems to be very straightforward. Nevertheless, I think this does not diminish the main value here.
questions: Is the result sensitive to the quantization scheme used in the paper?
limitations: No
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper empirically evaluates the ability of the recently proposed decision transformer model to generalize across Atari games. It also investigates other empirical questions (scaling behavior, transfer learning, baselines, etc.). The main empirical result is that the decision transformer model outperforms other generalist agents but loses to single-game agents. The paper demonstrates how an improved policy can be extracted using guided generation approaches similar to those used in language modeling and shows that scaling trends similar to those seen in vision and NLP hold for Atari games.

---------

UPDATE: I thank the authors for their detailed responses. After reading the other reviews and the responses, I've increased my score to "Accept".
strengths_and_weaknesses: Strengths
  + The paper investigates an important question (scaling behaviors of decision transformers) and conducts a detailed empirical investigation of the decision transformer in Atari games. The experiments are well constructed and nicely detailed.
  + The experimental results are novel (to my knowledge). The multi-game decision transformer seems to be the state-of-the-art in general game playing (although restricted to Atari games).
  + The paper is well written. It's easy to follow the main ideas and experiments. The paper also does a good job of placing its ideas into the larger body of work in sequence modeling, upside-down RL, transfer learning, etc.
  + There appears to be some algorithmic novelty in the guided action generation.

Weaknesses
  - As the paper mentions, it's unclear if these results hold outside of Atari games. However, evaluating this seems outside the scope of this paper.
  - I didn't fully understand the test-time action-selection described in Section 3.4. For example, what happens if P(expert|R) is very small? Full algorithmic pseudo-code might make it clearer.
  - It sounds like a single training run was used for each model. Is this correct? The answers in the checklist also mention error bars but I didn't see any in the figures. Also, the appendix mentions a single seed was used. Overall, it's unclear to me if these experiments are reproducible across seeds, hyper-params, etc. That said, given the large dataset and large computational cost, this may not be a major issue but perhaps this needs to be addressed somewhere in the paper.
  - Given that the main contributions of the paper are empirical, the above points make the overall impact of this paper somewhat limited. I'd be inclined to revise my opinion based on the authors response to the questions below.
questions: 1. How does the test-time action selection work exactly? In particular, what happens if the P(expert|R) is very small? Can the full test-time procedure be included in the paper or appendix?

2. How sensitive are the empirical results to different seeds, hyper-params, etc?
limitations: The authors have described the main limitations. Based on the response to a question above, it might need to be expanded to include sensitivity wrt seeds, hyper-params, etc.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes training a single model on a lot of games, combining the RL approach with the same token prediction approach we've seen enable the recent very capable LLM results. The model they train does very well on the suite of Atari games and lets us ask questions like:

1. What is learned from playing all of those games? 
2. Do the trends in vision and language hold for generalist RL agents?
3. Does this next-step tokenization prediction approach work for these games?
4. How do different online and offline methods work in this multi-game etup?
5. Does this idea of expert action inference, which comes from decomposing the next-step tokenization to explicitly model the expert prediction, improve upon BC?

The model is trained on trajectories collected in prior work and includes both expert and non-expert behavior. There are many baseline methods that they compare against. All in all, this is an extensive work which demonstrates that, while the model does not beat single-game performance (as expected), it is competitive across the majority of games.
strengths_and_weaknesses: Originality:

This paper is not particularly original. The only part of the algorithm that could be original is the Expert Action Inference inspired by Krause et al.

Clarity:

It's very clear. There's no issues here.

nit - L249: "We discuss these results *in* the Appendix".

Quality:

The quality is high as well. This paper sets out to do a goal - figure out how to train a single agent to a high level on a lot of games - and does it. It then supports that finding with relevant questions about whether it is capable of learning more (transfer), how it compares with online/offline methods, how it scales with size, and and how optimal vs suboptimal actions influence the learning. These all make sense as experiments to do and bolster the argument.

Significance:

The paper is significant in so far as it sets a bar for what comes next. It's somewhat of a successor (although not multi-modal) to One Model to Rule Them All, which previously was significant in showing that we can do a lot more in a single model than previously thought. It's not significant in being world-shaking. I think of this as being important in so far as a group showing the rest of the world that it's possible.

questions: 1. What happens if you do actually filter for only expert level trajectories? I get that you designed a policy that can skirt around this issue, but I think this would be good for understanding what happens when we use human experts as the behaviors to clone. That is largely our eventual purpose, so would be helpful to do that as a follow up. I suspect it wouldn't be that much extra work either on this paper ;).

2. Where is the Appendix? Some things are listed as being there, but I don't see it attached, nor is there a Supplemental... I might be being blind to this but other papers included their appendix explicitly in the paper.
limitations: The authors are honest about their limitations in the blurb at the end.
review_title: null

Meta-review: metareview: This paper demonstrates the generalization abilities of decision transformers relative to other approaches in multi-task settings. The reviewers found the topic interesting and the results compelling, unanimously supporting its inclusion in the conference programme. Accept.
confidence: Certain
Decision: Accept