Multi-Game Decision Transformers
Authors: Kuang-Huei Lee
Ofir Nachum
Sherry Yang
Lisa Lee
C. Daniel Freeman
Sergio Guadarrama
Ian Fischer
Winnie Xu
Eric Jang
Henryk Michalewski
Igor Mordatch
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
Generalist Agent
Multi-Environment RL
Upside-Down RL
Decision Transformers

Abstract: A longstanding goal of the field of AI is a method for learning a highly capable, generalist agent from diverse experience. In the subfields of vision and language, this was largely achieved by scaling up transformer-based models and training them on large, diverse datasets. Motivated by this progress, we investigate whether the same strategy can be used to produce generalist reinforcement learning agents. Specifically, we show that a single transformer-based model – with a single set of weights – trained purely offline can play a suite of up to 46 Atari games simultaneously at close-to-human performance. When trained and evaluated appropriately, we find that the same trends observed in language and vision hold, including scaling of performance with model size and rapid adaptation to new games via fine-tuning. We compare several approaches in this multi-game setting, such as online and offline RL methods and behavioral cloning, and find that our Multi-Game Decision Transformer models offer the best scalability and performance. We release the pre-trained models and code to encourage further research in this direction.

Multi-Game Decision Transformers
Kuang-Huei Lee"
Ofir Nachum
Mengjiao Yang
Lisa Lee
Daniel Freeman
Winnie Xu
Sergio Guadarrama
lan Fischer
Eric Jang
Henryk Michalewski
Igor Mordatch
Google Research
Abstract
A longstanding goal of the field of Al a method for learning a highly capable, generalist agent from diverse experience. In the subfields of vision and language, this was largely achieved by scaling up transformer-basee models and training them on large, diverse datasets. Motivated by this progress, we investigate whether the same strategy can be used to produce generalist reinforcement learning agents. Specifically, we show that a single transformer- based model with a single of weights trained purely offline can play suite of up to 46 Atari games simultane- ously at
performance. When trained and evaluated appropriately, we find that the same trends observed in language and vision hold, including scaling of performance with model size and rapid adaptation to new games via fine-tuning. We compare several approaches in this multi-game setting, such as online and offline RI. methods and behavioral cloning, and find that our Multi-Game Decision Transformer models offer the best scalability and performance. We release the pre-trained models and code to encourage further research in this direction]
Introduction
Building large-scale generalist models that solve many tasks by training massive task-agnostic datasets has emerged as a dominant approach in natural language processing [18] 12], computer vision 199 (). and their intersection [61 []. These models can adapt to tasks (such as translation
663
78) ). make use of unrelated data (such as using high-resource language to improve translations of incorporate
modalities by projecting images into language space [6)] 75 The success of these methods largely derives from a combination of scalable model architectures (77). abundance of unlabeled task-agnostic data, and continuous improvements in high performance computing infrastructure. Crucially, scaling laws 38 31 indicate that performance gains due scale have not yet reached a saturation point.
low-resource languages (17),
In this work, we argue that similar progression is possible in the field reinforcement learning, and take initial steps toward scalable methods that produce highly capable generalist agents. In contrast to vision and language domains, reinforcement learning has seen advocacy for the use smaller models [6 49 8 and is usually either used solve single tasks, or multiple tasks within the same environment. Importantly, training across multiple environments with very different dynamics, rewards, visuals, and agent embodiments has been studied less significantly.
'Equal contribution. [leekh, ofirnachum, imordatch] @google.oooooooooooooooooooom.lo com Additional information, vidcos and code be seen /sstes.googll.cooooo/ooooo www.:ggoooloccccooe. http:///wwwwwiiiiggamrrrnnnnnnno
sites,
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

144%
Sonan-Nomaabed IMM Soore
185%
126%
100%
65%
0%
Single-Gome DGN (Onlnn)
Single-Game BCQ (Oiline)*
Mutt-Game Decision
Mult Game BC Transformer (Oflline
Mult-Game Game DON Impota (Orline)
Mubs Game COL Impala (Offline)
Transformer (Ofline)
Figure I: Aggregates www.auumuunania normalized scores catter-uarttll Mean) across 41 Atari games. Grey bars are single-game specialist models while blue are generalists. Single-game BCQ (2] results are from Gulcehre et [25]. Multi-game models are all trained on dataset [ with inter-quartile mean noman-normalized score of 101 %, which Multi-Game DT notably exceeds.
Specifically, we investigate whether single model with single set of parameters can be trained to act in multiple environments from large amounts of expert and non-expert experience. We consider training on a suite of 41 Atari games [9] 25 for their diversity, informally asking "Can models learn something universal from playing many video games?". To train this model, we use only the previously-com collecced trajectories from Agarwal et al. but we evaluate our agent interactively. We striving for mastery efficiency that game-specine agents offer, still in early stages this research agenda. Rather, we investigate whether the same trends observed in language and vision hold for large-scale generalist reinforcement learning agents.
believe
We find that we can train a single agent that achieves 126% of human-level performance simul- taneously across all games after training offline expert and non-expert datasets (see Figuree T). Furthermore, we see similar trends that mirror those observed language and vision: rapid fine- tuning to never-before-See games with very little data (Section 4.5), scaling relationship between performance and model size (Section 4.4), and faster training progress for larger models (Appendix G).
Notably, not all existing approaches to multi-environmenn training work well. We investigate several approaches, including treating the problem as offline decision transformer-base sequence modeling [14 35,, online RL [53]. offline temporal difference methods (42), contrastive representations [56],
and behavior cloning (60) We find that decision transformer based models offer the best performance and scaling properties in the multi-envine multi-environment regime. However, to permit training on both expert and non-expert trajectories, we find is necessary to use guided generation technique from language modeling to generate expert-level actions, which is important departure from standard decision transformers.
Our contributions are threefold: First, we show that it is possible to train single high-performing generalist agent to act across multiple environments from offline data alone. Second, we show that scaling trends observed in language and vision hold. And third, we compare multiple approaches for achieving this goal, finding that decision transformers combined with guided generation perform the best. It is our hope this study can inspire further research in generalist agents. To aid this, we make our pre-trained models and code publicly available.
pre-train on non-expert to expert trajectories
expert play on all training games
Multi-Game
Decision
Transformer
fine-tune games
Figure 2: An overview of the training and evaluation setup. We observe expert-level game-play the interactive setting after offline learning from trajectories ranging from beginner expert.

---

2 Related Work
generalist agent for solving variety of environments has been goal for artificial intelligence (AI) researchers since the inception of Al as a field of study [50]. This same reason motivated the introduction of the Atari suite (the Arcade Learning Environment, or ALE) as a testbed for learning algorithms [00;; in their own words, the ALE is for "empirically assessing agents designed for general competency." While the celebrated deep Q-learning [52] and actor critic [54] agents were among the first to use a single algorithm for all games, they nevertheless required separate training and hyperparameters for each game agent. Later works have demonstrated the ability to learn single neural network agent on multiple Atari games simultaneously. either online (20) or via policy distillation [59] 67]. The aim of our work similar to learn single agent for playing multiple Atari games with a focus offline learning. We demonstrate results with human-level competency on up to 46 games, which is unseen in the literature.
A closely related setting is learning to solve multiple tasks within the same or similar environments. For example in the robotics field, existing works propose to use language-conditing conditioned tasks 48
34
while others posit goal-reaching as a way to learn general skills [5]]. among other proposals [7 82]
In this work, we tackle the problem of learning to act in a large collection of environments with distinctively different dynamics, rewards, and agent embodiments. This complicated but important setting requires a different type of generalization that has been studied significantly less.
concurrent work (65] also aims to train a wransformer-based generalist agent based on offiie data including for the ALE. This work differs from ours in that the offline training data is exclusively near-optimal and it requires prompting by expert trajectories at inference time, In contrast, we extend decision transformers 14 from the Upside-Down RL family [71 68 learn from a diverse dataset (expert and non-ex non-expert data), predict returns, and pick optimality-conditonne conditioed returns. Furthermore, we provide comparisons against existing behavioral cloning, online and offline RL methods, and contrastive representations [80] [36]. Other works that also consider LLM-like sequence modeling for
a variety of single control tasks include 66 84
57
3 Method
We consider a decision -aaking agent that at every time t receives an observation of the world o', chooses an action a', and receives a scalar reward r², Our goal to learn a single optimal policy distribution P7 (a")osSt "aatstaat,aa (,tt,, with parameters 0 that maximizes the agent's total future return R = =k> r on all the environments we consider.
3.1 Reinforcement Learning as Sequence Modeling
Following 14]
pose the problem of offline reinforcement learning a sequence modeling problem where we model the probability of the next sequence token i; conditioned on all tokens
prior to it: (a (z))), similar to contemporary decoder-only sequence models 12 15 62) sequences we consider have the form:
The
* (f.,,o, ....,. o' ft',
where represents time-step, M is the number of image patches per observation (which we further discuss in Section 3.2). and R' is the agent's 's target return for the rest of the sequence. Such sequence order respects the causal structure of the environment decision process. Figure 3 presents an overview of model architecture.
Returns, actions, and rewards are tokenized (See Section 3.2 for details), and we train the model to predict the next return, action, and reward discrete token in sequence via standard cross-entropy loss. The sequence consider is different from Chen et al. [4], which has (..., Rtf o', a' ...). Our design allows predicting the return distribution and sampling from it, instead of relying on a user to manually select an expert-level return at inference time
3.4).
Predicting future value and rewards have been shown to be useful objectives for learning better representations in artificial reinforcement learning agents [47] 69 44 and important signals for representation learning in humans [5]. Thus, while we may not directly use all of the predicted quantities, the task of predicting them encourages structure and representation learning of our
3

---

R
all our experiments as we find it covers most of the causal transformer decoder
0
R
a
I
Σ r
-1
0
+1
k>t
observation
return
action
reward
Figure 3: An overview of our decision transformer architecture.
environments. In this work, we do not attempt to predict future observations due to their non-discrete nature and the additional model capacity that would be required generate images. However, building mrage-based forward prediction models of the environment has been shown to be a useful We leave for future investigation.
representation objective for RL 228 27 29
3.2 Tokenization
To generate returns, actions, and rewards via multinomial distributions similarly to language gen- eration, we convert these quantities to discrete tokens. Actions a are already discrete quantities in the environments consider. We convert scalar rewards ternary quantities {-1, 0, +1}, and uniformly quantize returns into a discrete range shared by all our environments
Inspired by the simplicity and effectiveness of transformer architectures for processing images [19], we divide each observation image into collection of M patches (see Figure 3). Each patch is additively combined with trainable position encoding and linearly projected into the input token embedding space. We experimented with using image tokenizations coming from a convolutional network, but did not find it to have significant benefit and omitted it for simplicity.
We chose our tokenization scheme with simplicity in mind, but many other schemes are possible. While all our environments use shared action space, varying action spaces when controlling different agent morphologies can still tokenized using methods of (33 43 26 And while we used uniform quantization discretize continuous quantities, more sophisticated methods such as VQ- TQ-VAE (76 can be used learn more effective discretizations.
3.3 Training Dataset
To train the model, we use an existing dataset of Atari trajectories (with quantized returns) introduced in [1] The dataset contains trajectories collected from the training progress of a DQN agent [53]. Following [25]. we select 46 games where DQN significantly outperforms a random agent. 41 games are used for training and games are held out for out-of-distributio generalization experiments.
We chose 5 held-out games representing different categories including Alien and MsPacman (maze based), Pong (ball tracking), Space Invaders (shoot vertically), and StarGunner (shoot horizon- tally), to ensure out-of-distribution generalization can be evaluated different types of games.
For each of 41 games, we use data from 2 training runs, each containing roll-outs from 50 policy checkpoints, in turn each containing million environment steps. This totals 4.1 billion steps. Using the tokenization scheme in previous sections, the dataset contains almost 160 billion tokens.
As the dataset contains agent behaviors at all stages of learning, it contains both expert and non-expert behaviors. We do not perform any special filtering, curation, or balancing of the dataset. The
'The training datasets guantization, use range (-20, ..., 100J with returns we observe in the datasets.
SSccion 3.3) contains scalar reward values clipped to [-1, 1]. For retum
We use 6x6 patches, where each patch corresponds 14x14 pixels, in all experiments.

---

motivation to train such data instead of expert-only behaviors is twofold: Firstly, sub-optimal behaviors are more diverse than optimal behaviors and may still be useful for learning representations of the environment and consequences of poor decisions. Secondly, it may be difficult to create single binary criteria for optimality as it is typically a graded quantity. Thus, instead of assuming only task-relevant expert behaviors, we train our model on all available behaviors, yet generate expert behavior at inference time as described in the next section.
3.4 Expert Action Inference
As described above, our training datasets contain a mix of expert and non-expert behaviors, thus directly generating actions from the model imitating the data is unlikely to consistently produce expert behavior (as we confirm in [Section 4.7). Instead, we want to control action generation to consistently produce actions of highly-rewarding behavior. This mirrors the problem of discriminator guiidrd generation in language models, for which variety of methods have been proposed [40 79 58
We propose inference- time method inspired by [40 and assume binary classifier P(expert"]..) that identifies whether or not the behavior is expert-level before taking an action at time Following Bayes' rule, the distribution of expert-level returns time / is then:
P(expert' | [R....) x exp(x( Rtow)/ (Rnigh - RRow)
where Riow is the return lower bound and Rhigh is the return upper bound. Similarly to 70 73 74
39
we define binary classifier to be proportional to future return with inverse temperature +
P(expert' R ......) exp(xR')
This results a simple auto-regressive procedure where we first sample high-but-plausiblc target returns Rs according to log-probability log (o + k(R* Rtow)/ (Rnigh Rtow). and then sample actions according to Po(a') R² ...,.... See Figure 4 for an illustration of this procedure and Appendix B.3 for implementation details. can be seen variation of retum-conditioned policies 41 that automatically generates expert-level (but likely) returns at every timestep, instead of manually fixing them for the duration of the episode.
exp(R)
P(expert
P(
P(R expert ...)
R
P(a R
a
Figure 4: An illustration of our expert-level return and action sampling procedure. (R...... and Po(a) are the distributions learned by the sequence model.
Importantly, this formulation only affects the inference procedure of the model training is entirely unaffected and can rely on standard next-token prediction frameworks and infrastructure. While we chose this formulation for its simplicity, controllable generation is an active area study and expect other more effective methods to be introduced in the future. As such, contribution is to point out a connection between problems of controllable generation in language modeling and optimality conditioning in control.
4 Experiments
We formulate our experiments to answer a number of questions that are addressed in following sections:
How do different online and offline methods perform in the multi multi-game regime?
How do different methods scale with model size?
How effective are different methods at transfer to novel games?
We use 10 in all experiments.
5

---

Does multi-game decision transformer improve upon training data?
Does expert action inference (Section 3.4p improve upon behavioral cloning?
Does training on expert and non-expert data bring benefits over expert-only training?
We also consider whether there are benefits to specifically using the transformer architecture in Appendix DJ and qualitatively explore the attention behavior of these models in Appendix H.
4.1 Setup
Model Variants and Scaling. We base decision transformer (DT) configuration GPT-2 [2 as summarized in Appendix B.1] We report results for DT-200M (a Multi-Game DT with 200M parameters) if not specified otherwise. Other smaller variants are DT-40MM and DT-10M, We set sequence length to 4 game frames for all experiments, which results in sequences of 156 tokens.
Training and Fine-tuning. We train all Multi-Game DT models TPUv4 hardware and the Jaxline (Babuschkin et al. (7]) framework for 10M steps using the LAMB optimizer [8] with 3 10 learning rate, 4000 steps linear warm-up, no weight decay, gradient clip 1.0, B1 0.9 and 83 = 0.999, and batch size 2048. For fine-tuning on novel games, we train for 100k steps with
10 learning rate, 10 weight decay and batch size of 256 instead. Both regimes used image augmentations as described in Appenddx B.5
Metrics. We measure performance on individual Atari games by human normalized scores (HNS) 53 i.e. (score Mone-Saa scoreramkmm) (scoreeuumnn scoreramkm)), or DON-normalized scores, i.e. normalizing by the best DQN scores seen in the training dataset instead of using human scores. To create an aggregate comparison metric across all games, use inter-quartile mean (IQM) of human- normalized scores across all games, following evaluation best practices proposed in (2). Due to the prohibitively long training times, we only evaluated one training seed. We additionally report median aggregate metric in Appendix
4.2 Baseline Methods
BC Our Decision Transformer (Section 3.1 can be reduced to transformer-ban based Behavioral Cloning (BC) [60] agent by removing the target return condition and return token prediction. Similar to what we do for Decision Transformer, we also learn BC models at different scales (10M, 40M, 200M parameters) while keeping other configurations unchanged.
C51 DQN As point of comparison for online performance. we use the C51 algorithm []] which a variant of deep Q-learning (DQN) but with a categorical loss for minimizing the temporal difference (TD) errors. Following improvements suggested in Hessel et al. (30) well our empirical observations, we use multi-step learning with n = 4. For the single-game experiments, we use the standard convolutional neural network (CNN) used in the implementation of C51 [13] For the multi-game experiments, we modify the C51 implementation based on a hyperparameter search use Impala neural network architecture (20] with three blocks using 64, 128, and 128 channels respectively with a batch size of 128 and update period of 256.
CQL For an offline TD-based learning algorithm we use conservative Q-learnig (CQL) [42] Namely, we augment the categorical loss of C51 with a behavioral cloning loss minimizing log TQ (als). where (s, a) is state-action pair sampled from the offline dataset and лo (-(ss
softmax( Q(s, (s.,)), -). Following the recommendations in Kumar et al. [22 we weight the contribution of the BC loss by when using 100% of the offline data (multi-game training) and when using 1%
(single-gane -agee finetuning). For scaling experiments, we vary the number of blocks and channels in each block of the Impala: the number of blocks and channels one of (5 blocks, 128 channels) Rs 5M params, (10 blocks, 256 channels) x 30M params, (5 blocks, 512 channels) z 60M params, (10 blocks, 512 channels) 120M params.
CPC, BERT, and ACL For rapid adaptation to new games via fine-tuning, we consider representa- tion learning baselines including contrastive predictive coding (CPC) [56], BERT pretraining [88] and attentive contrastive learning (ACL) [80]. All state representation networks are implemented

---

multi-game online as additional multi- perceptrons (MLPs) or transformer layers on top of the Impala CNN used in C51 and CQL baselines. CPC uses additional MLP layers with 512 units each interleaved with ReLU activation to represent ((s), which is optimized by maximizing o(x) W (ss))) ) of true transitions (s, s') and minimizing ((s) W ((3) where š is a state randomly sampled from the batch (including states from other games). For BERT pretraining,
use self-attention layers with 4 attention heads of 256 units each and feed-for worward dimension 512, and train Φ(s) using BERT's masked self self- predictoo loss on a trajectory of sequence length 16. ACL shares the same model parametrization as BERT, with the inclusion of action prediction in the pretraining objective.
4.3
How do different online and offline methods perform in the multi-game regime?
We compare different online and offline algorithms in the multi-game regime and their single-game counterparts in Figure We find that single-game specialists are still most performant. Among multi- game generalist models, our Multi-Game Decision Transformer model comes closest to specialist performance. Multi-game online RI. with non-transformer models comes second, while we struggled to get good performance with offline non- transformerr models. We note that C51 DQN median score of 68% (see Appendixx E} which compares similarly to multi-game median Impala score of 70%, which we calculated from results reported by [20] for suite of games.
We believe the apparent advantage offline DT compared to online multi-game game methods like C51 may be explained in part through classical differences between online and offline settings in RL [45] Online methods must balance exploration with the ability learn and generalize from experience, which could be challenging in the multi-game setting, whereas offiine DT only needs to learn to distill and generalize from the fixed multi-game experience given (collected by specialist DQN agents [I). Beyond the difference between online and offinn,, one could also argue that C51 suffers from more training instability than DT due to the use of a temporal difference (TD) loss, which we discuss in the next paragraph.
4.4 How do different methods scale with model size?
In large language and vision models, lowest-achievable training loss typically decreases predictably with increasing model size. Kaplan et al. (38] demonstrated an empirical scaling relationship between the capacity of language model (NLP terminology for next-token autoregressive generative model) and its performance (negative log likelihood on held-out data). These trends were verified over many orders of magnitude of model size, ranging from few-million parameter models hundreds of billion parameter models.
Decision Transfomer
150%
100%
9-08
CQL (Impala)
Human-Normsiized ICM Soore
126%
Human-Normaled IQM Score
75%
728
Online CS1 DQN (Impala)
100%
93%
Behavioral Cloning Transformer
68%
50%
60%
50%
5665
5.%%
38%
38%
25%
26%
23%
0%
0%
6M
400
80M
200M
10M
40M
80M
200M
Number of Model Parameters
Number of Model Parameters
(a) Scaling of IQM scores training games with different model sizes and architectures.
(b) Scaling of IQM scores for all novel games after fine-tuning DT and CQL.
Figure 5: How model performance scales with model size, on training set games and novel games. (Impala) indicates using the Impala CNN architecture.
We investigate whether similar trends hold for interactive in in-game performance - not just training loss and show a similar performance scaling trend in Figure 5al Multi-Game Decision Transformer
7

---

performance reliably increases over more than an order of magnitude of parameter scaling, whereas the other methods either saturate, or have much slower performance growth.
In contrast, in Figures 5 and 5b, we find that CQL does not improve with increased model size, and actually shows a sharp drop in the performance of larger models the fine-tuning tasks. Temporal Difference (TD) methods suffer greater instability with larger model size in the multi-game setting, leading to this "inverse" scaling. Indeed, our attempts other objectives closer to pure TD (C51, DQN, DDQN) led to even worse results (which we do not report). We note that similar conclusions about instability with respect to network size have been made by other work [22]
We also find that larger models train faster, in the sense of reaching higher in in-game performance after observing the same number of tokens. We discuss these results in Appendix G
4.5 How effective are different methods at transfer to novel games?
Pretraining for rapid adaptation to new games has not been explored widely Atari games despite being a natural and well-motivated task due to relevance to how humans transfer knowledge to new games. Nachum and Yang (55] employed pretraining small expert data for Atari and compared to set of state representation learning objectives based on bisimulation 24 83,, but their pretraining and fine-tuning the same game. We are instead interested in the transfer ability of pretrained agents to new games.
large offline data and fine-tunining
We hence devise our own evaluation setup by pretraining DT, CQL, CPC, BERT, and ACL on the full datasets of the 41 training games with 100M steps each, and fine-tuning one model per held-out game using 1% (IM steps) from each game. The I 1% fine-tuning data is uniformly sampled from 50M step dataset without quality filtering. DT and CQL use the same objective for pretraining and fine-tuning, whereas CPC, BERT, and ACL each use their own pretraining objective and are fine-tuned using the BC objective. All methods are fine-tuned for 100,000 steps, which is much shorter than training any agent from scratch. We additionally include training CQL from scratch on the 1 % held-out data to highlight the benefit of rapid fine-tuning.
Pretraining with the DT objective performs the best across all games. All methods with pretraining outperform training CQL from scratch, which verifies our hypothesis that pretraining on other games should indeed help with rapid learning new game. CPC and BERT underperform DT, suggesting that learning state representations alone is not sufficient for desirable transfer performance. While ACL adds an action prediction auxiliary loss to BERT, showed little effect, suggesting that modeling the actions in the right way on the offline data is important for good transfer performance. Furthermore, we find that fine-tuning performance improves as the DT model becomes larger, while CQL fine-tuning performance is inconsistent with model size (sce Figure 5b).
Fine-tuning performance on the held-out games is shown in Figure 6
125%
CQL no pretraining
DON Now.Nozaazzdddee Score
100%
DT no pretraining
75%
BERT pretraining
ACL pretraining
50%
COL pretraining
25%
CPC pretraining
0%
Alien
Misaccaa
Pong
Spacelivvaerr
StarGunner
DT pretraining
Figure 6: Fine-tuning performance on 1% of 5 held-out games' data after pretraining on other 41 games using DT, CQL, CPC, BERT, and ACL. All pretraining methods outperform training CQL from scratch on the 1% held-out data, highlighting the transfer benefit of pretraining on other games. DT performs the best among all methods considered.
4.6 Does multi-game decision transformer improve upon training data?
We want to evaluate whether decision transformer with expert action inference is capable of acting better than the best demonstrations seen during training. To do this, we look at the top performing decision transformer model rollouts. We use top 3 rollouts instead of the mean across all rollouts
8

---

to more fairly compare to the best demonstration, rather than an average expert demonstration. We show percentage improvement over best demonstration score for individual games in Figure 7 We see significant improvement over the training data in number of games.
400%
Improvement over beel score delaast
300%
100%
0%
-100%
Adteni
Allantee
BankHeist
BalleZone
BeamFoder
Bosng
Breakout
Cantall
Centipade
ChopperCo
Creryllioo
DemonAliic
DoutieDun
Enduro
Fishing
Freewey
Dopher
Gravtar
Hero
Icchockky
Jamesbond
Kangaroo
Kungf aargFalls
allas
NameT
Phoenik
Obert
Roveraid
ReadRurne
Robstank
Seaqueel
Tima Pnt
UpNDoww
Vidco
WizarDDWWs
YanoReveng
Zexon
Figure 7: Percent improvement of top decision transformer rollouts over the best score in the training dataset. 0% indicates improvement. Top-3 metric (instead of mean) is used to more fairly
compare to the best rather than expert average demonstration score.
4.7 Does optimal action inference improve upon behavior cloning?
Behavioral Claning
aipeeke waw.eaaeeeeea..o....ooo. Score
Amidar:
Asssult
hsterix
Allanti
Baricteistt
Battlezane.
BeamRidor
Boking
Breakout:
Carrival:
Conúipdd
ImopprrCommannn
CrazyClimber
DemenAttack
DoubleDunkk
Enduro
FishingDerby
Freeway
Frestbite
Gopher-
Grovitar
Hero.
IceHookey..
Jamesbond.
Kangaroe-
Krull.
Kung Fuggstr
DamaThiccamme
Phoenix
Pooyan-
Qbort.
Rivernld:
RoadRurmer
Robotank.
Seaquest.
TimaPilot
Dew:
VidroPirball.
WizardotVier.
YaraRevenge.
zawwon-
Figure 8: Comparison of per-game for decision transformer behavioral cloning. Bars
indicate + standard deviation around the mean across 16 trials. We show DON-normalized scores in this figure for better presentations.
In Figure 1
that IQM performance across all games is indeed significantly improved by actions. FFiuure 8ishows the mean and standard deviation of scores across all games. While behavior cloning may sometimes produce highly- highly-rewarding episodes, it is less likely to do so. We find decision transformer outperforms behavioral cloning in 31 out of 41 games.
generating
4.8 Does training on expert and non-expert data bring benefits over expert-only training?
We believe that, comparing to learning from expert demonstrations, learning from large, diverse datasets that include some expert data but primarily non-expert data help learning and improve performance. To verify this hypothesis, we filter our training data [ from each game by episodic returns and only preserve top 10% trajectories to produce an expert dataset (see Appendix for details). We use this expert dataset to train our multi-game decision transformer (DT-40M) and the transformer-bart based behavioral cloning model (BC-40M). Figure compares these models trained on expert data and our DT-40M trained on all data.

---

We observe that (1) Training only on expert data improves behavioral cloning; (2) Training on full data, including expert and non-expert data, improves Decision Transformer, (3) Decision Transformer with full data outperforms behavioral cloning trained on expert data.
100%
Human Nomnn--mmaaadd IOM Boore
93%
89%
73%
50%
54%
25%
0%
Decision Transfrmer (Full Data)
Decisio Transformer (Eapert Data)
BC Transforer (Ful Dats)
BC Transformer (Fapert Data)
Figure 9: Comparison of 40M transformer models trained on full data and only expert data.
Conclusion
In the quest to develop highly capable and generalist agents, we have made important and measurable progress. Namely, our results exhibit a clear benefit using large transformer- models in multi-game domains, and the general trends in these results performance improvements with larger models and the ability to rapidly fine-tune to new tasks mirror the successes observed for large-scale vision and language models. Our results also highlight difficulties of online RL algorithms in handling the complexity of multi-game training on Atari. It is interesting to note that our best results achieved by decision transformers, which essentially learn via supervised learning sequence data, compared to alternative approaches such as temporal difference learning (more typical in reinforcement learning), policy gradients, and contrastive representation learning. This begs the question of whether online learning algorithms can be modified to be as "data-absorben"" as DT-like methods. While even our best generalist agents times fall short performance achieved by agents trained on a single task, this is broadly consistent with related works that have trained single models on many tasks [36 65 Still, our best generalist agents are already capable of outperforming the data they are trained on. We believe the trends suggest clear paths for future work that, with larger models and larger suites of tasks, performance is likely scale up commensurately.
Limitations. We acknowledge reasons for caution in over-generalizing our conclusions. Our results are based largely on performance in the Atari suite, where action and observation spaces are aligned across different games. It is unclear whether offline RL datasets such as Atari of sufficient scale and diversity that we would see similar performance scaling as observed NLP and vision benchmarks. Whether well as whether our conclusions hold for other settings, remains unclear.
observe other forms of generalization, such zero-shot adaptation, as
Societal Impacts. In the current setting, do not foresee significant societal impact the models are limited to playing simple video games. We emphasize that our current agents are not intended interact with humans or be used outside of self-contained game-playing domains. One should exercise increased caution if extending our algorithms and methods to such situations in order to ensure any safety and ethical concerns are appropriately addressed. At the same time, the capability of decision making based on reward feedback rather than purely imitation of the data has the potential to be easier to align with human values and goals.
Acknowledgements
We would like to thank Oscar Ramirez, Roopali Vij, Sabela Ramos, Rishabh Agarwal, Shixiang (Shane) Gu, Aleksandra Faust, Noah Fiedel, Chelsea Finn, Sergey Levine, John Canny, Kimin Lee, Hao Liu, Ed Chi, and Luke Metz for their valuable contributions and support for this work.
10

---

References
[1] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi, An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pages 104-114. PMLR, 2020.
[2] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34, 2021.
[3] Michael Ahn, Authony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn. Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, al. Do i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
[4] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. arXiv preprint arXiv:2204. 14/98, 2022.
[5] William H Alexander and Samuel J Gershman. Representation learning with reward prediction errors. arXiv preprint arXiv:2108. /2402, 2021.
[6] Anurag Arnab, Mostafa Dehghani, Georg Heigold. Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6836 6846, 2021.
[7] Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai. Aidan Clark, Ivo Danihelka. Claudio Fantacci, Jonathan Godwin, Chris Jones, Tom Hennigan, Matteo Hessel, Steven Kapturowski, Thomas Keck, lurii Kemaev, Michael King, Lena Martens, Vladimir Mikulik. Tamara Norman, John Quan, George Papamakarios, Roman Ring, Francisco Ruiz, Al- varo Sanchez, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Wojeicch Stokowiee, and Fabio Viola. The DeepMind JAX Ecosystem, 2020. URL http:tttt////ttuute httptttthw.ccocemmmeeeeemmme
[8] Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. Advances in neural information processing systems, 31, 2018
[9] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013.
[10] Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-211 2013.
[11] Mare Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement learning. In International Conference on Machine Learning, pages 449-458. PMLRR 2017.
[12] Tom Brown, Benjamin Mann. Nick Ryder. Melanie Subbiah. Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot leamerrs. Advances in neural information processing systems, 33:1877-1901, 2020.
[13] Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Mare Bellemare. Dopamine: A research framework for deep reinforcement learning. arXiv preprint arXiv:1812.06110 2018.
[14] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information prucessing systems. 34, 2021.
[15] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
[16] Giuseppe Cuccu, Julian Togelius, and Philippe Cudré-Mauroux. Playing atari with six neurons. arXiv preprint arXiv: 001880000001..33 2018.
[17] Rishabh Dabral, Ganesh Ramakrishnan, Preethi Iyothi, et al. Rudder: A cross lingual video and text retrieval dataset. arXiv preprint arXiv:2 055557 2021.
[18] Jacob Devlin, Ming-Wes Mei Chang, Kenton Lee, and Kristina Toutanova, Bert: Pre Preitraining of deep bidirec- tional transformers for language understanding, arXiv preprint 1rX 2018.
11

---

[19] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai. Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1
2020.
[20] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, al. Impala: Sealable distributed deep-rl with importance weighted actor-leamer architectures. In International Conference on Machine Learning, pages 1407-1416. PMI.R, 2018.
[21] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without explo- ration. In International Conference on Machine Learning, pages 2052-2062. PMLR, 2019.
[22] Hiroki Furuta, Tadashi Kozuno, Tatsuya Matsushima, Yutaka Matsuo, and Shixiang Shane Gu. Co- adaptation of algorithmic and implementational innovations in inference-based deep reinforcement learning. Advances in neural information processing systems, 34:9828 9842, 2021.
[23] Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for offline hindsight information matching, arXiv preprint (rXiv:2111.10300 2021.
[24] Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare G Bellemare. Deepmdp: Learning continuous latent space models for representation learning. In International Conference on Machine Learning, pages 21 2170-2179. PMLR, 2019.
[25] Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, et al. RI unplugged: A suite of benchmarks for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:7248-7259, 2020.
[26] Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature communications, 12(1):1-22 2021.
[27] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: I Learning behaviors by latent imagination. International Conference on Learning Representations, 2019.
[28] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pages 2555-2565. PMLR, 2019.
[29] Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmmy Ba. Mastering atari with discrete world models. International Conference on Learning Representations, 2020.
[30] Matteo Hessel. Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In shirtt-second AAAI conference on artificial intelligence, 2018.
[31] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
[32] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint
2019.
[33] Wenlong Huang, Igor Mordatch, and Deepak Pathak. One policy to control them all: Shared modular policies for agent-agnostic control. In International Conference on Machine Learning, pages 4455 4464. PMLR, 2020.
[34] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pages 991-1002. PMLR, 2022.
[35] Michael Janner, Qiyang Li, and Sergey Levine. Offine reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34, 2021.
[36] Lukasz Kaiser. Aidan N Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit. One model to learn them all. arXiv preprint 17XXv:1706.05137 2017.
[37] Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi task robotic reinforcement learning at scale. arXiv preprint arXiv:2104.08211 2021.
12

---

[38] Jared Kaplan, Sam McCandlish, Tom Henighan. Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray. Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint urXiv:2001.0883 2020.
[39] Hilbert Kappen, Vicenç Gómez, and Manfred Opper. Optimal control a graphical model inference problem. Machine learning, 87(2):159-183 2012.
[40] Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. Gedi: Generative discriminator guided sequence generation. arXiv preprint arXiv:2009.06366 2020.
[41] Aviral Kumar, Xue Bin Peng, and Sergey Levine. Reward-conditioned policies. arXiv preprint arXiv: 13465, 2019.
[42] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-1191, 2020.
[43] Vitaly Kurin, Maximilian Igl. Tim Rocktäschel, Wendelin Bochmer, and Shimon Whiteson. My body cage: the role of morphology in graph-based incompatible control. arXiv preprint arXiv:2010.0/856, 2020.
[44] Kuang-Huci Lee, Ian Fischer, Anthony Liu, Yijie Guo, Honglak Lee, John Canny, and Sergio Guadarrama. Predictive information accelerates learning in rl. Advances in Neural Information Processing Systems, 33: 11890-11911, 2020.
[45] Sergey Levine Aviral Kumar, George Tucker, and Justin Fu, Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint (iti::20000016443 2020.
[46] Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained transformers as universal computa- tion engines. arXiv preprint 103Y:::033.524444 2021.
[47] Clare Lyle, Mark Rowland, Georg Ostrovski, and Will Dabney. On the effect of auxiliary tasks on representation dynamics. In International Conference on Artificial Intelligence and Statistics, pages 1-9. PMLR, 2021.
[48] Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data, arXiv preprint arXiv:2005.07664 2020.
[49] Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search provides a competitive approach to reinforcement Icarning. arXiv preprint arXiv: 18X3:18003 2018
[50] John McCarthy, Marvin L Minsky, Nathaniel Rochester, and Claude E Shannon. A proposal for the dartmouth summer rescarch project on artificial intelligence, august 31, 1955. Al magazine, 12-4) 2006.
[51] Russell Mendonca, Oleh Rybkin, Kostas Danilidiss, Danijar Hafner, and Deepak Pathak. Discovering and achieving goals via world models. Advances Neural Information Processing Systems, 34, 2021.
[52] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 13 2013.
[53] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-333 2015.
[54] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning. pages 1928-1937. PMLR, 2016.
[55] Ofir Nachum and Mengjiao Yang. Provable representation learning for imitation with contrastive fourier features. Advances in Neural Information Processing Systems, 34, 2021.
[56] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint 1007::::007.33455 2018
[57] Pedro A Ortega, Markus Kunesch, Grégoire Delétang. Tim Genewein, Jordi Grau-Moya, Joel Veness, Jonas Buchli, Jonas Degrave, Bilal Piot, Julien Perolat, et al. Shaking the foundations: delusions in sequence models for interaction and control. arXiv preprint 100 2021.
13

---

[58] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin. Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. 2022.
[59] Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint 15Iiv::511.06:0:0 2015.
[60] Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88 97, 1991.
[61] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMIR, 2021.
[62] Jack W Rac. Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112..0 1446, 2021.
[63] Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:19100 10683, 2019.
[64] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34:12116-12128 2021.
[65] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent, 2022. URL https:///rxii org/ abs/a/s///2 22650061556
[66] Machel Reid, Yutaro Yamada, and Shixiang Shane Gu. Can wikipedia help offline reinforcement learning? urXiv preprint arXiv:2201. 12122, 2022.
[67] Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. arXiv preprint 15 2015.
[68] Juergen Schmidhuber. Reinforcement learning upside down: Don't predict rewards-just map them actions. arXiv preprint 1rXiv:19/ 2019.
[69] Julian Schritwieser, Toannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassahis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(78839):604-669) 2020.
[70] Ross D Shachter. Probabilistic inference and influence diagrams. Operations research, 36(4):589-604, 1988.
[71] Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojeiech Jaškowski, and Jürgen Schmidhuher. Training agents using upside-down reinforeement learning, arXiv preprint arXiv: arXiv: 1912.02877 2019.
[72] Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, arXiv preprint arXiv:2104.0988 2021.
[73] Emanuel Todorov. Linearly-solvable markov decision problems. Advances in neural information processing systems, 19, 2006.
[74] Mare Toussaint. Robot trajectory optimization using approximate inference. In Proceedings of the 26th annual international conference on machine learning, pages 1049- 1056, 2009.
[75] Maria Tsimpoukelli, Jacob L. Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34; 200-212,77 2021.
[76] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6309-6318 2017.
14

---

[77] Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[78] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou. Aditya Siddhant, Aditya Barua, and Colin Raffel, mt5: massively multilingual pre-trained text-to-text transformer. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 483-498, 2021.
[79] Kevin Yang and Dan Klein. Fudge: Controlled text generation with future discriminators. arXiv preprint mxXvv2104.05211 08, 2021.
[80] Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential decision making. In International Conference on Machine Learning, pages 11784-11794, PMLR, 2021.
[81] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large hatch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv: / 0904994000000 009620 2019.
[82] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning, pages 1094-1100. PMLR, 2020.
[83] Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint 10X1:22006.10722, 2020.
[84] Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. 20Xi::2002.05607,0 2022.
arXiv preprint
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section S
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
15

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "c1xE_vdZsId", "reviewer": "Reviewer_ZHCn", "summary": "This paper introduces the possibility of having a generalist RL agent that is based on the transformer. The paper demonstrate the effectiveness of a single RL agent, and that its scaling trend follows those of language and vision. The idea is validated on Atari games. ", "questions": "Is the result sensitive to the quantization scheme used in the paper?", "limitations": "No", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper is easy to follow and the problem is well motivated, What really stands out the detailed description and the solid experiments performed. I think the results could have great impact on RL, and potentially making large transformer based offline RL training the new dominant paradigm for the field, especially with the scaling behavior.\n\nWeakness\nI guess the main weakness is the novelty since going from decision transformer to multitask decision transformer seems to be very straightforward. Nevertheless, I think this does not diminish the main value here.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "_Ih9FOJ-C6I", "reviewer": "Reviewer_wP61", "summary": "The paper empirically evaluates the ability of the recently proposed decision transformer model to generalize across Atari games. It also investigates other empirical questions (scaling behavior, transfer learning, baselines, etc.). The main empirical result is that the decision transformer model outperforms other generalist agents but loses to single-game agents. The paper demonstrates how an improved policy can be extracted using guided generation approaches similar to those used in language modeling and shows that scaling trends similar to those seen in vision and NLP hold for Atari games.\n\n---------\n\nUPDATE: I thank the authors for their detailed responses. After reading the other reviews and the responses, I've increased my score to \"Accept\".", "questions": "1. How does the test-time action selection work exactly? In particular, what happens if the P(expert|R) is very small? Can the full test-time procedure be included in the paper or appendix?\n\n2. How sensitive are the empirical results to different seeds, hyper-params, etc?", "limitations": "The authors have described the main limitations. Based on the response to a question above, it might need to be expanded to include sensitivity wrt seeds, hyper-params, etc.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n  + The paper investigates an important question (scaling behaviors of decision transformers) and conducts a detailed empirical investigation of the decision transformer in Atari games. The experiments are well constructed and nicely detailed.\n  + The experimental results are novel (to my knowledge). The multi-game decision transformer seems to be the state-of-the-art in general game playing (although restricted to Atari games).\n  + The paper is well written. It's easy to follow the main ideas and experiments. The paper also does a good job of placing its ideas into the larger body of work in sequence modeling, upside-down RL, transfer learning, etc.\n  + There appears to be some algorithmic novelty in the guided action generation.\n\nWeaknesses\n  - As the paper mentions, it's unclear if these results hold outside of Atari games. However, evaluating this seems outside the scope of this paper.\n  - I didn't fully understand the test-time action-selection described in Section 3.4. For example, what happens if P(expert|R) is very small? Full algorithmic pseudo-code might make it clearer.\n  - It sounds like a single training run was used for each model. Is this correct? The answers in the checklist also mention error bars but I didn't see any in the figures. Also, the appendix mentions a single seed was used. Overall, it's unclear to me if these experiments are reproducible across seeds, hyper-params, etc. That said, given the large dataset and large computational cost, this may not be a major issue but perhaps this needs to be addressed somewhere in the paper.\n  - Given that the main contributions of the paper are empirical, the above points make the overall impact of this paper somewhat limited. I'd be inclined to revise my opinion based on the authors response to the questions below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "W091wt6GL6W", "reviewer": "Reviewer_EPhF", "summary": "This paper proposes training a single model on a lot of games, combining the RL approach with the same token prediction approach we've seen enable the recent very capable LLM results. The model they train does very well on the suite of Atari games and lets us ask questions like:\n\n1. What is learned from playing all of those games? \n2. Do the trends in vision and language hold for generalist RL agents?\n3. Does this next-step tokenization prediction approach work for these games?\n4. How do different online and offline methods work in this multi-game etup?\n5. Does this idea of expert action inference, which comes from decomposing the next-step tokenization to explicitly model the expert prediction, improve upon BC?\n\nThe model is trained on trajectories collected in prior work and includes both expert and non-expert behavior. There are many baseline methods that they compare against. All in all, this is an extensive work which demonstrates that, while the model does not beat single-game performance (as expected), it is competitive across the majority of games.", "questions": "1. What happens if you do actually filter for only expert level trajectories? I get that you designed a policy that can skirt around this issue, but I think this would be good for understanding what happens when we use human experts as the behaviors to clone. That is largely our eventual purpose, so would be helpful to do that as a follow up. I suspect it wouldn't be that much extra work either on this paper ;).\n\n2. Where is the Appendix? Some things are listed as being there, but I don't see it attached, nor is there a Supplemental... I might be being blind to this but other papers included their appendix explicitly in the paper.", "limitations": "The authors are honest about their limitations in the blurb at the end.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\n\nThis paper is not particularly original. The only part of the algorithm that could be original is the Expert Action Inference inspired by Krause et al.\n\nClarity:\n\nIt's very clear. There's no issues here.\n\nnit - L249: \"We discuss these results *in* the Appendix\".\n\nQuality:\n\nThe quality is high as well. This paper sets out to do a goal - figure out how to train a single agent to a high level on a lot of games - and does it. It then supports that finding with relevant questions about whether it is capable of learning more (transfer), how it compares with online/offline methods, how it scales with size, and and how optimal vs suboptimal actions influence the learning. These all make sense as experiments to do and bolster the argument.\n\nSignificance:\n\nThe paper is significant in so far as it sets a bar for what comes next. It's somewhat of a successor (although not multi-modal) to One Model to Rule Them All, which previously was significant in showing that we can do a lot more in a single model than previously thought. It's not significant in being world-shaking. I think of this as being important in so far as a group showing the rest of the world that it's possible.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HSk1LwBznQb", "reviewer": "Reviewer_4PtS", "summary": "This paper investigates the properties of decision transformer models trained on multiple Atari games using off-line RL. The model is found to be superior to multi-game models using other approaches, for example online non-transformer models or transformer models trained with behavioral cloning. The performance gap between single-game trained models vs. multi-game trained models, observed in some earlier works, was found in this study as well. ", "questions": "Please see the limitations section below.", "limitations": "I understand that these are ultimately empirical results, but the authors could perhaps do a better job explaining why these observed results might make sense: for example, why online C51 doesn’t seem to do as well as the offline DT or why non-transformer models in general don’t seem to do as well as transformer models. The lack of any scaling with model size for CQL in Fig 5 (in fact inverse scaling with model size in Fig 5b!) also seems surprising a priori, for example. What is the hypothesized explanation for this?  \n\nIn Figure 5a, the decision transformer is claimed to exhibit power law scaling with model size, but this doesn’t seem correct. First of all, it's not really possible to conclude a power law or anything like that from three data points only with any degree of confidence. Secondly, the figure shows a roughly linear increase in performance for the decision transformer in a semilogx plot. This corresponds to a logarithmic improvement in performance with model size (in fact it seems like the scaling on a linear-linear plot will be worse than logarithmic because the trend seems slightly slower than linear in the semilogx plot in Fig 5a). A power law would be linear in a log-log plot.\n\nAlso relatedly, were separate hyperparameter searches performed for different model sizes in the scaling experiments in Figure 5 and was the amount of compute (FLOPs) controlled between different model sizes (need to train smaller models for a larger number of iterations)? If not why not? If not, wouldn't that possibly underestimate the performance of the smaller size models in these experiments, as demonstrated, for example, in the new scaling laws paper by Hoffmann et al. (2022)?\n\nIn section 4.6, the best model rollouts are compared against the best training data. But, I’m really not sure if this is a fair comparison. With enough samples, one is essentially guaranteed to surpass the top training data eventually. Shouldn’t the model be expected to do better on average than the expert rollouts in the training data ideally? The performance gap between single-game and multi-game models shown in Fig. 1 (lack of positive transfer between games on average), also observed in other works, suggests this is not really the case yet with multi-game models. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\nThe experiments here are interesting and insightful, likely to be of general interest to the ML community. The work also seems technically sound.   \n\nWeaknesses:\n\nPlease see the limitations section below.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept