A Survey on Transformers in Reinforcement Learning
Authors: Wenzhe Li
Hao Luo
Zichuan Lin
Chongjie Zhang
Zongqing Lu
Deheng Ye
Venue: Accepted by TMLR ()

Abstract: Transformer has been considered the dominating neural architecture in NLP and CV, mostly under supervised settings. Recently, a similar surge of using Transformers has appeared in the domain of reinforcement learning (RL), but it is faced with unique design choices and challenges brought by the nature of RL. However, the evolution of Transformers in RL has not yet been well unraveled. In this paper, we seek to systematically review motivations and progress on using Transformers in RL, provide a taxonomy on existing works, discuss each sub-field, and summarize future prospects.

Published in Transactions on Machine Learning Research (09/2023)
A Survey on Transformers in Reinforcement Learning
Wenzhe Li
Iwz21 @ aswggiss.ingha.dcee. edu..n 1+2000 pku.edu. zichuanlin @ tencent.com chongjie @ wustl. edu zongqing. lu pku. pku.edu.cr dericye @ tencent.com
Hao 2.3+
Zichuan Lin
Chongjie Zhang
Zongqing Lu
Deheng 4T
'Tsinghua University
Pekinn University
BBAII
*Tencent Inc.
WWahington University in St.Louis
Reviewed on http:///////eeeeeeoe.//////////a
Abstract
Transformer has been considered the dominating neural architecture in NLP and CV, mostly under supervised settings. Recently, a similar surge of using Transformers has appeared in the domain of reinforcement learning (RL), but it is faced with unique design choices and challenges brought by the nature of RL. However, the evolution of Transformers RL has not yet been well unraveled. In this paper, seek to systematically review motivations and progress on using Transformers in RI., provide taxonomy existing works, discuss each sub-field, and summarize future prospects.
1 Introduction
Reinforcement learning (RL) provides a mathematical formalism for sequential decision making.. By utilizing RL, we can acquire intelligent behaviors automatically. While RL has provided a general framework for learning- control, deep neural networks, as a way of function approximation with high capacity, have been enabling significant progress along wide range of domains (Silver al., 2016; Vinyals et al., 2019; Ye al., 2020a;b).
While the generality of deep reinforcement learning (DRL) has achieved tremendous developments in recent years, the issue of sample efficiency prevents its widespread use in real-world applications. To address this issue, an effective mechanism is to introduce inductive biases into the DRL framework. One important inductive bias in DRL is the choice of function approximator architectures, such as the parameterization of neural networks for DRL agents. However, compared to efforts on architectural designs in supervised learning (SL), how to design architectures for DRL has remained less explored. Most existing works on architectures for RL motivated by the success of the (semi-) supervised learning community. For instance, common practice to deal with nigh-dimensional image-based input in DRL is introduce convolutional neural networks (CNN) (LeCun al., 1998; Mnih et al., 2015); another common practice to deal with partial observability is to introduce recurrent neural networks (RNN) (Hochreiter & Schmidhuber, 1997; Hausknecht & Stone, 2015).
In recent years, the Transformer architecture (Vaswani et al., 2017) has revolutionized the learning paradigm across wide range of SL tasks (Devlin al., 2018: Dosovitskiy et al., 2020; Dong al., 2018) and demonstrated superior performance over CNN and RNN. Among its notable benefits, the Transformer architecture enables modeling long dependencies and has excellent scalability (Khan al., 2022). Inspired by the success of SL, there has been surge of interest in applying Transformers in RL, with the hope of carrying the benefits of Transformers to the RL field.
Equual contribution; Equal advising.

---

Published in Transactions on Machine Learning Research (09/2023)
The use of Transformers in RL dates back to Zambaldi et al. (2018), where the self-attention mechanism is used for relational reasoning over structured state representations. Afterward, many researchers seek to apply self-attention for representation learning to extract relations between entities for better policy learning (Vinyals et al., 2019; Baker et al., 2019). Besides leveraging Transformers for state representation learning, prior works also use Transformers to capture multi-step temporal dependencies to deal with the issue of partial observability (Parisotto et al., 2020; Parisotto & Salakhutdinov, 2021). More recently, offline RL (Levine et al., 2020) has attracted attention due to its capability to leverage large-scale offline datasets. Motivated by offline RL, recent efforts have shown that the Transformer architecture can directly serve model for sequential decisions (Chen et al., 2021; Janner et al., 2021) and generalize to multiple tasks and domains (Lee et al., 2022; Carroll et al., 2022).
The purpose of this survey is to present the field of Transformers in Reinforcement Learning, denoted as "Transformer- based RL". Although Transformer has been considered one of the most popular models in SL research at present (De- vlin et al., 2018; Dosovitskiy al., 2020; Bommasani et al., 2021; Lu et al., 2021), it remains to be less explored in the RL community. In fact, compared with the SL domain, using Transformers in RL as function approximators faces unique challenges. First, the training data of RL is collected by an ever-changing policy during optimization, which induces non-stationarity for learning Transformer. Second, existing RL algorithms often highly sensitive to design choices in the training process, including network architectures and capacity (Henderson et al., 2018). Third, Transformer-basee architectures often suffer from high computational and memory costs, making it expensive in both training and inference during the RI. learning process. For example, in the case of AT for video game-playing, the training performance closely related to the efficiency of sample generation, which is restricted by the computa- tional cost of the RL policy network and value network (Ye al., 2020a; Berner al., 2019). Fourthly, compared to models that rely on strong inductive biases, Transformer models typically need much larger amount of training data to achieve decent performance, which usually exacerbates the sample efficiency problem of RL. Despite all these challenges, Transformers are becoming essential tools in RL due to their high expressiveness and capability, However, they are utilized for various purposes stemming from orthogonal advances in RL, such as a) RL that requires strong representation or world model (e.g., RL with high-dimensional spaces and long horizon); b) RL as a sequence modeling problem; and c) Pre-training large-scale foundation models for RL. In this paper, we seek to provide comprehensive overview of Transformer-
RI, including taxonomy of current methods and the challenges. We also discuss future perspectives. as we believe the field of Transformer-basee RL will play an important role in unleashing the potential impact of RL, and this survey could provide starting point for those looking to leverage its potential.
We structure the paper as follows. Section 2 covers background on RL and Transformers, followed by a brief intro- duction on how these two are combined together. In Section 3, we describe the evolution of network architecture in RL and the challenges that prevent the Transformer architecture from being widely explored in RL for a long time. In Section 4, we provide a taxonomy of Transformers in RL and discuss representative existing methods. Finally, we summarize and point out potential future directions in Section 5.
2 Problem Scope
2.1 Reinforcement Learning
In general, Reinforcement Learning (RL) considers learning in a Markov Decision Process (MDP) M = (S,A, P,r,y, Po),, where S and A denote the state space and action space respectively. P(s'ls, a) is the transition dynamics, r(s, a) is the reward function, 7 E (0, 1) the discount factor, and Po is the distribution of initial states. Typ- ically, RL aims learn policy (a]ss) maximize the expected discounted return (=r) E, [E, 7'rss, a.).. To solve an RL problem, we need to tackle two different parts: learning to represent states and learning to act. The first part can benefit from inductive biases (e.g., CNN for image-based states, and RNN for non- non-Markovian tasks), The second part can be solved via behavior cloning (BC), model-free or model-based RI. In the following part, we introduce several specific RL problems related to advances in Transformers in RL.
In offline RL (Levine et al., 2020), the agent cannot interact with the environment during training. Instead, it only has access to a static offline dataset D {(s, a, s)rr collected by arbitrary policies. Without exploration, modern offline RL approaches (Fujimoto et al., 2019; Kumar et al., 2020; Yu et al., 2021b) constrain the learned policy close to the data distribution, to avoid out-of-distribution actions that may lead to overestimation.
Offline RL.
2

---

Published in Transactions on Machine Learning Research (09/2023)
Recently, in parallel with typical value-based methods, popular trend in offline RL RL via Supervised Learning (RvS) (Emmons et al., 2021), which learns an outcome-conditioned policy to yield desired behavior via SL..
Goal-conditioned RL. Goal-conditioeed RL (GCRL) extends the standard RL problem to goal-augmented setting, where the agent aims learn conditood policy (Taas,)) g) that can reach multiple goals. Prior works propose to various techniques, such as hindsight relabeling (Andrychowicz et al., 2017), universal value function (Schaul et al., 2015), and self-imitatee imitation learning (Ghosh et al., 2019), to improve the generalization and sample efficiency of GCRL. GCRL is quite flexible as there are diverse choices goals. We refer readers to (Liu et al., 2022) for detailed discussion around this topic.
Model-based RL. In contrast to model-free RL which directly learns the policy and value functions, model-based RL learns an auxiliary dynamic model of the environment. Such model can be directly used for planning (Schrit- twieser et al., 2020), or generating imaginary trajectories to enlarge the training data for any model-free algo- rithm (Hafner et al., 2019). Learning a model is non-trivial, especially large or partially observed environments where we first need to construct the representation of the state. Some recent methods propose use latent dynam- ies (Hafner et al., 2019) or value models (Schrittwieser al., 2020) to address these challenges and improve the sample efficiency of RL.
2.2 Transformers
Transformer (Vaswani al., 2017) one of the most effective and scalable neural networks to model sequential data. The key idea of Transformer is incorporate self-attention mechanism, which could capture dependencies within long sequences in efficient manner. Formally, given sequential input with n tokens (x, R RR), J==r where d is the embedding dimension, the self-attention layer maps each token x; to query q, E RR a key k, Rd, and a value V, E Rdd via linear transformations, where dg = dg. Let the sequence of inputs, queries, keys, and values be X Rgad Q E Raxd, K Raxd2, and V Rxxd:= respectively. The output of the self-attention layer Z Raxd, is weighted sum of all values:
QKT ddg
Z = softmax
V.
With the self-attention mechanism (Bahdanau et al., 2014) well as other techniques, such multi-head attention and residual connection (He et al., 2016), Transformers can learn expressive representations and model long-term interactions.
Due the strong representation capability and excellent scalability, the Transformer architecture has demonstrated superior performance over CNN and RNN across a wide range supervised and unsupervised learning tasks. So natural question is: act) in RL?
Transformers to tackle the problems (i.e., learning to represent states, and learning to
2.3 Combination of Transformers and RL
We notice that a growing number of works are seeking combine Transformers and RL diverse ways. general, Transformers can be used as component for RI algorithms, e.g., representation module dynamic model. Transformers can also serve as one whole sequential decision-maker. Figure I provides sketch of Transformers' different roles in the context of RL.
3 Network Architecture in RL
Before presenting the taxonomy of current methods in Transformer-basee RL, we start by reviewing the early progress of network architecture design in RL, and summarize their challenges. We do this because Transformer itself is an advanced neural network and designing appropriate neural networks contributes to the success of DRL.
3

---

Published in Transactions on Machine Learning Research (09/2023)
7
Policy
Transformer
Model
Environment
Transformer
R
5g
22
Egt
Agt
Perception
Single task
Transtormer
Multiple
Environment
Entities
Cross domains
Figure I: An illustrating example of Transformer-hasee RI. On the one hand, Transformers can he used as one component in RL. Particularly, Transformers can encode diverse sequences, such entities, agents, and stacks of historical information; and is also an expressive predictor for the dynamics model. On the other hand, Transformers can integrate all subroutines in RL and act as sequential decision maker. Overall, Transformers can improve RL's leamning efficiency single-task, multi task, and cross domain settings.
3.1 Architectures for function approximators
Since the seminal work Deep Q-Network (Mnih al., 2015), many efforts have been made in developing network architectures for DRL agents. Improvements in network architectures in RL can be mainly categorized into three classes. The first class is to design policy or value functions. For example, Wang et al. (2016) propose dueling network architecture with one for the state value function and another for the state-dependent action advantage function. This choice of architecture incorporates the inductive bias that generalizes learning across actions. Other examples include the value decomposition network, which has been used to learn local Q-values for individual agent (Sunchag et al., 2017) or sub-reward (Lin et al., 2019). The second class is investigate whether general techniques of neural networks (e.g., regularization, skip connection, batch normalization) can be applied RL. To name a few. Ota al. (2020) find that increasing input dimensionality while using online feature extractor boost state representation learning, and hence improve the performance and sample efficiency of DRL algorithms. Sinha et al. (2020) propose deep dense architecture for DRL agents, using skip connections for efficient learning, with an inductive bias to mitigate data processog inequality. Ota et al. (2021) use DenseNet (Huang et al., 2017) with decoupled representation learning to improve flows information and gradients for large networks. The third class is to scale DRL agents for distributional learning. For instance, IMPALA (Espeholt et al., 2018) have developed distributed actor-learner architectures that can scale to thousands of machines without sacrificing data efficiency. Recently, due the superior performance of Transformers, some researchers have attempted to apply Transformers architecture in policy optimization algorithms, but found that the vanilla Transformer design fails to achieve reasonable performance in RL tasks (Parisotto et al., 2020).
structure that incorporates RL inductive biases to ease the difficulty of training
3.2 Challenges
While the usage Transformers has made rapid progress in SL domains in past years, applying them in RL is not straightforward with the following unique challenges in folds!.
Namely Transformers RL. Transformers in SI., and Transforners RI. other neural network architectures RI.
4

---

Published in Transactions on Machine Learning Research (09/2023)
local peo-imedrr requence
Agents
Tempora sequence GTIXL
Marning bosedt VOM
Rapresentation leaming
Model learning
Transform RL
2018
2021
Sequential decision-makkng
Generalist agents
Laarreed janrrdd-cootttored GOT
Mudtpie takee GatoJAn
Ren Retom-cooooiioeee Shurhir
Mudipl domsins UNEMASIVONNT
Beyand offine RL. COT/MATT
Boom search Stucture SPLT
Figure 2: The taxonomy Transformer-base RL (Transform RL in short). The timeline based on the first work related the branch.
On the one hand, from the view of RL., many researchers point out that existing RL. algorithms are incredibly sensitive to architectures of deep neural networks (Henderson et al., 2018; Engstrom al., 2019; Andrychowicz et al., 2020). First, the paradigm of alternating between data collection and policy optimization (i.e., data distribution shift) in RL induces non-stationarity during training. Second, RL algorithms often highly sensitive to design choices in the training process. In particular, when coupled with bootstrapping and off off-policy learning, learning with function approximators can diverge when the value estimates become unbounded (i.e., "deadly triad") (Van Hasselt et al., 2018). More recently, Emmons et al. (2021) identify that carefully choosing model architectures and regularization are crucial for the performance of offline DRL agents.
On the other hand, from the view of Transformers, the Transformer-based architectures suffer from large memory footprints and high latency, which hinder their efficient deployment and inference. Recently, many researchers aim to improve the computational and memory efficiency of the original Transformer (Tay et al., 2022), but most of these works focus on SL domains. In the context of RL, Parisotto & Salakhutdinov (2021) propose to distill learning progress from a large Transformer-baser learner model to a small actor model to bypass the high inference latency of Transformers. However, these methods are still expensive in terms of memory and computation. So far, the idea of efficient or lightweight Transformers has not yet been fully explored in the RL community. In addition to considerable memory usage and high latency. Transformer models often require significantly larger amount of training data to achieve comparable performance models that rely on strong inductive biases (e.g., CNN and RNN). Given that DRL algorithms often struggle with sample efficiency, it can be challenging for DRL agents to gather sufficient data to train the Transformer models. As we shall see later, such challenge inspires Transformer's 's prosperity in offline RI..
4 Transformers in RL
Although Transformer has become a popular model most supervised learning research, it has not been widely used in the RI. community for a long time due to the aforementioned challenges. Actually, most early attempts of Transformer-based RL apply Transformers for state representation learning or providing memory information, but still use standard RL algorithms such temporal difference learning and policy optimization for agent learning. Therefore, these methods still suffer from challenges from the conventional RL framework. Until recently, offline RL allows learning optimal policy from large-scale offline data. Inspired by offline RL, recent works further treat the RL problem as conditional sequence modeling problem on fixed experiences, which bypasses the challenges of bootstrapping error in traditional RI,, allowing Transformer to unleash its powerful sequential modeling ability.
In this survey paper, we retrospect the advances of Transformer-based RL, and provide taxonomy to present the current methods. We categorize existing methods into four classes: representation learning, model learning, sequential decision-making. and generalist agents. Figure 2 provides taxonomy sketch with subset of corresponding works.
5

---

Published in Transactions on Machine Learning Research (09/2023)
4.1 Transformers for representation learning
One natural usage of Transformers is to use it as a sequence encoder. In fact, various sequences in RL tasks require processing'. such as local per-timestep sequence (multi-entity sequence (Vinyals et al., 2019; Baker et al., 2019), multi-agent sequence (Wen et al., 2022), etc.), temporal sequence (trajectory sequence (Parisotto et al., 2020; Banino et al., 2021)), and so on.
4.1.1 Encoder for local per-timestep sequence
The early notable success of this method is to process complex information from variable number of entities scattered in the agent's observation with Transformers. Zambaldi et al. (2018) first propose to capture relational reasoning over structured observation with multi-head dot-product attention, which is subsequently used in AlphaStar (Vinyals et al., 2019) to process multi-entity observation in the challenging multi-agent game StarCraft II. The proposed entity Transformer encodes the observation as:
Emb "rannsoomerrei,,". ,.),
where e, represents the agent's observation on entity i, either directly sliced from the whole observation or given by an entity tokenizer.
Several follow-up works have enriched entity Transformer mechanisms. Hu et al. (2020) propose a compatible decou- pling policy to explicitly associate actions to various entities and exploit an attention mechanism for policy explanation. Wang et al. (2023b) learn entity Transformer with general knowledge and feature-space-agnostie tokenization via transfer learning within different types of games. To solve the challenging one-shot visual imitation, Dasari & Gupta (2021) use Transformers to learn a representation focusing on task-specific elements.
Similar to entities scattered in observation, some works exploit Transformers to process other local per-timestep se- quences. Tang Ha (2021) leverage the attention mechanism process sensory sequence and construct a policy that is permutation invariant
inputs. In the incompatible multi-task RI. setting, Transformers proposed extract morphological domain knowledge (Kurin et al., 2020). Regarding the presence of multimodal information (e.g. image and language) in local per-timestep observations, Team et al. (2021) utilize Transformer-based structure to integrate these multimodal information and represent the state of the agent.
Moreover, recent RL algorithms are trying to incorporate vision inductive biases into policy learning. For instance, Vision Transformer (ViT) uses patch sequences to process images the visual domain, which can be used for rep- resentation learning in RL. Tao et al. (2022) test the effectiveness of ViT and its variants combined with various self-supervised techniques (Data2vec, MAE, and Momentum Contrastive learning) for visual control tasks. However, notable performance gain is shown in their experiments with less complex tasks. On the other hand, Hansen et al. (2021) find ViT-based architecture is prone to overfitting and address the promblem with data-augmentation. More- over, Kalantari et al. (2022) use ViT architecture to learn Q value with vision inputs, showing its potential to improve the sample efficiency of RL algorithms. Moreover, Seo et al. (2022a) combine VIT with an improved ceature-mask MAE to learn image features that are better suited for dynamies, which can benefit decision-making and control.
As summary, the Transformer, as choice for local per-timestep sequence encoding, is being applied in various RL scenarios. When the RL task itself requires attention the relationships among various parts of observations, such as entities and morphological domain sequences, the attention mechanism inherent in Transformers is suitable for their encoding process. When complex observation information needs to be processed, some works desire to transfer the expressive power demonstrated by Transformers in vision or language domains to the representation learning in RL. Additionally, there are some works indicating a trend towards using pre-trained encoders in RL, further establishing a deeper connection between the choice of representation learning structures in RI. and the highly acclaimed architectures in the vision and language domains.
PIn the context of Transformers, the term sequence refers to the manner in which data is processed. While the local per-timestep information may he represented a or graph structure specific tasks, the information conventionally aggregated specific order either input as post-procsssing step output.

---

Published in Transactions on Machine Learning Research (09/2023)
4.1.2 Encoder for temporal sequence
Meanwhile, it is also reasonable to process temporal sequence with Transformers. Such a temporal encoder works as memory module:
Emby: = Transformer(oo,." .., ,t);
where On represents the agent's observation at timestep and Embor represents the embedding of historical observa- tions from initial observation to current observation.
In early works, Mishra et al. (2018) fail to process temporal sequence with vanilla Transformers and find it even worse than random policy under certain tasks. Gated Transformer-Xi (GTXLL) (Parisotto et al., 2020) is the first efficacious scheme to use Transformer as the memory. GTrXL provides gated 'skip' path with Identity Map Reordering to stabilize the training procedure from the beginning. Such architecture can also incorporate language instructions to accelerate meta RI. (Bing et al., 2022) and multi-task RI. (Guhur et al., 2023). Furthermore, Loynd et al. (2020) propose a shortcut mechanism with memory vectors for long-term dependency, and Irie et al. (2021) combine the linear Transformer with Fast Weight Programmers for better performance. In addition, Melo (2022) proposes to use the self-attention mechanism to mimic memory reinstatement for memory-based meta RL. Esslinger et al. (2022) combine Transformer with Bellman loss to process observation history as the input of Q-network.
The attention mechanism in Transformers avoids the need for recurrent context input, making it superior to recur- rent models in encoding long dependencies. While Transformer outperforms LSTM/RNN as the memory horizon grows and parameter scales, it suffers from poor data efficiency with RL signals. Follow-up works exploit some aux- iliary (self-) supervised tasks to benefit learning (Banino et al., 2021) or use a pre-trained Transformer as a temporal encoder (L.i et al., 2022; Fan et al., 2022).
4.2 Transformers for model learning
In addition to using Transformers as the encoder for sequence embedding, Transformer architecture also serves as the backbone of the world model in model-based algorithms. Distinct from the prediction conditioned on single-step ob- servation and action, Transformer enables the world model to predict transition conditioned on historical information.
Practically, the success Dreamer and subsequent algorithms (Hafner et al., 2020; 2021; ; 2023; Seo et al., 2022b) has demonstrated the benefits of world models conditioned on history in partially observable environments or in tasks that require memory mechanism. A world model conditioned on history consists of an observation encoder capture abstract information and transition model to learn the transition in latent space, formally:
2t Penc (zelo))
14 Prt1, 1e41 Prnns (2 9e4 (e+1l age);
where 2, represents the latent embedding of observation Ot, and Peney Pirans denote observation encoder and transition model, respectively.
There several attempts to build world model conditioned on history with Transformer architecture instead of RNN in previous works. Concretely, Chen et al. (2022) replace the RNN-based Recurrett State-Space Model (RSSM) in Dreamer with Transformer-basee model (Transformer State-Space Model, TSSM), IRIS (Imagination with auto- Regression over an Inner Speech) (Micheli et al., 2022) and Transformer-based World Model (TWM) (Robine et al., 2023) learns Transformer- based world model simply via auto-regressive learning on rollout experience without KL balancing like Dreamer and achieves considerable results on the Atari (Bellemare et al., 2013) 100k benchmark.
Besides, some works also try to combine Transformer-based world models with planning. Ozair al. (2021) verify the efficacy of planning with Transformer-base world model tackle stochastic tasks requiring long tactical look- ahead. Sun et al. (2022) propose a goal-conditionel Transformer-baat world model which is effective in visual- grounded planning for procedural tasks.
It is true that both RNN and Transformer are compatible with world models conditioned on historical information. However, Micheli et al. (2022); Chen et al. (2022) find that Transformer is more data-efficient world model com- pared with Dreamer, and experimental results of TSSM demonstrate that Transformer architecture is lucrative in tasks that require long-term memory. In fact, although model-based methods are data-efficient, they suffer from the
7

---

Published in Transactions on Machine Learning Research (09/2023)
Method
Setting
Hindsight Info
Inference
Additional Strieturo/Usage
DT (Chen et al., 2021)
Offline
conditioning
basic Transformer stricture
TT (Jamner et al., 2021)
IL/GCRL/Odline
return to
beam search
basic Transformer structure
(Shafiullah
2022)
conditioning
hasic Transformer stricture data augmentation
BooT (Wang et al. 2022)
Offline
return-to-go
beam search
GDT (Furuta et al., 2021)
HIM
arbitrary
conditioning conditioning conditioning
anti-causal aggrogator adversarial clustering
ESPER (Paster
2022)
Offline (stochastic)
expected
DoC (Yang al., 2022a)
Offline (stochastic)
learned representation
additional latent value func. additional func.
QDT (Yamagata al., 2022)
Offline
relabelled return-t-go
conditioning conditioning conditioning conditioning min max search comfitioning conditioning conditioning
StARformer (Shang ot al., 2022)
IL/Offline
scurn-to-gofewwad
Step Transformee & Sequence Transformer Transforeer Outer Transformer
(Mao et al., 2022)
Online/Offline
return- to-ghhhnne learned representation none go return-tn- tn-go
ConDT (Konan et 2022)
Offlinc
cetun-dependent transformation
SPLT (Villaflor et 2022)
Offline
separate models for world and policy drup span embeekling
DeFog (Ilu al. 2023)
Oflline
ODT (7heng et al., 2022)
Online finetune
trajectory-hann based entropy
MADT (Meng ol., 2021)
Online finetune (mulltiagentt)
separate models for and critic
Table
summary of Transformers for sequential decision makion..
compounding prediction error increasing with model rollout length, which greatly affects the performance and limits model rollout length (Janner al., 2019). The Transformer-basee world model can help alleviate the prediction error longer sequences (Chen al., 2022; Janner et al., 2021).
4.3 Transformers for sequential decision-making
In addition to being expressive architecture to be plugged into components of traditional RL algorithms, Trans- former itself can serve as model that conducts sequential decision decision-making making directly. This is because RL can be viewed as a conditional sequence modeling problem generating a sequence actions that can yield high returns.
4.3.1 Transformers as milestone for offline RL
One challenge for Transformers to be widely used RL is that the non-stationarity during the training process may hinder its optimization. However, the recent prosperity in offline RL motivates a growing number of works focusing on training Transformer model on offline data that can achieve state-of-the of--th-artttttt performance. Decision Transformer (DT) (Chen et al., 2021) first applies this idea by modeling RL as an autoregressive generation problem to produce the desired trajectory:
T = (RI, a1 Rg. 82, ....... Rir, ST, or
where RG Et - ae) is the return-to-go. By conditioning on proper target return values the first timestep, DT generate desired actions without explicit TD learning or dynamic programming. Concurrently, Trajectory Transformer (TT) (Janner et al., 2021) adopts similar autoregressive sequenee modeling:
M log , Po(rela, 86, +lg Pa Pa Pa(Rerre 0t, St, Tet)s
log (o(T)TTe))= Σ log Po(s) +ed)+ Σ
8t,
where M and N denote the dimension of state and action, respectively. In contrast to selecting target return values, TT proposes to use beam search for planning during execution. The empirical results demonstrate that TT performs well on long-horizon prediction. Moreover, TT shows that with mild adjustments on vanilla beam search, TT can perform imitation learning, goal-conditioned RL, and offline RL under the same framework. Regarding the behavior cloning setting, Behavior Transformer (BeT) (Shafiullah et al., 2022) proposes a similar Transformer structure as TT to learn from multi-modal datasets.
In light of Transformer's superior accuracy sequence prediction, Bootstrapped Transformer (BooT) (Wang et al., 2022) proposes to bootstrap Transformer to generate data while optimizing it for sequential decision making. For- mally, given a trajectory from the original dataset, BooT resamples the last T < T timesteps, and concatenates the generated 7" steps with the original - T"' steps as a new trajectory 7':
+' (TST Seg FST- D(TST-TITST-n)
8

---

Published in Transactions on Machine Learning Research (09/2023)
Bootstrapping Transformer for data augmentation expand the amount and coverage of offline datasets, and hence improve the performance. More specifically, this work compares two different schemes to predict the next token Un:
Dn ~ TST-T:: (autoregressive generation, sampling conditioned on previously generated tokens)
In ~
(teacher-foreing generation, sampling conditioned on previously original tokens)
and the results show that using two schemes together can generate data consistent with the underlying MDP without additional explicit conservative constraints.
4.3.2 Different choices of conditioning
While conditioning on return-to-go is practical choice to incorporate future trajectory information, one natural ques- tion is whether other kinds of hindsight information can benefit sequential decision-making. To this end, Furuta et al. (2021) propose Hindsight Information Matching (HIM), a unified framework that can formulate variants of hindsight RL. problems. More specifically, HIM converts hindsight RI. into matching any predefined statistics of future trajectory information w.r.t. the distribution induced by the learned conditional policy:
min Ex-p(t) (D
where is the parameter that policy {[ass, is conditioned on, D is a divergence measure such as KL divergence or f-divergences, and the information statistics (T) can be any function of the trajectory via the feature function t Furthermore, this work proposes Generalized DT (GDT) for arbitrary choices of statistics I (7)² and demonstrates its applications in two HIM problems: offline multi-task state-marght matreialll matching and imitation learning.
Specifically, one drawback of conditioning on retur-tt-go is that it will lead to sub-optimal actions in stochastic envi- ronments, as the training data may contain sub-optimal actions that luckily result in high rewards due the stochas- ticity of transitions. Paster et al. (2022) identify this limitation in general RvS methods. They further formulate RvS as an HIM problem and discover that RvS policies can achieve goals in consistency if the information statistics are inde- pendent of transitions' stochasticity. Based on this implication, they propose representations (ESPER), an algorithm that first clusters trajectories and estimates average returns for each cluster, and then trains a policy conditioned on the expected returns. Alternatiell., Dichotomy of Control (DoC) (Yang et al., 2022a) proposes to learn representation that is agnostic stochastic transitions and rewards in the environment via minimizing mutual information. During inference, DoC selects the representation with the highest value and feeds it into the conditioned policy.
sta.eumme-tttocaaaiiitt----ii indeeeeddn
Yang et al. (2022b) proposed a method by annotating task-specific procedure observation sequences in the training set and using them to generate decision actions. This approach enables the agent to learn decision- maainng based on prior planning, which beneficial for tasks that require multi-step foresight.
In addition to exploring different hindsight information, another approach to enhance return-to-go conditioning to augment the dataset. Q-learning DT (QDT) (Yamagata et al., 2022) proposes to conservative value function to relabel return-to-go in the dataset, hence combining DT with dynamic programming and improving its stitching capability.
4.3.3 Improving the structure of Transformers
Apart from studying different conditioned information, there are some works to improve the structure of DT or TT. These works fit into two categories: a) Introduce additional modules loss functions to improve the represen- tation. Shang et al. (2022) argue that the DT structure is inefficient for learning Markovian-like dependencies, as it takes all tokens as the input. To alleviate this issue, they propose learning an additional Step Transformer for local state-action-reward representation and using this representation for sequence modeling. Similarly, Mao et al. (2022) use an inner Transformer to process the single observation and an outer Transformer to process the history, and cas- cade them as a backbone for online and offline RL. Konan et al. (2022) believe that different sub-tasks correspond to different levels of return-to-oo, which require different representation. Therefore, they propose Contrastive Decision Transformer (ConDT) structure, where return-dependent transformation is applied to state and action embedding
3 special case, 1 (r) = Ri DT.

---

Published in Transactions on Machine Learning Research (09/2023)
before putting them into causal Transformer. The return-eependent transformation intuitively captures features spe- cific to the current sub-task, and it is learned with an auxiliary contrastive loss to strengthen the correlation between transformation and return. b) Design the architecture to improve the robustness. Villaflor et al. (2022) identify that TT structure may produce overly optimistic behavior, which is dangerous in safety-critical scenarios. This is because TT implements model prediction and policy network in the same model. Therefore, they propose SeParated Latent Trajectory Transformer (SPLT Transformer), which consists of two independent Transformer-basee CVAE structures of the world model and policy model, with the trajectory as the condition. Formally, given the trajectory 7k (st, (tt I,+1, 0t+1, SttK (++*) and its sub-sequence T' (8t; Ot, 8t+1, at+1,---- St+k), SPLT Trans- former optimizes two variational lower bounds for the policy model and the world model:
K
Beg Add
log po, (ac+klT's th: 25)
DKL (2 )e )(=(=))
K log pe.. (st+k+1, T14ks Be-k+117711 22"") DKL (2 (T( )(((2)))
Exp ~d
where ls, and pv, are the encoder and decoder of the policy model, and qo.. and pe.. are the encoder and decoder of the world model, respectively. Similarly to the min-mat mix-max search procedure, SPLT Transformer searches the latent variable space to minimize return-to-go in the world model and to maximize return-to-go in the policy model during planning. Hu et al. (2023) consider the possible frame dropping in the actual application scenario where states and rewards at some timesteps are unavailable due to frame dropping and the information at previous timesteps is left to be reused. They propose Decision Transformer under Random Frame Dropping (DeFog), a DT variant which extends the timestep embedding by introducing an additional drop-span embedding to predict the number of consecutive frame drops. In addition to frame dropping, DT may also suffer from forgetting, as it merely relies on parameters to "memorize" the large-scale data in an implicit way. Therefore, Kang et al. (2023) introduce an internal working memory module into DT to extract knowledge from past experience explicitly. They also incorporate low-rank adaptation (LoRA) (Hu et al., 2022) parameters to adapt to unseen tasks.
4.3.4 Extending DT beyond offline RL
Although most of the works around Transformers for sequential decision-making focus on the offline setting, there are several attempts to adapt this paradigm to online and multi-agent settings. Online Decision Transformer (ODT) (Zheng et al., 2022) replaces the deterministic policy in DT with a stochastic counterpart and defines a trajectory-level pol- icy entropy help exploration during online fine-tuning. Pretrained Decision Transformer (PDT) (Xie al., 2022) adopts the idea of Bayes' rule in online fine-tuning control the conditional policy's 's behavior in DT. Besides, such two-stage paradigm (offline pre- pre-training with online fine-tuningl is also applied to Multi-Agent Decision Transformer (MADT) (Meng et al., 2021), where a decentralized DT is pre-trained with offline data from the perspective of indi- vidual agents and is used as the policy network in online fine-tuning with MAPPO (Yu et al., 2021a). Interestingly, to the best of our knowledge, we do not find related works about learning DT in the pure online setting without any pre-training. We suspect that this because pure online learning will exacerbate the non-stationary nature of training data, which will severely harm the performance of the current DT policy. Instead, offline pre- trainininn can help to warm up well-behaved policy that is stable for further online training.
4.4 Transformers for generalist agents
In view of the fact that Decision Transformer has already flexed its muscles various tasks with offline data, several works turn to consider whether Transformers can enable generalist agent to solve multiple tasks or problems, as in the CV and NLP fields.
4.4.1 Generalize to multiple tasks
Some works draw on the ideas of pre-training on large-scale datasets CV and NLP, and try to abstract a gen- eral policy from large-scale multi-task datasets. Multi-Game Decision Transformer (MGDT) (Lee al., 2022), a variant of DT, learns DT on a diverse dataset consisting of both expert and eon-expert data and achieves close-to-human performance multiple Atari games with single of parameters. In order obtain expert-level performance with
10

---

Published in Transactions on Machine Learning Research (09/2023)
dataset containing non-expert experiences, MGDT includes an expert action inference mechanism, which calculates an expert-level retum-to-go posterior distribution from the prior distribution of returnto-go and preset expert-level retur-to-go likelihood proportional according to the Bayesian formula. Likewise, Switch Trajectory Transformer (SwitchTT) (Lin et al., 2022), a multi-task extension to TT, exploits a sparsely activated model that replaces the FFN layer with a mixture-of-exper layer for efficient multi-task offline learning. More specifically, such "switch layer" consists of n expert networks Ei,...; ; En and router to select a specific expert for each token:
y = (z)) i= arg max (x = arg max
Σ3 ch(m),
where h(x) denotes the logits produced by the router. Besides, a distributional trajectory value estimator is adopted to model the uncertainty of value estimates. With these two enhanced features, SwitchTT achieves improvement over TT across multiple tasks terms of both performance and training speed. MGDT and SwitchTT exploit experiences collected from multiple tasks and various performance- policies to learn a general policy. To solve multi-objective RL problem, Zhu al. (2023) build multi-objective offline RL dataset and extend DT to preference and return conditioned learning. As for meta RL across diverse tasks, Team et al. (2023) demonstrate that Transformer model- based RL can improve adaptation and scalability with curriculum and distillation.
However, constructing large-scale multi-task dataset is non-trivial. Unlike large-scale datasets in CV or NLP, which are usually constructed with massive public data from the Internet and simple manual labeling, action information is always absent from public sequential decision-making data and not facile to label. Thus, Baker et al. (2022) propose semi-supervised scheme utilize large-scale online data without action information by learning a Transformer- based Inverse Dynamic Model (IDM) small portion of action-labeled data, which predicts the action information with past and future observations and is consequently capable of labeling massive unlabeled online video data. IDM is learned on small-scale dataset containing manually labeled actions and is accurate enough to provide action labels of videos for effective behavior cloning and fine-tuning, Further, Venuto et al. (2022) conduct experiments where they train IDM with action-labeled data from other tasks, which reduces the need of lction-labeled data specifically for the target task.
The efficacy of prompting (Brown et al., 2020) for adaptation new tasks has been proven in many prior works in NLP. Following this idea, several works aim at leveraging prompting techniques for DTT-based methods to en- able fast adaptation. Prompt-based Decision Transformer (Prompt-DT) (Xu et al., 2022) samples sequence of transitions from the few-shot demonstration dataset as prompt, and shows that it can achieve few-shot policy gener- alization on offline meta RL tasks. Reed et al. (2022) further exploit prompt-based architecture to learn a generalist agent (Gato) via regressgres sequence modeling on a super large-scale dataset covering natural language, image, temporal recision-making and multi-modal data. Gato is capable of a range of tasks from various domains, including text generation and makiin.. Specifically, Gato unifies multi-modal sequences in shared tokenization space and adapts prompt-based inference in deployment generate task-specific sequences. Similarly to Gato, RT-1 (Bro- han et al., 2022) and PaLM-E (Driess et al., 2023) leverage large-scale multimodal datasets to train Transformers that can achieve high performance on downstream tasks. VIMA (Jiang et al., 2022) combines textual and visual tokens as multimodal prompts to build scalable model that can generalize well among robot manipulation tasks.
Despite being effective, Laskin et al. (2022) point out that one limitation of the prompt-based framework is that the prompt is demonstrations from a well-behaved policy, as contexts in both works are not sufficient to capture policy improvement. Inspired by the in-context learning capabilities (Alayrac et al., 2022) of Transformers, they propose Algorithm Distillation (AD) (Laskin et al., 2022), which instead trains Transformer of the learning progress of single-task RL algorithms. Therefore, even in new tasks, the Transformer can learn to gradually improve its policy during the auto-regressive generation.
across-episode sequences
4.4.2 Generalize to multiple domains
Beyond generalizing to multiple tasks, Transformer is also a powerful "universal" model to unify a range of domains related to sequential decision -aking.. Motivated by advances in masked language modeling (Devlin et al., 2018) technique NLP, Carroll et al. (2022) propose Uni[MASK], which unifies various commonly-studied domains, in- cluding behavioral cloning, offline RL, GCRL, past/future inference, and dynamics prediction, as one mask inference problem. Uni[MASK] compares different masking schemes, including task-specific masking, random masking, and
11

---

Published in Transactions on Machine Learning Research (09/2023)
finetune variants. It is shown that one single Transformer trained with random masking can solve arbitrary inference tasks. More surprisingly, compared to the task-specific counterpart, random masking can still improve performance in the single-task setting.
In addition to unifying sequential inference problems in the RL domain, Reid et al. (2022) find it beneficial to fine- tune DT with Transformer pre-trained on language datasets or multi-modal datasets containing language modality, Concretely, Reid al. (2022) find out that pre-training Transformer with language data while encouraging similarity between language and RL-based representations can help improve the performance and convergence speed of DT. This finding implies that even knowledge from non-RL fields can benefit RL training via Transformers. Li et al. (2022) fur- ther show that policy initialized with pre-trained language models can be finetuned to accommodate different tasks and goals in interactive decision-making. Additionally, several works (Huang et al., 2022a;b; Raman et al., 2022; Yao et al., 2022; Ahn et al., 2022; Wang et al., 2023c; Du et al., 2023; Wang et al., 2023a) discover that pre-trained large- scale language models are capable of generating reasonable high-level plans to accomplish complex tasks without further finetuning. However, even with well-behaved low-level policies, directly applying large language models for execution is inefficient even infeasible in particular scenarios or environments. Therefore, these works propose to generate valid action sequences with affordance guidance (Ahn et al., 2022), autoregressive correction (Huang et al., 2022a), corrective re-prompting (Raman et al., 2022), interleaved reasoning and action traces generation (Yao et al., 2022), and description feedback (Huang et al., 2022b; Wang et al., 2023c; Du et al., 2023; Wang et al., 2023a). Even without RL modules, Wu et al. (2023) find that GPT-4 (OpenAI, 2023). when prompted with related RI, paper and chain-of-thought, can produce a competitive performance in RL benchmark tasks.
5 Summary and Future Perspectives
This paper briefly reviews advances in Transformers for RL. We provide a laxonomy of these advances: a) Transform- serve powerful module of RL, e.g., acting as a representation module world model; b) Transformers can serve sequential decision-maker, and c) Transformers can benefit generalization across tasks and domains. While we cover representative works on this topic. the usage of Transformers in RL is not limited to our discussions. Given the prosperity of Transformers in the broader Al community, we believe that combining Transformers and RL is a promising trend. To conclude this survey, in the following, we discuss future perspectives and open problems for this direction.
Bridging Online and Offfine Learning via Transformers. Stepping into offline RL is a milestone in Transformer- based RL. Practically, utilizing Transformers to capture dependencies in decision sequence and to abstract policy is mainly inseparable from the support of considerable offline demonstrations. However, it is unfeasible for some decision-making tasks to get rid of the online framework in real applications. On the one hand, it is not that easy to obtain expert data in certain tasks while substantial amount of less related data holds potential valuable information. On the other hand, some environments are open-ended (e.g., Minecraft), which means the policy has to continually adjust to deal with unseen tasks during online interaction. Therefore, we believe that bridging online learning and offline learning is necessary. However, most research progress following Decision Transformer focuses on the offline learning framework. We thereby expect some special paradigm designs to address this issue via transferring the generalization capabilities demonstrated by Transformer structures in large vision or language models to decision tasks and to potentially significantly enhancing the utilization of less related offline data. In addition, how train a well-performed online Decision Transformer from scratch is an interesting open problem.
Combining Reinforcement Learning and (Self-) Supervised Learning. Retracing the development of Transformer-based RL, the training methods involve both RL and (self-) supervised learning. When trained under the conventional RL framework, the Transformer architecture is usually unstable for optimization (Parisotto et al., 2020). RL algorithms inherently require imposing various constraints to avoid issues like the deadly triad (Van Has- selt et al., 2018), 8), which could potentially exacerbate the training difficulty of the Transformer architecture. Meanwhile, the (self-) supervised learning paradigm, while improving the optimization of the Transformer structure, significantly constrains the performance of the policy based on the quality of the experienee/deemonsttatiion data. Therefore, a better policy may be learned when we combine RL and (self-) supervised learning in Transformer learning. Some works (Zheng et al., 2022; Meng et al., 2021) have tried out the scheme of supervised pre-training and RL-involved fine-tuning. However, exploration can be limited with a relatively fixed policy (Nair al., 2020), which is one of the
12

---

Published in Transactions on Machine Learning Research (09/2023)
bottlenecks to be resolved. Also, along this line, the tasks used for performance evaluation are relatively simple. It is worthwhile to further explore whether Transformers can scale such (self-) supervised learning up to larger datasets, more complex environments, and real-world applications. Further, we expect future work to provide more theoretical and empirical insights to characterize under which conditions this (self-) supervised learning is expected to perform well (Brandfonbrener et al., 2022; Siebenborn et al., 2022; Takagi, 2022).
Transformer Structure Tailored for Decision-making Problems. The Transformer structures in the current DT- based methods are mainly vanilla Transformer, which is originally designed for the text sequence and may not fit the nature of decision-making problems. For example, is it appropriate to adopt the vanilla self-attention mechanism for trajectory sequences? Whether different elements in the decision sequence or different parts of the same elements need to be distinguished in position embedding? In addition, as there are many variants of representing trajectory sequence in different DT-based algorithms, how to choose from them still lacks systematic research. For instance, how to select robust hindsight information when deploying such algorithms in the industry? Furthermore, the vanilla Transformer is a structure with huge computational cost, which makes it expensive at both training and inference stages, and high memory occupation, which constrains the length of the dependencies it captures. We notice that some works (Zhou al., 2021; Fedus al., 2022; Zhu et al., 2021) have improved the structure from these aspects and have been validated in various domains. It is also worth exploring whether similar structures be used in the decision-making problem.
Towards More Generalist Agents with Transformers. Our review has shown the potential of Transformers as general policy (Section 4.4). In fact, the design of Transformers allows multiple modalities (c.g., image, video, text, and speech) to be processed using similar processing blocks and demonstrates excellent scalability to large-capacity networks and huge datasets. Recent works have made substantial progress training agents capable of performing multiple and cross-domain tasks. However, given that these agents are trained on huge amounts of data, it is still uncertain whether they merely memorize the dataset and if they can perform efficient generalization. Therefore, how to learn agent that can generalize unseen tasks without strong assumptions is well worth studying (Boustati et al., 2021). Moreover, we are curious about whether Transformer is strong enough to learn general world model (Schubert et al., 2023) for different tasks and scenarios.
Connections to Other Research Trends. While we have discussed how RL can benefit from the usage of Trans- formers, the reverse direction, i.e., using RL. to benefit Transformer training is an intriguing open problem yet less explored. We see that, some works model language/dialogue generation tasks with offline RL setting and learn gen- eration policy with relabeling (Snell al.. 2022b) or value functions (Verma al., 2022: Snell et al., 2022a; Jang et al., 2022). Recently, Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022) learns a reward model and uses RL algorithms to finetune Transformer for aligning language models with human intent (Nakano et al., 2021; OpenAI, 2023). In the future, we believe RI. can be useful tool to further polish Transformer's performance in other domains. In parallel with Transformers, diffiusion model (Song et al., 2020; Ho et al., 2020) is also becoming mainstream tool for generative tasks and demonstrates its applications in RL (Janner et al., 2022). While no signif- RL (Janner et al., 2019; Ajay
icant performance gap is observed between Transformer-basee RL and Diffusion- et al., 2022), we speculate that each model may have advantages in the domain they are most commonly used, i.e., Transformer in language RL tasks and diffusion model in vision RL tasks. Moreover, diffusion models may perform better in stochastic environments as they can model complex distributions. However, the relatively slow inference may hinder diffusion models' usage in real-time socision-maaing problems. We hope future work can conduct rigorous experiments and provide more empirical evidence to verify our conjectures.
13

---

Published in Transactions on Machine Learning Research (09/2023)
References
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. conditional gener- ative modeling you need for decision-making? arXiv preprint arXiv:2211 15657, 2022.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23736, 2022.
Marcin Andrychowicz. Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Picter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
Marcin Andrychowicz. Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al. What matters for on-policy deep actor-critic methods? a large-scale study. In International conference on learning representations, 2020.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0477 2014.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emer- gent tool use from multi-agent autocurricula. In International Conference on Learning Representations, 2019.
Bowen Baker, 1lge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampe- dro, and Jeff Clune. Video pretraining (VPT): Learning to act by watching unlabeled online videos. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Andrea Banino, Adria Puigdomenech Badia, Jacob Walker, Tim Scholtes, Jovana Mitrovic, and Charles Blundell. Coberl: Contrastive bert for reinforcement learning. In International Conference on Learning Representations, 2021.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 20133.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota with large scale deep reinforcement learning. arXiv preprint 1912v 066800 2019.
Zhenshan Bing, Alexander Koch, Xiangtong Yao, Fabrice O Morin, Kai Huang, and Alois Knoll. Meta-reiifforcmmnnt learning via language instructions. arXiv preprint (xXiv:2209.04224, 2022.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney Arx. Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models, arXiv preprint arXiv:2108.0723 072558 2021.
Ayman Boustati, Hana Chockler, and Daniel McNamee. Transfer learning with causal counterfactual reasoning decision transformers. arXiv preprint arXiv:2110.14355 2021.
David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return- conditioned supervised learning work for offline reinforcement learning? arXiv preprint 2tt:2200600179,, 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakr- ishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint 2r.iv:2212.0663 2022.
14

---

Published in Transactions on Machine Learning Research (09/2023)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1900 2020.
Micah Carroll, Orr Paradise, Jessy Lin, Raluca Georgeseu, Mingfei Sun, David Bignell, Stephanie Milani, Katja Hofmann, Matthew Hausknecht, Anca Dragan, et al. Unimask: Unified inference in sequential decision problems. arXiv preprint arXiv2211..08862 2022.
Chang Chen, Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. Transdreamer: Reinforcement learning with transformer world models. arXiv preprint arXiv:2202.094481, 2022.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 10:11500455999 2021.
Sudeep Dasari and Abhinav Gupta. Transformers for one-shot visual imitation. In Conference on Robot Learning, pp. 2071 PMLR, 2021.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint arXiv: (a0:::81000400 04805, 2018.
Linhao Dong, Shuang Xu, and Bo Xu. Speech-transformeer no-recurrence sreence-t--seqeence model for speech recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5884-5888. IEEE, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11922 2020.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 2023.
Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob An- dreas. Guiding pretraining in reinforcement learning with large language models. arXiv preprint 2rX6622302.06662 2023.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:2112.1075/ 2021.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry, Implementation matters deep rl: A case study on ppo and trpo. In International conference on learning representations, 2019.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pp. 1407-1416. PMLR, 2018.
Kevin Esslinger, Robert Platt, and Christopher Amato. Deep transformer g-networks for partially observable rein- forcement learning. arXiv preprint arXiv:2206.01078, 2022.
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang,
Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowl- edge. In Thit Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling trillion parameter models with
simple and efficient sparsity. The Journal of Machine Learning Reseurch, 23(1):5232-5270, 2022.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2062. PMLR, 2019.
15

---

Published in Transactions on Machine Learning Research (09/2023)
Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for offline hindsight infor- mation matching, arXiv preprint urXiv:2111.10366 2021.
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning reach goals via iterated supervised learning. arXiv preprint arXiv: 10Xiv:112200088 2019.
Pierre-Louis Guhur, Shizhe Chen, Ricardo Garcia Pinel, Makarand Tapaswi, Ivan Laptev, and Cordelia Schmid. Instruction-drrien history-aware policies for robotic manipulations. Conference on Robot Learning, pp. 175- 187. PMLR, 2023.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: 1912.0160, 2019.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. International Conference on Learning Representations, 2020.
Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2021. 2021.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.0411 2023.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 aaai fall symposium series, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 13:6840-6851 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-tere term memory. Neural computation, 9 1997.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Kaizhe Hu, Ray Chen Zheng, Yang Gao, and Huazhe Xu. Decision transformer under random frame dropping. arXiv preprint (tti::203300399) 2023.
Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. Updet: Universal multi-agent rl via policy decoupling with transformers. International Conference on Learning Representations, 2020.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extract- ing actionable knowledge for embodied agents. arXiv preprint arXiv:2201.07207, 2022a.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models, arXiv preprint arXiv:2207.05608 2022b.
16

---

Published in Transactions on Machine Learning Research (09/2023)
Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. Going beyond linear transformers with recur- rent fast weight programmers. Advances in Neural Information Processing Systems, 34:7703-7717, 2021.
Youngsoo Jang, Jongmin Lee, and Kee-Fung Kim. Gpt-critic: Offline reinforcement learning for end-to-end task- oriented dialogue systems. In International Conference on Learning Representations, 2022.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy opti- mization. Advances in Neural Information Processing Systems, 32, 2019.
Michael Janner, Qiyang Li, and Sergey Levine. Reinforcement learning one big sequence modeling problem. In ICML 2021 Workshop on Unsupervised Reinforcement Learning, 2021.
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.00 09991, 2022.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anand- kumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:22/0.33094, 2022.
Amir Ardalan Kalantari, Mohammad Amini, Sarath Chandar, and Doina Precup. Improving sample efficiency of value based models using attention and vision transformers. arXiv preprint wrXiv:2202.00710 2022.
Jikun Kang, Romain Laroche, Xindi Yuan, Adam Trischler, Xue Liu, and Jie Fu, Think before you act: Decision transformers with internal working memory. arXiv preprint arXiv:2305. 16338, 2023.
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR), 54(10s):1-41, 2022.
Sachin G Konan, Esmaeil Seraj, and Matthew Gombolay. Contrastive decision transformers. 6th Annual Conference Robot Learning, 2022.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179-1191, 2020.
Vitaly Kurin, Maximilian Igl, Tim Rocktäschel, Wendelin Boehmer, and Shimon Whiteson. My body is cage: the role of morphology in graph-based incompatible control. arXiv preprint arXiv:2010.01856, 2020.
Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Hansen, Angelos Filos, Ethan Brooks, al. In-context reinforcement learning with algorithm distillation. arXiv preprint arXiv:22100 10. 14215, 2022.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-bass learning applied to document recog- nition. Proceedings of the IEEE, 86(1 (6(11):22782324 1998.
Kuang-Hus Huei Lec, Ofir Nachum, Sherry Yang, Lisa Lee, C. Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, and Igor Mordatch. Multi-game decision transformers. In Advances in Neural Information Processing Systems, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint arXiv:2005.01643, 2020.
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. Pre-trained language models for interactive decision-making. Advances Neural Information Processing Systems, 35:31199 1113119991122222 2022.
Qinjie Lin, Han Liu, and Biswa Sengupta. Switch trajectory transformer with distributional value approximation for multi-task reinforcement learning. arXiv preprint arXiv:2203.07413 2022.
Zichuan Lin, Li Zhao, Derek Yang, Tao Qin, Tie-Yan Liu, and Guangwen Yang. Distributional reward decomposition for reinforcement learning. Advances in neural information processing systems, 32, 2019.
17

---

Published in Transactions on Machine Learning Research (09/2023)
Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Problems and solutions. arXiv preprint crXiv:2201.08299 2022.
Ricky Loynd, Roland Fernandez, Asli Celikyilmaz, Adith Swaminathan, and Matthew Hausknecht. Working memory graphs. In International conference machine learning, pp. 6404-6414, PMLR, 2020.
Kevin Lu, Aditya Grover, Picter Abbeel, and Igor Mordatch. Pretrained transformers as universal computation engines. arXiv preprint arXiv:2103.05247, 2021.
Hangyu Mao, Rui Zhao, Hao Chen, Jianye Hao, Yiqun Chen, Dong Li, Junge Zhang, and Zhen Xiao. Transformer in transformer as backhone for deep reinforcement learning. arXiv preprint arXiv:2212. 14538, 2022.
Luckeciano C Melo. Transformers are meta-reinforcemmnt learners. International Conference on Machine Learn- ing, pp. 15340-15359, PMLR, 2022.
Linghui Meng, Muning Wen, Yaodong Yang, Chenyang Le. Xiyun Li, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, and Bo Xu. Offline pre-trained multi-agent decision transformer: One big sequence model conquers all starcraftii tasks. arXiv preprint 1220022 2021.
Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample efficient world models. arXiv preprint htli::2209000888 2022.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-leamer. In Interna- tional Conference Learning Representations, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 (187750):529-55555 533, 2015.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.099:: 2020.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arxtv:2112.003332. 2021.
OpenAL. Gpt-4 technical report, 2023.
Kei Ota, Tomoaki Oiki, Devesh Jha, Toshisada Mariyama, and Daniel Nikovski. Can increasing input dimensionality improve deep reinforcement learning? In International Conference on Machine Learning, pp. 7424-7433. PMLR, 2020.
Kei Ota, Devesh K Jha, and Asako Kanezaki. Training larger networks for deep reinforcement learning. arXiv preprint arXiv:2102.07920, 2021.
Long Ouyang, Jeff Wu, Xu Jiang. Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02/55. 2022.
Sherjil Ozair, Yazhe Li, Ali Razavi, Ioannis Antonoglou, Aaron Van Den Oord, and Oriol Vinyals. Vector quantized models for planning. In International Conference on Machine Learning, pp. 3302-8313. PMLR, 2021.
Emilio Parisotto and Ruslan Salakhutdinov. Efficient transformers in reinforcement learning using actor-learner dis- tillation. arXiv preprint arXiv:2104.01655, 2021.
Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg. Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. In International conference machine learning, pp. 7487-7498. PMLR, 2020.
Keiran Paster, Sheila Mcllraith, and Jimmy Ba. You can't count on luck: Why decision transformers fail in stochastic environments. arXiv preprint arXiv:2205. 2022.
18

---

Published in Transactions on Machine Learning Research (09/2023)
Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, and Stefanie Tellex. Planning with large language models via corrective re-prompting. arXiv preprint arXiv:2211.099:: 2022.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175, 2022.
Machel Reid, Yutaro Yamada, and Shixiang Shane Gu. Can wikipedia help offline reinforcement learning? arXiv preprint arXiv:2201.1122,, 2022.
Jan Robine, Marc Höftmann, Tobias Uelwer, and Stefan Harmeling. Transformer-basee world models are happy with 100k interactions. arXiv preprint arXtv:2303.07009 2023.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference on machine learning, pp. 1312-1320. PMLR, 2015.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):004-611 2020.
Ingmar Schubert. Jingwei Zhang, Jake Bruce, Sarah Bechtle, Emilio Parisotto, Martin Riedmiller, Jost Tobias Sprin- genberg, Arunkumar Byravan, Leonard Hasenclever, and Nicolas Heess. A generalist dynamics model for control. arXiv preprint arXiv:2305.10912 2023.
Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In Conference on Robot Learning, pp. 1332-1344. PMLR, 2022a.
Younggyo Seo, Kimin Lee, Stephen I. James, and Pieter Abbeel. Reinforcement learning with action-free pre-training from videos. In International Conference on Machine Learning, pp. 19561-19579. PMLR, 2022b.
Nur Muhammad Mahi Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning & modes with one stone. arXiv preprint 2rXiv:226..11251, 2022.
Jinghuan Shang. Kumara Kahatapitiya, Xiang Li, and Michael S Ryoo. Starformer: Transformer with state-action- reward representations for visual reinforcement learning. In European Conference Computer Vision, pp. 462- 479. Springer, 2022.
Max Siebenborn, Boris Belousov, Junning Huang, and Jan Peters. How crucial is transformer in decision transformer? arXiv preprint arXiv:2211. 14655, 2022.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrit- twieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
Samarth Sinha, Homanga Bharadhwaj, Aravind Srinivas, and Animesh Garg. D2rl: Deep dense architectures in reinforcement learning. arXiv preprint 1rXiv:20100009/16,, 2020.
Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language learning, arXiv preprint arXiv:2206.1187/ 2022a.
Charlie Snell, Sherry Yang, Justin Fu, Yi Su, and Sergey Levine. Context-aware language modeling for goal-oriented dialogue systems. arXiv preprint arXiv:2204. 10/98, 2022b.
Yang Song, Jascha Sohl-Dick Dich-Diicktten,, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole, Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13450 2020.
Jiankai Sun, De-An Huang, Bo Lu, Yun-Hui Liu, Bolei Zhou, and Animesh Garg. Plate: Visually-grounded planning with transformers in procedural tasks. IEEE Robotics and Automation Letters, 7(2):49244493,, 2022.
19

---

Published in Transactions on Machine Learning Research (09/2023)
Peter Sunehag, Guy Lever, Audrumas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi- agent learning. arXiv preprint
2017.
Shiro Takagi. On the effect of pre-training for transformer in different modality on offline reinforcement learning. arXiv preprint arXiv:2211.00 1.09817, 2022.
Yujin Tang and David Ha. The sensory neuron a transformer: Cermutation-invariie neural networks for reinforce-
ment learning. Advances in Neural Information Processing Systems, 0122574--2588) 2021.
Tianxin Tao, Daniele Reda, and Michiel van de Panne. Evaluating vision transformer methods for deep reinforcement learning from pixels. arXiv preprint arXiv:2204.04903 2022.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys, 55(6):1-28, 2022.
Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmice Michael Chang, Natalie Clay, Adrian Collister, et al. Human- adaptation in an open- ended task space. arXiv preprint arXiv:2301.07608 2023.
DeepMind Interactive Agents Team, Josh Abramson, Arun Ahuja, Arthur Brussee, Federico Carnevale, Mary Cassin, Felix Fischer, Petko Georgiev, Alex Goldin, Mansi Gupta, et al. Creating multimodal interactive agents with imita- tion and self-supervised learning. arXiv preprint arXiv:21
2021.
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep rein- forcement learning and the deadly triad. arXiv preprint arXiv: 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
David Venuto, Sherry Yang, Pieter Abbeel, Doina Precup, Igor Mordatch, and Ofir Nachum. Multi-environment pretraining enables transfer action limited datasets. arXiv preprint arXiv:2211.33377 2022.
Siddharth Verma, Justin Fu, Mengjiao Yang, and Sergey Levine. Chai: A chatbot ai for task-oriented dialogue with offline reinforcement learning. arXiv preprint 27Xi::2204.08426. 2022.
Adam R Villaflor, Zhe Huang, Swapnil Pande, John M Dolan, and Jeff Schneider. Addressing optimism bias in sequence modeling for reinforcement learning. In International Conference on Machine Learning, pp. 22270- 22283. PMLR, 2022.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et Grandmaster level in stareraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019.
Guanzhi Wang, Yugi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anand- kumar. Voyager: An open-ended embodied agent with large language models, arXiv preprint 17X1 2023a.
Kerong Wang, Hanye Zhao, Xufang Luo, Kan Ren, Weinan Zhang, and Dongsheng Li. Bootstrapped transformer for offline reinforcement learning. arXiv preprint (00.:22266.88699) 2022.
Rundong Wang, Weixuan Wang, Xianhan Zeng, Liang Wang, Zhenjie Lian, Yiming Gao, Feiyu Liu, Siqin Li, Xian- liang Wang, QIANG FU, al. Multi-agent multi-game entity transformer. 2023b.
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint 56Xiv:2302.01166., 2023c.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanetot, and Nando Freitas Dueling network architec- tures for deep reinforcement learning. In International conference on machine learning, pp. 1995-2003. PMLR. 2016.
20

---

Published in Transactions on Machine Learning Research (09/2023)
Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is a sequence modeling problem. arXiv preprint arXiv:2205.44900 2022.
Yue Wu, So Yeon Min, Shrimai Prabhumoye, Yonatan Bisk. Ruslan Salakhutdinov, Amos Azaria, Tom Mitchell, and Yuanzhi Li. Spring: Gpt-4 out-performs rl algorithms by studying papers and reasoning. arXiv preprint arXiv:2305. 15486, 2023.
Zhihui Xie, Zichuan Lin, Junyou Li, Shuai Li, and Deheng Ye. Pretraining in deep reinforcement learning: A survey, arXiv preprint arXiv:2211.03955 2022.
Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In International Conference on Machine Learning, pp. 24631-24645, PMLR, 2022.
Taku Yamagata, Ahmed Khalil, and Raul contos-Rodriguez, Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. arXiv preprint arXiv:2209,00 03093,, 2022.
Mengjiao Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum. Dichotomy of control: Separating what you can control from what you cannot. arXiv preprint arXiv:2210. 13435, 2022a.
Mengjiao Sherry Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum. Chain of thought imitation with procedure cloning. Advances in Neural Information Processing Systems, 35:36366-36381, 2022b.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
Deheng Ye, Guibin Chen, Wen Zhang, Sheng Chen, Bo Yuan, Bo Liu, Jia Chen, Zhao Liu, Fuhao Qiu, Hongsheng Yu, et al. Towards playing full moba games with deep reinforcement learning. Advances in Neural Information Processing Systems, 33:621-632, 2020a.
Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang. Xipeng Wu. Qingwei Guo, al. Mastering complex control in moba games with deep reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 6672-6677 2020b.
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games, arXiv preprint arXiv:2103.01955, 2021a.
Tianhe Yu. Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conserva- tive offline model-based policy optimization. Advances in neural information processing systems, 34:28954-28967,
2021b.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Re- ichert, Timothy Lillicrap, Edward Lockhart, et al. Deep reinforcement learning with relational inductive biases. In International conference on learning representations, 2018
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. arXiv preprint arXiv:2202.05607. 2022.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11106-11115, 2021.
Baiting Zhu, Mcihua Dang, and Aditya Grover. Scaling pareto-efficient decision making via offline multi-objective arXiv preprint arXiv:2305. 00567, 2023.
Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. Long-short transformer: Efficient transformers for language and vision. Advances in neural information processing systems, 34:17723-177888 2021.
21


Reviews:
[{"review_id": "9j9VE8NsuZ", "reviewer": "Reviewer_KsT8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- This area is clearly of interest to many in the TMLR audience, as transformers are increasingly the go-to modeling architecture of choice across a variety of ML domains, including RL. This paper should serve as a reasonably recent and comprehensive reference for practitioners \n- I largely agree with the authors' taxonomy of uses of various Transformer architectures, though I disagree with the framing (see Weaknesses)\n- Good survey of future directions and outstanding challenges in the field.\n\n# Weaknesses\n\n## Framing\n\nI feel conflicted about the paper—on the one hand, Transformers have upended RL (and every other ML subfield) and so it's obviously important for people to understand how they're used in RL, and towards that end this paper does a good job at surveying the recent work and latest advancements (it may have missed a few papers, since it's hard to catch every paper that gets released in this field, but it feels reasonably comprehensive). In providing a reasonably comprehensive overview of the recent literature, this paper does a good job.\n\nOn the other hand, I find the framing of the paper a bit reductive, as it simplifies and combines several distinct research advances and thrusts under the generic umbrella of \"use Transformers for RL\". Specifically, the authors identify three \"pillars\" of recent advances in RL:\n\n1. Using Transformers to model high-dimensional, abstract sequential and/or graph-structured data as components in more \"traditional\" RL systems.\n2. The view of reinforcement learning as sequential decision making + the hindsight information matching framework. Note that this advance is *not* inherent to Transformers, though obviously the first work pushing this view indeed used a Transformer as its modeling architecture of choice.\n3. The gradual trend towards large-scale, generalist, pretrained (\"foundation\") models that learn to transfer and share parameters across a variety of tasks for improved performance, and the use of already successful pretrained models (such as large language models) as backbones for RL systems. Note that this advance is *also* not specific to Transformer models, though obviously the overwhelming majority of currently successful foundation models are Transformer-based.\n\nNote that of these three thrusts, all 3 involve transformers to some degree, but arguably only one of these (the first pillar) is actually specific to the Transformer architecture. The other I do believe should be included in a Transformers for RL review paper, **however**  I find the paper takes a bit of a reductive framing by describing everything as Transformer-first. For example, section 4.3 is described as \"using Transformers for sequential decision making\", and is framed primarily as \"let's use Transformers to solve a SDM problem.\" OTOH I believe the framing should be something like: \"A recent advance in RL has been viewing RL as sequence modeling...Transformers are a particularly useful architecture for this approach because they are particularly good sequence models.\" Similarly, I feel like Section 4.4 is less about \"Transformers for generalist agents\" and more about \"Transformers as (one choice) of a model architecture, towards the general trend of large-scale pretrained models and transfer learning for RL.\"\n\nThis framing can be seen in the authors' term \"TransformRL\". I disagree with the authors' decision to define a new name, TransformRL, to refer to the use of Transformers in RL, given the discussion on reductiveness above, and especially given how many overloaded and confusing terms the fields of RL/ML/DL have already. In my opinion the term conflates the field (RL) with current tools that we find useful to use in the field (Transformers) and the recent central advances dominating the RL conversation, of which Transformers play varying degrees of importance (RL as sequence modeling, large-scale pretraining). It's possible Transformer hype will die down, yet these other ideas will persist, and I don't think that RL should currently be defined by the use of Transformers. Imagine, for example, if we had analogous terms like RNNNLP or CNNCV—I view them all as rather unnecessary.\n\nI think to improve the framing under my view, the paper could use a more nuanced discussion at the beginning, by saying that transformers are now a workhorse in RL because they are such expressive and capable models, but they are used for very (different) reasons stemming from orthogonal advances in RL, including (1) RL in high-dimensional spaces, (2) RL as a sequence modeling problem, and (3) the trend towards large-scale, pretrained foundation models.\n\n## Could better summarize advantages and disadvantages *specific to Transformers* in RL\n\nI find the paper could be more detailed and nuanced when describing the reasons why one might specifically use Transformers over other models in RL, and what features of problems make Transformers a particularly good choice. Most of the sections of the paper basically show that Transformers have seen success when slotting them into basically every possible part of the RL problem: when processing observations (section 4.1.1), memory (section 4.1.2), world models (section 4.1.3). The authors catalog dozens upon dozens of apparent success stories where Transformers successfully replace existing deep architectures for RL.\n\nWhat should the takeaway be for the readers reading this paper? Is it that one should now use  Transformers basically unilaterally, in every possible component of an RL model? The paper could emphasize *particular problems* of an RL task that make Transformers a particularly suitable solution.\n\nFor example, in Section 4.1, it seems like Transformers have seen success in processing \"complex information\", including multi-agent representations (zambaldi et al 2018), morphological domain knowledge (kurin 2020), \"sensory sequences\" (tang and ha), multimodal information (team 2021), and a variety of vision settings. What is the overall recommendation or understanding from the authors, after surveying this work? Is it that one should no longer use any other more domain specific architectures such as CNNs and RNNs? Does it seem like Transformers be the go-to architecture for representation learning in RL? Are there any commonalities among domains where Transformers consistently outperform or underperform, or is more empirical work needed? I imagine that this work shows that Transformers have the most success in RL when applied the same kinds of representation learning problems as the supervised learning problems that Transformers are so good at. Assuming this is the case, the authors could more clearly illustrate this by unifying the case studies in this section, and tying it back to the general advantages of transformers as mentioned e.g. in 3rd paragraph of the intro (long dependencies, scalability).\n\nSimilarly, why should one use a Transformer instead of a more traditional memory mechanism such as an RNN/LSTM in RL agents (section 4.1.2)? Should it only be for environments with extremely long-distance dependencies? How should one think about the tradeoffs of (1) increased sequence modeling capacity and (2) larger model capacity? There is some description of the relevant tradeoffs in the final paragraph, but the writing could try to better aggregate and summarize the trends in each subarea here.\n\n---\n\nThe paper also lacks nuance in describing the current outstanding challenges in using Transformers in RL. For example, end of page 1/beginning of page 2, and Section 3.2 describes \"unique challenges\" for Transformers in RL:\n1. Sensitivity to architecture selection\n2. Sensitivity to design choices (e.g. hyperparameters)\n3. Large memory footprints and high latency\n\nIn my view, only (3) is a challenge that is arguably somewhat unique to Transformers, especially the large-scale transformers that have seen success across vision, NLP, and RL tasks. (1) and (2) seem like general issues across the entire field of RL.\n\n## Minor\n\n- I don't think \"foundation model\" is used correctly e.g. bottom of page 1, first sentence of section 4—I don't think a particular architecture can be categorized as a \"foundation model\" since you can still have small domain-specific transformers. Whether something is a \"foundation model\" is primarily determined by the model capabilities and usage intent rather than the particular architecture\n- I don't think Transformers are strictly sequence models, since sequence implies linear order and one can apply transformers to various richly ordered or unordered representation learning problems. However section 4.1 seems to cast Transformers as such. For example it's not clear to me that all of the uses of Transformers in section 4.1.1 are **sequence encoding** problems—e.g. the multi-entity StarCraft modeling problem is not necessarily a sequence modeling problem but rather a set representation learning problem.\n- End of section 4.2 \"The Transformer-based world model can help alleviate the prediction error on longer sequences.\" Can authors elaborate? Is this an empirical finding from the Janner paper or just a general statement about improved sequence modeling capacity from transformers?\n- I think in every review paper it's difficult to avoid the \"laundry list paragraph of several different papers\" trope, but minimizing this to whatever extent possible is ideal. Section 4.3.3 sticks out as a particular \"laundry list\" paragraph that outlines a dozen or so advance sin Transformer architectures without really unifying or describing the main shortcomings people are trying to work around, and the main trends in this research. There are also a lot of terms that are not fully explained and digestible to the average reader (e.g. CVAE, \"drop-span embedding\", etc)\n\n## Overall\n\nI don't think any of the weaknesses I described above are blockers to publication, and co-reviewers may not see as much issue with the framing as I do, and I am happy to be overriden. However in my opinion there is some additional work that would elevate this paper beyond the typical review paper in providing a more nuanced and unified view of the various new advances that have defined modern (2020+) RL research, and how precisely the Transformer architecture plays a role in all of this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Transformers have emerged as a general and powerful model for learning from high-dimensional, rich, and sequential data that has seen success across variety of domains of ML, including NLP and Computer Vision. This paper surveys the use of Transformers in Reinforcement Learning, categorizing uses of transformers into the following:\n\n1. as one component of a more traditional RL agent system, e.g. as an observation encoder, especially when processing rich visual or hierarchical/abstract data; as a memory module that processes past states (as an alternative to other memory mechanisms such as RNNs); or as a world model;\n2. as an \"end-to-end\" sequential decision maker, by treating RL as a sequence modeling problem, in the line of decision transformer and other approaches to hindsight RL;\n3. as very large, generalist models that can solve many tasks and domains, especially when combined with pretrained models such as large language models\n\nWithin these three taxonomies, the authors survey a fairly comprehensive set of recent work, including core ideas (e.g. RL as sequence modeling, generalist models), as well as additional work attempting to build and generalize such ideas. They conclude with several promising directions for future research.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As I described above, I would be happy to be overriden if my co-reviewers believe the paper is fine as is, but I think to be an excellent review paper, the paper could (1) slightly change the wording to be more clear about the different roles that Transformers play across different recent advances in RL, and (2) more clearly explain the advantages and shortcomings of the Transformer architecture specifically (rather than listing challenges specific to RL as a whole), and how recent work (e.g. section 4.3.3) is trying to get around such limitations.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0Uuzhef3nS", "reviewer": "Reviewer_e3ki", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n  - many important recent works are listed \n  - the paper makes the reader questioning about challenges and potential of these architectures\n  - some interesting discussions\n\nWeakness : \n   - No experiments to support discussions\n   - Not enough details in many places. It would have been very useful to choose one important approach in each section, to be detailed, or at least a bit formalized (e.g., GDT, SPLT, SwitchTT, BooT, etc). It would greatly help the reader to organize the given information, which is a bit dense in the current form, and focus on main aspects of the discussion (I was a bit lost in the textual flow, some illustrations and equations usually help to clarify).\n- Sometimes a lack of structure. Giving bold titles to paragraphs could help gathering the main idea of each. Section 4.4.1 is an example of section that would greatly benefit from a  more highlighted structure.  \n- Difficult to understand why challenges given in the paper are specific to transformers. Need of massive training data and latency are well-known challenges related to any deep architectures (including eg RNN) \n - The survey devotes a very large part to offline learning, specifically following decision transformers (and GDT) principles. While it is true that it is mainly used in that setting, it would have been very useful to discuss further the reasons that limit their  applicability in online RL (for instance by giving return-to-go as input as DT does for offline data). Section 4.3.4 is really too short on this, please develop.   ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is a survey on the use of Transformers in Deep Reinforcement Learning. After discussing challenges of the use of such architectures (notably related w.r.t. to their need of massive training data and their latency), authors present works for different use of Transformers, namely for representation learning in RL, model learning and as the actor of the learned policy. The latter is the most developed, mainly in the scope of offline learning. Finally authors discuss advantages of Transformers for learning generalist agents and give some insights about their view on important future works in that scope. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I would really appreciate if authors could give more details at least about the following approaches : GDT, SPLT, SwitchTT, BooT. Giving some formalized components of their principles. \n- A trending emerging architecture is decision diffusers for RL tasks. It would have been useful if authors could discuss pros and cons of transformers w.r.t. these new competitors. \n- develop section 4.3.4\n- I have difficulties to understand the core differences between the two first paragraphs given as perspectives in 5. For me it is very close... \n- Authors discuss in two different places about the \"deadly triad problem\" that hinders the use of Transformers for online learning  of RL policies. In the first para of perspectives (section 5), it is argued that self-supervised learning of sequences is thus better. I would like to point out that the policy gradient paradigm is fully equivalent to such a log-likelihood maximization when all rewards are set to 1 in every trajectory. For me, appart the onpolicy exploration process of policy gradients,  there is no real difference between both paradigms. So this does not look very \"new\". Could authors elaborate on this ? By the way, policy gradient is fully ignored in the survey...\n- The last perspective paragraph (RL for transformers) should be removed : while it suggests that RL could be used to learn better transformer architectures, the discussion is about the use of Transformers  for sequences modelling, which has no difference with imitation learning (and RLHF is simply RL finetuning as it is done for any other decision-making problem, I cannot understand why authors treat that tasks differently here).", "broader_impact_concerns": ".", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZkIXkjzLfA", "reviewer": "Reviewer_XCTM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- This is a timely survey for an emerging and promising research area. The works mentioned in the paper are comprehensive and representative, and cover all the main areas (as far as I know) in this filed. \n- The authors provide a brief historical perspective on the development of model architectures in RL, and discussed challenges in applying transformer in the RL domain, e.g. the problem of non-stationary training data. This sets up an adequate context for the main discussion.\n\nWeakness (For each bullet, please see \"requested changes\" for details.)\n- There are a few inaccurate description of cited papers. \n- A few claims are not well supported or need rephrasing.\n- A few missing citations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Main contributions:\n- This survey provides a taxonomy of different applications of the transformer architectures in RL, including transformer as state encoder, as world model, or as decision maker (in the decision-transformer style).\n- The authors also reviewed works that apply transformer models to train generalist agents, either from scratch or by fine-tuning or prompting models.\n- The authors discussed future perspectives and open problems. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I only have a few non-critical requests:\n\nInaccurate description of cited papers:\n- In section 3.2, \"Emmons et al. (2021) identify that carefully choosing ... are crucial for the performance of DRL agents.\" That paper only discusses offline RL.\n- In section 5, under \"Transformer Structure Tailored for Decision-making Problems\": Zhou et al., 2021 is not an NLP paper. If the authors want to mention efficient transformer variants, Tay et al., 2022 (already cited) should have pointers to representative works.\n\nQuestionable claims\n- In the introduction, \"the training objective of RL agents is typically a function of the current policy, which induces nonstationarity\nduring learning a Transformer\". The non-stationary is not caused by using the policy in the training objective. It's caused by the training data being collected using an ever-changing policy.\n- Section 4.2, \"The Transformer-based world model can help alleviate the prediction error on longer sequences.\" Please add citations to support this claim.\n\nA few missing citations\n- When discussing architectures in DRL, please consider cite IMPALA [1]\n- Deep Transformer Q-Networks [2] uses transformer with observation history as input, and the bellman loss is trained for multiple time-steps together.\n- [3] discusses applying ViT to RL\n\nTypos:\n- Section 4.4.2: \"directly applying large language models for execution is inefficient or even feasible\". feasible -> infeasible or unfeasible.\n\nAdditional suggestion:\n-  In the introduction and section 3.2, the authors discuss a few challenges of adopting transformer to RL problems. I think another possible reason is that transformer models typically need much more training data to be effective compared to models with strong inductive biases (CNN, RNN). In traditional DRL, sample efficiency is a key benchmark, which limits how much data the agent can collect to fit the model. It might be helpful to discuss this aspect as well.\n\n[1] IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures. Espeholt et al., 2018\n\n[2] Deep transformer q-networks for partially observable reinforcement learning. Esslinger et al., 2022\n\n[3] Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation. Hansen et al., 2021\n\n", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all left a number of requested changes to particular points, and the framing in general. I see the authors have clarified in the responses, and I would expect them to update the paper accordingly before the final version is submitted.
Decision: Accept (poster)