Decision Transformer: Reinforcement Learning via Sequence Modeling
Authors: Lili Chen
Kevin Lu
Aravind Rajeswaran
Kimin Lee
Aditya Grover
Michael Laskin
Pieter Abbeel
Aravind Srinivas
Igor Mordatch
Venue: NeurIPS 2021 Poster (2021)
Keywords: transformers
reinforcement learning
deep learning
generative modeling

Abstract: We introduce a framework that abstracts Reinforcement Learning (RL) as a sequence modeling problem. This allows us to draw upon the simplicity and scalability of the Transformer architecture, and associated advances in language modeling such as GPT-x and BERT. In particular, we present Decision Transformer, an architecture that casts the problem of RL as conditional sequence modeling. Unlike prior approaches to RL that fit value functions or compute policy gradients, Decision Transformer simply outputs the optimal actions by leveraging a causally masked Transformer. By conditioning an autoregressive model on the desired return (reward), past states, and actions, our Decision Transformer model can generate future actions that achieve the desired return. Despite its simplicity, Decision Transformer matches or exceeds the performance of state-of-the-art model-free offline RL baselines on Atari, OpenAI Gym, and Key-to-Door tasks.

Decision Transformer: Reinforcement Learning via Sequence Modeling
Lili Chen 11 Kevin Lu Aravind Rajeswaran', Kimin Lee',
Aditya Grover Michael Laskin'
Pieter Abbeel', Aravind Srinivas Igor Mordatch 1.5
equal contribution
lequal advising
l UC Berkeley Facebook AI Research UUCA *OpenAl GGoogle Brain
{lilichen, kzl)@berkeley.ce edu
Abstract
We introduce framework that abstracts Reinforcement Learning (RL) as se- quence modeling problem. This allows to draw upon the simplicity and scalabil- ity of the Transformer architecture, and associated advances in language modeling such as GPT-x and BERT. In particular, we present Decision Transformer, an architecture that casts the problem of RI. as conditional sequence modeling. Un- like prior approaches to RL that fit value functions or compute policy gradients, Decision Transformer simply outputs the optimal actions by leveraging causally masked Transformer. By conditioning an autoregressive model on the desired return (reward), past states, and actions, our Decision Transformer model can gen- erate future actions that achieve the desired return. Despite its simplicity, Decision Transformer matches exceeds the performance of mtte-offae fe-thh--rtt art model-free offline RL baselines on Atari, OpenAl Gym, and Key-to-Doo to-ooor tasks.
a
a
t-1
linear decoder
causal transformer
emb. pos. enc.
R
a
R
a
t-1
21
return
state
action
Decision Transformer architecture]] States, actions, and returns are fed into modality- specific linear embeddings and a positional episodic timestep encoding is added. Tokens are fed into GPT architecture which predicts actions autoregressively using causal self-attention mask.
Figure 1:
Our code is available at: https://sitte. www.tttte..ooolll.ooo//ooooeeeee ootbb//rrrllllelyeeeeeeeeeiiiiaa
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

graph
training dataset (random walkss)
generation
goal
00
goal
-3
start
Figure Illustrative example of finding shortest path for a fixed graph (left) posed as reinforcement learning. Training dataset consists of random walk trajectories and their per-node returns-to-go (middle). Conditioned on a starting state and generating largest possible return at each node, Decision Transformer sequences optimal paths.
1 Introduction
Recent work has shown transformers can model large-scale distributions of semantic concepts, including capable zero-shot generalization in language [2] and impressive outotf-distribution image generation [3]. This stands in sharp contrast to much work in reinforcement learning (RL), which learns a single policy to model particular narrow behavior distribution. Given the diversity of applications and impact of transformer models, we seek to examine their application to sequential decision making problems. In particular, instead of using transformers as an architectural choice for traditional RL algorithms 5 we seek to study if trajectory modeling (analogous to language modeling) can serve as replacement for conventional RL algorithms.
We consider the following shift in paradigm: instead of training a policy through conventional RL algorithms like temporal difference (TD) learning 6 the dominant paradígm in RL, will train transformer models on collected experience using a sequence modeling objective. This will allow us to bypass the need for bootstrapping to propagate returns thereby avoiding one of the "deadly triad" [6] known to destabilize RL. It also avoids the need for discounting future rewards, as typically done in TD-Ieaming, which can induce undesirable short-sighted behaviors. Additionally, we can make use of existing transformer frameworks widely used in language and vision that are easy scale, utilizing a large body of work studying stable training of transformer models; this approach removes the need for specialized RL frameworks by appealing only to commonplace supervised learning systems. Given their demonstrated ability to model long sequences and wide data distributions, transformers also have other advantages. Transformers can perform credit assignment directly via self-attention, contrast to Bellman backups which slowly propagate rewards and prone to "distractor" signals 77 This enable transformers still work effectively in the presence of sparse or distracting rewards. Furthermore, a transformer modeling approach can model wide distribution of behaviors, enabling better generalization and transfer. While "upside-down" reinforcement learning (UDRL) [ [9 [0] also uses a supervised loss conditioned on a target return, work is motivated by sequence modeling rather than supervised learning and seeks benefit from modeling long sequences of behaviors. See Section for more discussions about related works.
We explore our hypothesis by considering offline RL, where we will task agents with learning policies from suboptimal data - pproducing maximally effective behavior from fixed, limited experience. This task is traditionally challenging due to error propagation and value overestimation [T.. However, it is a natural task when training with a sequence modeling objective. By training an autoregressive model on sequences of states, actions, and returns, we reduce policy sampling to autoregressive generative modeling. We can specify the expertise of the policy which "skill" to query by manually setting the return tokens, acting prompt for generation.
Illustrative example. To get an intuition for our proposal, consider the task finding a shortest path on a directed graph posed as an RL problem. The reward is 0 when at the goal node and - otherwise. We train GPT [12] model to predict next token in a sequence of returns-to-go (sum of future rewards), states, and actions. Training only on random walk data with no expert demonstrations we can at test time generate optimal trajectories by adding a prior to generate highest possible returns (see more details and empirical results in the Appendix) and subsequently generate actions

---

conditioned on that. Thus, by combining the tools of sequence modeling with hindsight return information, we achieve policy improvement without the need for dynamic programming.
Motivated by this observation, we propose Decision Transformer, where we use the GPT architecture autoregressively model trajectories (shown in Figure We study whether sequence modeling can perform policy optimization by evaluating Decision Transformer on offline RL benchmarks in Atari [3]. OpenAI Gym (14). and Key-to-Door [15] environments. We show that without using dynamic programming - Decision Transformer performs comparably on these benchmarks to state-of of-the-art model freee offline RL algorithms [16 [7]. Furthermore, in tasks where long-term credit assignment is required, Decision Transformer capably outperforms RL algorithms. With this work, we hope bridge vast recent progress in transformer models with RL problems.
2
Preliminaries
2.1 Offline reinforcement learning
We consider learning in Markov decision process (MDP) described by the tuple (S, A, P, R). The MDP tuple consists of states s E S, actions a E A, transition dynamics P(s'|s, (s, a), and a reward function r R(s, a). We use 8t, az and r, R(s,, a,) denote the state, action, and reward at timestep I, respectively. The goal in reinforcement learning is to learn a policy which maximizes the expected return E
Σn N' in an MDP. In offline reinforcement learning, instead of obtaining data via environment interactions, only have access to some fixed limited dataset consisting of trajectories from the environment. This setting is harder as it removes the ability for agents to explore the environment and collect additional feedback.
2.2 Transformers
Transformers were proposed by Vaswani et al. [ architecture efficiently model sequences. They consist of stacked self-attention layers with residual connections. Each self-attention layer receives n embeddings {= corresponding to unique input tokens, and outputs embeddings { preserving the input dimensions. The i-th token is mapped via linear transformations to key hi, query qi, and value u, The i-th output of the self-attention layer is given by weighting the values Uj by the normalized dot product between the query qi and other keys ky:
24 Σ
ky =1) Vy.
(1)
This allows the layer to assign "credit" by implicitly forming state-return associations via similarity of the query and key vectors (maximizing the dot product). In this work, we use the GPT architec- ture [12], which modifies the transformer architecture with causal self-attention mask enable autoregressive generation, replacing the summation/softmax over the n tokens with only the previous tokens in the sequence E [1) We defer the other architecture details the original papers.
3 Method
In this section, we present Decision Transformer, which models trajectories autoregressively with minimal modification to the transformer architecture, as summarized in Figure [ and Algorithm []
Trajectory representation. The key desiderata in our choice of trajectory representation are (a) should enable transformers to learn meaningful patterns and (b) we should be able to conditionally generate actions at test time. It is nontrivial to model rewards since we would like the model to generate actions based on future desired returns, rather than past rewards. As a result, instead
of modeling the rewards directly, model the returns-to-go R. Er=z Pv. This leads to the following trajectory representation which is amenable to autoregressive training and generation:
T T=
[ (1,,81,00 a1, R2, 82, a2,..........................a.. Ri, 8T, (7,a7)
(2)
Architecture, We feed the last K timesteps into Decision Transformer, for a total of 3K tokens (one for each modality: return-to-go, state, or action). To obtain token embeddings, we learn linear
3

---

layer for each modality, which projects raw imputs to the embedding dimension, followed by layer normalization [8] For environments with visual inputs, the state fed into convolutional encoder instead of linear layer. Additionally, token note this is different than the standard positional embedding used by transformers, as one timestep corresponds to three tokens. The tokens are then processed by GPT [12 model, which predicts future action tokens via autoregressive modeling.
embedding for each timestep is learned and added to each
Training. We sample minibatches of sequence length K from the dataset. The prediction head corresponding the input token 8 trained to predict a, either with cross-entropy loss for discrete actions or mean-squas -qquree error for continuous actions and the losses for each timestep are averaged. We did not find predicting the states or returns-t--g go to necessary for good performance, although it is possible (as shown in Section 5.3) and would be an interesting study for future work.
Evaluation. During evaluation rollouts, we specify target return based (e.g., specify maximum possible return to generate expert behavior) as well as the environment starting state, initialize generation. After executing the generated action, decrement the target return by the achieved reward and obtain the next state. We repeat this process of generating actions and applying them obtain the next return-to-oo and state until episode termination.
desired performance
Algorithm Decision Transformer Pseudocode (for continuous actions)
R, S, t: returns-tar to states, actions, # K: context length (length of each input to DecisionTransformerr) transformer: transformer with causal masking (GPT) embed_s, embed_a, embed_R: linear embedding layers embed_t: learned episode positional embedding pred_a: linear action prediction layer
timesteps
main model
def DecisionTramsforrer (R, 8, a, t):
compute embeddings for tokens
pos_ eese_mmbeding embed_t(t) # per-timestep (note: not per-token) s_embedding embed_s(s) pos_embedding a a_embedding embed_a(a) R R_embedding embed_R(R)
pob_endedding
poa_embedding
interleave tokens as (R.1, s_1, a_l, R_K, s_K)
input_embeds stack (R__ceRR_eeedie (R_embedding, s_embedding, a_embedding)
use transformer get hidden states
hidden_states = transformer (input_enbeds- ilaptteebbd-iimtttaaabb input_embees))
# select hidden states for action prediction tokens
a a_hidden = unstack (hidden,, statansstttttatttt.e.......... actions
predict action return pred_a(a_hidde)
training 1oop for (R, a, t) in dataloader: dims: (batch_size, dim) a_preds DecisionTransformer (R, a, t)
loss mean ((a_preds a)++2) # 12 loss for continuous actions
optimizer. (r) loss. backvard (); optimizer. step()
evaluation loop
target return R, 8, a, t, done [target_return]] [env [env. reset ()], [], [1], False
# for instance, expert-leved return
autoregressive caneratiin/samplinn while not done: action sample action Deiisionransfomerr (R, a, t)[-1] for cts actions new_s, r, done,
env mtv.ste((action)
# append new tokens to sequence
R R + [R[-1] - r] # decrement returns- -orrns-to-ae to- with revard s, a, s [new_s], a [action], t + [len (R)]
R, 8, a, t R[-K]],
# only keep context length of

---

Decision Transformer (Ours)
TD Learning
Behavior Cloning
Performance
100
50
Atari
OpenAl Gym
Key-To-Door
Figure Results comparing Decision Transformer (ours) to TD learning (CQL) and behavior cloning across Atari, OpenAl Gym, and Minigrid. On diverse set of tasks, Decision Transformer performs comparably or better than traditional approaches.
Evaluations on offline RL benchmarks
In this section, investigate if Decision Transformer can perform well compared standard TD and imitation learning approaches for offline RL. TD learning algorithms represent the conventional state- of-the-art, while imitation learning algorithms have similar formulations Decision Transformer. The exact algorithms depend on the environment but our motivations are as follows:
TD learning: most of these methods use an action-space constraint or value pessimism, and will be the most faithful comparison to Decision Transformer, representing standard RI. methods. A state-of-the-an model-free method Conservative Q-Learning (CQL) [17] which serves as our primary comparison. In addition, we also compare against other prior model-free RL algorithms like BEAR [19] and BRAC [20).
Imitation learning: this regime similarly uses supervised losses for training, rather than Bellman backups. We use behavior cloning here, and include more detailed discussion in Section 5.]
We evaluate on both discrete (Atari (3I) and continuous (OpenAI Gym [4] former requires long-term credit assignment, while the latter requires fine-grained continuous control, representing a diverse set of tasks. Our main results are summarized in Figure where we show averaged expert normalized performance for each domain.
control tasks. The
4.1 Atari
The Atari benchmark challenging due to its high- high-dimensional visual inputs and difficulty of credit assignment arising from the delay between actions and resulting rewards. We evaluate method on % all samples in the DQN-replay dataset as per Agarwal et al. [6], representing 500 thousand of the 50 million transitions observed by online DQN agent [2]] during training; report the mean and standard deviation of 3 seeds. We normalize scores based on a professional gamer, following the protocol of Hafner et al. [22] where 100 represents the professional gamer score and represents random policy.
We compare to CQL 17 REM [6]. and QR-DQN 23 on four Atari tasks (Breakout, Qhert, Pong.
and Seaquest) that are evaluated in Agarwal et [16]. We use context lengths of 30 for Decision Transformer (except K 50 for Pong); for results with different values of K see the supplementary material. We also report the performance of behavior cloning (BC), which utilizes
Game
DT (Ours)
CQL
QR-DQN
REM
BC
Breakout
267.5 97.5
211.1
21.1
32.1
138.9
61.7
Qbert
15.1 + 11.4
104.2
1.4
17.3 14.7
Pong
106. 8.1
111.9
20.0
39.1
85.2
20.0
Scaquest
2.4 0.7
1.7
1.4
1.0
2.1 - 0.3
Table 1:
Gamer-normalized scores for the 1% DQN-replay Atari dataset. We report the mean and variance across seeds. Best mean scores are highlighted in bold. Decision Transformer (DT) performs comparably to CQL on 3 out of games, and outperforms other baselines in most games.

---

the same network architecture and hyperparameters as Decision Transformer but does not have eeturn-to-go conditioning" For CQL, REM, and QR-DQN baselines, we report numbers directly from the CQL paper. We show results in Table I] Our method is competitive with CQL in 3 out of games and outperforms or matches REM, QR-DQN, and BC on all games.
4.2 OpenAl Gym
In this section, we consider the continuous control tasks from the D4RL benchmark [24] We also consider 2D reacher environment that not part of the benchmark, and generate the datasets using similar methodology to the D4RL benchmark. Reacher is a goal-conditioned task and has sparse rewards, so represents different setting than the standard locomotion environments (HalfCheetah, Hopper, and Walker). The different dataset settings are described below.
Medium: million timesteps generated by "medium" policy that achieves approximately one-third the score of expert policy.
2. Medium Medium-Replay. the replay buffer of an agent trained to the performance of a medium policy (approximately 25k-400k timesteps in our environments).
3. Medium-Fxpert: million timesteps generated by the medium policy concatenated with million timesteps generated by an expert policy.
We compare to CQL [17]. BEAR 19, BRAC 20 and AWR 225 CQL represents the state-of- the-art in model-free offline RL, an instantiation of 'TD learning with value pessimism. Score are normalized so that 100 represents an expert policy, as per Fu al. 24 CQL numbers are reported from the original paper; BC numbers are run by us; and the other methods are reported from the D4RL paper. Our results are shown in Table 2
Decision Transformer achieves the highest scores in majority of the tasks and is competitive with the state of the art in the remaining tasks.
Dataset
Environment
DT (Ours)
CQL
BEAR
BRAC-
AWR
BC
Medium Expert
HalfCheetah
86.8 + 1.3
62.4
53.4
41.9
52.7
59.9
Medium-Exppe Expert
Hopper
107.6 1.8
111.0
96.3
0.8
27.1
79.6
Medium- Medium-Expert
Walker
108. 0.2
98.7
40.1
81.6
53.8
36.6
Medium- Exppert
Reacher
89.1 + 1.3
30.6
T3.3
Medium
HalfCheetah
42.6
44.4
41.7
46.3
37.4
43.1
Medium
Hopper
67.6
58.0
52.1
31.1
35.9
63.9
Medium
Walker
74.0 1.4
79.2
59.1
81.1
17.4
77.3
Medium
Reacher
5..2
26.0
48.9
Mediium-Replan
HalfCheetah
36.6
46.2
38.6
47.7
40.3
4.3
Medium- Replay
Hopper
82.7 1 7.0
48.6
33.7
0.6
28.4
27.6
Medium- Replly
Walker
66.6 3.0
26.7
19.2
0.9
15.5
36.9
Rediumm-Reply
Reacher
18.0 1 2.4
19.0
5.4
Average
(Without Reacher)
74.7
63.9
48.2
36.9
34.3
46.4
Average (All Settings)
69.2
54.2
17.7
Table 2: Results for D4RL d We report the mean and variance for three seeds. Decision Transformer (DT) outperforms conventional RL algorithms almost all tasks.
5
Discussion
5.1 Does Decision Transformer perform behavior cloning on a subset of the data?
In this section, seek gain insight into whether Decision Transformer performing imitation learning on subset of the data with a certain return. To investigate this, we propose a new method, Percentile Behavior Cloning (%BC), where we run behavior cloning on only the top X% of timesteps in the dataset, ordered by episode returns. The percentile X % interpolates between standard BC (X 100%) that trains on the entire dataset and only cloning the best observed
be thought of as
We also tried using an MI.P with K 1 as in prior work, but found this was worse than transformer. 'Given that COL is generally the strongest TD learning method, for Reacher only the CQL baseline.

---

Dataset
Enviromment
DT (Ours)
10%BC
25% BC
40%BC
100% BC
CQL
Medium
HalfChectah
42.6
42.9
43.0
43.1
43.1
44.4
Medium
Hopper
67.6 + 1.0
65.9
65.2
65.3
63.9
58.0
Medium
Walker
74.0 1.4
78.8
80.9
78.8
77.3
79.2
Medium
Reacher
51.2
51.0
48.9
58.2
58.4
26.0
Medium Replay
HalfChectah
0.8
40.8
40.9
41.1
4.3
46.2
Sedium-Replas
Hopper
82.7 7.0
70.6
58.6
31.0
27.6
48.6
Medium-RRepaa
Walker
66.6 - 3.0
70.4
67.8
67.2
36.9
26.7
Medium-Replay
Reacher
18.0 2.4
33.1
16.2
10.7
5.4
19.0
Average
56.1
56.7
52.7
49.4
39.5
43.5
Table 3: Comparison between Decision Transformer (DT)) and Percentile Behavior Cloning (%BC).
trajectory (X - 0%), trading off between better generalization by training more data with training a specialized model that focuses on desirable subset the data.
We show full results comparing %BC to Decision Transformer and CQL in Table 3 sweeping over X [10%, 25%, 40%. 100%]. Note that while both %BC and DT introduce hyperparameters, returns are human interpretable and is relatively natural for humans to specify a desired return compared to choosing an optimal subset for cloning. When data is plentiful as the D4RL regime we find %BC can match or beat other offline RI methods. On most environments, Decision Transformer is competitive with the performance of the best &BC, indicating it can hone on particular subset after training the entire dataset distribution.
In contrast, when we study low data regimes succ as Atari, where as the dataset- - %BC is weak (shown in Table 4 This suggests that in scenarios with relatively low amounts of data, Decision Transformer dataset to improve generalization, even if those trajectories are dissimilar from the return conditioning target. Our results indicate that Decision Transformer can be more effective than simply performing imitation learning on subset of the dataset, On the tasks we considered, Decision Transformer either outperforms or competitive BCC, without the confound of having to select the optimal subset.
1% of a replay buffer
outperform %BC by using all trajectories in the
Game
DT (Ours)
B0%BC
40%BC
100%
Breakout
267.5 + 97.5
28.5 8.2
73.5 6.4
108.2 + 67.5
138.9 61.7
Qbert
15.1 + 11.4
6.6 1.7
16.0 13.8
11.8
17.3 - 14.7
Pong
106. 1 + 8.1
2.5 0.2
13.3 2.7
72.7 + 13.3
85.2 = 20.0
Seaquest
2.4 0.7
1.1 0.2
1.1 1 0.2
1.6 0.4
2.1 + 0.3
Table 4: &BC scores for Atari. We report the mean and variance across seeds. Decision Transformer (DT) outperforms all versions of %BC in most games.
5.2 How well does Decision Transformer model the distribution of returns?
We evaluate the ability of Decision Transformer to understand return-to-ge tokens by varying the desired target return over wide range evaluating the multi-task distribution modeling capability of transformers. Figure shows the average sampled return accumulated by agent over the course of the evaluation episode for varying values of target return. On every task, the desired target returns and the true observed returns are highly correlated. On some tasks like Pong, HalfCheetah and Walker, Decision Transformer generates trajectories that almost perfectly match the desired returns (as indicated by the overlap with the oracle line). Furthermore, on some Atari tasks like Seaquest, we can prompt the Decision Transformer with higher returns than the maximum episode return available the dataset, demonstrating that Decision Transformer is sometimes capable extrapolation.
5.3 Does Decision Transformer perform effective long-term credit assignment?
To evaluate long-term credit assignment capabilities of our model, we consider variant of the Key-to-Door environment proposed in Mesnard et [15). This is grid-based environment with sequence of three phases: (1) in the first phase, the agent is placed in room with key; (2) then, the

---

Breakout
Obert
Pong
Seaquest
Perfcimance
200
100
100
203
350
20
40
63
80
100
50
100
133
HalfChectah
Hopper
Walker
Reacher
Performance
100
20
50
75
Return (Normaized)
Return (Normalized)
Target (Normalized)
Target Return (Normalized)
Decision Transformer
Oracle
Best Trajectory in Dataset
Figure 4: Sampled (evaluation) returns accumulated by Decision Transformer when conditioned on the specified target (desired) returns. Top: Atari. Bottom: D4RL medium-replay datasets.
agent is placed in empty room; (3) and finally, the agent is placed in room with door. The agent receives a binary reward when reaching the door in the third phase, but only if it picked up the key in the first phase. This problem is difficult for credit assignment because credit must be propagated from the beginning to the end of the episode, skipping over actions taken in the middle.
We train on datasets of trajectories generated by applying random actions and report success rates T Furthermore, for the Key-to-Door environment we use the entire episode length as the context, rather than having fixed content window in the other environments. Methods that use highsight return information: our Decision Transformer model and %BC (trained only on successful episodes) are able to learn effective policies producing near-optimal paths, despite only training on random walks. TD learning (CQL) cannot effectively propagate Q-values over the long horizons involved and gets poor performance.
Dataset
DT (Ours)
CQL
BC
% BC
Random
1K Random Trajectories 71.8%
13.1%
1.4%
69.9%
3.1%
10K Random Trajectories
94.6%
13.3%
1.6%
95.1%
3.1%
Table 5: Success rate for Key-to- -ooo environment. Methods using hindsight (Decision Transformer, %BC) can learn successful policies, while TD learning struggles to perform credit assignment.
5.4 Can transformers be accurate critics in sparse reward settings?
In previous sections, we established that decision transformer can produce effective policies (actors). We now evaluate whether transformer models can also be effective critics. We modify Decision Transformer to output return tokens addition action tokens on the Key-to-Door environment. We find that the transformer continuously updates reward probability based on events during the episode, shown in Figure 5 (Left). Furthermore, we find the transformer attends to critical events in the episode (picking up the key or reaching the door), shown in Figure 5 (Right), indicating formation of stwae-reward associations discussed in Raposo al. (26 and enabling accurate value prediction.
5.5 Does Decision Transformer perform well in sparse reward settings?
A known weakness of TD learning algorithms is that they require densely populated rewards in order perform well, which can be unrealistic and/or expensive. In contrast, Decision Transformer can improve robustness in these settings since it makes minimal assumptions on the density of the reward. To evaluate this, we consider delayed return version of the D4RL benchmarks where the agent does not receive any rewards along the trajectory, and instead receives the cumulative reward of the trajectory the final timestep. Our results for delayed returns are shown in Table Delayed returns minimally affect Decision Transformer: and due to the nature of the training process, while imitation learning methods are reward agnostic. While TD learning collapses, Decision Transformer and %BC still perform well, indicating that Decision Transformer can be more robust to delayed rewards.

---

not nick up key
ptck up and roach door
Frestded reward pwrrdepbbllii
0.8
and neach
Attention weight
0.6
0.66
0.4
92
key room
d strractor FOOnT mode time
door room
pick key
neach door
Up sode time
Figure 5: Left: Averages of running return probabilities predicted by the transformer model for three types of episode outcomes. Right: Transformer attention weights from all timesteps superimposed for particular successful episode. The model attends to steps near pivotal events in the episode, such picking up the key and reaching the door.
Delayed (Sparse) DT (Ours)
Agnostic
Original (Dense) DT (Ours)
Dataset
Environment
CQL
BC
BBC
CQL
Medium-Fxpert
Hopper
107.3 3.5
9.0
59.9
102.6
107.6
111.0
Medium
Hopper
60.7 4.5
5.2
63.9
65.9
67.6
58.0
Meeium-Reepay
Hopper
78.5 3.7
2.0
27.6
70.6
82.7
48.6
Table 6: Results for D4RL datasets with delayed (sparse) reward. Decision Transformer (DT) and imitation learning are minimally affected by the removal of dense rewards, while CQL fails.
5.6
Additional Discussions
For more discussions see the supplementary material.
Related work
Offline reinforcement learning. To mitigate the impact of distribution shift in offline RL, prior
algorithms either (a) constrain the policy action space [27 28 29 or (b) incorporate value pes- simism (27 17 or (c) incorporate pessimism into learned dynamics models 30 31. Since we do Decision Transformers explicitly learn the dynamics model, primarily compare against model-free algorithms; adding a dynamics model tends to improve the performance of model-free algorithms. Another line of work explores learning wide behavior distribution from an offline dataset by learning a task- aggostic set of skills, either with likelihood-basee approaches [32 33 34 335 or by approaches, maximizing mutual information [36] 37 38). Our work similar to the likelihood-basse which do not use iterative Bellman updates although we use a simpler sequence modeling objective instead of variational method, and use rewards for conditional generation of behaviors.
Supervised learning in reinforcement learning settings. Some prior methods for reinforcement learning bear more resemblance to static supervised learning, such Q-learning (39 40), which still
uses iterative backups, or likelihood-based methods such as behavior cloning, which do not (discussed in previous section). Recent work [ TQ studies "upside-down" reinforcement learning (UDRL), which are similar to our method in seeking to model behaviors with supervised loss conditioned on the target return. key difference our work the shift of motivation to sequence modeling rather than supervised learning: while the practical methods differ primarily in the context length and architecture, sequence modeling enables behavior modeling even without access the reward, in similar style to language [12] or images [1]. and is known to scale well 2 The method proposed by Kumar al. [9] is most similar to our method with K 1, which we find sequence modeling/long contexts to outperform (see supplementary material). Ghosh et al. [22 extends prior UDRL methods to use state goal conditioning, rather than rewards, and Paster et al. 44 further use an LSTM with state goal conditioning for goal-conditoned online RI. settings. Concurrent to our work, Janner et al. 44 propose Trajectory Transformer, which similar to Decision Transformer but additionally uses state and return prediction, well discretization, which incorporates model-based components.

---

We believe that their experiments, in addition to our results, highlight the potential for sequence modeling to be a generally applicable idea for reinforcement learning.
Credit assignment. Many works have studied better credit assignment via state-association, learning an architecture which decomposes the reward function such that certain "important" states comprise most of the credit [45 46 15]. They use the learned reward function to change the reward of an actor- critic algorithm to help propagate signal over long horizons. In particular, similar our long-term setting, some works have specifically shown such state-associative architectures can perform better in 48 26]. In contrast, we allow these properties to naturally emerge in transformer architecture, without having to explicitly learn reward function or critic.
delayed reward settings 47
Conditional language generation. Various works have studied guided generation for images [9] and language 500 51. Several works 12
53
54
55
56 57 have explored training or fine-tuning
of models for controllable text generation.
language models can also be used to 59 60). However, these approaches mostly assume constant "classes", while in reinforcement learning the reward signal is time-varying, Furthermore, is more natural to prompt the model desired target return and continuously decrease it by the observed rewards over time, since the transformer model and environment jointly generate the trajectory.
learn disciminators to guide generation [58 50
Attention and transformer models, Transformers [ have been applied successfully to many tasks in natural language processing [2] and computer vision 66 63). However, transformers are relatively unstudied RL, mostly due to differing nature of the problem. such as higher variance in training. Zambaldi et al. (5] showed that augmenting transformers with relational reasoning improve performance in combinatorial environments and Ritter et al. [4 showed iterative self- attention allowed for RL agents better utilize episodic memories. Parisotto et al. 4 discussed design decisions for more stable training of transformers in the high-variance RL setting. Unlike our work, these still use actor-critic algorithms for optimization, focusing on novelty in architecture. Additionally, in imitation learning, some works have studied transformers as replacement for LSTMs: Dasari and Gupta [65] study one-shot imitation learning, and Abramson al. 66] combine language and image modalities for coxticondi behavior generation.
7 Conclusion
We proposed Decision Transformer, seeking to unify ideas language modeling and RL. On standard offline RL benchmarks, we showed DT can match or outperform strong algorithms designed explicitly for offline RL with minimal modifications from standard language modeling architectures.
Societal impact. For real-world applications, it is important to understand the types of errors transformers make in MDP settings and possible negative consequences. It will also be important to consider the datasets we train on, which can potentially add destructive biases, particularly as we consider studying augmenting RL agents with more data which may come from questionable sources.
Linitations. We introduced our paradigm shift and showed its potential in our experiments, but there significant room for more research in this direction. The current architecture requires considerations of context length and return-to-go hyperparameters, and we show results on standard RL benchmarks; future work could improve the architecture and demonstrate results in more complex environments and tasks. We used a simple supervised loss that was effective in our experiments. but applications to large-scale datasets could benefit from self-supervised pretraining tasks. In addition, one could consider more sophisticated embeddings for returns, states, and actions. While we do not directly evaluate scaling and generalization, we utilize method known to scale generalize well in domains such as language and vision, and we are excited about larger RL systems built upon our framework.
8 Acknowledgements
This research was supported by Berkeley Deep Drive, Open Philanthropy, and the National Science Foundation under NSF:NRI #2024675. Part of this work was completed when Aravind Rajeswaran was a PhD student at the University of Washington, where he was supported by the J.P. Morgan PhD Fellowship in AI (2020-21). We also thank Luke Metz, Daniel Freeman, and anonymous reviewers for valuable feedback and discussions, as well as Justin Fu for assistance in setting up D4RL benchmarks, and Aviral Kuman for assistance with the CQL basclines and hyperparameters.
10

---

References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- tion Processing Systems, 2017.
[2] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
[3] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102. 120.2 2021.
[4] Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayaku- mar, Max Jaderberg, Raphael Lopez. Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. In International Conference on Machine Learning. 2020.
[5] Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Deep reinforcement learning with relational inductive biases. In International Conference on Learning Representations, 2018.
[6] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT Press, 2018.
[7] Chia-Chun Hung, Timothy Lillicrap. Josh Abramson, Yan Wu, Mehdi Mirza, Federico Carnevale, Arun Ahuja, and Greg Wayne. Optimizing agent behavior over long time scales by transporting value. Nature communications, 10(1):1-12, 2019.
[8] Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojeiech Jaskowski, and Jürgen Schmidhuber. Training agents using upside-down reinforcement learning. arXiv preprint arXiv:1912.0283 022777777 2019.
[9] Aviral Kumar, Xue Bin Peng, and Sergey Levine. Reward-conditioned policies. arXiv preprint 1rX1::191..13465, 2019.
[10] Acting without rewards. acting-without-rewarrs
2019.
URL
nttts://ogma.ai/000
[11] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0 2020.
[12] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training 2018.
[13] Marc Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-270 2013.
[14] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 1606.0/4000014400 2016.
[15] Thomas Mesnard, Théophane Weber, Fabio Viola, Shantanu Thakoor, Alaa Saade, Anna Harutyunyan, Will Dabney, Tom Stepleton, Nicolas Heess, Arthur Guez, et al. Counterfactual credit assignment in model-free reinforcement learning. arXiv preprint arXiv:2011.09440 2020.
[16] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, 2020.
[17] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2020.

---

[18] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 16X.:1607.004500 2016.
[19] Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. arXiv preprini arXiv:1906.0094) 2019.
[20] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11300 2019.
[21] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533 2015.
[22] Danijar Hafner, Timothy Lillicrap. Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020.
[23] Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Conference on Artificial Intelligence, 2018.
[24] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[25] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weiighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 2019.
(26) David Raposo, Sam Ritter, Adam Santoro, Greg Wayne, Theophane Weber, Matt Botvinick, Hado van Hasselt, and Francis Song. Synthetic returns for long-term credit assignment. arXiv preprint arXiv:2102.1 /2425, 2021.
[27] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforeement learning without exploration. In International Conference on Machine Learning, 2019.
[28] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, 2019.
[29] Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. In International Conference on Learning Representattinss 2020.
[30] Rahul Kidambi, Aravind Rajeswaran, Pranceth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
[31] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In Advances in Neural Information Processing Systems, 2020.
[32] Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Of- fline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010. 13611, 2020.
[33] Victor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giro-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills, In International Conference on Machine Learning, 2020.
[34] Karl Pertsch, Youngwoon Lee, and Joseph J Lim. Accelerating reinforcement learning with learned skill priors. arXiv preprint arXiv:2010. / 1944, 2020.
[35] Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. International Conference on Learning Representations, 2021.
12

---

[36] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. In International Conference on Learning Representations, 2019.
[37] Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Reset-free lifelong learning with skill-space planning. arXiv preprint arXiv:2012.0333 2,,.00388, 2020.
[38] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamies- aware unsupervised discovery of skills. In International Conference on Learning Representa- rions, 2020.
[39] Christopher Watkins. Learning from delayed rewards. 01 1989.
[40] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1rXiv:1312.5602, 2013.
[41] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pages 1691-1703. PMLR, 2020.
[42] Dibya Ghosh, Abhishek Gupta, Justin Fu, Ashwin Reddy, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals without reinforcement learning. arXiv preprint arXiv:1912.0600: 060888 2019.
[43] Keiran Paster, Sheila A Mcllraith, and Jimmy Ba. Planning from pixels using inverse dynamics models. arXiv preprint arXiv:20/2.02419, 2020.
[44] Michael Janner, Qiyang Li, and Sergey Levine. Reinforcement learning as one big sequence modeling problem. urXiv preprint arXiv:2106.020333 2021.
[45] Johan Ferret, Raphaël Marinier, Matthieu Geist, and Olivier Pietquin. Self-attentional credit assigmment for transfer in reinforcement learning. arXiv preprint arXiv:1907.08027, 2019.
[46] Anna Harutyunyan, Will Dabney, Thomas Mesnard, Mohammad Azar, Bilal Piot, Nicolas Heess, Hado van Hasselt, Greg Wayne, Satinder Singh, Doina Precup, et al. Hindsight credit assignment. arXiv preprint arXiv: 19/2.0///200200: 2019.
[47] Jose A Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. arXiv preprint arXiv: 07857, 2018.
[48] Yang Liu, Yunan Luo, Yuanyi Zhong, Xi Chen, Qiang Liu, and Jian Peng. Sequence mod- eling of temporal credit assignment for episodic reinforcement learning. arXiv preprint 1905. 13420, 2019.
[49] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Conference on Computer Vision and Pattern Recognition, 2019.
[50] Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. Hafez: an interactive poetry generation system. In Proceedings of ACI. System Demonstrations, 2017.
[51] Lilian Weng- log, 2021.
Controllable neural text generation. filiunweng.giiluu..ooo io/li- URL h 2iihab..oo/11-11o2200////// html
[52] Jessica Ficler and Yoav Goldberg, Controlling linguistic style aspects in neural language generation. arXiv preprint N.Kiv:/777.0263: 2017.
[53] Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. Toward controlled generation of text. In International Conference Machine Learning, 2017.
[54] Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv: 1906.9906666 2019.
13

---

[55] Lantao Yu, Weinan Zhang, Jun Wang. and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. AAAI conference on artificial intelligence, 2017.
[56] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08530 2019.
[57] Nitish Shirish Keskar, Bryan McCann, Lav Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint WwXiv:/909.0588 2019.
[58] Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv:
2019.
[59] Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators. arXiv preprint /805.0 2018.
[60] Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. Gedi: Generative discriminator guided sequence generation. arXiv preprint (tt::20000063667 2020.
[61] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 18/::8100000 0.805, 2018.
[62] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. European Conference on Computer Vision, 2020.
[63] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint
arXiv:2010.11922 2020.
[64] Sam Ritter, Ryan Faulkner, Laurent Sartran, Adam Santoro, Matt Botvinick, and David Raposo. Rapid task-solving in novel environments. arXiv preprint arXiv:2006.03662 2020.
[65] Sudeep Dasari and Abhinav Gupta. Transformers for one-shot visual imitation. arXiv preprint arXiv:2011.05970 2020.
[66] Josh Abramson, Arun Ahuja, lain Barr, Arthur Brussee, Federico Carnevale, Mary Cassin, Rachita Chhaparia, Stephen Clark, Bogdan Damoe, Andrew Dudzik, al. Imitating interactive intelligence. arXiv preprint 2rXiv:20/2.05677 2020.
14


Reviews:
[{"review_id": "vHA7htDA2F", "reviewer": "Reviewer_dbvQ", "summary": "This paper uses the Transformer model as an auto-regressive model to predict the actions (decisions) givens the state inputs, past actions, and more importantly, the return-to-go. Training such a policy becomes a supervised-learning problem, similar to behavior cloning. The paper applies this method to offline RL problems and demonstrates that policy trained in this supervised manner achieves competitive results compared to SOTA model-free offline RL algorithms.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is a combination of existing techniques (Transformer + Upside-down RL) and shows that this combination yields better or comparable performance compared to BC and some SOTA offline RL algorithms.\n\nStrengths:\n* The method is clearly explained. The paper proposed a simple method to improve behavior cloning performance by conditioning the policy on the return-to-go. Based on the paper writing, one can easily implement the method.\n\n* I like the idea of conditioning the policy on the return-to-go value. In behavior cloning, typically the policy only depends on the states (sometimes also the previous actions). If the training dataset contains multiple state-action pairs on the same state but with different actions, then the policy learning can be hurt as it cannot capture multi-modality. The paper proposes to condition the policy with the return-to-go value as well. With the return value, now the model can have some capacity to fit multiple state-action pairs in the same state because it is very likely that these state-action pairs will have different return values even though the states are the same. This technique can be useful when there is multiple action supervision in the same state or when there is suboptimal action supervision. Note that as the authors also point out in the related work, there are prior works that use this idea such as the Upside-down RL paper.\n\n* The paper does a thorough comparison with SOTA offline RL methods and provides experiments on several environments.\n\n\nWeaknesses:\n\n* I cannot get why %BC is not a realistic approach. A better explanation would be helpful. In Decision Transformer (DT), one would need to know the return-to-go value to deploy the policy as it's part of the policy input. Then BC can also use this value to filter out the good trajectories on which BC is performed. So it seems that both methods can have the same assumption. If the authors mean that it requires policy deployment to determine whether to use 10%, 25%, 40%, or 100% of the training data, then one can use a threshold return value to filter out the trajectories. So instead of showing the BC performance with %BC, one can show the BC performance trained with datasets that are filtered with different threshold return values.\n\n* This paper seems to over-emphasize the role of the Transformer. From my understanding, conditioning the policy on the return-to-go value seems to be the key to why it can outperform BC. If the authors think that Transformer is the key, then it is necessary to show this with a baseline comparison on LSTM/GRU policies that have the same policy input as the ones used in the paper.\n\n* I would like to see more analysis on why the method works. While there are many experiments showing the performance of DT, little analysis is actually trying to explain why DT works better than BC. Section 5.1 gives some analysis that BC trained with top $X$% of data has similar performance with DT. This could be that DT learns to do BC only on good trajectories and ignore the rest, or DT is able to fit all the trajectories by conditioning the policy on return-to-go value which can tell how good the action is. Also, Is it because DT has the capacity to better deal with multi-modality via the return-to-go policy input? \n\n* An ablation study on the value of the sequence length $K$ is necessary to understand how important this hyperparameter is and is it very sensitive to each game.\n\n* If it is the case that policy in DT can fit all trajectories because the return-to-go value can help the policy distinguish different action supervision even if the state input is very similar, then I would like to see experiments on binary sparse-reward environments. In these environments, the return values for all the successful episodes will be the same (e.g., $+1$) and the return values for all failed episodes will also be the same (e.g., $0$). How does DT perform in these sparse environments?\n\n* The details of the Key-to-Door environments are missing.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper did not state clearly the limitations of the proposed method. Possible limitations can be on the type of environments where the method can work, the episode length constraint due to the use of attention module, the sensitivity of the hyperparameters such as the training sequence length $K$, etc.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "b9PI0KxSIvh", "reviewer": "Reviewer_8P8v", "summary": "This paper proposes a sequence-model approach for offline RL, which does not need to solve cumbersome dynamic programming subproblems. It utilizes a transformer architecture to effectively model state-action sequence which is conditional to the return-to-go. The proposed method shows competitive or superior performance compared to the previous state-of-the-art in various offline RL datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n- The paper provides a novel and valuable direction to solve RL problems without TD learning which have been fundamental to conventional RL algorithms despite being hard to optimize.\n- Transformers have shown remarkable performance across NLP, vision, and GNN. This work introduces a nice way to integrate this architecture to the RL domain.\n- The proposed method performs exceptionally well on long-term credit assignments, which seems plausible considering the origin of the transformer architecture.\n\nCons\n- Based on the comparisons with %BC, it seems the proposed method is just a better (or more sample-efficient) way to perform behavior cloning. Can the authors provide some evidence that DT is more than a BC? For example, can the method discover behaviors not directly included in the dataset?\n- The authors argue that %BC is not a realistic approach as it needs online evaluations to choose the optimal subset size. However, this goes the same with DT as it needs to tune its hyperparameters ($K$, return-to-go). The appendix shows that the optimal hyperparameters vary between environments.\n- The paper does not list the hyperparameter search procedure (search range, selection criteria) at all, which is especially important for offline RL as online evaluations are (conceptually) impossible.\n- Sensitivity to the hyperparameters ($K$, return-to-go conditioning) is not given.\n- Std values for %BC results are missing in Table 3.\n- Exploration is a crucial part for RL algorithms. However, it is unclear how to expand this work to perform explorations with DT.\n\nOther comments\n- Does the method fail when it is given an extremely OOD return-to-go value (e.g., $\\times 0$, $\\times 5$, $\\times 10$, ...) ?\n- How does the method perform on other D4RL datasets such as *-random and *-full-replay?\n- The algorithm is explained only in high-level phrases. Including a pseudo-code of the training and the inference procedure may help the readers understand the method concretely.\n\n========== Update ==========\n\nThe author response addresses most of my concerns, and I think the updated manuscript will be valuable to the machine learning community. I am increasing my rating to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations\n- Noted above\n\nSocietal Impact\n- N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "MQaS0B2X9AQ", "reviewer": "Reviewer_BnKN", "summary": "The paper investigates the power of Transformers in sequential decision-making tasks. The authors suggest to use trajectory modeling in contrast to using Transformers as a model architecture in conventional RL algorithms. The proposed algorithm is called decision transformer and outperforms model-free offline RL algorithms in tasks requiring long-term credit assignment. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed reading this paper. The paper is beautifully organized and very well written. Decision Transformer is well motivated and positioned. \n\nThe only concern that I have which I think is extremely important is the following: \n\nThe authors did not compare Decision Transformers to other related works where Transformers are used as a model architecture within an existing RL method. For instance, the authors did in fact mention (very briefly) the differences of their method compared to reference 43 of the paper in the related works section, but there is no experiment comparing the tradeoff between “Transformers as a model architecture in RL” vs “the paradigm shift proposed here”. Could you please comment? The paper can benefit significantly from an additional experiment here.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed partially the limitations of their approach.\nI do not see any potential negative societal impact from this work.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "2xjq8xjE9qB", "reviewer": "Reviewer_SzBr", "summary": "* This paper introduces Decision Transfomer: a transformer based model that treats RL as a simple sequence modelling problem, and thus is able to eschew dynamic programming bootstrapping involved in contemporary RL algorithms.\n* The experiments on Atari and gym environments show that by using an appropriate return as “prompt”, DTs can model the desired behavior, including perform comparably with state-of-the-art offline RL algorithms. \n", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a really compelling idea: Can we treat Reinforcement Learning as just another sequence modelling problem? Doing so allows us to re-use all the machinery already developed for modelling sequences in NLP (such as transformers), and removes the need to carefully design RL algorithms. \n\nThe core idea is to let transformers model trajectories in an environment where each trajectory is a sequence of returns, states and actions. Interestingly, only predicting actions is found to be necessary (as opposed to predicting other tokens of states and returns as well). Modelling the full trajectory could be a nice direction for future work: something like predicting discretized embeddings of states using a VQ-VAE could work well.  Either way, the idea is indeed simple in hindsight but quite effective: a hallmark of influential papers. \n\nThe writing is exceptionally clear throughout, and the authors have put in a good amount work to make the exposition as clear as possible. Figure 1 succinctly describes the idea, and Figure 2 along with the illustrative example does a great job of walking the reader through how the idea would work in a toy setting. \n\nThe experiments on Atari are on only 4 games, it would be great if future versions of the paper can expand on this. Some aspects of the experiments do seem problematic:\n\n* **Per-game hyper-parameter tuning**: the context length and desired return ratio are tuned individually for each environment. On Atari, this is not kosher, and the norm is to use a single set of hyper-parameters for all games. The goal of-course is not to build specialized agents for each Atari game, but rather a single algorithm that can master all games, and per-game hyperparam tuning goes against this ethos. This makes comparisons to the baselines unfair / uncontrolled, since I don’t think CQL / REM did per-game hyper-param tuning. \n* **Problematic model selection**: It’s not mentioned in the paper, but I imagine the way hyperparams were chosen was by doing *online* policy evaluation. This violates the offline learning assumption as assuming access to online evauluation is often not realistic. [1] and [2] point out these issues and recommend evaluation protocols, so I encourage the authors to show results with offline selection as well. \n\nMinor:\n* The introduction should mention upside-down RL since its very similar in similar in spirit, instead of delegating it as another entry in the related work. DTs are a natural evolution of the initial idea proposed in upside-down RL, so it deserves a more central discussion.\n* line 109 should say section 5.4 instead of section 5.3\n\nThe analysis section is quite thorough and well done, and answers a lot of questions the reader might have about DTs. The sections on modelling diverse returns and sparse rewards are quite informative. \n\nOverall, I think this paper is great fresh new perspective on RL and will undoubtedly be influential. There are some problems with the experimental evaluation in the current version, but nothing that can’t be fixed. I am therefore evaluating assuming these issues can be ironed out. \n\n[1] Gulcehre, C., Wang, Z., Novikov, A., Le Paine, T., Gomez Colmenarejo, S., Zolna, K., ... & de Freitas, N. (2020). Rl unplugged: Benchmarks for offline reinforcement learning. NeurIPS 2020\n\n[2] Paine, Tom Le, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. \"Hyperparameter selection for offline reinforcement learning.\" arXiv preprint arXiv:2007.09055 (2020).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After an involved discussion on several topics regarding experimental practices and relation to prior art, the reviewers have settled on a range of positive scores. The cited strengths of the paper include the potential impact of introducing modern sequence modeling tools for RL problems, both the offline-RL domain in the paper's experiments and potentially more areas in the future. I personally found several areas where further technical detail and discussion would be desireable in the final version:
A) Expand your analysis of the method's limitations. Note the requests of reviewer dbvQ, but additionally, consider the balance and fairness in your text between identifying the "deadly triad" of deep TD compared to your speculations on the peformance of transformers for the task. The empirical comparison shows DT is strong but not overwhelming and therefore we may find several of the same, or new, problems specific to modeling the relations of reward/return/state/actions over time are present. Help the reader to understand the limits you expect for your method and the community to know when to use this and where to push new methods next. 
B) While the training procedure is well documented, some reviewers mentioned the lack of detail about using your model at test/inference time. Even in offline RL benchmarks, one must execute evaluation rollouts where novel states can be encountered and the future return is not available. I'd like to see a full description of the algorithm to perform these rollouts with your model (perhaps in picture form analagous to Fig 1).
 
Decision: Accept (Poster)