Semi-supervised Offline Reinforcement Learning with Pre-trained Decision Transformers
Authors: Catherine Cang
Kourosh Hakhamaneshi
Ryan Rudes
Igor Mordatch
Aravind Rajeswaran
Pieter Abbeel
Michael Laskin
Venue: ICLR 2022 Submitted (2022)
Keywords: Multi-task RL
Decision Transformer
self-supervised RL
Pretraining

Abstract: Pre-training deep neural network models using large unlabelled datasets followed by fine-tuning them on small task-specific datasets has emerged as a dominant paradigm in natural language processing (NLP) and computer vision (CV). Despite the widespread success, such a paradigm has remained atypical in reinforcement learning (RL).
In this paper, we investigate how we can leverage large reward-free (i.e. task-agnostic) offline datasets of prior interactions to pre-train agents that can then be fine-tuned using a small reward-annotated dataset. To this end, we present Pre-trained Decision Transformer (PDT), a simple yet powerful algorithm for semi-supervised Offline RL. By masking reward tokens during pre-training, the transformer learns to autoregressivley predict actions based on previous state and action context and effectively extracts behaviors present in the dataset. During fine-tuning, rewards are un-masked and the agent learns the set of skills that should be invoked for the desired behavior as per the reward function. We demonstrate the efficacy of this simple and flexible approach on tasks from the D4RL benchmark with limited reward annotations.

Under review conference paper at ICLR 2022
SEMI-SUPERVISED OFFLINE REINFORCEMENT LEARNING WITH PRE- TRAATRD DECISION TRANSFORMERS
Anonymous authors Paper under double-blind review
ABSTRACT
Pre-training deep neural network models using large unlabelled datasets followed by fine- tuning them on small task-specific datasets has emerged as a dominant paradigm in natural language processing (NLP) and computer vision (CV). Despite the widespread success, such paradigm has remained atypical in reinforcement learning (RL). In this paper, we investigate how we can leverage large reward-free (i.c. task-agnostic) offline datasets of prior interactions to pre-train agents that can then be fine-tuned using a small reward- annotated dataset. To this end, we present Pre-trained Decision Transformer (PDT), simple yet powerful algorithm for supervupe Offline RL. By masking reward tokens during pre-training, the transformer learns to autoregressivley predict actions based on previous state and action context and effectively extracts behaviors present in the dataset. During fine-tuning, rewards are un masked and the agent learns the set of skills that should be invoked for the desired behavior as the reward function. We demonstrate the efficacy of this simple and flexible approach on tasks from the D4RL benchmark with limited reward annotations.
1 INTRODUCTION
Over the past decade, we have seen tremendous progress in artificial intelligence, due in large part to our ability to train deep neural network models using massive amounts of data. The initial success of training such deep models was achieved by using large labeled datasets (e.g. ImageNet (Deng et al., 2009)), which provided rich and direct supervision for various tasks (e.g. in CV and NLP). However, the need for human annotations limited the scalability of such an approach, especially for tasks requiring detailed annotations like instance segmentation machine translation (Lin et al., 2014; Bojar et al., 2016). Self-supervision has emerged as dominant paradigm to overcome this limitation, where deep models are pre-trained using large unlabeled datasets, followed by fine-tuning with a small labelled dataset, consideaaby reducing the annotation requirements (Chen et al., 2020; He et al., 2020; Grill et al., 2020).
Analogously, progress in RL has required human annotations in the form of detailed reward functions. While some exceptions exist where the environment computes the rewards (e.g. games like Atari and Go); in most real-world applications of RL, a human is required design reward functions or annotate frames for rewards. Such well-engineered reward functions are crucial for an agent's s ability to learn successful policies, which hard to scale. At the same time, collecting reward-free and task-agnostic datasets for RL is becoming increasingly easier through use of offline datasets (Fu al., 2020), tele-operation and play datasets (Rajeswaran et al., 2018; Lynch et al., 2019), or reward-free exploration (Pathak et al., 2017; Eysenbach et al., 2018; Liu & Abbeel, 2021). In this work, we aim to design RL agents that utilize large reward- free datasets for unsupervised pre-training as well as small finetuning solve variety of downstream tasks.
datasets for supervised

---

Under review conference paper at ICLR 2022
pre pre-training
fine-tuning
unlabeled dataset D
labeled dataset
causal transformer
causal transformer
D
Figure 1: (left) The pre-traine pre-training phase: A transformer predicts actions conditioned on states and rewards that are masked and is trained on the un-annotated large offline dataset. (right) The fine-tuning phase where small fraction of the data is reward-labeled, we unmask the reward tokens and predict the reward as well as actions during fine-tuning the downstream task.
To this end, we consider the problem of semi-supervised offline RL, which is conducive for investigating our research question. In this setting, the agent has access large reward-free (task-agnostic) offline dataset of environment interactions. Such datasets are casier to scale than reward-labeled data and can be re-used for several downstream tasks with different rewards, similar to the trend of pre-training on unlabeled data and adapting to supervised downstream tasks in CV (Dosovitskiy et al., 2020) and NLP (Devlin et al., 2018 8; Brown et al., 2020). The agent further has access to a small dataset containing reward annotations, which anchors the agent to learn the desired behavior or task.
For this setting, we introduce a new approach: Pre-trained Decision Transformers (PDT), an architectural extension to the recently introduced Decision Transformer (DT; Chen et al. (2021)), to make it amenable to supervisd (offline) RL. DT frames RL as a sequence modeling task using an autoregressive and causally masked transformer model, and was shown to be competitive with traditional RL algorithms based on temporal difference learning. As shown in Figure 1, our framework extends DT and consists of two steps: pre-training and fine-tuning. First, from a large and fixed offline data, trajectories to predict the next actions autoregressively with the DT architecture. To accommodate fine-tuning with reward signals, during this pre-training step, we mask the reward tokens by zeroing them out well modifying the causal mask of the self-attention layers to generate actions without reward tokens. This can enable the model to implicitly learn the skills present in the offline un-labeled dataset. Then in the fine-tuning step, we un-mask the reward tokens and fine-tune the transformer end-to-end, predicting both reward and action tokens. This allows the agent to use the information present in the small reward-annotated dataset to adapt its set of skills towards achieving high rewards in the downstream task.
the state and action context of
We summarize our contributions as follows:
1. We propose a new semi-supervised offline Rl. setting to bring RI. closer to the self-supervised learning paradigm in CV and NL.P.
2. We introduce Pre-trained Decision Transformers (PDT)- an extension of the recently proposed Decision Transformers which masks the reward tokens during pre-training and can automatically generate reward-to-ger during evaluation.
3. We show how PDT can efficiently be fine-tuned for a range of diverse downstream tasks using the same pre-trained model in few-shot manner, achieving leading aggregate performance on D4RL environments in the low-label regime.
4. We provide insight into how different parts of PDT contribute the final performance.
2

---

Under review conference paper at ICLR 2022
2 RELATED WORK
Behavioral Cloning (BC). When a reward-free but expert-quality dataset is available, imitation learning is often considered as a simple and yet powerful baseline for learning effective policies Osa et al. (2018); Rahmatizadeh et al. (2018). Our work differs in the motivation to utilize sub-optimal reward-free experience in the pre-training phase of the agent.
Offline RL. Offline RL is a paradigm for RL where agents are trained on offline datasets rather than directly interacting with the environment. The main issue in utilizing existing value-base off-policy RL algorithms in the offline setting is that these methods often erroneously produce optimistic value function estimates that can encourage the policy to pick actions that are out-of-distribution (OOD) compared to the offline dataset, resulting in failure. One way mitigate this problem is to constraint the learned policy to be "close" the data generating policy (also known as behavior policy). by means of KL-divergence (Jaques et al., 2019; Wu al., 2019a; Siegel et al., 2020; Peng al., 2019a), Wasserstein distance (Wu et al., 1a), or MMD (Kumar et al., 2019), and then using the sampled actions from this constrained policy in Bellman backup or applying a value penalty based on these divergence measures. Another way be conservative on value estimation by adding penalty terms that enforces the models to learn lower bound on the value functions (Kumar et al., 2020; Fujimoto et al., 2019). All these methods however require the offline datasets to reward-annotated, and rely crucially on rewards for learning.
Another line of work extends Model Based Reinforcement Learning (MBRL) methods to offline settings, Similar to model-free methods, existing MBRL methods cannot be readily used in offline settings due to the distribution shift problem and require some safeguard to mitigate exploiting model inaccuracies. To mitigate this issue, Kidambi et al. (2020) and Yu et al. (2020) both incorporate pessimism in learning the dynamics and estimating uncertainties in reward learning. While the dynamics models themselves can be learned without reward functions, planning policy learning using the learned model again relies crucially having access to a reward function or a large reward-annotated dataset.
Skill Extraction with Behavioral Priors. Methods that leverage behavioral priors utilize reward-free envi- ronment interactions to learn the set of task-agnostic skills via either maximizing likelihood estimates (Pertsch al., 2020; Ajay et al., 2020; Singh et al., 2020) or maximizing mutual information (Eysenbach et al., 2018; Sharma et al., 2019; Campos et al., 2020). Behavioral priors learned through maximum likelihood latent variable models have been used for structured exploration in RL (Singh et al., 2020), to solve complex long-horizon tasks from sparse rewards (Pertsch et al., 2020), and regularize offline RL policies (Ajay et al., 2020; Wu et al., 2019b; Peng et al., 2019b; Nair et al., 2020). However, these skill extraction algorithms have high architectural complexity, requiring a generative model to learn a latent skill space and an RI. agent to learn policies over skills as its action space. contrast, PDT has one simple architecture for both skill extraction by modeling state-action sequences and control by conditioning the desired reward.
3 APPROACH
3.1 PRELIMINARIES
Offline Reinforcement Learning. We consider the standard RL setting of a Markov Decision Process (MDP) defined by the tuple M (S, T). The MDP tuple consists of state E S, actions a E A. rewards r = R(s, a), and transition dynamics Tss'(s,a) a). A policy in RL aims to pick actions that will maximize the
expected total return of the MDP E
ΣT- R(s at) One episode can be represented as a trajectory in the MDP consisting of states, actions and rewards (81, a1 1 82; a2 r2, ..: ST, QT, TT).
3

---

Under review conference paper at ICLR 2022
Offline RL is setting of RL that seeks find the optimal policy from fixed offline dataset of interactions.
'D = [s;, a;, ri) ) As result, this setting is more difficult than the standard RL setting due to lack of access the environment, and therefore is prone distribution shift.
Decision Transformers. The usage of transformers (Vaswani et al., 2017) to tackle reinforcement learning problems has been a topic of recent study Chen et al. (2021) and Janner et (2021). Decision Transformer, rather than learning explicit policies or value functions, trajectories are modeled as sequences of state, actions, and reward-to-go A reward-to-2e (RTG) is defined as the sum of future rewards R, = E E'= Pr- Then the problem setting becomes a sequence modeling problem. where actions are generated conditioned on the past context, current state, and the desired reward-to-go. With context of length H current time step / (i.e. )) the policy becomes:
p(a, St, Re- 1, (1--01110 18 84- by ... Rs- H1 ag H1 St- H
c(
The architecture of Decision Transformer first applies linear token embedding to states, actions, and RTGs, and then adds a learned positional embbedding (shared across tokens within the same time-step) to differentiate between similar tokens at different positions along the sequence. uses GPT (Radford et al., 2019) architecture to learn a policy that auto-regressively generates actions based on prior context. The loss during training DT is a mean-squared error loss between predicted actions and ground truth actions.
3.2 OFFLINE RL WITH PRE-TRAINED DECISION TRANSFORMERS
Our approach is composed of two stages: Pre-training and Fine-tuning. During pre- pre-training the model learns extract the common temporally extended sequences of actions from the pre-training dataset. Our hope that by fine-tuning this model on the downstream reward-annotated dataset it will be able to quickly learn task-specific knowledge including a reward prediction model and a reward-conditioned policy. We now claborate each part in more details:
3.2.1 PRE-TRAINING DT
During pre-training, a sequence of trajectory modeled as:
[11], [ai), [RTG]] [52]. [22), [RTG). [ST], [ar], [RTG]
Pos.
Pos
PoxT
Where [RTG] is the placeholder token for reward-to-go which is hard-coded zero during pre-training. Note that the same positional encoding is added to state, RTG, and action of the same time-step. We also modify the causal mask the self-attention layer to disregard any attention paid the [RTG] tokens. This is crucial for the model to be reward-agnostric, and therefore. allowing for a quicker adaptation to downstream RTGs fed in fine-tuning time. Pre-training is done by computing the maximum- maximum-likelihodd estimates (MLE) of the neural network parameters.
LPr(O) = E(s,,,.C)-DPr [-log log po (aelse, C. )
(1)
3.2.2 FINE-TUNING PDT
Similar to the fine-tuning strategy of Devlin et al. (2018), simply fine-tune all PDT parameters the downstream reward-annotated data. Since we have access to reward signals, we can now include them in the input sequence:
4

---

Under review conference paper at ICLR 2022
[s1]
[Ri]]
[s2] [aa); [Raa]
[ST]. [ar]] [RT]]
Pos.1
Pos.2
Pos.T
During fine-tuning instead of just predicting actions conditioned on the current state and context, i.e. aa Se, Ct c(Hl) we also predict the RTGs conditioned on the current state, action, and context at every step, i.e. po Re f8t, dz, C/ c(41) We found that empirically, this helps the model infer the RTG distribution which can help with the quality of the fine-tuned policy.
Overall, the training process in this step is similar to the training in the Decision Transformer, except for the addition of reward-prediction loss term. The fine-tuning loss is modified contain both the MLE term for actions and the prediction loss for rewards, as described in Eq. 2. Note that the historical context c( also contain the now unmasked reward-to-ge tokens.
CFT (0) Cr, [Log (aolst, log pe (Ru ae C. c
(2)
Return Prediction. One limitation of the reward-to tot-to--ooodiiioeedd structure of Decision Transformer that each environment and each task requires a hand-picked initial reward-to-go to feed in. This can be un-intuitive and difficult: picking the best initial value can require prior knowledge about the training dataset and what rewards are feasibly achievable. Additionally, a fixed initial reward-to-go does not work well in environments that have random resets, such as D4RL's (Fu et al., 2020) pointmaze environment, since the distances to goal are different based on the randomized initial locations. In such environments, the best feasible reward-to-an to-go depends on the initial state, and cannot be a fixed value. To overcome this while still learning reward-to-go -ot-tt----ooiiiioneeee actor, learn to predict the best feasible reward-te swward-to-oo go based on the context and state. The model is trained to predict a probability distribution over the possible RTG tokens from the training dataset. At evaluation time, we pick the next RTG by sampling N RTGs (e.g. N 250) from the outputted probability distribution and picking the one with the highest value.
R₂ max R.. R2 ... R,
po (Relss, de, C. c (1....NN
(3)
One detail necessary for return prediction in continuous-reward environments (e.g. locomation mujoco tasks) is that we discretize the reward- sward-to-gor to-gos into bins in order to learn a probability distribution over discerete values RTGs rather than predicting scalar real-value returns.
3.2.3 EVALUATION
To rollout a trajectory using the model trained from PDT, we model it as a sequence of state-action-rewarde We alternate between two prediction phases for each time-step in the environment: (1) Using the con- text (st-1, H, O4-H H: R4- Ri-HH .... predict the return-to-go Rd (see: Eq. 3). (2) Using the updated context (st- H: at- ; Ri-H, 8t, RR) predict the action dz In the first H steps of the trajectory when there is not enough context,
zero padding.
4 EXPERIMENTS
the experiments we are interested answering the following questions: (i) Can PDT be efficiently fine- tuned on downstream offline datasets where reward-labeled data is limited? (ii) How does PDT compare
5

---

Under review conference paper at ICLR 2022
OT
PDT (ound)
CGL COL+BG
0.8
Nomolzod Score
0.9
0.4
622
0.0
Number downstream trajectoiiss
Figure 2: Aggregate results of the normalized scores across all domains and datasets. PDT achieves leading aggregate scores in the low-label regimes and is competetive with BC regularized CQL despite not learning a value funcction.
TD learning offline RL algorithms, despite not fitting any explicit value function? (iii) What are the most important design choices for adapting DTs to the semi-supervised offline RL setting?
4.1
ENVIRONMENTS AND DATASETS
We evaluate the performance of PDT on three simulated mujoco (HalfCheetah, Walker2d, and Hopper) and one robotic manipulation environment (FrankaKitchen) from the D4RL benchmark.
4.1.1 DATASETS
Mujoco. For the mujoco locomotion datasets, consider three simulated locomotion agents and three datasets per agents (medium, medium-replay, and medium-expert), summing to a total of nine datasets. The datasets collected in the following ways:
The medium dataset collected with partially trained SAC agent.
The medium-replay dataset the entire replay buffer of SAC agent throughout training.
The medium-expert is mix between trajectories from the medium dataset and an expert policy.
FrankaKitchen. This environment consists 7-DOF kitchen robotic arm that is tasked to manipulate objects in particular order. We use kicthen-partial--0 which is collected with tele-operation and includes manipulating all objects, present in the environment, in different orders. However, each trajectory in the dataset is reward-labeled according the order the target task.
For pre-trained methods, we use the entire dataset in the D4RL benchmark to learn the behavioral prior. For fune-tuning we evaluate on different number of trajectories sampled from the dataset. In our experiments, full refers to using the entire reward-labeles dataset, 25 refers to using 25 randomly sampled trajectories, and 10 refers to using 10 randomly sampled trajectories. By doing so, we can investigate whether PDT can efficiently fine-tune the downstream offline tasks with small reward-labeled offline datasets?
4.1.2 EVALUATION PROTOCOL
For evaluating each of the results, we calculate each algorithm's performance by taking the average returns of the last 5 epochs for training the downstream dataset. For each epoch, evaluate 10 trajectory rollouts in the environment and report the average return based on the downstream task reward function.
We use the following approaches for comparison:
6

---

Under review conference paper at ICLR 2022
Table 1: Normalized scores for each dataset and each algorithm. PDT results are competative with other more
sophisticated TD-learning methods that leverage the reward-free as well as the reward-annotated datasets. (i.c. CQL+BC)
Domain
Dataset
Num trajectories
BC
DT
POT (ours)
CQL
CQL+BC
202
0.82
0.79 0.1
1.55
1.02
medium--repla--v2
25
0.85
0.54
0.80 0.06
0.80
0.93
10
0.13
0.55 0.05
-0.09
0.82
1.00
Halfcheetah
medium-v2
25
0.93
0.89
0.91 0.04
0.92
0.96
10
0.84
0.93 0,032
0.83
0.92
2000
0.74
0.75 0.067
0.26
0.50
macdium-expert--2
25
0.63
0.41
0.75 1: 0.0769
0.27
0.42
0.33
0.63 0.077
0.34
0.43
2039
0.50
0.34 0.025
0.80
0.55
medium--eplyy-v2
25
0.24
0.20
0.17 + 0,0230
0.03
0.12
0.027
2187
0.54
0,55 0.055
0.64
0.61
Hopper
medium-v2
25
0.43
0.44
0.36 0.08
0.54
0.57
0.40
0.56 0.042
0.57
0.56
3214
0.44
0.59 + 0.0761
0.59
0.62
medium expeutm-xpper-v22
0.32
0.48 +-0.66
0.41
0.28
1093
0.48
0.13
0.80
0.76
medium--cphny--2
25
0.69
0.17
0.15 0.09
0.05
0.04
10
0.08
0.20 0.05
0.03
0.02
1191
0.80
Walker2d
medium-v2
25
0.70
0.56
0.77 0.05
0.04
0.50
10
0.30
0.44 0.09
0.02
0,80
2191
0.92
modium--xxpert-22
25
0.89
0.62
0.98 0.01
0.63
0.86
0.22
0.97 0.02
0.48
0.75
601
0.39
0.57
0.06
0.50
0.17
FrunkaKitchen
kitehen-partillv00
25
0.00
0.31
0.19 0.07
0.25
0.00
full
0.66
0.64
0.73
0.69
Average
25
0.58
0.44
0.55
0.39
0.52
0.31
0.53
0.29
0.49
PDT (ours): Our approach for pre-training DT reward-free data and fine-tuning on the reward- annotated dataset.
DT:
Decision Transformer which uses transformers to auto-regressively predict actions conditioned current state, reward-to-go. and context.
BC: This baseline simply behavioral cloning on the reward-free dataset. Since the trajectories in these datasets are not entirely collected from experts, we do not expect this baseline to be superior and be able achieve beyond the average of the returns in the dataset.
CQL: Conservative Q-Learning (Kumar al., 2020), which a popular model-free offline RL algorithm. This baseline is only trained on the downstream data but uses the reward annotations.
BC regularized CQL. This is extension of CQL that also regularizes the policy's distribution to match that the pre-training dataset. We simply apply CQL on the downstream dataset while regularizing the policy imitate the actions in the pre-training dataset with an additional loss term to the policy updates.
CQL+BC:
We train BC the entire reward-free dataset; CQL, and DT only the reward-annotated dataset, and CQL+BC on both reward-free and reward-annotated dataset. CQL+BC, similar to PDT, leverages both the reward-free and
datasets.
7

---

Under review conference paper at ICLR 2022
Results. Figure 2 shows the aggregate of the normalized returns across all tasks as function of the number of reward-labeled fine-tuning trajectories. As we can see, compared to DT and CQL, the algorithms that leverage reward-free free datasets (i.e. PDT and CQL+BC) perform significantly better low data regimes. is also noted that PDT's performance is competitive with CQL+BC, but the simplicity and flexibility of its architecture makes PDT a suitable candidate for wide-spread adoption semi-supervised offline RL settings.
To illustrate the granular results on each environment, we have included Table 1. The scores are normalized the maximal return the trajectories each dataset. As we can see, even though both DT and CQL mostly achieve good results in full data regimes, their performance significantly drops in settings where labeled-data is limited. In the kitchen environment in particular, CQL+BC completely collapses as the desired multi modal behavior of the agent cannot be captured by simple BC regularization, while PDT can recover the task-specific behavior in low data regimes.
4.2 ABLATION STUDIES
this section we study different components of the PDT algorithm to provide insight on the contribution of each part. In particular,
interested in performing the following ablations:
The effect of reward prediction during fine-tuning, During the fine-tuning step of our framework we not only predict the next actions conditioned on the current state, reward-to-go. and context, but we also predict the distribution of possible remaining reward-to-gos To this end, we study the effect of this component by comparing to where it is not used. We an experiment with the maze2d-large-v D4RL in which the agent needs to know how far it is from the goal location to be able to navigate to it properly.
Trajectories
PITT rew.
DYT
10
0.69
20
0.71
0.40
50
0.69
0.67
0.60
0.72
0.70
Figure 3: Left: Ablation on effect of reward prediction on pointmass maze task, Right: Prediction of returns-to-g go of different discrete states in the PointMaze environment
this experiment, we use our model to output the best possible return-to-ge tourn-to-go given different starting locations in the maze. In Figure 3 (right) darker square indicates a bin that represents a better return-to sourn-to-go. This means that if the point mass starts in that state, will be able quickly reach the goal square of (9, 1). As expected, states closer to (9, 1) have better reward-to-go.
The effect of masking reward token attention during pre-training. During pre-training we mask the reward attention to force the model to not attend to reward tokens, and to auto-regressively model the next actions conditioned on the current state, and context without rewards. Here, we compare the post fine-tuning performance of PDT with a version that does not apply attention masks during pre-training. We show three experiments on the medium-exper suite in mujoco, since the effect is seen most clearly in this dataset type. Including masking during pre- pre-training, improves results regardless of how many trajectories the fine-tuning dataset includes.
Comparison to other choices of fine-tuning. PDT's strategy for fine-tuning is the most straight-forward instantiation of this idea which is fine-tuning all pre-trained parameters. Here, we compare this particular design choice to other common fine-tuning patterns seen in transformer literature. In particular compare,
8

---

Under review conference paper at ICLR 2022
Table 2: Ablation on effect of reward masking during pre- pre-training
task
number of trajoctories
PDT (w/ masking)
PDT (wo/ masking)
3214
0.59
0.47
hogge--nsdiun-expeert222
25
0.45
0.45
10
0.48
0.38
2000
0.75
0.78
halfcheetah medium expert -2
10
0.63
0.59
2191
0.99
0.97
Walleer2-mmddium-xxxrrt222
0.98
10
0.97
0.94
fine-tuning with frozen self-attention layer and fine-tuning with both frozen self- self-attention and feed-forward layers. The later choice was proposed recently on al. (2021).
Prosen Ads-Pt
Frozen Ade
#Tal
3.
0.8
Nomated
04
Namber Injestoriee
Figure Ablation on the effect which PDT parameters to fine-tune, Numbers presented show the average normalized score over all medium and medium-expert mujoco experiments presented in our main results.
Empirically, we that fine-tuning all of the parameters, on average, outperforms fine-tuning only a subset of the parameters, although the difference overall is not extreme. Interestingly, in the very-low data domain of 10 trajectories, it does appear that only freezing the attention parameters does lead the best performance, but for 25 trajectories and the full trajectories, fine-tuning all parameters does best.
5 CONCLUSION
We presented PDT, a simple and effective architecture amenable to semi-supervised offline RL settings where the architecture is pre-trained on reward-free and task-agnostic datasets and then fine-tuned a task-specific downstream dataset. Our technical contribution is extending Decision Transformers to this setting by masking reward tokens during pre-training and predicting remaining reward-to-gos during fine-tuning. We show how PDT can outperform existing offline RL methods in low data regimes. The simple training procedure of PDT makes it great candidate for wide-spread adoption in semi-supervised offline RL settings.
REFERENCES
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010. 13611, 2020.

---

Under review conference paper at ICLR 2022
Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al. Findings of the 2016 conference on machine translation. Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pp. 131-198, 2016.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. arXiv preprint vrXiv:2005.14165 2020.
Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning, pp. 1317-1327. PML.........RRR 2020.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv:2 /06. 106.01355 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. 1597-1607.PMLRR 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv: 04800, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint arXiv:2010. /929, 2020.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. arXiv preprint arXiv: 1802. 06070, 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, arXiv preprint (rXiv:2004.072199 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 20622 PMLR, 2019.
lean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.077 0777333,. 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and 2020.
Pattern Recognition, pp.
Michael Janner, Qiyang Li, and Sergey Levine Reinforcement learning as one big sequence modeling problem. arXiv preprint arXiv:2106. 02036,0220...00 2021.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv: 190.000400 2019.
10

---

Under review conference paper at ICLR 2022
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. arXiv preprint S0i::2000,0595/,, 2020.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off off-policy q-learning via bootstrap- ping error reduction. arXiv preprint arXiv: 190.000990 00949, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce- ment learning, arXiv preprint arXiv:2006.04779, 2020.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference computer vision, pp. 740-755. Springer, 2014.
Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In International Conference on Machine Learning, pp. 6736-674 6747. PMLR, 2021.
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained transformers as universal computation engines, arXiv preprini arXiv:2103.05247 2021.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In CoRL, 2019.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine, Awac: Accelerating online reinforcement learning with offline datasets. 2020.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, and Jan Peters. An algorithmic perspective on imitation learning. arXiv preprini arXiv:181.066111 2018.
2018.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self- supervised prediction. In International conference on machine learning, pp. 2778-2787. PMLR. 2017.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage--wiighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00117 2019a.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weightee regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019b.
Karl Pertsch, Youngwoon Lee, and Joseph J Lim. Accelerating reinforcement learning with learned skill priors. arXiv preprint arXiv:2010.1 /1944, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Rouhollah Rahmatizadeh, Pooya Abolghasemi, Ladislau Bölöni, and Sergey Levine. Vision-basee based multi- task mani manipulation for inexpensive robots using end-to-end learning from demonstration. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 3758-3765. IEEE, 2018.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsuper- vised discovery of skills. arXiv preprint 197.01697 2019.

---

Under review conference paper at ICLR 2022
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08399, 2020.
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinchart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. arXiv preprint arXiv:2011.1002: 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp- 5998-6008 2017.
Yifan Wu, George Tucker, and Ofrr Nachum. Behavior regularized offline reinforcement learning. arXiv preprini 191. 133611 2019a.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1rXin:1911.11311,200 2019b.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu
Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005.13239 2020.
12


Reviews:
[{"review_id": "yn7m90RgbRo", "reviewer": "Reviewer_z2Wf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1.\tThe semi-supervised setting seems new in offline reinforcement learning. \n2.\tThe proposed method is simple. \n3.\tAblations have been provided to study the effects of reward prediction during fine-tuning, masking reward token attention during pre-training, and other choices of fine-tuning. \n\nWeakness:\n1.\tThe importance of fine-tuning is not well justified. The pre-trained decision transformer model has already achieved sufficiently good performance without reward annotations. Why do we need to fine-tune the pretrained model on a subset of the pretrained dataset with reward annotations? The significance of fine-tuning can be shown on more diverse and challenging datasets. \n2.\tPre-training a decision transformer on a large dataset is expensive. Fine-tuning a full pre-trained decision transformer on downstream dataset is not efficient. It would be better if a light-weight fine-tuning method can be applied. \n3.\tDespite the authors provided ablations, the discussion and conclusion on which factor is most effective is sufficient. For example, Why PDT with masking reward token during pre-training works slightly better than PDT without masking?\n\nOther questions:\n1.\tTable 1 shows the fine-tuning results of PDT. How does the pre-trained decision transformer perform? Is it better than the reported PDT results? If the pretrained decision transformer outperform the PDT fine-tuned on smaller downstream datasets, then the fine-tuning is not that significant. \n2.\tSome important experimental details are missing. For example, how many epochs were used for fine-tuning and how many steps in each epoch? If the fine-tuning is expensive, then why not train the DT on the small dataset with reward annotations from scratch? \n", "summary_of_the_paper": "This paper presents Pre-trained Decision Transformer (PDT) for semi-supervised offline reinforcement learning. PDT first pre-train a decision transformer model on the trajectory dataset without rewards, and then fine-tune on a smaller dataset with reward annotations. Empirically, PDT achieves compatible performance with offline reinforcement learning baselines. ", "summary_of_the_review": "Overall, I vote for weak rejection. Despite its simplicity, the novelty and significance of the proposed method are limited. My major concern is the importance of fine-tuning is not well supported in the experiments. It can be enhanced by showing the significance of fine-tuning on more diverse downstream datasets that collected from related but more challenging tasks. Hopefully the authors can address my concern in the rebuttal period. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fyL0m2EceVu", "reviewer": "Reviewer_xxc6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Extending decision transformer to work on unlabelled datasets is an interesting research direction, one can imagine a large scale DT trained on a large-scale unlabelled dataset can enable efficient transfer to unseen tasks and envs in a similar nature to GPT-3. The specific setting that this paper operates in though is quite odd. \n\nThe paper proposes a way to pre-train Decision Transformers on unlabelled  or reward-free data, but the experiments are conducted on data collected by expert/semi-expert agents which did use reward during their training. This also means that the offline datasets are biased towards high reward trajectories. So, calling the pre-training setup “reward-free” is misleading. There could be several alternative experimental setups which can make more realistic reward-free setups:\n\n  * Pre-training on data collected by a random policy\n  * Pre-training on data collected by an exploratory policy which doesn’t use extrinsic reward (like curiosity etc.) \n  * Pre-training on data collected from a different task or reward function. (Task-transfer) \n  * Pre-training on in-the-wild videos. \n\nOn experimental results, PDT doesn’t show much benefit over simple BC or CQL+BC. \n\nThe most aspects of the paper are on how to improve fine-tuning performance, and which design choices matter. I would recommend authors to use a more appropriate experimental setup, or show results for transfer, which would make a much more compelling case. ", "summary_of_the_paper": "This paper proposes pre-trained decision transformers (PDT) that are trained on an offline dataset collected by an expert/semi-expert agent with reward information masked out. PDT is then evaluated by fine-tuning on the same task. ", "summary_of_the_review": "The paper has major shortcomings with the experimental setup considered to evaluate the efficacy of pre-trained decision transformers, and falls short of outperforming simple baselines. For these reasons, I recommend rejection at this point.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RGhUe9gMCOv", "reviewer": "Reviewer_fBou", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Key Strengths\n  1. I think this is a good problem area -- improving efficiency of RL systems and providing a vehicle for generalizing RL knowledge across settings seems very valuable.\n  2. I think the method presented here is intuitive; withholding reward information while giving the model a better ability to model action sequences seems like it ought to help.\n\n### Key Weaknesses\n  1. I'm confused by something key in your results. Firstly, what does the \"-v#\" suffix indicate in your data tables? You don't make that clear anywhere as far as I can tell. This is relevant because your performance #s don't appear to match those in the decision transformer paper (for the DT or CQL runs). Why are these different?\n  2. I'm not as convinced by your results. In the context of comparing across a wide variety of tasks, I think using a metric like the # of wins or average rank is superior to average performance across tasks. In your setting, vs. CQL+BD on FULL you win 3 / 10 times, on 25 you win 5 / 10 times, and on 10 you win 5 / 10 times with 2 / 10 ties. In terms of overall rank, you take first 3/10, 3/10, and 5/10 (with 1/10 ties) times for full, 25, and 10 respectively. These don't suggest to me that you beat CQL+BD or the overall suite of baselines in a compelling fashion. I also think it is a problem if your method does worse than existing approaches in the full-data setting. At the very least, it should be matching those approaches, otherwise why would I use it in real-world settings? If it will only over advantages in a narrow range of # of observed trajectoreis (where the # allowed likely varies by setting/task), using it could risk yielding worse test set performance vs. standard methods.\n     A related point to this is that dropping the uncertainty in your average performance cell in table 1 isn't acceptable -- you have all the variances for the PDT model listed in the individual cells above; propagate those through the averaging operation so we can see to what extent things are meaningfully separated. Similarly, there need to be error bars in Figure 2. I am definitely concerned that in some cases (e.g., 25 and 10) the propagated uncertainty will be so large that even in the average comparison there isn't clear separation between your method and CQL+BC.\n    Note that this concern is what motivates the score of \"2\" for the correctness column. If the presentation of results could be cleaned up and a more thorough comparison across methods performed, I'd improve that numerical score.\n  3. While I don't know of any works that do this exactly, I do think there is only limited novelty/technical sophistication here. It seems to me that you have largely just split the training phase of the DT into a PT piece where rewards were occluded then a FT piece with rewards present. While that contribution does have merit, I'm not sure it is sufficient for an ICLR submission.\n\n### Minor Weaknesses:\n  1. In your setting, what are the pre-training datasets vs. the fine-tuning datasets? Or are they the same and only the reward values are occluded during PT? An important part of PT/FT in the general domain is the inherent question of domain generalizability in going from a PT dataset to a FT dataset, which come from different distributions. It seems like this may be missing from your setting? If I am correct and there is no question of domain generalizability / task transfer, then I feel your paper is slightly misrepresenting its problem. Rather than doing PT/FT, this is really more like classical semi-supervised learning, where you have instances of labels (or in this case, rewards) being missing at random from your training data. This isn't really a big problem, as you compare to semi-supervised methods as appropriate, but re-framing it to be more in line with that perspective would help make the paper tighter.\n", "summary_of_the_paper": "### What is the Problem / Question?\nPre-training has not been thoroughly explored within RL, so it is unknown how effectively task-free unlabeled datasets can be used for PT in an RL context. The authors present a simple strategy for such PT for RL that shows benefits with limited reward annotations.\n\n### Why is it impactful?\nRL is a famously expensive discipline of ML which nonetheless has huge application areas and potential in theoretial and deployment contexts. Being able to translate the improvements offerred by pre-training to the RL context would help empower this already important technique significantly.\n\n### Why is it hard? Why have previous approaches failed?\nThere are several previous approaches to this problem, including some the authors readily identify in their study, such as the CQL+BC method which uses unlabeled data to perform semi-supervised offline RL by adapting policy prior data via unlabeled data. \n\n### How do they solve it?\nThe authors propose using the (already published) decision transformer model, but pre-training that model first on a dataset with only action sequences, without any reward information.\n\n### How do they validate their solution?\nThe authors compare their proposed framework against existing models, including both the traditional DT, a BC baseline, and the CQL and CQL+BC methods (the latter of which is a semi-supervised approach). They find slight improvements over existing methods in the extreme few-shot settings (10 observed trajectories at FT time), with further reduced gains at larger #s of observed trajectories, culminating in worse performance than traditional methods at full data scale.\n", "summary_of_the_review": "I think this is a great start, but insufficient for ICLR, so I'm recommending reject here. This is motivated by two key problems: (1) The results don't convince me there is real improvement here, in particular vs. the CQL+BC baseline, and I do think the degradation in performance at the full setting is a major issue, and (2) I'm not convinced there is sufficient technical novelty / sophistication here. This submission in its current form (modulo some edits to improve the presentaition of the results and more accurately reflect the performance relationship between the various methods) might be a great fit for a workshop, and improvements to the PT/FT method to increase significance of results and/or expand the technical sophistication of the method would make it a better fit for ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0swsDhWBzTN", "reviewer": "Reviewer_qGs8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "== Strengths ==\n\n- Transfer learning like pre-train & fine-tune and self-supervised in deep reinforcement learning is an important and interesting problem.\n- PDT can leverage reward-free trajectories and improve the agent in fine-tune stage with limited 'labeled data'. It could be a promising solution to the problem of data efficiency in deep RL.\n- The proposed approach achieves improvements on D4RL datasets.\n\n== Weaknesses ==\n\n- I think the contribution and novelty of this work is not enough, the main difference from Decision Transformers is the MLE loss, which is minor, and the reward prediction. So the ablation study on effect of reward prediction during fine-tuning on mujoco or FrankaKitchen in D4RL is also important.\n- As an improved version of decision transformers, I noticed that the order of trajectory representation (s_t, a_t, \\hat{R}_t} is different from that in DT (\\hat{R}_t, s_t, a_t). As a results, in Section 3.1, \\hat{R}_t is not included in the C_t^{(H)}, which may be different from the DT version. There should be an explanation for this change. \n- In Figure 1(Right), the \\hat{R}_{t-1} is both of the output and input of transformer, so the model is using the predicted RTG as input? Why not use the real rewards from environment at evaluation time?\n- 'For pre-trained methods, we use the entire dataset in the D4RL benchmark to learn the behavioral prior.', so the entire dataset means all trajectories for one task or all tasks? As listed in contributions, 'PDT can efficiently be fine-tuned for a range of diverse downstream tasks using the same pre-trained model', how different tasks with different sizes of states and actions share the same embedding layers? I think more details of pre-train stage are required.\n- As for the experiments results, I noticed that in some cases, when there are 25 trajectories, the performance is worse than 10 and full, and the gap is quite big, what is the reason for this?", "summary_of_the_paper": "This paper introduces a self-supervised approach in deep reinforcement learning with decision transformers, which is widely used in CV and NLP tasks. The algorithm can achieve better performace than traditional methods like DT, BC and CQL, especially when data is insufficient.", "summary_of_the_review": "The paper introduces a self-supervised approach to improve the performance of decision transformers in deep RL when data is limited. But the contribution and novelty is not enough. So I give the score of 5.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explored pre-training for deep offline reinforcement learning, developing a method that first pre-trained decision transformers on trajectories without rewards, and then fine-tuned on limited data with rewards. The reviewers were pleased with the overall research questions and directions, but found that they were substantial shortcomings in the experimental setup and results that make this paper not yet suitable for inclusion. The approach is relatively simple and straightforward, which is actually a good thing, but that means that it must be correspondingly investigated and developed with convincing empirical results. Unfortunately, there are a number of open questions about the experimental set up, and the results are not convincing that the method is effective against alternatives, as detailed in the reviews. There was no author rebuttal.
Decision: Reject