Can Wikipedia Help Offline Reinforcement Learning?
Authors: Machel Reid
Yutaro Yamada
Shixiang Shane Gu
Venue: Submitted to ICLR 2023 (2023)
Keywords: offline rl
language models
transfer learning

Abstract: Fine-tuning reinforcement learning (RL) models has been challenging because of a lack of large scale off-the-shelf datasets as well as high variance in transferability among different environments. Recent work has looked at tackling offline RL from the perspective of sequence modeling with improved results as result of the introduction of the Transformer architecture. However, when the model is trained from scratch, it suffers from slow convergence speeds. In this paper, we look to take advantage of this formulation of reinforcement learning as sequence modeling and investigate the transferability of pre-trained sequence models on other domains (vision, language) when finetuned on offline RL tasks (control, games). To this end, we also propose techniques to improve transfer between these domains. Results show consistent performance gains in terms of both convergence speed and reward on a variety of environments, accelerating training by 3-6x and achieving state-of-the-art performance in a variety of tasks using Wikipedia-pretrained and GPT2 language models. We hope that this work not only brings light to the potentials of leveraging generic sequence modeling techniques and pre-trained models for RL, but also inspires future work on sharing knowledge between generative modeling tasks of completely different domains.

Under review as a conference paper at ICLR 2023
CAN WIKIPEDIA HELP OFFLINE REINFORCEMENT LEARNING?
Anonymous authors Paper under double-blind review
ABSTRACT
Fine-tuning reinforcement learning (RL) models has been challenging because of lack of large scale off-the-shelf datasets as well as high variance in transferability among different environments. Recent work has looked at tackling offline RL from the perspective of sequence modeling with improved results as result of the introduction of the Transformer architecture. However, when the model is trained from scratch, it suffers from slow convergence speeds. In this paper, we look to take advantage of this formulation of reinforcement learning as sequence modeling and investigate the transferability of pre-trained sequence models other domains (vision, language) when finetuned on offline RL. tasks (control, games). To this end, we also propose techniques to improve transfer between these domains. Results show consistent performance gains in terms both convergence speed and reward a variety of environments, accelerating training by 3-6x and achieving state- of-the-an art performance in a variety of tasks using Bakkipedia-pretriinn language models. We hope that this work not only brings light to the potentials of leveraging generic sequence modeling techniques and pre-trained models for RI., but also inspires future work on sharing knowledge between generative modeling tasks of completely different domains.
and GPT2
INTRODUCTION
Large pre-trained language models have shown impressive performance in natural language (De vlin et al., 2019. Radford et al. 2018) and vision (Dosovitskiy et al. 2021) tasks. Furthermore, Transformer- based autoregressive language models (Vaswani et al. [017]] 7: Baevski Auli 2019) Radford et al. 2019) have shown to be powerful sources of zero-shot and few-shot performance (Brown et al. 2020), with notable rapid adaptation in low resource settings, demonstrating their easy adaptability and transferability
number of tasks in their respective domains. Adapting autoregressive language models has also been extended to the multimodal setting (Tsimpoukelli et al. 202 for tasks such visual question answering.
Concurrently, offline reinforcement learning (RL) has been seen as analogous to sequence modeling (Chen et al., 2021 Janner al. 2021 Furutaa et al. 202T), framed as simply supervised learning to fit complexities commonly associated with reinforcement learning (Watkins Dayan 1992, Kakade 2001). allows us to take advantage of techniques popularized in sequence modeling tasks for RL.
trajectories in an offline dataset. This relaxation, doing away with many of the
Pre-training, particularly, is an essential technique for alleviating higher compute costs from using more expressive models such as Transformers. However, such concept is still relatively fresh in RL (Singh et al.) 2020, Tirumala et al.) [2020), due to the difficulty parameterizing different scenes and tasks through single network (Wang et al.. 2018b Jiang et al.] 2019 Zeng al., 2020) as well as the lack of large off-the-shelf datasets for pre-training (Cobbe et al. al.] 2020 Zhu et al. 2020 Yu
al. 2020). Adopting pre-training
default option for recent Transformer based methods (Chen 2021 Janner et al. 2021 Furuta et al. 2021 appears far away if we only look within RL.
ctt al.
Unified under the umbrella of sequence modeling, we look at whether Transformer-basee pre-trained language models are able to be adapted to standard offline reinforcement learning tasks that have no relations to language. Given the setting of having a single model pre- pre-trained natural language to finetune on each offline RL task individually, we demonstrate drastic improvements in convergence speeds and final policy performances. We also consider further techniques (e.g. extension of positional

---

Under review as a conference paper at ICLR 2023
Jelpps
Antoregressive Language Model Pre-training (o.g. GPT)
Trajectory Modeling
(finetuning offline RL)
Jempps
R
Rx
AN
Figure 1: Adapting pre-trained language models (e.g. from Wikipedia) to offline RL (e.g. in continuous control and games).
embeddings, embedding similarity encouragement) in order to better take advantage of the features learned by the pre-trained language model and demonstrate greater improvements.
We demonstrate that pre-training on autoregressively modeling natural language provides consistent performance gains when compared to the Decision Transformer (Chen et al. (021) on both the popular OpenAl Gym (Brockman et al. (0016) and Atari (Bellemare et al. 2013) offline RL benchmarks. We also note significantly faster convergence speed, with 3-6x improvement over a vanilla Decision Transformer turning hours of training to tens of minutes, indicating long-term computational efficiency benefits on language pre-training.
Our findings allude to the potential impact of large scale pre-training for reinforcement learning, given its surprising efficacy when transferring from a distant sequence modeling domain such as natural language. Notably, unlike other work on multi-task offline RL, our model provides consistent results in terms of both reward and convergence regardless of environment and setting, indicating a forseeable future where everyone should use pre-trained language model for offline RL.
2 BACKGROUND
Offline Reinforcement Learning We consider standard Markov Decision Process (MDP) with state space E S and action space A, specified by initial state distribution p(s1), dynamics distribution p($t+1]8 (t), and a scalar reward function r(s, a). The goal of reinforcement learning (RL) is to find the optimal policy '²" (als) which maximizes the y-diicounted expected return the agent interacts in the environment,
Σ Ert(s) at)
max E 01 -,p
In offline RL, the objective remains the same, but has to be optimized with no interactive data collection on fixed set of trajectories Ti, each of the form below with horizon N,
T (ri, $1, Q1, 72, $2, .2,..., TN, SN, a.).
Common approaches include value-based or model-based objectives with regularization (Fujimoto et al. 2019. Levine et al. (020). and more recently, direct generative modeling of these trajectories conditioned on hindsight returns (Chen et al. 202T Janner et al. 2021 Furuta et al. 202T).
Transformer model In this subsection, we briefly review the Transformer architecture (Vawwani ctt 2017) used to model sequences. The Transformer is comprised of stacks of identical Trans- former layers. Each of these layers takes in set of n-dimensional vectors that are fed through the two main building blocks: a multi-head self-attention sublayer and a feedfoward MLP as shown below:
((x)
Attention(2) softmax
Vv())
(3)
Vn
Feedforward (x) I2(g(Lr())))
(4)
2

---

Under review as a conference paper at ICLR 2023
where Q, K and V represent linear projections that parameterize the projection of input into the query, key and value spaces; while L1. Le and represent the first linear projection, second linear projection, and activation function that comprise the feedforward MLP. This is followed by residual connection (He et al. 2015) and layer normalization (Ba et al. 2016).
Autoregressive Language Model Pre-training Although there are multiple techniques for language model pre-training (e.g. masked language modeling: Devlin et al.) 2019), we will review DeeeiinggDDeiinnnne
autoregressive language modeling given its correspondence with the sequence modeling objective we employ for our offline reinforcement learning tasks.
Given a sequence x [x X2....X. comprised of tokens x,, we look model the likelihood of the sequence P(x) by way of modeling the probability of predicting each token x; in step-by htep or autoregressive fashion (commonly left-to-right). Naturally, it follows that each token's prediction will be conditioned on all the previous elements in the sequence X shown below (Bengio et al. 2001
P(x) II | X
(5)
3 METHODOLLOGY
In this section we discuss our proposed methodology and techniques to better adapt pre-trained language models to model trajectories, in the case of offline RL tasks with minimal modification to architecture and objectives shown in Figure
3.1 MODELING
Following (Chen et al. 2021 model trajectories autoregressively by representing them in the following manner:
t = (R1, a1, R2. 82, .,. RN, RR,...,,...... SN SN,aN)
(6)
where trajectory t is modeled analogously sequence shown in in Equation and R, Ex It, St, a, represent the returns-to-go. state and action for each timestep given N timesteps, respectively.
3.2 TECINIQUES
Encouraging similarity between language representations and offline RL input representations We find the issue of lack of alignment between state, action and reward input representations and language representations - partially holding back further extraction of the capabilities model. To this end, we use similarity-based objective in order to maximize the similarity between the set of language embeddings E = [Ei...., Ev] with vocabulary size V and the set of input representations I Ii.... LAN. The input representations parameterized by linear projections Lr, L.,, I,, corresponding to the target reward projection, action projection and state projection, respectively.
language
Given the following cosine similarity function:
C(21, 22)
2|
|
we compute the negative (as we use gradient descent to optimize this objective) the sum of maximum similarity value for each embedding E1...., Eg...., Ey and each input representation Jo. Jo....,Iy Ii.... IN as follows:
3N Σ
Lcos
max C(1,, ET)
(8)
WWe looked at using mean pooling instead of max pooling for this objective and found that models with the mean pooling objective did converge.
3

---

Under review as a conference paper at ICLR 2023
This allows encourage the input embeddings to become similar to their language coun- terparts. However, due to computational cost of computing this loss for large values of V, we propose use K mmmanss clustering over the embeddings reduce the size of V to number of clusters K. We then treat the cluster centers akin to the original embeddings in order to compute our loss. Furthermore, optimize this computation with vectorization.
Language model co-training We also experiment with continuing to train jointly on language modeling and trajectory modeling. This allows to encouraging the model's transformer backbone to be able to handle both language and trajectories simultaneously. We refer to the standard negative log likelihood loss over each predicted token used for this objective as LLM-
3.3 FINAL OBJECTIVE
We now combine the objectives into the final objective L LMSE + À Loos + A2L1M. Where LMSE represents the mean squared error loss (calculated between the predicted continuous actions, and continuous actions contained in the dataset) used for the primary trajectery modeling objective (Chen itt al. 2021). LLM represents the negative log likelihood-bate token prediction language modeling objective, and λι, λ2 represent hyperparameters control the weight of the cosine similarity loss and language modeling loss, respectively.
EXPERIMENTS
Game
ChibiT
GPT2
DT
CQL
QR-DQN
REM
BC
Breakout
280.3 = 63.7
287.8 + 78.5
267.5
211.1
21.1
32.1
138.9
Qhert
22.3 + 9.3
22.5 12.8
15.4
104.2
1.7
1.4
17.3
Pong
112.3 7.2
111.0 + 5.7
106.1
111.9
20.0
39.1
85.2
Seaquest
2.9 0.3
3.0 0.2
2.5
1.7
1.4
1.0
2.1
Table 1:
Gamer-normalized scores for the 1% DQN-replay Atari dataset. variance across three seeds. Highest mean scores are highlighted in bold.
We report the mean and
Dataset
Emvronmenn
ChibiT
GPT2
CLIP
IGPT
DT
CQL
TD3-BC
BRAC-v
AWR
BC
Medium Expert
HallCheetahi
Hopper
110.0 1.2
91.7 91.8=05
1109-
1.0.2400.1
01.3+: 044
199:..1
6.9
107.6
56.8
111.0
62.4
90.7
98.0
41.9
0.8
32.7
27.1
50.9
70.6
Walker
108.1
1.3
1.8
0.7
108.1
98.7
110.1
81.6
53.8
36.6
HallCheetahi
43.3 - 0.1
4.8110.1
12320.2
1.11
12.6
44.4
48.3
48.3
37.4
43.1
Medium
Hopper
82.1 4.6
79.1
0.9
5.7.
67.6
58.0
59.3
31.1
35.9
63.9
Walker
77.8
1.53
T1110.
0.4
71.0
79.2
81.1
17.1
77.3
Medium
Replay
HalCCheetah
Walker
Hopper
71.3
81.3 5.0
39.7 2:
94.4 2.5
72.7 1.
40.3
89.9
37.9
45.8 45.8 0.3
0.3
9.1 7.7
5.7 + 0.9
1.1:
06.6
36.6
82.7
26.7
46.2
48.6
81.8
60.9
41.0
47.7
0.6
28.4
40.3
15.5
36.9
27.6
Average (ABl Settings)
78.3
80.1
76.3
3.7
74.7
63.9
75.3
36.9
34.3
46.4
Table 2: Results for D4RL 5 We report the mean and variance for three seeds. Language model pre-trainined models are consistently better than the Decision Transformer, and outperform/are competitive other baselines.
4.1 MODELS
Pre-trained Models We use the popular GPT2-small model to benchmark the impact of language- only pre-training. For direct comparison with the Decision Transformer (Chen et al.) (021). also pre-train a language model with the same parameter count on the popular language modeling Wikitext-103 dataset (Merity al.] 2016), consisting of over 100 million tokens from full Wikipedia articles. We refer to this model as Chibi Note that when we transfer a pre-trained model towards trajectory modeling embeddings, while replacing the language token embeddings with the projections of the action, state and reward representations.
offline RI. dataset, transfer all the Transformer layers and positional
"CChiii" means "small" "mini" in Japanese.
4

---

Under review as a conference paper at ICLR 2023
To explore the effect of pre-training vision datasets, we also study CLIP (Radford et al.) 2021 and ImageGPT (Chen et al., 2020). CLIP is comprised of an image encoder and a text encoder, and trained to predict which caption matches with which image. While the text encoder is an autoregressive Transformer, the image encoder a Vision Transformer, which is not autoregressive. Therefore, for the autoregressive setup of offline reinforcement learning. we use the pre-trained text encoder as our initializer, while discarding the image encoder part. ImageGPT is based on the same Transformer architecture as GPT2, but instead of language, is trained on images unrolled into long sequences of pixels in an autoregressive manner.
RL Baselines state-of-the-art offline RL algorithms as follows: Decision Transformer (DT) (Chen et al. (02I), CQL (Kumar et al., 2020). TD3+BC (Fujimoto & Gu 2021). BRAC (Wu et al. 2019). and AWR
addition to benchmarking our pre-trained language models, we compare to popular
baselines (Peng al. 2019).
Hyperparameters We use the following hyperparameters for language model pre-training: the architecture is the same as that of (Chen et al. 2021) (128 model dim, 1 attention head, layers), learning rate of 3e-4, batch size 65536 tokens, for 6 hours (80000 steps), using warmup schedule over the first 10000. We the same byte-pair encoding (BPE: (BPE:[Sennric) Sennrich et al.] 2016)) Kudo & Richardson 2018) as that used by GPT-2 (Radford al., 2019). For our offline RL tasks, we follow the hyperparameters used by (Chen et al. 2021). For our additional objectives, we decay X1, A2. to reach 0.0 each after 5000 steps. We tune initial values of X for values of {0.1, 0.2) } and X2 for values of (0.0, 0.2, 0.4). We include additional details in the appendix.
We benchmark our models against the D4RL offline RL benchmark datasets (Fu et al. 2020) for the OpenAl Gym MuJoCo (Brockman et al. 2016) and Atari (Bellemare et al. 2013) tasks.
4.2
ATARI
We run our ChibiT and GPT2 models on the challenging Atari dataset (Bellemare et al. 2013). We use the four Atari tasks evaluated in (Agarwal et al. 2020), namely Breakout, Qbert, Pong and Seaquest. Baseline numbers used are provided by (Chen et al. 2021) for behavior cloning and Decision Transformer models, while CQL, REM, and QR-QDN baseline numbers are provided by (Kumar al., 2020 Agarwal al. 2020). Following (Hafner et al. 2021), normalize scores based on that of professional gamer on the evaluation set.
We show results in Table I It can be seen that ChibiT and GPT2 results consistently improve over/match a strong vanilla Decision Transformer baseline. Our models are competitive with the Decision Transformer on all four games and competitive with CQL on 3/4 games.
4.3 GYM
In this section, we consider results on the OpenAl Gym tasks (HalfCheetah, Walker2d, and Hopper) from the D4RL benchmark (Fu et al. 2020).
We train our models for total of 100k timesteps and evaluate every 5000 timesteps, with each evaluation consisting of 10 episodes. Bascline results are obtained directly from the D4RL paper (Fu
al. 2020) and Decision Transformer results are directly taken from (Chen et al., (02T). Similarly, following (Fu et al.) 2020), we compute the normalized score over returns, computed by taking EdorCRTEh score
100 X expert score rendem scoze"
We show results comparing ChibiT, GPT2, and CLIP with state-of-the-arr offline RL algorithms in Table Pre-training improves the Decision Transformer by large margins in an overwhelming majority of tasks, clearly demonstrating that language pre-training improves over random initialization using sequence modeling techniques in terms of reward. We also take note of the minimal difference between ChibiT, CLIP, and GPT2, showing that that at this scale, improvements on offline RL are not necessarily strongly correlated with model size as has been shown on both large-scale vision and language tasks. We note that CLIP, while improving over a vanilla DT model, is often slightly less competitive that our pure language modeling objectives. Our ChibiT and GPT2 models achieve and average performance of 78.3 and 80.1, respectively, showing strong competitiveness on all settings with all baselines. These pre-trained language models acheive ofate-of-the-an results by
5

---

Under review as a conference paper at ICLR 2023
GPT2
1GPT
Random
Farly
10
Middle
33
10-
Last
375
Figure 2: Attention analysis. We visualize early, middle and last attention weights computed by GPT-2, iGPT, and randomly initialized DT models on Hopper-medium to study how pre-training on different modalities affects how the model attends to previous timesteps. The x-axis represents keys (representations that are being "looked at") while the y-axis represents queries (i.e. representations that are "looking at" other representations) for given timestep. Ligher colors represent higher attention weights, while darker colors represent lower weights.
outperforming the strong Decision Transformer and TD3+BC baselines by significant 3.0-5.4 points.
5 ANALYSIS
In this section, we look at more fine-grained details and properties of various aspects of adapting pre-trained language models to offline RL tasks with ablations on OpenAl Gym.
Convergence Speed We evaluate tinetoo-oonvergenc GPT2, ChibiT and DT using the our implementations of the former two and the author- author-provided implementation of the latter.
Results are reported in Table We find that pre-training on language allows us to speed up training process of Transformer-based offline RL models, measured in wall-clock time. Convergence defined as the point where average performance attains score within (normalized score) of the best score.
Interestingly, we also find that GPT2, despite its larger model size at 84M model parameters, still manages to train faster than DT. This points towards potential benefits of pre-training scale and increased efficiency during finetuning. We run experiments on a single NVIDIA V100 16GB GPU and Intel Xeon Gold 6148 Processor.
Model
Walker2d
HalfCheetah
Hopper
Language initialization versus vision initialization As we establish that Transformers pre-trained on lan- guage data are surprisingly effective for accelerating training convergence time on offline reinforcement learning tasks, tempting ask if this phenomenon is inherent to language pre-training or does it extend to vision pre-training as well. To answer this question, we compare two GPT models, ImageGPT-smaal (iGPT) and GPTl-small (GPT2), pre-trained on language and vision data, respectively. Since Transformer architec- tures are domain-ape aggostic, these models can be trained on ID sequences of any form. Hence,
DT (GitHub)
3h14m
3h23m
2h47m
ChibT (ours)
43m
48m
36m
GPT2 (ouns)
1h27m
1h32m
1h2m
Table Training time comparison (measured hours and minutes on a single V100 GPU on the medium medium-expert setting) between the Deci- sion Transformer and two pre-trained models: ChibiT and GPT2 OpenAl gym tasks. Note GPT2 144x larger than other models with 84M model parameters.
compare GPT2, which pre-trained many

---

Under review as a conference paper at ICLR 2023
Sca anguage pro toincd models vorsus randomly 100.0
300.5
Avg. rowardd
300-0
200.5
Model
Avg. Reward
200.3
ChibiT (context 20)
67.7
ChibiT (context = 60)
67.3
10.3-
DT (context = 20)
61.4
Lerguoge holtannn
10
16-
DT (context = 60)
61.2
million parameters
Figure 3: Comparison of Average Medium-Exper reward for various model sizes on OpenAl Gym.
Table 4:
Experiment increased con-
text length with pre-trained models on the medium setting
sequences of discrete language tokens, and iGPT, which was pre-trained on autoregressive image generation at the pixel level (note that both models were trained on ~ 10t0 tokens). Given the results in Table 2 for iGPT, we found that the model had extremely low returns, and did not reach convergence. Notably, on some seeds, the model even performed worse than a random score after training on Walker medium, with normalized score of -0.1, in contrast with GPT-2 pre-training which gives us an average increase of 5.1 points (measured terms of normalized reward) over the Decision Transformer.
Furthermore, when we turn our attention to the difference between GPT2 and CLIP, we see that GPT2, which is based on pure-language based pre-training, performs better. While the text encoder of CLIP is also an autoregressive Transformer pre-trained on text, the objective of CLIP is different from GPT2 in that the former attempts to match text with a corresponding image, while the latter is pre-trained on pure autoregressive language modeling. Given this, we hypothesize that generative (versus discriminative) training objective is more useful for transfer to a generative task.
We believe that this alludes to underlying similarities between language modeling and trajectory modeling, whereas a large difference between image modeling and trajectory modeling. Perhaps this can be attributed to the "natural" sequential nature of language and trajectories, versus the forced 2D->1D nature that was used pre-train iGPT.
Attention Analysis To further understand the discrepancy between language- based and vision- based pre-training, we visualize attention weights, extracted from GPT2 and iGPT after fine-tuning on Hopper medium, as an example offline RL task. As reference, also extract attention weights from randomly initialized networks of Decision Transformers. In Figure [.2 we plot the attention weights averaged over all attention heads in each model, and present the visualizations for early, middle, and last layers, respectively. Due to the autoregressive nature of our task, attention weights in the upper right triangle are masked out, so that the model can only attend past sequences.
As general trend, we see that in earlier layers GPT2 and the randomly initialized model tend to attend to positions with multiples of timesteps behind the current position. This indicates that actions attend to previous actions, states attend to previous states, and returns-to to-go attend to previous returns-to-go. to-go. Constrasted with this, iGPT's attention less interpretable, however showing notably stronger recency bias. In the middle layers, DT continues the trends of its early layers, whereas iGPT tends to fixate on a single state (given the overwhelming brightness of timestep 2), GPT2 starts showing a stronger preference for previous returns to go (given that lighter colors are consistently timestep 1, 4, etc..... Finally, in the models' last layer, while iGPT and random initialization tend to exhibit behaviour closer to mean pooling over all previous inputs, GPT's final prediction seems to be heavily reliant on the initial returns-to to-go. This perhaps indicates that goal conditioning is stronger in GPT2.
How important is the model size of Transformer? We explore how pre-training changes the impact on model size for these offline RL tasks. We train randomly initialized models with various parameter counts (approx. 600K, 3M, 18M, 84M) as well as enguuge-pre-trainee models on WikiText-103 103 with the same parameter counts. Exact hyperparameters for this experiment are given in the Appendix
NNoe that when pre-training language models with 600K, 3M, and 18M parameters, control that pre-training takes exactly 6 hours 4 V100 GPUs.
7

---

Under review as a conference paper at ICLR 2023
Model
HallCheetah
Walker2d
Hopper
Model
HalfCheetah
Walker2d
Hopper
ChibiT
43.3 0.1
17.8 +.1
82.1 + 4.6
Chibil (wlo Gooa)
43.1
ChibiT (FT)
43.3 0.1
77.8 0.1
82.1 + 4.6
ChibiT (wlo Eixt)
43.3 0.2
17.8::022
81.1
ChibiT (Frozen)
26.4 + 1.2
63.3
57.7 + 7.0
ChibiT (rand. emb.)
43.0 0.4
76.5 1.2
78.4 = 2.0
Table 5:
Experiment on freezing model weights versus finetuning them on OpenAl Gym.
Table Ablation of proposed techniques
We visualize the average (over Hopper, Walker2d, and HalfCheetah) of Medium Expert results in Figure 3 Unsurprisingly, we observe that randomly initialized Decision Transformer, tends to have lower relative returns as parameter sizes increase likely due to overfitting on finite data. Interestingly, however, pre-trained language models tend to increase performance parameter count increases, despite diminishing returns with increasing parameter count. Nonetheless, this is exciting as demonstrates that even language pre-training may be beneficial at scale, especially for larger and more diverse offline RL datasets in the future.
Context length We try various context lengths with pre-training and not pre-training: context 20 (following (Chen et al. (02TT) and context = 60. Results are shown in Table It can be seen that additional context does not seem to help even when pre-training on long range language modeling, perhaps alluding to the limited utility of long-range context for the OpenAl Gym tasks.
Can we freeze model parameters? We also look at how ChibiT performs when model weights (transformer blocks: self-attention and feedforward) are frozen with only action, state and return projections La, L,, L, being trained. Previous work (iimpouklll et al. 2021 Lu et al. (202]) has demonstrated how frozen language models have the capability extend the vision domain with respectable performance, which we aim to test with this experiment. We show results on Table 5 on the D4RL medium setting OpenAl Gym. When freezing model weights, performance is underwhelming with performance drastically reducing much as ~40%. We conjecture this is due to our tasks being complex generative modeling as opposed to discriminative classification (Lu et al.) 2021), where the output distribution is of higher dimension hence the need for more intensive finetuning.
Ablation of proposed techniques We perform an ablation study of our proposed auxiliary tech- niques and compare the impact of including and not including pre-trained positional embeddings. Results are shown in T It can seen that the combination of our objectives are able to increase performance consistently. We also note that the removal of pre-trained positional embeddings results in the largest average decrease in performance over ChibiT, alluding to the fact that this positional information is important and transferable offline RL.
RELATED WORK
Transformer Pre-training Pre-training Transformer-basee models (Vaswani al.) 2017) was
initially proposed by (Radford et al. 2018) with their Generative Pre-trained Transformer (GPT). They performed autoregressive language modeling on a relatively large dataset, showing promising initial success not only on its ability to scale to large models sizes, but also for its impressive performance when fine-tuning sieetuning on task-specific natural language understanding (NLU; Wang et al. (2018)) datasets. paradigm with their masked language modeling objective. Furthermore, recently this paradigm has extended to computer vision with the Vision Transformer (ViT; Dosovitskiy al. 2021) and iGPT (Chnn al. 2020)
BERT (Devlin et al., 2019), extended this tre-train->finttue
Sequence Modeling for Offline RL Offline RL became popular starting from a simple observation
that many performant off-policy algorithms (Mnih et al.] 2015 Lillicrap et al., 2015 Gu et al.)
2016 Haarnoja et al.] 2018; Fujimoto et al. (0188) fail to learn in fully off-policy, i.e. offline, batch setting (Fujimoto et al., 2019). Numerous algorithmic work ensued (Wu et al. 2019 Jaques ett al.) 2020; Ghasemipour
al. 2021 Kumar al., 2020; Fujimoto & Gul 202 with various applications (Jaques et al. 2020 Chebotar et al. 2021). Building on reward- coward-conditioned imitation
learning (Srivastava al."" 2019 Kumar et al. 2019) Transformers (Parisotto et al. 2020) have been
8

---

Under review as a conference paper at ICLR 2023
recently adopted for replacing offline RL with sequence modeling (Chen et al. 2021| Janner et al. 2021; Furuta et al. 2021). Despite initial successes, many techniques popular in language modeling have yet to be experimented in these offline RL benchmarks, and our work constitutes an initial step toward bridging the two communities.
Pre-training for RL Contrary to language or vision (Devlin et al.
2019 Dosovitskiy et al. 202 1
successes deep RL have largely focused on isolated tasks/ domains (Mnih et al. 2015 Silver
eet al.. 2016; Gu et al. 2017: Kalashnikov et al. 2018; Vinyals et al. 2019).
Pre-training results are often limited to vision or language processing (Yen-Chen et al.) 2020: Lynch & Sermanei 202T)
or specially-crafted domains (Singh et al. 22200] Tirumala et al., 2020). Arguably, fundamental
bottleneck for pre-training in RL the difficulty in reusing single network across vastly different tasks, observation spaces, action spaces, rewards, scenes, and agent morphologies. Preliminary work explored various aspects of this problem through graph neural networks for morphology generalization (Wang al. 2018b Pathak et al.] [0099] Chen al. 2018 Kurin al.) (2020). language for universal reward specification (Jiang et al.
2019: Lynch & Sermanet 2021 1 [Shridhar
al. 2022), and object-centric action spaces (Zeng et al.
2020 Shridhar al. 2022; Noguchi et al.
2021). Our work is orthogonal to these as we essentially amortize RL algorithm itself, expressed
as sequence modeling with Transformer, instead of specific RL domain information, and can be
combined with domain-specific pre-training techniques YennChhnn et al. 2020
Lynch & Sermanet
2021 Banino et al. 2021 effortlessly.
Adapting language models to new modalities and domains Within language modeling recently there has been interest in adaptation of pre-trained language models by way of continued pre-training (Gururangan et al., 2020). Furthermore, (Tsimpoukelli et al.] (021) looked at adapting frozen language models for few-shot question answering by adding auxiliary vision encoder. Other (concurrrent) work has proposed using language as semantically meaningful way of communicating between modalities directly using frozen pre-trained language models for planning (Zeng al.
2022 Li et al. 2022: Huang et al. 2022). More related to our work is that of (Lu et al. 20021) where they look adapting frozen language models to various tasks such as image classification. Concurrent work (Reed et al. (2022) has looked at multi-tasking using generic sequence modeling for transformer-basee RL agents, while other concurrent work has shown that language pre-training is helpful for in-context learning as a result of having long-tailed distribution (Chan et al. 2022). Our work extends on the spirit of these works by adapting language models to a new domain of RL, however, as far was we know, we the first to propose leveraging generative model (in language) for generation in another domain (RL) as opposed to a discriminatory task such as classification.
CONCLUSION
We investigate how pre-trained models can improve generic offline RL problems, recently casted as sequence modeling. To our surprise, we discover that fine-tuning from a Wikipedia-trainee small transformer (ChibiT) GPT2 model outperforms the basic Decision Transformer (DTT and other RI.-based offline baselines by a large margin in terms of policy performance and convergence, establishing state-of-the-as art scores on the competitive D4RL benchmark in both Gym and Atari and cutting down the DT training time by 3-6x. We perform extensive ablation studies and analyses, and found how language pre-training (as opposed vision pre-training), model size, and fine-tuning (as opposed to freezing parameters) play critical roles in the final performances. We hope our work can accelerate the adoption of pre-training in RL and leads more interest in applying other sequence modeling techniques from language and vision into RL.
Beyond RI,, our work constitutes the first successful transfer, to the best of our knowledge, of a pre-trained generative model in one domain (language) to a generative modeling task in a completely different domain (RL on continuous control and games). This hints at some underlying universal structure across sequence modeling domains, and could perhaps lead to unified generative modeling pre- training for better transferability among them. In future work, we look to investigate in more depth which properties of language structure are useful for reinforcement learning and sequence modeling in other domains, and whether previous work studying language structure (Hupkes et al. 2019) does indeed relate to compositional generalization of neural networks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning, 2020.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Mare G. Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference Neu- ral Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 29304 29320, 2021. URL https: httpttpp:s//ppccceddenntt hew.:edddnnnnnn................. 9cf4 15 42 Abstract -html
wwtpppppppppppppppppppcppppp.epp 2022/02222222 //hash
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint arXiv: Arxiv-1607,0644 06450, 2016.
Alexei Baevski and Michael Auli. Adaptive imput representations for neural language modeling. In International Conference Learning Representations, 2019. URL URL httpss://opeeeeeiiee
Andrea Banino, Adrià Puidomenech Badia, Jacob Walker, Tim Scholtes, Jovana Mitrovic, and Charles Blundell. Coberl: Contrastive bert for reinforcement learning. arXiv preprint arXiv: Arxiv-2107.05431, 2021.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013.
Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. In T. Leen. T. Dietterich, and V. Tresp (eds.), Advances in Neural Information Processing Systems, volume 13. MIT Press, 2001. URL htt.:: httpptppppppppp.apapggggoooo///e
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz I Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Stephanie C. Y. Chan, Adam Santoro, Andrew K. Lampinen, Jane X. Wang, Aaditya Singh, Pierre H. Richemond, Jay McClelland, and Felix Hill. Data distributional properties drive emergent few-shot learning in transformers. arXiv preprint arXiv: Arxiv-2205
2022.
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, et Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint arXiv:2/04.07749 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv: Arxiv-2 Priv-21066.01 106.0115..,.... 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu. Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1691-1703. PMIR, 13-18 Jul 2020. /
ings.
mlr.
html
Tao Chen, Adithyavairavan Murali, and Abhinav Gupta. Hardware conditioned policies for multi- robot transfer learning. arXiv preprint arXiv:181 1,08664, 2018.
10

---

Under review as a conference paper at ICLR 2023
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pP- PMLR, 2020.
Jacob Devlin, Ming-Was Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv: Arxiv-2004.072199 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. arXiv preprint urXiv:2106.0660 2021.
Scott Fujimoto. Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pp. 2062-2062. PMLR, 2019.
Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for offline hindsight information matching. arXiv preprint arXiv:2111.00366 2021.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference Machine Learning, pp. 3682-3691. PMLR, 2021.
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International conference on machine learning, pp. 2829-2838. PMLR, 2016.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off- off-policy updates. In 2017 IEEE international conference robotics and automation (ICRA), pp. 3389-3396. IEEE, 2017.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't t stop pretraining: Adapt language models to domains and tasks, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp.
PMLR, 2018.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models, 2021.
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv: Arx00-15/2.03385,, 2015.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models zero- shot planners: Extracting actionable knowledge for embodied agents. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 9118-9147. PMLR, 2022. URL mlr. www.sssssse........ v 62/ ht/huangg2200 htmml
Dicuwke Hupkes, Verna Dankers, Mathijs Mul. and Elia Bruni. Compositionality decomposed: how do neural networks generalise? arXiv preprint arXiv: Arxiv-1908 08351, 2019.
Michael Janner, Qiyang Li, and Sergey Levine. Reinforcement learning as one big sequence modeling problem. arXiv preprint arXiv:2106.02039, 2021.

---

Under review as a conference paper at ICLR 2023
Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Shane Gu, and Rosalind Picard. Human-centrie dialog training via offline reinforcement learning. arXiv preprint arXiv:2010.05848 2020.
Yiding Jiang, Shixiang Gu, Kevin Murphy, and Chelsea Finn. Language abstraction for hierarchical deep reinforcement learning. arXiv preprint arXiv:1906.07344 2019.
Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 14, 2001.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv: 1806. 10293, 2018.
Taku Kudo and John Richardson. Sentencepiece: simple and language independent subword tokenizer and detokenizer for neural text processing, 2018.
Aviral Kumar, Xue Bin Peng, and Sergey Levine. Reward-conditioned policies. arXiv preprint arXiv:1912.13465, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0477 2020.
Vitaly Kurin, Maximilian Igl. Tim Rocktäschel, Wendelin Boehmer, and Shimon Whiteson. My body is cage: the role of morphology in graph-based incompatible control. arXiv preprint arXiv:2010.01856, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement Iearning: Tutorial, review, and perspectives open problems. arXiv preprint arXiv:2005.0/66 2020.
Shuang Li, Xavier Puig, Yilun Du, Clinton Wang, Ekin Akyürek, Antonio Torralba, Jacob An- dreas, and Igor Mordatch. Pre-trained language models for interactive decision makiin.. ArXiv, abs/2202.01 01 2022.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: (ttp:::500099999 02597, 2015.
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained transformers as universal computation engines, 2021.
Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. Proceedings of Robotics: Science and Systems. doi, 10, 2021.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 8177550):5295533, 2015.
Yuki Noguchi, Tatsuya Matsushima, Yutaka Matsuo, and Shixiang Shane Gu. Tool as embodiment for recursive manipulation. arXiv preprint arXiv:2 /2.00399 2021.
Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphaël Lopez Kaufman, Aidan Clark, Seb Noury, Matthew Botvinick, Nicolas Heess, and Raia Hadsell. Stabilizing transformers for reinforcement learning. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 1 119 of Proceedings of Machine Learning Research, pp. 7487-7498. PMLR, 13-18 Jul 2020. URL.tttpa /ttp///ooooeecce/// ngs mlrr. www.sprsssssep...p.p. / sotto20a html
Deepak Pathak, Chris Lu, Trevor Darrell, Phillip Isola, and Alexei A Efros. Learning con- trol self-assembling morphologies: a study of generalization via modularity. arXiv preprint arXiv: 190 05546, 2019.
12

---

Under review as a conference paper at ICLR 2023
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. advantage-weighted regression: Simple and scalable off- oofico reinforcement learning. arXiv preprint arXiv: Arxi--1910.00177 2019.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Suskeveter. Improving language under- standing by generative pre-training, 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley D. Edwards, Nicolas Manfred Otto Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. ArXiv, abs/2205.06175. 2022.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers). pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 10.11885/1//11-111 1162. URL http:: hattps://aclanthhll.s
org Dog/P16-220 1162
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning, pp. 894-906. PMLR, 2022.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):4844489 2016.
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. arXiv preprint arXiv:2011.1002. 2020.
Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojciech Jaškowski, and Jürgen Schmidhuber. Training agents using upside-down reinforcement learning. arXiv preprint arXiv: wrXiv:192..02877, 2019.
Dhruva Tirumala, Alexandre Galashov, Hyeonwoo Noh, Leonard Hasenelever, Razvan Pascanu, Jonathan Schwarz, Guillaume Desjardins, Wojciech Marian Czarnecki, Arun Ahuja, Yee Whye Teh, et al. Behavior priors for efficient reinforcement learning. arXiv preprint arXiv:2010.1422 2020.
Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, 2017.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning, Nature, 575(7782):350-354, 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: Arxiv-1804.00 07461, 2018a.
Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. In International Conference on Learning Representations, 2018b.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):27--292. 1992.
13

---

Under review as a conference paper at ICLR 2023
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's 's transformers: State-of-the-art natural language processing, 2020.
Yifan Wu, George Tucker, and Ofrr Nachum. Behavior regularized offline reinforcement learning. arXiv preprint
2019.
Lin Yen-Chen, Andy Zeng, Shuran Song, Phillip Isola, and Tsung-Y. Yi Lin. Learning to see before learning to act: Visual pre-training for manipulation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 7286-7293. IEEE, 2020.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning, pp. 1094-1100. PMLR, 2020.
Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong. Ivan Krasin, Dan Duong, Vikas Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. arXiv preprint arXiv:2010. 14406, 2020.
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Fed- erico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprini arXiv: Arxiv-2204.00590 2022.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martin-Martin. robosuite: A modular simulation framework and benchmark for robot learning, arXiv preprint arXiv:2009.12290 12290, 2020.
14

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.I HYPERPARAMETERS & TRAINING DETAILS
Hyperparameter
Value
Layers
3
Attention Heads
1
Activation fn
ReLU
Batch size
64
Context
20
Return-to-go conditioning
6000 HalfCheetah
3600 Hopper
5000 Walker
Dropout
0.2
Learning rate
le-4
LR Warmup for GPT2
5000 steps
500
K for ChibiT
1000
0.1
0.2
Hopper A
0.2
Table 7: Hyperparameters used for OpenAl Gym
On choosing the value of K We base the choice of the value of K based on GPU memory constraints. For K 1000 and K 500, we find that they perform similarly in practice (both time and performance wise), albeit K 1000 performing slightly better performance wise. However the memory requirements of K 1000 tend double which often leads to OOM errors our NVIDIA V100 16GB GPUs for GPT-2 (motivating our reason to use K = 500 for GPT-2 and K 1000 for ChibiT).
Other implementation details Pre-trained models are trained with and taken from the HuggingFace Transformers library (Wolf et al., 2020). The model code for our GPT2 model gpt CLIP ietiippiittii apenai/clip vit base patch32 and iGPT openai /ma/gegptteptt small.
Model
Parameter Count
Num. Tokens
DT
596K
ChibiT
596K
107
iGPT
84M
10
GPT-2
84M
10
CLIP
38M
10
Table 8: Model parameter counts and number of unique pre-training tokens
Language Model Pre-training with larger sizes For our large sized pre-trained models in model scale experiments, we use the following dimensions:
Param. Count
Model Dim.
Num. Heads
Num. Layers
3M
256
4
18M
512
6
84M
768
12
12
Table 9: Parameter count for various pre-trained models used in our model scale experiments.
15

---

Under review as a conference paper at ICLR 2023
B ATTENTION VISUALIZATION
We visualize the attention weights with temperature 0.1 to improve visual interpretation.
C REPRODUCTION OF DT RESULTS VERSUS DT RESULTS IN (CHEN ET AL.
2021
We re-run the results in (Chen et al.
2021 and include them for reference in Table TO
Dataset
Enviromment
DT
DT((uur))
HalfCheetah
86.8 + 1.3
86.5 + 0.8
Medium Expert
Hopper
107.6 + 1.8
107.4 2.0
Walker
108.1 0.2
108.4 0.1
HalfCheetah
42.1 0.1
Medium
Hopper
67.6 1.0
3.1
Walker
74.0 1.4
74.4 1.9
HalfCheetah
36.6 - 0.8
36.2 + 1.4
Medium Replay
Hopper
82.7 +: 7.0
80.4 6.3
Walker
66.6 = 3.0
67.0 2.4
Average (All Settings)
74.7
74.5
Table 10: Re-implementation of Decision Transformer using their codebase
D PERFORMANCE PROFILES
We compute statistical significance tests using rliable (Agarwal al. (021] OpenAl Gym. Specifically, as we are only comparing two algorithms DT (Chen et al. (2021) and ChibiT, we only plot performance profiles and the boostrapped confidence interval measure.
16

---

Under review as a conference paper at ICLR 2023
100
190
130
030
300
690
000
17.33
10.00 RES.
MED US00 BGS 5010
200
810
02.
61.0
109
1:00
1.00
050
030
0:00
022
020
8:0
0.0
6:00
100
122
124
11C
111
112
18mm Hibmalaladd
1.00
1:00
Packa Kw 8a0e
a00
053
8:30
325
020
2.5
0.0
1279 1017 120 7125
112.5
Manaaiid Rose (wwwllocdocccnee eapai
Humon Normatze
Figure 4: Performance profiles on D4RL datasets. Yellow colors represent ChibiT and blue colors represent Decision Transformer (DT). We report the profiles based on score distributions over 10 runs using different random seeds. Language model pre-trained models are consistently better than DT.
(lumsn homalaut Dook for unbtreecu
Haman Nama Door wultvvealnddiinnn
1DM
20T3
CHIIT
Humas Cooe rntociindanviele
Scoee wathheeltrraeeeeeem
Orai
X00
TACE 1005 HIT
RED 2005 TRF 300 0X0 013
Figure 5: Bootstrapped confidence intervals (CIs) D4RL datasets. Yellow colors represent ChibiT and blue colors represent Decision Transformer (DT), We report the intervals based on score distribu- tions over 10 runs using different random seeds. Language model pre-trained models consistently better than DT.
17


Reviews:
[{"review_id": "msyFIYkTxUl", "reviewer": "Reviewer_gaVR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies the effects of a using a pretrained language model for the offline reinforcement learning problem. It proposes techniques to improve upon the feature learning component of the pretrained models. Experimentally, this paper shows performance gains in accuracy as well as decreased training times.", "summary_of_the_review": "The overall idea is interesting but I think the paper can be further strengthened by making the evaluations more thorough.", "strength_and_weaknesses": "Strength\nThe overall question asked by the paper is an interesting one - whether pretraining on different modalities can be helpful for offline RL. The paper studies this problem from two perspectives: the final reward obtained by the trained policies and the computational time to achieve those rewards.   \n\nWeakness\nWhile the idea is interesting, the paper in its current draft has several weaknesses:\n- The evaluation section of the experiments is quite weak. The paper only compares the method and baselines on a Atari and a subset of the D4RL benchmarks. I would like to see a more comprehensive evalaution of the proposed method on the full suite of D4RL and RL unplugged dataset to validate the claims.\n- Additionally, while the paper claims that the results are consistently better than baselines, in Table 1 for instance only for 1 (Seaquest) dataset can the proposed method be seen to perform statistically better than baselines. Bolding the means in this table is quite misleading. Similarly, for Table 2, only 3/9 cases does the method perform better.\n- One of the main advantages of the proposed method is the superior convergence times as compared to exisiting benchmarks. This advantage, which is where the paper cna possibly shine, is only shown via a table on 3 environments. It would be great to actually see training curves (am not sure right now with the high variance in rewards, how the thresholding is done) across a range of environments to convince the reader that this advantage is indeed maintained.\n- Section 5 (analysis section) while addresses quite a few interesting hypotheses, however, it again falls short in execution. For instance, Figure 3 is plotted for average reward across environments and the y-axis scale is from 101-104, without any error bars. I am not sure how to make a conclusive inference from this plot. Similarly, Table 4 has only two different context lengths and compares average reward (which is not an indicative measure of performance) on one particular setup. Overall, i think the all the hypotheses need to be thoroughly tested before making sweeping conlusions.   \n- I am surprised that setting lambda_2 to zero leads to a big degradation in performance. Does this indicate that pretraining on wikipedia is not helpful from a statistical perspective but somehow helps stabilize optimization? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OJUy0NBJAh8", "reviewer": "Reviewer_y7eL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper verifies that large-scale pre-training has an impressive impact on offline reinforcement learning. Accordingly, it proposes an auto-regressive based understandable language model co-training strategy ChibiT. The model has impressive performance on both Atari and GYM.", "summary_of_the_review": "This submission is well-motivated and well-studied. The empirical results support the hypothesis that pre-trained sequence models on other domains (e.g. vision) can be transferred to other domains when fine-tuned well. Even though there are only limited technical contributions, this timely empirical investigation would be welcomed by the offline RL community. ", "strength_and_weaknesses": "# Strength\n\n- The problem is well defined, with a clear mathematical formulation.\n- The performance improvement is impressive.\n- The paper investigates an interesting question that has a large audience.\n\n\n# Weakness\n\n- The paper fails to explain how different pretrained models influence off-line RL. \n- I don't see the point of including Wikipedia in the name. I assume that it's not a key element of their model and conclusions. They may also train their model on other corpora, e.g. c4. And the paper doesn't conduct an ablation study of using a different language-based corpus.\n- The interpretability of using a language model to help offline RL is weakly explained in the paper. It seems that the authors believe that the offline RL only benefits from a similar sequential structure. If so, I think of an experiment setting that may verify it. If a pretrained vision model is trained using continuous images with low resolutions, which can be constructed by dealing with videos crawled from youtube or other online channels, how will the model perform to help the off-line reinforcement learning?\n- The author should provide experiments based on language models like XL-Net and RoBerta with a similar parameter size. It may help to illustrate the interpretability of using language models to help offline RL as well.\n- This submission has relatively limited technical contributions, as most of the algorithmic components were proposed in previous papers. \n\n# Questions\n\n- The experiment results in Table. 3 are impressive but counter-intuitive. I wonder whether there is a probability that the GPU utilization is not fairly equal when running these experiments with DT, ChibiT, and GPT2. Because the experiments are done with a single V100 and models with very different parameter sizes. I hope that the authors can double-check it and provide the analysis based on steps with equal batches. The training time comparison experiment is not convincing enough for me.\n- The interpretability of why the pure language-based pretraining model performs better than the pretrained vision-language multimodal is very interesting but relatively shallow.  Are there any possibilities that an experiment can be conducted to verify that the “natural” sequential nature of language and trajectories helps?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "D7zVn5SjXHx", "reviewer": "Reviewer_3DmF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper evaluates the impact of using language and vision pretrained transformers as an initialization for a Decision Transformer style model used in offline RL settings.  It provide methods for improving the transferability of the pretrained models and demonstrate that pretraining consistently matches or outperforms random initialization on Atari and substantially outperforms random initialization on D4RL, in addition to accelerating training speed.  It also includes thorough ablations demonstrating the value of the proposed methods and additional insight into the impact of design decisions like pretraining modality and model size among others.", "summary_of_the_review": "A nice paper that demonstrates the effectiveness of language model pretraining for offline RL applications by proposing methods to facilitate this transfer and providing thorough and compelling experiments supporting the success of their approach.", "strength_and_weaknesses": "Strengths:\n- Provides strong evidence that pretraining on language at least matches but more often improves performance over random initialization, often by a large margin.\n- Good experimental design and thoroughness, specifically: Training ChibiT to give a comparably sized model to DT which enabled apples to apples comparison and an understanding of the impact of model size on transferability, reproducing DT results to ensure comparability, using rliable for statistical significance tests, including the attention analysis & thorough ablations\n\nWeaknesses:\n- Not much motivation is given for using the LM objective. I find it surprising that this improves performance, do you have intuition for why this is?  I see in Table 6 that removing either the LM objective or the representation alignment objective hurts performance, and I wonder at the impact of removing both.  A hypothesis could be that these two objectives work in conjunction, that pushing the RL representations to align with the original word embeddings makes it beneficial for the transformer to continue to be able to act on the original word embeddings.  If this is the case then removing one or the other alone would clearly harm performance, but it wouldn’t necessarily be the case that removing both would harm performance.\n\nSmall notes & questions (that don’t impact decision):\n- In Section 4.1, Hyperparameters paragraph, sentence 2 is missing a word: “We [use] the same byte-pair encoding…”\n- In Section 5, “Can we freeze model parameters?” You conjecture that freezing hurt performance in your setting in contrast to Lu et al 2021 because your task is more complex (a plausible conjecture).  However, note that a follow-up to the Lu et al 2021 paper (Arxiv: https://arxiv.org/abs/2107.12460) indicates that when freezing model weights, performance reduces on the subset of tasks from Lu et al 2021 tested (if the learning rate is tuned). This suggests that a difference in task complexity may not be the sole reason for freezing hurting performance in your case.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ASX-1abuv6G", "reviewer": "Reviewer_F11T", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a technique to improve inter-domain migration after investigating how pre-trained models can improve the general offline RL problem, using reinforcement learning as a paradigm for sequence modeling, and investigating the transferability of pre-trained sequence models on other domains (vision, language) for fine-tuning on offline reinforcement learning tasks (control, games). The authors' fine-tuning from a Wikipedia-trained small transformer (ChibiT) and GPT2 model proves to be substantially better than the basic Decision Transformer (DT) and other RL-based offline baselines in terms of policy performance and convergence, building state-of-the-art scores on competitive D4RL benchmarks from Gym and Atari, and reducing DT training time by a factor of 3-6, clearly demonstrating that language pre-training outperforms random initialization using sequence modeling techniques in terms of reward. The work is somewhat innovative, transferring a pre-trained generative model in one domain (language) to a generative modeling task in another completely different domain (RL on continuous control and games), which is an effective attempt in terms of experimental results. This is a recommended paper that reveals the potential of pre-trained models using generic sequence modeling techniques and RL.", "summary_of_the_review": "This paper address offline reinforcement learning from a sequence modeling perspective with an eye to introducing the Transformer architecture, and the results are improved to address the problem of high variability in migrability between different environments. When models are trained from scratch, convergence is slow, and the authors use this reinforcement learning as sequence modeling to investigate the transferability of pre-trained sequence models on other domains (vision, language) for fine-tuning offline reinforcement learning tasks (control, games). I have some knowledge of the relevant techniques and the overall architecture is reasonable.", "strength_and_weaknesses": "Strengths\n1. The pre-trained model is built under the Decision Transformer framework, which is a very novel attempt, and the scheme is feasible from the experimental results.\n2. The experimental parameters are comprehensive, reproducible and technically very solid, which shows that the authors have sufficient technical reserves and good insights into the field, and the information provided represents a fair effort to enable the reader to reproduce the results.\n3. The authors demonstrate the excellent performance of the two proposed pre-training models through comprehensive experiments, revealing the potential of pre-training models using generic sequence modeling techniques and RL, inspiring future work on knowledge sharing between generative modeling tasks in completely different domains.\n\nWeaknesses\n1. too little is shown about the model architecture, can a more specific discussion be given?\n2. the convergence speed of the model proposed in this paper is significantly reduced relative to the Decision Transformer, which proves the effectiveness of pre-training but does not consider the time required for pre-training, can this be provided?\n3. In terms of the final results, the reward boost is not groundbreaking, can you explain why this is the case? Is it due to data quality or model architecture?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject