Pre-training with Synthetic Data Helps Offline Reinforcement Learning
Authors: Zecheng Wang
Che Wang
Zixuan Dong
Keith W. Ross
Venue: ICLR 2024 poster (2024)
Keywords: Deep Reinforcement Learning
Offline Reinforcement Learning
Pretraining
Primary area: reinforcement learning
TL;DR: We show pre-training with synthetic Markov Chain and MDP data can significantly improve offline DRL performance.

Abstract: Recently, it has been shown that for offline deep reinforcement learning (DRL), pre-training Decision Transformer with a large language corpus can improve downstream performance (Reid et al., 2022). A natural question to ask is whether this performance gain can only be achieved with language pre-training, or can be achieved with simpler pre-training schemes which do not involve language. In this paper, we first show that language is not essential for improved performance, and indeed pre-training with synthetic IID data for a small number of updates can match the performance gains from pre-training with a large language corpus; moreover, pre-training with data generated by a one-step Markov chain can further improve the performance. Inspired by these experimental results, we then consider pre-training Conservative Q-Learning (CQL), a popular offline DRL algorithm, which is Q-learning-based and typically employs a Multi-Layer Perceptron (MLP) backbone. Surprisingly, pre-training with simple synthetic data for a small number of updates can also improve CQL, providing consistent performance improvement on D4RL Gym locomotion datasets. The results of this paper not only illustrate the importance of pre-training for offline DRL but also show that the pre-training data can be synthetic and generated with remarkably simple mechanisms.

Published as a conference paper at ICLR 2024
PRE- WITH SYNTHETIC DATA HELPS OFFLINE REINFORCEMENT LEARNING
Zecheng Wang! +f
Che Wang?.4e
Zixuan Dong".4.
Keith Ross
1 New York University Abu Dhabi New York University Shanghai 3 SFSC of Al and DL, NYU Shanghai 4 New York University
ABSTRACT
Recently, it has been shown that for offline deep reinforcement learning (DRL), pre-training Decision Transformer with a large language corpus can improve downstream performance (Reid al. 2022). A natural question ask is whether this performance gain can only be achieved with language pre-training, or can be achieved with simpler pre-training schemes which do not involve language. In this paper, we first show that language is not essential for improved performance, and indeed pre- pre-training with synthetic IID data for a small number of updates can match the performance gains from pre-training with large language corpus; moreover, pre-training with data generated by one-step Markov chain can further improve the performance. Inspired by these experimental results, we then consider pre-training Conservative Q-Learning (CQL), a popular offline DRL algorithm, which is Q-learning- and typically employs a Multi-Layer Perceptron (MLP) backbone. Surprisingly, pre-training with simple synthetic data for a small number of updates can also improve CQL. providing consistent performance improvement on D4RL Gym locomotion datasets. The results of this paper not only illustrate the importance of pre-training for offline DRI. but also show that the pre-training data can be synthetic and generated with remarkably simple mechanisms.
INTRODUCTION
It is well-known that vre-training can provide significant boosts in performance and robustmess for downstream tasks, both for Natural Language Processing (NLP) and Computer Vision (CV). Recently, in the field of Deep Reinforcement Learning (DRL), research on pre-training is also becoming increasingly popular. An important step in the direction of pre-training DRL models the recent paper by Reid et al. (2022), which showed that for Decision Transformer (DT) (Chen al. 2021). pre-training with the Wikipedia corpus can significantly improve the performance of the downstream offline RL task. Reid et al. (2022) further showed that pre-training on predicting pixel sequences hurts performance. The authors state that their results indicate "a foreseeable future where everyone should use pre-trained language model for offline RL". In a more recent paper. Takagi (2022) explores more deeply why pre-training with a language corpus can improve DT. However, it remains unclear whether language data is special in providing such benefit, or whether more naive pre-training approaches can achieve the same effect. Understanding this important question can help us develop better pre- pre-training schemes for DRL algorithms that are more performant, robust and efficient.
We first explore pre-training Decision Transformer (DT) with synthetic data generated from simple and seemingly naive approach. Specifically, we create a finite-state Markov Chain with small number of states (100 states by default). The transition matrix of the Markov chain is obtained randomly and is not related to the environments or the offline datasets. Using the one-step MC, we generate sequence of synthetic MC states. During pre-training, we treat each MC state in the sequence as a token, feed the sequence into the transformer, and employ autoregressive next-state (token) prediction, as is often done in transformer-base LLMs (Brown et al. 2020). We pre-train
"Equal contribution. *Corresponding author, email: 2w2374 @ @nyu.edu This work done prior to Che Wang joining Amazon.

---

Published as a conference paper at ICLR 2024
with the synthetic data for a relatively small number of updates compared with that of language pre- pre-training updates in (Reid et al. 2022). After pre pre-training with the synthetic data, we then fine- tune with specific offline dataset using the DT fifline-DRL algorithm. Surprisingly, this simple approach significantly outperforms standard DT (i.e., with no pre-training) and also outperforms pre-training with large Wiki corpus. Additionally, show that even pre-training with Independent and Identically Distributed (IID) data can still match the performance of Wiki pre-training.
Inspired by these results, we then consider pre-training Conservative Q-Learning (CQL) (Kumar et al. 2220) which employs a Multi-Layer Perceptron (MLP) backbone. Here, we randomly generate a policy and transition probabilities, from which we generate a sequence of Markov Decision Process (MDP) state-action pairs. We then feed the state-action pairs into the Q-network MLPs and pre-train them by predicting the subsequent state. After this, we fine-tune them with a specific offline dataset using CQL. Surprisingly, pre-training with III) and MDP data both can give boost to CQL..
Our experiments and extensive ablations show that pre-training offline DRL, models with simple synthetic datasets can significantly improve performance compared with those with no pre-training, both for transformer- and MLP-based backbones, with a low computation overhead. The results also show that large language datasets are not necessary for obtaining performance boosts, which sheds light on what kind of pre-training strategies are critical to improving RL performance and argues for increased usage of pre-training with synthetic data for an easy and consistent performance boost.
RELATED WORK
Many practical applications of RL constrain agents to learn from an offline dataset that has already been gathered, without further interactions with the environment (Fujimoto et al., 2019:] Levine et al.,
2020). The early offiie DRL papers often employ Multi-Layer Perceptron (MLP) architectures
(Fujimoto et al.) 2019 Chen et al. 2020 Kumar al. 2020 Kostrikov et al., 202T). More recently,
there has been sigmificant interest in transformer-bast based architectures for offline DRL, including DT
(Chen et al. 2021). Trajectory Transformer (Janner et al. 2021) and others (Furuta al., 202 Li
et al. 2023). In this paper, we study both transformer-ban based and the more conventional Q-learning- based methods to understand how different pre-training schemes can affect their performance.
is well-known that pre- tre-traiiing can provide significant improvements in performance and robustness for downstream tasks, both for Natural Language Processing (NLP) (Devlin et al. [0088] Radford
et al. [2018] Brrwn et al. 2020) and Computer Vision (CV) (Donahue et al. [014] Huh et al. [016]]
Kornblith et al., 2019p. p In offline DRI, pre-training is becoming an increasingly popular research topic. An important step in the direction of pre- pre-training offline DRL models is Reid et al. (022)),
which shows that for DT. pre- pre-training on Wikipedia can significantly improve the performance of the downstream RL task. Takagi (002)) further explores why such pre-trate pre-training improves DT. Inspired by these recent findings, we aim for a more comprehensive understanding of pre-training in DRL.
There are also works that pretrain representations and then use them to learn offline or online DRL tasks (Yang & Nachum 2021 Zhan
generic image data or use offline DRL data itself learn
et al. 2021 Wang et al. 2022 Shah & Kumar 2021 Hansen al. [202 Nair et al. [022 Parisi et al.
2022: Radosavovic et al. 2023 Karamcheti et al. 2023). Xie et al. (2023) shows funure-conditioned
unsupervised pretraining leads to superior performance in the offline-online setting. Different from these works, we focus on understanding whether language pre- pre-training is special in providing a performance boost and investigate whether synthetic pre-training can help DRL.
Pre-training with synthetic data has been shown to benefit a wide range of downstream NLP tasks
(Papadimitriou & Jurafsky| (2020) Krishna et al., 202T Ri & Tsuruoka, (2022, Wu et al. 2022 Chiang & Lee 2022). CV tasks (Kataoka et al. [220 Anderson & Farrell| (022)) and mathematical reasoning tasks Wu et al. [02T]. There are also works that study the effect of different properties of synthetic NLP data Ri & Tsuruoka (2022); Chiang & Lee (2022) He et al. (2022b). In particular, we provide results that show the Identity and Mase-ng synthetic data schemes from He et al. (2022b) can also improve offline RL performance in Appendix F While these works focus on CV and NLP applications, we study the effect of pre-training from synthetic data with large domain gaps in DRL.
To the best of our knowledge, this is the first paper that shows pre-training on simple synthetic data can be a surprisingly effective approach to improve offline DRL performance for both raanformer-basee and D-learning-hasee approaches.
2

---

Published as a conference paper at ICLR 2024
3 PRE-TRAINING DECISION TRANSFORMER WITH SYNTHETIC DATA
3.1 OVERVIEW OF DECISION TRANSFORMER
Chen et al. (2021) introduced Decision Transformer (DT), transformer-based algorithm for offline RL. An offline dataset consists of trajectories 81, a1.7':.... ............... .N,, aN, where Sn, an and I' is the state, action, and reward timestep n.. DT models trajectories by representing them as
[Ri, 81, 01,......... RN, SN, UN);
where Rn = D "t is the return-to-go at timestep n. The sequence a modeled with a transformer in an autoregressive manner similar to autoregressive language modeling except that Ra Sn, On at the same timestep n are first projected into separate embeddings while receiving the same po- sitional embedding. In Chen et al. (202T). the model is optimized to predict each action on from (Ri, 81, di.... ....... Rn-1, Sn-1, On-1, Rr 5,). After the model is trained with the offline trajectories, at test time, the action at timestep is selected by feeding into the trained transformer the test trajectory (R1, $1, aj..... Re, st). where RG is now an estimate of the optimal return-to-go.
In the original DT paper (Chen et al. 2021), there is pre-trate pre-training, i... training starts with random initial weights. Reid et al. (2022) consider first pre-training the transformer using the Wikipedia corpus, then fine-tuning with the DT algorithm to create policy for the downstream offline RL task.
3.2 GENERATING SYNTHETIC MARKOV CHAIN DATA
We explore pre-training DT with synthetic data generated from a Markov Chain. For the synthetic data, we simply generate sequence of states (tokens) using a finite-state Markov Chain with small number of states. The transition probabilities of the Markov chain are obtained randomly (as described below) and are not related to the environment or the offline dataset. After creating the synthetic sequence data, during pre- trainininng we feed the sequence into the transformer and employ next state (token) prediction, as is often done transformer- bassed LLMs (Brown et al. 2020). After pre-training with the synthetic data, we then fine-tune with the target offline dataset using DTT.
We generate the MC transition probabilities as follows. Let .,,,..., M denote the MC's finite state space, with M/ 100 being the default value. For each state in S, we draw M independently and uniformly distributed values, and then create a distribution over the state space S by applying softmax to the vector of M values. In this manner, we generate M probability distributions, one for each state, where each distribution is over S. Using these fixed transition probabilities, we generate the pre-training sequence Ho, Fi...... Fr follows: randomly sample from S to get the initial state zo in the sequence; after obtaining It, we generate T++1 using the MC transition probabilities.
During pre-training, we train with autoregressive next-state prediction (Brown et al. 2020):
[(D,, T,,..., Tr; 0) = - log Po (Fo. ;1,...,,TT)
.T) = E log (1 (o, T1,..., 11-))
As the states are discrete and analogous to the tokens in language modeling tasks, the embeddings for the states are learned during pre- pre-training as is typically done in the NLP literature.
3.3 RESULTS FOR PRE-TRAINING DT WITH SYNTHETIC DATA
We first compare the performance of the DT baseline (DT), DT with Wikipedia pre-training (DT+Wiki), and DT with pre-training on synthetic data generated from a I-step MC with 100 states (DT+Synthetic).. We consider the same three MuJoCo environments and D4RL datasets (Fu ct al. 2020) considered in Reid et al. (002)) plus the high-dimensional Ant environment, giving total of 12 datasets. For a fair comparison, we the authors' code from Reid et al. (2022) when running the downstream experiments for DT, and we keep the hyperparameters identical to those used in (Chen et al., 2021 Reid al. 2022) whenever possible (Details in Appendix A.T. For each dataset, we fine-tune for 100,000 updates. For DTk-ikk,, we perform 80K updates during pre-training following the authors. For DT+Synthetic, however, we found that we can achieve good performance with much fewer pre-training updates, namely, 20K updates. After every 5K updates during fine-tuning, we run 10 evaluation trajectories and record the normalized test performance"
This evaluation metric follows D4RI. Fu et
2020 We provide review in Appendix A.3
3

---

Published as a conference paper at ICLR 2024
We report both final performance and, in Appendix B best performance. For best performance, we use the best test performance seen over the entire fine-tuning period. Best performance is also employed in (Chen et 2021 Reid et al. 2022). In practice, determine when the best performance occurs (and thus the best model parameters), interaction with the environment is required, which is inconsistent with the offline problem formulation. The final performance can be better metric since it does not assume we can interact with the environment. For the final performance, we average over the last four sets of evaluations (after 85K, 90K, 95K, and 100K updates for DT). When comparing the algorithms, the two measures (final and best) lead to very similar qualitative conclusions. For all DT variants, we report the mean and standard deviation of performance over 20 random seeds.
Table I: Final performance for DT, DT pre-trained with Wikipedia data, and DT pre-trained with synthetic data. Synthetic data is generated from a one-step MC with a state space size of 100.
Average Last Four
DT
DT+Wiki
DT+Syuthcti
hulfcheetah-meddie mediumm-experr
4..9
43.9 +
49.5 + 9.9
sappo-mmdiim--xpert
81.0 11.8
94.0 + 8.9
99.6 + 6,5
walker2d medium--mxpp expert
1.5.0:1:3.5
ant-medium-crte expert
8.7
113.9
+ 10.5
117.9 = 8.7
halfcheetal-meddee mediuum--amm
37.5 1.3
39.1
1.6
39.3 + 1.1
hopper medium replay
46.7
10.6
51.4
13.6
61.8 13.9
walker2d medium-replay
49.2 10.1
7.7
56.8 1: 5.1
ant-mediumm-rplay
8.99
5.3
88.4
halfchectab-mccdimm
4.54
42.6
42.5 + 0.2
reeper-medium
58.2
walker2d-medium
70.4
70.8
3.0
71.5 4.1
ant unt-medium
8..0
88.5
+ 4.2
87.8 4,2
Averago datascts
67.7
73.6
Table [ shows the final performance for DT, DT pre-trained with Wiki, and DT pre-trained with synthetic MC data. We see that, for every dataset, synthetic pre-training does well better than the DT baseline, and provides an overall average improvement of nearly 10%. Moreover, synthetic pre-training also outperforms Wiki pre-training by 5% when averaged over all datasets, and this is done with significantly fewer pre-training updates. Compared to DT+ Wiki, DT+SSythhetcc is much more computationally efficient, using only 3% of computation during pre-training and 67% during fine-tuning (Details in Appendix A). Figure shows the normalized score and training loss for DT, DT with Wiki pre-training, and DT with MC pre-training. The curves are aggregated over all I2 datasets, each with 20 different seeds (per-environment curves in Appendix B.2). To account for the pre- training updates, we also offset the curve for DT+Synthettc to the right by 20K updates. Note that in practice, the pre- pre-training only needs to be done once, but the offset here helps to show that even with this disadvantage, Syntheti still quickly outperforms the other two variants.
75
DT
Nomalizedd Score
DT Training Loss
Di+
70
DT Synthetic
65
10
60
DT
DT+i Synthhei
55
1x13%
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Number of Updates
Number of Updates
(a) Fine-tuning learning curve.
(b) Fine-tuning training loss curve.
Figure 1: Performance and loss curves, averaged over 12 datasets for DT. DT+Wiki. DT+Synthetic.
Our synthetic data uses a small state space (vocabulary) and carries no long-term contextual or semantic information. From Table [ and Figure [ we can conclude that the performance gains obtained by pre-training with the Wikipedia corpus are not due to special properties of language, such the large vocabulary the rich long-term contextual and semantic information in the dataset, as conjectured in Reid et al. (2022) and Takagi (2022). In the next subsection, we study how different properties of the synthetic data affect the downstream RL performance.
4

---

Published as a conference paper at ICLR 2024
3.4 ABLATIONS FOR PRE-TRAINING DT WITH SYNTHETIC DATA
In the above results, we employed one-step MC to generate the synthetic data. In natural language, token dependencies are not simply one-step dependencies. We now consider whether increasing the state dependencies beyond one step can improve downstream performance. Specifically, we consider using& multi-step Markov Chain for generating the synthetic data Zo, ................................ JT. In an n-step Markov chain, depends on 1t-11, ................................ It-n- For an n-step MC, we randomly construct the fixed n-step transition probabilities, from which we generate the synthetic data. Table[[show the final performance averaged over the final four evaluation periods for the DT baseline and for MC pre-training with different numbers of MC steps. We see that synthetic data with different step values all provide better performance than the DT baseline; however, increasing the amount of past dependence in the MC synthetic data does not improve performance over a one-step MC.
Table 2: Pre-training with different numbers of MC steps. For example, 2-MC means the MC data is generated from 2-step Markov Chain. Other hyper-parameters remain the default values.
Average Last
I.CC
2.MC
5.MC
44,9 = 3.4
4.9
44.3
43.8
Copper-modium-oxpee
81.0 11.8
99.6
+ 6.5
99.1 6,5
98.2 5.7
walker2d 2d--mddimm---peee
105.0
0.8
3.1
105.9
3.1
ant medium expert
107.0 8.7
117.9
1: 8.7
122.2
1: 5.3
108.9
1 11.7
feheettt medium-replle replay
37.5 1.3
19.3 1.1
39.5
39.4
popper-medium-roee
46.7 + 10,6
61.8
13.9
59.8
11.0
60.1
11.4
walker2d-medine medduuuinn--
49.2
10.1
5..
3.9
5 5.8
ant medium replay
80.9 :: 3.9
2.7
1.
86,1
4.4
halfchectal-medium
42,4 = 0.5
42,5
0.2
42.6
42,5
0.3
hopper hopper-mediute mediam
3.2
60.2
2.1
3.3
59.6
walker2d medium
70.4 1: 29
71.5
4.1
70.7 4.2
70.1
1 4.0
ant- ont-medium
= 4.7
87.8
87.0
4.1
Average datasets
67.7
1
5.4
73.6
4.9
73.0
4.2
718
We now investigate whether increasing the size of the MC state space (analogous to increasing the vocabulary size in NLP) improves performance. Table the final performance for DT baseline and DT pre-trained with MC data with different state space sizes. The results show that state space sizes improve the performance over the baseline, with 100 and 1000 giving the best results.
Table 3: Pre-training with synthetic MC data with different state space sizes. For example, S=10 means the MC data is generated from 10-state MC. Other hyper-parameters remain default.
Average Last Four halfcheetah mesiii--eiiim--eee
DT
$10
$100
$1000
$10000
$100000
44.9 = 3.4
43,
= 2.6
49.5
9.9
45.4
1:4::00 4.5
44.0
43.6 2.7
hopper medium expert
81.0 :1 11.8
98.8
1 8.4
99.6
1.611... 6.5
102.2
1: 5.7
99.8
6.2
99.4
1.6.6. 6.7
105.0 3 3.5
105.4
4
107.4
+ 0.8
107.1
+ 1.9
105.9
3.1
103.
+ 5.0
ant aut--nedium--e expert
107.0
114.6
117.9
8.7
118.7
6.7
10.5
123.2
1 6.3
halfcheetah medium
seniiuuu--epayy replay
37.5 1.3 40.0
: 0.9
39.3
1.1
40.0
1: 0.8
39.6
19.6110
1.2
39,9
1 0.9
46.7 10.6
58.6
13.2
61.8
68.0
10.8
62.0
5
12.6
malker2d-medium--e
49.2 10.1
52.6
10.1
56.8
5.1
6.2
60.1
58.8
8,5
ant- aut-medium-repla
80.9 3.9
87.1
= 444
88,4
2.7
8788
84.5
488
86.8
3.6
halfcheetah medium
42.4 1.5. 0.5
42.5
1 0.4
42.5
0.2
42,4
0.3
42.5
15 0.3
42.4
0.4
hopper-medium walker2d medium
5.2
59.6
3.0
60.2
+
60.4
2.7
58.7
3.8
$7.3
33
70.4
3.8
71.5
72.8
72.4
72.4
ant- mediam
89.0 +: 4.7
88.9
- 3.7
87.
4.2
87.1
+ 2.8
88.8
1 4.2
88.3
+ 3,2
Average over dltuseets
67.7
1:
5.4
71.9
3
5.3
73.6
1
4.9
74.0
1
4.0
72.91.44 4.6
72.4
We now consider how changing the temperature parameter in the softmax formula affects the results. (Default temperature is 1.0.) lower temperature leads to more deterministic state transitions, while a higher temperature leads to more uniform state transitions. Table shows the final performance for DT with MC pre-training with different temperature values. The results show that all temperatures provide a performance gain, with temperature of 1 being the best. In this table, generating synthetic data with Independent and Identically Distributed (IID) states with uniform distributions over state space of size 100. Surprisingly, even this scheme performs significantly better than both the baseline and the Wiki pre-training. This provides further evidence that the complex token dependencies in the Wiki corpus are not likely the cause of the performance boost.
also consider
Table 5 shows the final performance for DT with MC pre- pre-training with different numbers of pre- training updates. Our results show that with even just 1k updates, MC pre-training matches the
5

---

Published as a conference paper at ICLR 2024
Table 4: Pre-training with different temperature values. Other parameters remain the default values.
Average Last Four
T=0.01
T=0,1
T=10
r-100
ITD uniform
44.9 3.4
46.6
54
52.6
11.9
49.5
99
43.3
4.2
44,5 = 4.0
nogper-mediou--xperr
81.0 11.8
95.4
8.1
95.2
9.2
99.6
6.5
99.9
+ 6.3
98.7
walker2d medium expert
105.0
106.4 1:
106.6
1 2,9
107.4
0.8
106.3
1 366
103.2 4.2
ant medium expert
8.7
114.9
121.7
5.5
117.9
= 8.
118.6
10.1
108.2
9.6
105.8 2
naffehectah-mediume medium-replay
37.5
39.5
0.
39.3
39.7
0.8
40.
09
hogper-medium-eplay
46.7
10.6
52.5
12.0
$2.8
= 14.4
6188
+ 13.9
60.2
60.
61.6 10.8
walker2d medium replay
49.2 10.1
6..
57.0
57. 6.
ant ant-medium--eplay
80.9 3.9
86.7
3.5
3.7
88.4
+
85.8
86.1 8,2
hallcheetah- mediumm
42.4
+
0.2
42.5
+ 0.2
42.5 - 0.3
4.3
126002
hopper medium
38.2.:32
59.1
1.34
19.4..355
60.2
2.1
57.913
59.4
19.1332
walker2d-mes 2d-mediumm
70.4 2.9
71.7
28
3.1
71.5
+
70.7 + 3.6
71.7 4.1
69.1 54
ant-medium
89.0
3.
3,0
87.8
4.2
88.4
46
89
Average over datasets
67.7 4 5.4
71.7 4.7
73.6
1: 4.9
7.74
7.9
71.3
performance of Wiki pre-training. Using as few as 20k updates (onc-fourth of DT+Wiki), our method already obtains significantly better performance.
Table 5: Pre-training with synthetic MC data with different number pre-training updates.
Average
updates
10k updates
updates
updates
60k updates
updates
hullcheetal- expert
44.9 ::
45.5
4.1
45.9
49.5
50.4
30.4
11.5
50,4
+ 11.0 49,4 i 8.8
hopper-mode popprr-mddimmccppett
81.0 = 11.8
93.1
9.8
94.8 74
99.6 6,5
102.5 + 8.1
101.1
+ 788 100.7 + 6,3
walker2d medium-espee exppett
105.0
105.5
2.9
106.3
107.4
106.8
107.3
ant medium expert
107.0 8.7
113.1
1 11.8
112.4 8.5
117.9
1 8.7
1224
6.3
121.8
1 5.7
120.4
6.7
halfcheetah- medium replay
37.5
39.6
0.8
398
19.3
39.6
19,2
hopper-medium--e
46.7 - 10.6
53.9
11.1
56.7
12.1
61.8
13.9
6188 15.1
61.8
12.3
62.3 9.7
49.2 10.1
52.2 79
93
5.1
59.6
$6,9
6.6
medium replay
8.
83.1
4:88
4.72
2.7
88.6
3.7
89.1
1.1:3. 3,4
87.5
$7.5.3. 3.5
halfchectal-mcdium
0.5
0.4
4.
42,5 0.2
42.5
4.4
42,5
1 0.2
hopper medium
58.2
59.2
59.3
614
60.3
103:::1 233
walker2d medium
2:
70.8
1.9
6..61..11
71.5
71.0
L 39
71.
4.0
72.1
1: 3.5
ant-medium
89.0
87.8
4.4
9.
878
880
38
86
87.0
Average datasets
67.7 1.51155
70.5
5.5
5.1
7.96
74.4
1
52
742
1
5.1
73.8
These ablation results show that synthetic pre-training is robust over different settings of the synthetic data, including the degree of past dependence, MC state-space size, the degree of randomness in the transitions, and the number of pre-training updates.
PRE-TRAINING CQL WITH SYNTHETIC DATA
Given that pre-training with synthetic data can significantly increase the performance of DT, we now study whether synthetic data can also help other MLP-based offline DRL algorithms. Specifically, we consider CQL, which a popular offline DRL algorithm for the datasets considered in this paper. For the pre-training objective, we use forward dynamics prediction, as it has been shown to be useful in model-based methods (Janner et al. 2019) and auxiliary loss literature (He et al., 2022a). Since forward dynamics prediction will require both a state and an action as input, we generate new type of synthetic data, which we call the synthetic Markov Decision Process (MDP) data. Different from synthetic MC, when generating synthetic MDP data, also take actions into considcration.
4.1 GENERATING SYNTHETIC MDP DATA
To generate the synthetic MDP data, we first define discrete state space S. a discrete action space A, random policy distribution æ, and random transition distribution p. Similar to how we created an MC for the decision transformer, the policy and transition distributions are obtained by applying softmax function on vectors of random values, and the shape of the distributions is controlled by a temperature term For each trajectory in the generated data, we start by choosing state from the state space, and then for each following step in the trajectory, we sample an action from the policy distribution and then sample state from the transition distribution. Since CQL uses MLP networks and the state and action dimensions are different for each MuJoCo task, during pre- pre-training we map each discrete MDP state and MDP action to vector that has the same dimension as the MuloCo

---

Published as a conference paper at ICLR 2024
RL task. For each state and action vector, entries are randomly chosen from uniform distribution between -1 and 1, and then fixed.
We pre-train the MLP with the forward dynamics objective, i..., we predict the next state s' and minimize the MSE loss (8' =s')², where s' is the actual next state in the trajectory. After pre-training, then fine-tune the MLP with specific dataset using the CQL algorithm.
4.2 RESULTS FOR CQL WITH SYNTHETIC DATA PRE-TRAINING
In the experimental results presented here, for the CQL baseline, we train for 1 million updates. For CQL with synthetic MDP pre-training (CQL+MDP), we pre-train for only 100K updates and then train (i.e., fine-tune) for million updates. By default, we set the state and action space sizes to 100 and use a temperature
for both the policy and transition distributions. All results are over 20 random seeds. We do not tune any hyperparameters for CQL CQL with synthetic pre-training but directly adopt the default ones in the codebase recommended by CQL authors More details on CQL experiments can be found in Appendix A.2
Table Table compares the performance of CQL to CQL+MDP. The table includes a wide range of MDP state/action space sizes and shows that synthetic pre-training gives a significant and consistent perfor- mance boost. With 1,000 states/actions, synthetic pre-training provides a 10% average improvement over all datasets and up to 84% and 49% for two of the medium-expe expert datasets.
Table 6: Final performance for CQL and CQL+MDP pre-training with different state/action space sizes. The number after "S" indicates the size of the state/action space. Temperature is equal to 1.
Average Last Four
CQL
S=10
$=100
$=1.000
$=10,000
$=100,000
halicheetah expeum-espert
35.9 = 5.2
5 5.8
63.1
66.2
:: 7.3
65.6
9.1
63.7 : 6.8
hopper medium expert
59.3 21.4
90.4 1 15.5
90,2
1 13.2
88.1
1 10.6
89.8
1 13.0
84.9
1. 20.2
walker2d -e--rddd--dd-------e- medium-expert
107.8
109.8
0.3
109.8
0.3
110.1
0.4
110.1
+ 04
110.1
0.3
ant-mediunn-cxper ceppunt-xppett
118.8
124.0
5.4
1.1444.1
4.7
129.2
4.3
halfcheetah- haltchecullmcdduu---ppaay
46.6 0.3 0.3
46.5
0.3
46.8
0.4
46.5
0.3
46.6
02
46.5
0,3
hopper medium replay
94.2 ::
96.3
2.9
95.3
96.9
98.0
2.0
walker 2d medium-ree medium-replay
80.0 4,1
83.9
+ 3.0
83.9
+ 2.4
838
- 1.6
81.3
81.1
34
82.9
+ 1.9
and mediumm-replay
96.7 1 3.8
101.7
1: 4.0
102.0
1:
102.3
24
101.9
1 2.6
100.6
3
hhlfchhetah medium
48.3 0.2
48.6
48.7
1: 0.2
48.7
0.2
48.7
0.2
48.6
0.2
hopper-medium walker2d-medium ant ant-medium
68.2 4.0
64.6 2.6
66.9
4.1
66.2
2.
65.5
66.9
+ 3.3
82.1
2.3
83.4
83.7
0.6
83.2
1.3
98.7 - 4.0
102.4 3.6
103.2
+ 3,3
103.3
1 3.8
103.4
+: 2.9
101.2
+ 3.4
Average over datascts
4.1
83.7
3.8
3.7
85.6
3.0
35
84.6
Table 7 shows the final performance for CQL+MDP with different temperature values using the default state/action space size. The results show that either too small or too large of temperature can reduce the performance boost, while the default temperature (r = 1) gives good final performance averaged over all datasets. 7 shows the results with uniformly distributed IID synthetic data, equivalent to using an infinitely large temperature. Surprisingly, the IID data performs almost as well as the MDP synthetic data, indicating the robustness of synthetic pre-training regardless of state dependencies. We provide a partial theoretical explanation of this behavior in the next subsection.
Table shows the final performance for CQL+MDP with different numbers of pre-training up- dates. Even with only 1K updates, synthetic MDP pre-training outperforms the baseline. The best performance boost is obtained with more pre- pre-training updates of 100K and 500K.
Figure 2 shows the normalized score and training loss (Q loss plus CQL conservative loss) averaged over all datasets during fine-tuning. Similar to Figure [ our synthetic experiments (CQL+MDP and CQL+IID) have been offseted by 100k updates. Both pre-training schemes start to surpass the CQL baseline at around 400K updates and maintain a significant performance advantage onward. In addition, performing pre-training update is quite fast since the forward dynamics objective only involves calculating the MSE loss of predicting the next state and backpropagation of the -enttwor.. 100K pre-train can be done in 5 minutes single GPU
hhttps:/githubb. http:://gihuu..oo/yyngg--/ gon/yyung-geng/COII
Detailed led computation time discussion can be found in Appendix A.2
7

---

Published as a conference paper at ICLR 2024
Table Final performance for CQL and CQL+MDP pre-training with different temperature values.
Average Last Four
CQL
T=0.01
T-0.1
r-100
CQL
malrchoctal-meddmm--e
35.9
47.1
5.6
532=65
63.1
12
61.1
81
59
59.7
= 6.4
hopper-mediee Cupper-mediun-e-eeee expert
59.3 59.3 21.4 52.4
26.0
892 96
90.2
13.2
83.0
+ 18.8
74.9
23.2
83.4
17.5
walker2d medium expert
107.8 3.8 110.2
188
1.9.
109.8
0.3
109.9
1 0.3
109.9
0.8
109,9
ant medium expert
118.8 1:
117.0
62
125.1
126.0:544
125.7
4,4
129.6
125.0 4.4
46.6 0.3
4.3
03
46.8
04
46.7
0.
46.6
4..7
94.2 2.2
95.8
2.9
2
95.3 - 3.2
93.6
93.8
93.3 3.0
walker medium replay
82.8
82.7
ant unt-mediun--epla
16.7
101.0
3.3
1010
102.0
102.8
3.5
102.
- 3.9
1018
salfcheetah-mmediiee
48.3 01::::000
48.6
0.2
48.6
0.2
48.7
+ 02
48.6
0.2
48.7
+:
48.6 0.2
hopper mediumm
68.2.1.4.0
67.7
3.34
65.1
28
66.9
1 4.1
66.3
14.3..34 3.2
3.1
66,3 3:9
2aller24-mediam
12 1.8
83.0
+ 088
834
0.7
834
+ 1.1
83.2
08
83.1
1.1
833
ant-medium
98.7
100.9
103.1
102.7
102.2
3.9
103.7 29
Avernge over datasets
78.0 4.7
79.4
83.4 3.3
84.9
+ 3.7
84.0
4.2
83.4 4 4.7
83.7
3.9
Table 8: Final performance for CQL and CQL+MDP with different number of pre-training updates.
Average Last Four
CQL.
IK updates
10K
updates
40K
updates
100K
updates
500K updates
IM updates
halfcheetah mediimn-eeppee expert
35,9 5.2
41.
48.4
56.1
7.2
66.4
5.6
61.5
hopper medium expert
59.3 21.4
76.9 + 186
87.3 + 12.0
87.6 - 18.9
90.2 + 13.2
92.5 + 14.0
82.2 + 14.5
107.8 3.8
109.9
109.
110.0
109.7
110.2
medium expert
118.8 1: 5.2
121.0
4.7
119.0
126.7
4.9
126.0
54
127.8
544
126.9
4.5
salfchectah-medinge medium--elll
46.6
46.3
46.7 0.4
46.7 0.
468 = 0.4
46.6
46.6 0.3
hopper-meediun-cpp
2.2
2.7
4.6
33
9.2
96.9
walker malker2d-ment 24-mmmiimm-mppppa
80.0 8..0
80.6 1: 3.4
80.6
83.7 2.5
84.0 2.1
83,9 + 2.4
83.7 + 2.4
83.2 1.7
medium replay
3.
99.9
3.5
halfcheetah-medle medium
48.5 0.2 0.2
48.6 0.2
48.6
48.7 + 0.2
48.6 0.2
48.5 0.2
hopper-medium
68.2 4.0
66.0
66.
66.9 3.6
66.9
4.1
66.7
65
walker2d 2eeer2d-mdiimm
82.1
83.
83.0 + 1.4
8.7
83.4 1.1
83.
83.4 1.2
ant-medium
08.7 4.0
100.0
42
1018 32
103.3 4.8 4.8
103.2
1822:35 33
102.5 39
100,9 4.4
Average over datasets
78.0
3.7
90
100-
CQL
Normalize medmaiized Score
CQL Training Loss
CQL+MDP
70
80
CQL+IID
60
50
60
30
CQL
40
CQL+MDP
20
CQL+IID
20
10
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0 iet
Number Updates
let
Number of Updates
(a) Fine-tuning performance curve.
(b) Fine-tuning loss curve.
Figure 2: Performance and loss curves, averaged over 12 datasets for CQL, CQL+MDP and CQL+IID.
To summarize, these results show that for a wide range of MDP data settings, pre-training with synthetic data provides a consistent performance improvement over the CQL baseline. Due limited space, number of additional experiments and analyses are presented in Appendix E
4.3 ANALYSIS OF OPTIMIZATION OBJECTIVE
To gain some insight into why IID synthetic data does almost well MDP data, now take closer look at the pre-training loss function. Let fo(s, a) be an MLP that takes as input a state-action pair (s, a) and outputs a vector state s'. Let a (so, do, 81, 01,.... ; 8T- @r- 87) denote the pre-training data, where the states and actions come from finite state and action spaces S and A. For the given pre-training data o, we optimize 0 to minimize the forward-dynamic objective:
T Σ
J(O)= =
[ ac) - 11+1
8

---

Published as a conference paper at ICLR 2024
Let (ss, a) be the set of states s' that directly follow (s, a) in the pre-training dataset If s' directly follows (s, a) multiple times, we list repeatedly in A , a) for each occurrence. We can then rewrite [0)
(( = Σ E
a) s'11
(s,))Sx A (s)
Now let's assume that the MLP is very expressive so that we can choose & so that fo a) can take on any desired vector x. For fixed a). let
x" (s, a) arg min
Σ
| s'11
s'EA(s,a)
Note that x² (s,a) is simply the centroid for the data in A (s, Thus
((0)
Σ E
(s. s'12
(s.0)ESxA (EA(S.))
In other words, the each (s, a) E S x A. For each (s, a) we want the MLP to predict the centroid in A(s, a), that is, we want fo(s, a) (s, a). This observation is true no matter how the pre-training data is generated, for example, by an MDP or if each (s, a) pair is IID.
objective is equivalent to finding the centroid of A a) for
Now let's compare the MDP and IID pre-training data approaches. For the two approaches, the centroid values will be different. In particular, for the IID case, the centroids will be near each other and collapse single point for infinite-length sequence o. For the MDP case, the centroids will be farther apart and will be distinct for each (8, a) pair in the limiting case. From the results in Table 7 the performance after fine-tuning& largely insensitive the distance among the various centroids.
5 CONCLUSION
In this paper, we considered offline DRL and studied the effects of several pre-training schemes with synthetic data. The contributions of this paper are follows:
1. We propose a simple yet effective synthetic pre-training scheme for DT. Data generated from a one-step Markov Chain with small state space provides better performance than pre-training with the Wiki corpus, whose vocabulary is much larger and contains much more complicated token dependencies. This novel finding challenges the previous view that language pre-training can provide unique benefits for DRL.
2. We show that synthetic pre-training of CQL with an MLP backbone can also lead signif- icant performance improvement. This is the first paper that shows pre-training on simple synthetic data is a surprisingly effective approach to improve offline DRL performance for both transformer-basee and Q-learning-based algorithms.
3. We provide ablations showing the surprising robustness of synthetic pre-training over past dependence, state/action-space size, and the peakedness of the transition and policy distributions, giving a consistent performance gain across different data generation settings.
4. Moreover, we show the proposed approach is efficient and easy to use. For DT, synthetic data pre-training achieves superior performance with 4x less pre-train updates. taking only 3% computation time at pre-training and 67% at fine-tur tun-tunnn compared with DT+ Wiki. For CQL, the generated data have consistent state and action dimensions with the downstream RL task, making it easy to use with MLPs, and the pre-training only takes minutes.
5. Finally, provide theoretical insights into why IID data still achieve a good perfor- mance. We show the forward dynamics objective equivalent to finding the state centroids underlying the synthetic dataset, and CQL is largely insensitive their distribution.
The novel findings in this paper bring up a number exciting future research directions. One is to further understand why pre-training on data that is entirely unrelated to the RL task can improve performance. Here, it is unlikely the improvement comes from a positive transfer of features, so we suspect that such pre- training might have helped make the optimization process smoother during fine-tuning. Other interesting directions include exploring different synthetic data generation schemes and investigating the extent to which synthetic data can be helpful.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work is partially supported by Shanghai Frontiers Science Center of Artificial Intelligence and Deep Learning at NYU Shanghai.
REFERENCES
Connor Anderson and Ryan Farrell. Improving fractal pre-training. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1300-1309, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34, 2021.
Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best-action imitation learning for batch deep reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pp. 18353-18363, 2020.
Cheng-Han Chiang and Hung-yi Lee. On the transferability of pre-trained language models: study from artificial datasets. In Proceedings of the AAAl Conference on Artificial Intelligence, volume 36, pp. 10518-10525, 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.044: 04805, 2018.
Jeff Donahue, Yangging Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: deep convolutional activation feature for generic visual recognition. In International conference on machine learning, pp. 647-655. PMLR, 2014.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062. PMLR, 2019.
Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for offline hindsight information matching. arXiv preprint arXiv:21 2021.
Nicklas Hansen, Zhecheng Yuan, Yanjie Ze, Tongzhou Mu, Aravind Rajeswaran, Hao Su, Huazhe Xu, and Xiaolong Wang. On pre-training for visuo-motor control: Revisiting a feaming-from-seeee scratch baseline. arXiv preprint arXiv:2212.05744 2022.
Tairan He, Yuge Zhang, Kan Ren, Minghuan Liu, Che Wang, Weinan Zhang, Yuqing Yang, and Dongsheng Li. Reinforcement learning with automated auxiliary loss search. Advances in Neural Information Processing Systems, 35:1820-1834, 2022a.
Zexue He, Graeme Blackwood, Rameswar Panda, Julian McAuley, and Rogerio Feris, Synthetic pre-training tasks for neural machine translation. arXiv preprint 17X1v:2212.09864, 2022b.
Zexue He, Graeme Blackwood, Rameswar Panda, Julian McAuley, and Rogerio Feris. Synthetic pre-training tasks for neural machine translation. In Anna Rogers, Jordan Goyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 8080 8098, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 18.18653/ 220223.inningg-aac..55 URL https: :/ aaalllll ann ogy. ogg find ngs acl a/aaaaalaaoooooooooo
512
Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. What makes imagenet good for transfer learning? arXiv preprint arXiv: 1608.08611111 2016.
10

---

Published as a conference paper at ICLR 2024
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, pp. 12519-12530, 2019.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as big sequence modeling problem. Advances in neural information processing systems, 34, 2021.
Siddharth Karamcheti, Suraj Nair, Annie Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for robotics. arXiv preprint vrXiv:2302.12766 2023.
Hirokatsu Kataoka, Kazushige Okayasu, Asato Matsumoto, Eisuke Yamagata, Ryosuke Yamada, Nakamasa Inoue, Akio Nakamura, and Yutaka Satoh. Pre-training without natural images. In Proceedings of the Asian Conference on Computer Vision, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.990, 2014.
Simon Korbblihh, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVE conference on computer vision and patlern recognition, pp. 2661-2671, 2019.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning, arXiv preprint arXiv:2110.06169, 2021.
Kundan Krishna, Jeffrey Bigham, and Zachary C Lipton. Does pretraining for summarization require knowledge transfer? arXiv preprint (tt:::2090009553 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.00 2077:20006.4777, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0/643 2020.
Wenzhe Li, Hao Luo, Zichuan Lin, Chongjie Zhang, Zongqing Lu, and Deheng Ye. survey on transformers in reinforcement learning. arXiv preprint 2:X10:2301.03044, 2023.
Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Con-
ference on Learning Representations, 2017. FRR.hhttss://aag. http:://.ppp.ss........
semant. ssematticscho cscho or.g
[o
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203. 12601, 2022.
Isabel Papadimitriou and Dan Jurafsky. Learning music helps you read: Using transfer to study linguistic structure in language models. arXiv preprint arXiv:2004. 14601, 2020.
Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsurprising effectiveness of pre-trained vision models for control. arXiv preprint arXiv.2203.03580, 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. Conference on Robot Learning, pp. 416-426. PMLR, 2023.
Machel Reid, Yutaro Yamada, and Shixiang Shane Gu. Can wikipedia help offline reinforcement learning? arXiv preprint AXXi:2201.12122 2022.
Ryokan Ri and Yoshimasa Tsuruoka. Pretraining with artificial language: Studying transferable knowledge in language models. arXiv preprint arXiv:2203. 103.6 2022.
11

---

Published as a conference paper at ICLR 2024
Rutav Shah and Vikash Kumar. Rrl:
Resnet representation for reinforcement learning. In Self-Supervision for Reinforcement Learning Workshop-ICL& 2021, 2021.
Shiro Takagi. On the effect of pre-training for transformer in different modality on offline reinforce- ment learning. arXiv preprint arXiv:2211.09817. 2022.
Che Wang, Xufang Luo, Keith Ross, and Dongsheng Li. Vr13: data-driven framework for visual deep reinforcement learning. Advances in Neural Information Processing Systems, 35: 32974 32988, 2022.
Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. Transformers: State-of-the-art natural language processing. In Conference on Empirical Methods Natural Language Processing: System Demonstrations, 2020.
Yuhuai Wu, Markus N Rabe, Wenda Li, Jimmy Ba, Roger Grosse, and Christian Szegedy. Lime: Learning inductive bias for primitives of mathematical reasoning. In International Conference on Machine Learning, pp. 1221-111222. PMLR, 2021.
Yuhuai Wu, Felix Li, and Percy Liang. Insights into pre-training via simpler synthetic tasks. Advances in Neural Information Processing Systems, 35:21844-2188 2022.
Zhihui Xie, Zichuan Lin, Deheng Ye, Qiang Fu, Yang Wei, and Shuai Li. Future-conditooned unsupervised pretraining for decision transformer. In International Conference on Machine Learning, pp. 38187-38203. PMLR, 2023.
Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential decision making. arXiv preprint 10Xi:2202..08815. 2021.
Albert Zhan, Ruihan Zhao, Lerrel Pinto. Pieter Abbeel, and Michael Laskin. A framework for efficient robotic manipulation. In Deep RL Workshop NeurIPS 2021, 2021.
12

---

Published as a conference paper at ICLR 2024
A HYPERPARAMETERS & TRAINING DETAILS
A.1 DECISION TRANSFORMER
Table 9: Experiment settings during synthetic pre-training.
Hyperparameter
Value
Number of layer
Number of attention heads
Embedding dimension
128
Sequence length
1024
Batch size
65536 tokens/64 sequences
Steps
80000
Dropout
0.1
Learning rate
3 x 10 10-2
Weight decay
Learning rate decay
Linear warmup for first 10000 training steps
Table 10: Hyperparameters of Decision Transformer for OpenAl Gym experiments.
Hyperparameter
Value
Number layers
Number of attention heads
Embedding dimension
128
Nonlinearity function
ReLU
Batch size
64
Context length
20 HalfCheetah, Hopper, Walker, Ant
Return-to-go conditioning
6000 HalfCheetah
3600 Hopper
5000 Walker, Ant
Dropout
0.2
Learning rate
10
Grad norm clip
0.25
Weight decay
backhone, 10 elsewhere
Learning rate decay
Linear warmup for first 5000 training steps
Implementation Experiment details Pre-trained models are trained with the HuggingFace Transformers library (Wolf et al. 2020). We used AdamW optimizer (Loshchilov & Hutter 2017) for both pre-training and finetuning. Unless mentioned, we followed the default hyperparameter settings from Huggingface and PyTorch. Our model code is gpt Synthetic pre-tre pre-training done synthetic datasets generated to be about the size of Wikitext- (Merity et al.] (0166). Our hyperparameter choices follow those from Reid al. (002) for both pre-training and finetuning, which shown in detail in table 9 and I0] In Reid et al. 2022). it is shown that the additional kmeans auxiliary loss and LM loss provide only marginal improvement (An average score of 0.3). Without using these losses, our synthetic wre-training results outperform DT+Wiki by an average score of 3.7, as shown in Table
1
DT Computation Time Discussion Table shows the number of updates needed for each variant to reach 90% final performance of the DT baseline for individual datasets. Our synthetic models are about 27% faster compared Wikipedia pre-training in reaching the goal returns averaging all datasets.
In terms of pre-training computation time, run both Wikipedia and synthetic pre- tr-iiriiing rtx8000 GPUs. Synthetic pre- pre-traing traininin takes about 2 hours and 11 minutes to train for 80k updates while Wikipedia pre-training takes about 16 hours and 45 minutes to train for the same number of
13

---

Published as a conference paper at ICLR 2024
Table 11: Number of updates for DT, DT with Wikipedia pre-training, and DT with synthetic pre- training before reaching desired target return. For each dataset, the target return is 90% of the final performance by DT baseline.
Number of Updates
DT
DT+Wiki
DT+Synthetic
selicheeta--meduum--nn
15.3k
11.3k
9.3k
hopper-medium -eppett
23.8k
18.8k
13.3k
18.8k
20k
14k
unt-medium-expee exppert
15.5k
7.8k
7.8k
salfcheceta-mediine
8,5k
6.8k
5.3k
hopper-medium
9.3k
10.5k
8.5k
walker2d-medium
9k
9.5k
7.3k
ant-medium
8.5k
7k
6.3k
halfcheetah- sattheetaa--mdddu---eepppa
16.5k
11.8k
7.5k
hopper-mediiee mediiu--replay
23.3k
28.3k
17.3k
walker2d walker2d-medium-ree
13.3k
17.8k
12.3k
iill-mediuum-rpee -epplay
7.8k
9.5k
6.8k
Average over datasets
14.1k
13.2k
9.6k
Table 12: Computation time for DT, DT with Wikipedia pre-training, and DT with synthetic pre- training DT. We compare computation time over the medium-expert datasets only. All experiments a single rtx8000 GPU with the default settings for 100k updates.
Computation Time
Di+iiki
Dynthtttttic
malfcheetah-medium--ene
hrs 27 mins
3 hrs 50 mins
hrs 32 mins
hrs 55 mins
hrs 25 mins
2hrs 11 mins
walker2d-medium-eee expeert
hrs mins
hrs 45 mins
hrs 18 mins
ant-medium-cxpee expeert
2 hrs mins
3 hrs 52 mins
2 hrs 46 mins
Average over datasets
hrs mins
hrs 43 mins
hrs mins
updates. The 87% reduction in training time is achieved largely due to the reduced number of token embeddings. Furthermore, Table haas shown that synthetic pre-training reaches ideal performance with as few as 20k pre-training updates (in about 33 minutes), which means that synthetic pre-training obtains superior results with only about 3% of the computation resources needed for Wikipedia pre-training.
Table 12 shows the computation time comparison downstream RL tasks over the medium-expert datasets only. Without using auxiliary losses in Reid et al. (2022), our pre-trained model runs about same speed DT baseline which much faster than DT+ Wiki (we only use 67% of the time during fine-ture fine-tuning).
14

---

Published as a conference paper at ICLR 2024
A.2 CQL EXPERIMENTS DETAILS
We develop code based the implementation recommended by CQL authors Most of the hyperparameters used the training process or the dataset follow the default setting, and we list them in detail in Table 15 and Table 16. Also, provide additional implementation and experiment
details below.
CQL Computation Time Table 13 first shows the number of updates required for each default algorithm variant to reach 90% of the final performance of the CQL baseline for individual datasets. Compared with the CQL baseline, CQL-MDP takes about 34% and CQL-IID takes about 32% fewer fine-tuning updates in reaching the same target test returns when averaged over all datasets. In terms of the real wall-clock computation time. Table 14]shwws the time consumed on one single rtx8000 GPU to pre-train 100K updates with synthetic MDP data, and to fine-tune 1M updates with CQL algorithm for each downstream medium-expper dataset. Surprisingly, a few minutes of synthetic pre-training enough to efficiently improve downstream performance.
Table 13: Number of updates for CQL baseline and CQL with synthetie pre-training required reach 90% the final performance by CQL baseline on each dataset.
Number of Updates
CQL
CQL+MDP
CQL+IID
halfcheetah-mediim--e
575.0k
251.0k
250.2k
hopper-medium--xppe eeppet
78.5k
79.8k
80.2k
walker24-meeduu--eppp
222.5k
130.2k
126.2k
ant-medium-expprt
254.2k
232.5k
245.0k
siricheetah-mmdium--ree
60.5k
42.2k
44.5k
hoppee--eeium-reee
161.5k
88.5k
109.5k
walker2d-meedium-reee
114.5k
95.2k
106.0k
-etu
73.8k
85.2k
69.2k
halfcheetah--mddumm
111.0k
65.0k
58.8k
hopper-medium
96.5k
71.5k
97.5k
161.2k
104.5k
105.0k
ant- ant-medium
34.0k
38.8k
39.0k
Average over datasets
161.9k
107.0k
110.9k
Table 14: Computation time for 100K-update CQL synthetic pre-training, and IM-update CQL fine-tuning. We compare computation time over the medium-expert datasets only. All experiments a single rtx8000 GPU with the default settings.
CQL Computation Time halfcheetah malfcheetah-mediim-eee
Synthetic Pre-training
Fine-tuning
4.1 mins
4 hrs 52 mins
4.0 mins
hrs 25 mins
walker2d medium-exeieeeeeeeeeee ant- iil--medium-xxpe
4.0 mins
4 hrs 33 mins
4.3 mins
5 hrs 5 mins
Average over datasets
4.1 mins
hrs 44 mins
Generate Synthetic MDP Data When generating the synthetic MDP data, we make use of numpy. www.aando...m..... seed () to construct policy/transition distributions instead of storing those prob- abilities in huge table. For example, every time retrieve a transition distribution specified by (conditioned on) an integer pair (8, a), we first set numpy. random, seed (8 x 888 + x 777), then generate uniformly distributed (between 0 and I) vector with the same length as the state space size, and finally input this vector to the softmax function with temperature 7 to get a probability distribution. The next state transitioned from (s, a) can be sampled from this particular distribution.
15

---

Published as a conference paper at ICLR 2024
Synthetic Pre-training Following the common framework of pre-training and then fine-tuning LLMs, we always pre-train our MLP by using only one seed of 42, while fine-tuning the MLP on multiple seeds due to the algorithmic sensitivity to hyperparameter settings of CQL (Kostrikov et al. 2021
CQL Fine-tuning We adopt the Safe Targer technique (Wang et al. 2022 to alleviate the potential Q loss divergence due to RL training instability and distribution shift which has been proven to exist through our early experiments. When computing the target Q value //urget each update of the SAC algorithm, we simply set Utarget <- Qmix if Yurrett > Qmax. where Qmax is the safe Q value predefined for each dataset. Due to the robustness of this method (Wang et al. (022). we choose Qmax 100 X maxx given the discount factor of 0.99, where Tmax the maximum reward in the dataset. Note that we do not include a safe Q factor as proposed the original work. For more details, please refer to Wang et al. (2022).
16

---

Published as a conference paper at ICLR 2024
Table 15: Hyperparameters of synthetic pre-training for CQL experiments
Hyperparameter
Value
nets hidden layers
Architecture
hidden dim
256
Q nets activation function
ReLU
Optimizer
Adam KKngmma & 2014)
Criterion
MSE
Training Hyperparameterss
nets learning rate
3e-4
Total updates
100K (default)
Batch size
256
Seed
42
Number of trajectories
1000
Max length of each trajectory
1000
Action space size
100 (default)
Synthetic Data
State space size
100 (default)
Policy distribution temperature
(default)
Transition distribution temperature
(default)
Sampling distribution of action/state entries
Uniform(-1 -1, 1)
Table 16: Hyperparameters of fine-tuning CQL for D4RL, Locomotion Datasets.
Hyperparameter
Value
Q nets hidden layers
Q nets hidden dim
256
Architecture
Q nets activation function
ReLU
Policy net hidden layers
Policy net hidden dim
256
Policy net activation function
ReLU
Optimizer
Adam
(Kingma Ba 2014
Q nets learning rate
3e-4
Policy net learning rate
3e-4
Target Q update rate
5e-3
SAC Hyperparameters
Batch size
256
Max target backup
False
Target entropy
-1 Action Dim
Entropy in Q target
False
Policy update multiplier
Discount factor
0.99
Lagrange Q difference clip
False
False
CQL Hyperparameters
Importance sampling
True
Number of sampled actions
3
Temperature
Min Q weight
5.0
Epochs
200
Updates per epoch
5000
Others
Number of evaluation trajectories
10
Max length of evaluation trajectories
1000
Seeds
0~114,
42, 666. 1042, 2048, 4069
17

---

Published as a conference paper at ICLR 2024
A.3 EVALUATION METRIC
For each experiment setting, we record the Normalized Test Score which is computed as A95 TPST RETÚRN-KTH SCORS x 100, where (ES__TET_RETTTRE the test return averaged NAX SCORE XIN SCORE 10 undiscounted evaluation trajectories; STN__SCORE and WAX_SCORE, environment-specife constants predefined by the D4RL. dataset (Fu al. 2020). We summarize those constants of each environment Table 17
Table 17: Values of MIN__SCORE SCORE and MAX_MMORE SCORE used performance evaluation.
Environment
(MIN SCORE, MAX SCORE))
halfchectah-v2
178953, 12135.0)
walker2d-v2
(1.629008, 4592.3)
hopper-v2
(-20.272305, 3234.3)
ant-v2
(-325.6, 3879.7)
18

---

Published as a conference paper at ICLR 2024
B ADDITIONAL DT RESULTS
B.1 BEST SCORE RESULTS
Table 18: Best test score for DT, DT with Wikipedia pre-training, and DT with MC pre-training (DT+Synthetic). The synthetic data is generated from a one-step MC with state space size of 100, and temperature value of 1 (default values).
Best Score
DT
DT+Wiki
DT+Synthetic
halfcheetah- hethhreella-meiimm--- expert
67.2 112-911 9.1
64.2 11.5
72.7 + 16.8
hopper-mediuee wwe---mmdddddmeeeeeeee expeert
106.1 4.2
111.4 1.5
111.5 = 0.8
walker2d malker2d-medium- expprt
108.6
109.2
109.1 + 0.3
unt-medium-expen expert
127.6 4.3
1.8
131.5
malffhhettal-mediu--rree leplay
39,9 0.4
4.5
4..
hopper-mediun--reph
77.8 7.0
8.66
84.8
=
72.9
75.2 4.0
2.5
92.9
95.9
halfcheetah-meddiee mediiu
43.3 0.2
4.2
0.2
hopper-medium
68.3 4.0
7.14
70.9 4.4
walker2d-mede
78.9 = 1.5
1.23
79.0 + 1.2
ant-medium
100.4 + 1.8
100.6 1.2
100.5 + 0.8
Average over datasets
82.0 = 3.3
82.8 = 3,4
84.7
Table shows the best performance for DT, DT pre-trained with Wiki, and DT pre-trained with synthetic MC data. Similar to Table synthetic pre-training does as well or better than the DT baseline for every dataset. Synthetic pre-training also outperforms Wiki pre-training with significantly fewer pre-traie pre-training updates.
Table 19: Best score for pre-training with different number of MC steps.
Best Score
DT
1-MC 2-MC
5-MC
malfcheetah-mediinee expert
67.2
72.7 16.8
59.2 12.8
59.8 = 12.9
hopper-mediue medium-expert
106.1
111.5
111.5
1.5.5
walker2d-medee expeert
108.6
0.3
109.1 0.5
ant-medium-exper
127.6 4.3
1.1.5
133.7
6.0
hopper-mediun--ephe
77.8 7.0
8..2
84.7
walker2d-medine medium-replay
73.9 3.7
7.02
7..6
7..7
ant-medium-replay
92.4 2.5
1.0
9.17
9.
sitccheetah-mediumm
4.2
4.2
4.24
4.2
hopper-medium
68.3 4.0
70.9 1.944.
6..4
69.1
walker2d-medine
78.9 1.5
7.2
7.54
79.3
ant-medium
100.4:11.
0.8
1.5.7
1.7
39.9 0.4
4.4
4.4
41.3 + 0.3
Average datasets
82.0 3.3
84.7
8..6
83.1 + 3.3
Table 19 shows best score comparison for DT and DT + MC pre-training with different MC steps. 1 1-step MC gives the best performance, while all settings provide performance gain over the baseline.
Table 20 shows the best score comparison of DT bascline and DT + MC pre- pre-training with different state space sizes. All MC settings provide performance boost, while state space sizes of 100 and 1000 give the best performance.
Table 21 shows the best score comparison of DT baseline and DT + MC pre-training with different temperature values. All temperature settings provide some performance boost over the baseline.
Table 22 shows the best comparison for DT and DT MC pre-training for different gradient steps. this case, our synthetic pre-training experiments show a similar best score performance boost over the baseline.
19

---

Published as a conference paper at ICLR 2024
Table 20: Best score for pre-training with different state space sizes.
Best Score
DT
S10
S100
$1000
$10000
$100000
halfcheetah- watichretal-meduu----yee
medhum-expert
67.2
58.6 1 13.0
72.7
16.8
64.1
14.5
64.2
14.3
56.7 1
hopper-medium wondiiu---ppeer
106.1 1: 4.2
111.8 1 0.7
111.5
0.8
111.5 0.7
111.8
1 0.4
111.5 1
walker2d malker2d-medium-ee medium-experr
108.6 0.4
109.1 + 0.5
109.1
0.3
109.1 + 0.5
109.3 0.5
109.2 0.5
ant mee--medd--meee
127.6
131.6
131.5
133.9 2.0
130.9 5.9
133.7 2.5
halfcheetah- medium-reppie replay
0.4
41.6 0.3
0.4
41.4
1.44
41.4 1 0.3
hopper-medium-reee
77.8
81.6 5.5
84.8
87.1
86.3
80.9 + 6.8
walker2d valker2d-medium--eee -ee-imm-meedlllee replay
3.7
7.2
4.0
7.6
76.7
78.0 + 4.8
ant medimm--eemllleee
9.
96.4 1.3
95.9
96.5
95.3
95.1 +
bescheetah-medding
0.2
43.3
0.2
0.2
4.1
4..
43,3 1 0.2
hopper-medium
4.0
68.9
70.9
3.7
6.35
65.9 +
valker2d-mediuee
78.9 1.5
= 2.0
79.0 1.2
79.5
80.0
ant-medium
100.4 1.8
1.5
100.5 0.8
100.1
2.5
100.5
Average over datasets
82.0 3.3
3.0
84.7
84.6
84.0
83.0 3
Table 21: Best score for pre-training with different temperature values.
Best Score
T=001
T=0.1
[=1
r=10
1=100
uniform
halicheetah Mullichetaa--meddu--eeeg
exper
12.1
hopper mediumm expert
106.1 1:: 4.2
111.3
111.2
1
111.5
0.8
110.9
1.5
111.4
1.0
111.7 1: 1.0
walker22 mediimm-exxprrt
0.4
109.1
0.5
109.0
mnt-meeiiu--cpeet
227.6=4..3
131.8 3.0
134.2
2.2
131.5
337
126.2
125.3
hallcheetal- medilm--emllaay
39.9
41.4
41.6
1 0.3
41.5
41.3
0.3
41.5
0.4
0
hopper medium replay
77.8 7.0
80.2
82.1
6.5
84.8
82.8
1 6.9
85.9
1.3
85.4 5.6
walker2d mallke2d-medium-ree medium-replen
73.9
76.2 4.3
762
37
75.2
4.0
77.4 32
74.1
3.7
75.4
3.1
ant medium replay
halfcheetah-methe
43.
43.4
0.2
43.4 0.
43.3
0.2
43.4 - 0.2
4.2
43.4
0.2
hopoer-medium malker2d-mediue
4.0
70.0
68.0
70,9
49
5,
67.9
78.9
79.2
1.4
79.5
79.0
79.6
= 0.9
79.1
79.5
ant medium
100.4 1:
101.0
1.7
100.2
1
1.5
100.5
088
100.0
1:
101.1
2
100.6
2.
Average
3.30
84.1
848
Table 22: Best score for pre-training for different gradient updates, e.g. 10K means pre-trained for 10K gradient steps.
Best Score
DT
1k updates
10k updates
20k updates
40k updates
60k updates
80k updates
Malkhhoctall-mcdiu--xxxo
67.2
67.6
12.8
66.3
16.8
67.8
15.9
65.9
63.5
17.6
ampper-medium-ee expert
106.1
111.1
111.3
1.5
111.5
0.8
111.8
111.7
- 1.0
111.4
2alker2d-mcdimm-zpprr
108.6
0.4
109.3
1033000 0.7
109.1
= 0.6
109.1
+ 0.3
109.0
0.4
109.2
= 0.4
199.1 0.3
nut-meddum-eet expert
hulfsheetah medium replay
39.9 0.4
41.3 0.4 0.4
41.4 0.
41.5
0.4
41.2 1 0.3
41.4
4 0.3
41.3
0.4
hopper-medium. medium replay
846
5.11
walker2d-nat ded--miiim-rpppln
7.79
75.0
74
4.0
75.8
+ 3,8
75,3
337
76.2
ant- mad-umdiim--plaa
92.4
94.3
91.1
959
1.0
96.6
1.2
96.2 1.1
96.2
halfshectah-meet mediuum
4.2
43.3
433
02
43.3
43.3
43.3
0.2
43.4 0.2
hopper-modium walker2d medium
6..3
70.3 4.9
69.5
19.5360
36
70,9
44
69.4
33
71.0
1.00:44
69.5 338
79.5
789
ant medium
100.4 1: 1.8
100.5
1:
1.4
100.2
1
14
100.5
088
99.9
1
14
100.4
1
1.4
99.5
1 1.5
Average over datasets
82.0
83.6
=: 3.6
83.8
84.7
2 3.6
84.4
22 30
84
84.1
20

---

Published as a conference paper at ICLR 2024
B.2 DT TRAINING CURVES
Figure 3 and 4 shows the normalized score and training loss averaged over all datasets during fine- tuning for DT, DT with Wiki pre-training, and DT with synthetic pre-training. Each curve is averaged over 20 different seeds.
100
60
00
Nomalzzed Store
Normaized Score
58
30
56
40
DT
DT 4T Symthtttti
DT
DT-WiKi
30
DT+
50
Sy+Shnthetic
DT+ DT+Synthetic
48
20
0.2
0.4
0.6
0.8
1,0
0.2
0.4
0.6
0.8
0.2
0.4
Number Updates
Number of Updates
Number Updates
(a) Hopper- medium
(b) Hopperr-eediun--ellla
(c) Hopper--eddiumeepee
60
72.5
20.0
-50
Nomalizd Score
Rbmalized
67.5
Nomaltzed
85.0
62.3
DT
30
DT DT+
DT DT+
60.0
DT+Wiki Synthhtettic
57.5
DT Synthetic
DT Synthetic
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Number of Updates
Number of Updates
Number of Updates
(d) Walker-medium
(e) Walker-medium--eplay
(l) Walker-medium-eeppet
40
55.0
42.0
38
52.5
DT+
41.5
36
50.0
DynShttettic
41.0
Normatzee
34
47.5
32
45.0
40.0
39.5
Ulik
30
DI Wiki
42.5
Syntheti
28
Synthetic
40.0
0.2
0.4
0.2
0.4
0.6
0.8
1.0
0.4
Number Updates
Number Updates
Number Updates
(g) HaliCheetab-mmddimm
(h) HallChettah-mcdiuu-rrplaa
HallChhettah-mddduu-xpprrrr
92
90
120
90
Hermaltzed Scare
15
88
0 110
Rorma g
105
86
84
75
DT
100
DT
Di-i
D
DT+Wiki
82
Synthhetiic
70
DI Syythetic
D1+Synthetic
Updates
updates
Updates
(j) Ant-medium
(k) Ant-medium-replay
(I) Ant-medium-expprt
Figure 3: Learning curves for DT. DT with Wikipedia pre-training, and DT with synthetic pre-training. Our pre-training scheme (DT+Synthetic) has been offset for 20000 updates to represent the 20000 pre-training updates with synthetic data.
21

---

Published as a conference paper at ICLR 2024
OT+Wiki Synnheeticc
DT Wilki
DT Training Loos
DT+Wiki
+yythhetic
Synthetic
Training
Taining
10-3
0.4 Number Updates
0.2
0.6
0.8
Number Updates
Number Updates
(a) Hopper-medium
(b) Hopper-meciiu-rreplyy
Hopper-mecdum-exper
DT
DT
Loss
DT+
Traiiin Loss
DT training Loss
DT+
DT DT+Synthetic
Oyythtthtiic
DT+ Synthetic
10
10
0.5 Number Updates
Number Updates
Number Updates
(d) Walker-medium
(c) Walker- replay
(f) Walker-mediuu-exppe expprr
DT
DT
DT SynShhthetic
DT Trining mainingtoo Loss
DT Training Loss
DT Synthhtiic
DI Synthetic
10
10-
10-1
exai
awerr
0.2
0.4
0.5
1.0
0.2
0.4
0.8
1.0
0.2
0.4
0.8
0.8
Number Updates
Number Updates
Number of Updates
(g) HalfCheetah-medium
(h) HallCheetah-mcdduu-rrpaay
(i) HallChhetth-mmddium-xprrr
DT
DT
DT
training Lozs
DT+Wiki
DT+Wik DT Dy+Shythetic
Dr Draiiinn Trining Loss
DT+Wiki
10
DT SynShnthetic
10-1
Dy+Sytthetic
DT Training
0.2
0.4
0.5
0.8
1.0
0.2
0.4
0.6
0.B
1.0
0.2
0.4
0.5
0.8
1.0
Number Updatos
Number of Updotes
Number of Uodates
(i) Ant-medium
(k) Ant-medium-rephae
(l) Ant-medium-exper
Figure 4: Training loss curves for DT, DT with Wikipedia pre- pre-training, and DT with synthetic pre-training. Our pre-training scheme (DT+Synthetic) has been offset for 20000 updates represent the 20000 pre-training updates with synthetic data.
22

---

Published as a conference paper at ICLR 2024
C ADDITIONAL CQL RESULTS
C.1 BEST SCORE RESULTS
In this section, we present the best score obtained along the fine-tuning process for individual datasets and different algorithm variants. Table 23 shows the best performance of CQL baseline and CQL+Synthetic with range of different state space sizes, while setting the temperature to I and pre-training for 100K updates. 2able(24showe the best performance of CQL baseline, CQL+Synthetic with range of temperatures, and CQL+IID, while keeping state and action space sizes to 100 and pre- training for 100K updates. Table 8 shows the best performance of CQL baseline and CQL+ pre-trained with a different number of updates while setting state and action space size to 100 and the temperature to 1. Most of the synthetic pre-trained variants can outperform the CQL baseline in terms of the best score averaged over all datasets, while in general, the best score is not as sensitive to the hyperparameter settings of synthetic pre-training the final score. In addition, is worth noting that synthetic pre-training usually leads to a smaller standard deviation of the acquired best scores compared with the CQL baseline, which indicates more consistent appearance of high performance during fine-tuning
Table 23: Best test score for CQL and CQL+MDP pre-training with different state/action space sizes. The number after "S" indicates the size of the state/action space. Temperature equal to
Best Soore
CQL
S=10
S=100
$=1,000
$=10,000
$=100,000
hullcheeaal-mmdimmm-ee
48.2 2 6.2
75.3
8..3
88.2
= 2.9
87.9
85.9 + 3.9
hopper medium medium--sxpert
110.8 +: 1.0 111.6
1:
0.4
111.3
1.1 111.8 0.3
111.6
- 0.3
111.8
0.3
walker2d medium expert
112.9 1: 1.4
1128
0.9 112.2
1.221008
112.3
112.8
1: 1,3
112.5
1:
ant ont-medium-exppe
127.8 2.0 131.5
132.1
- 1.1
133.6
132.9
0.7
133.1
0.7
maffchccal-mmciun--eppe
mediun-replay
44
45.7
4.3
45.5
03
45.7
45.6
hopper- medium-repln
99.5 1.3
100.9
100.1
1.5 100.7
0.9
101.3
0.7
101.2
87.6= 1.4
90.2
89.2
1.1
89.0
1.2
89.1
89.0
aat-medium-ree replll
99.9
102.4
1.0
103,0
1.0
102.6
1.0
102.0
- 0.9
102.3
0.9
hulfcheetah medium
46.9 11 0.2
4.24
47.4
1: 0.2
47.5
0.2
47.5
0.22
47.4
0.1
hopper-medium medium
87.3
2.
84.3
3.2
834
2.4
24
85.0
1.9
walker2d-medium
85.7 0.5
8..1
86.2
0.6
86.2
0.4
86.2
86.2
0.6
86.2
0.4
ant-medium
104.2
104,8
104.7
0.5
Average over datasets
88.0 1 1.7
91.1
1.3
91.8
1
92.1
= 1.0
92.
0.9
92.1 E 1.0
Table 24: Best test score for CQL and CQL+MDP pre-training with different temperature values. The value after "7" indicates the temperature value.
Best Score
CQL
T=0.01
7-0.1
7=1
10
7=100
CQL
halfcheetahh medium expert
48.2 1 6.2 66.4
7.3 73.9
6.4
85.3
1
3.1
82.9 5.7
82.7
82.7
5.0
84.0
1 3.8
hopper-medingg medium- expert
110.8 1.0
109.0
= 5.4
111.7
111.3
110.7
109.5
400
111.0
+
malkerdd-medinm--yy
112.9 + 1.4
11311
= 0.6
1126
0.8
112.9
0:9
113.4
= 1.8
medium expert
127.8
128.
1.6
1.1.0.10
1
1.1
132.2
1326
131.4
halfchectah--medhe weeeetah-mmmim---eee medium-replay
45.7 +: 0.4
0,3
02
48.9
45.7
0.4
45.7
45.9 - 0.3
99.5 + 1.3
100.7
= 0.7
100.5
100.1
100.2
+ 1.2
100.0
12
99.3 1.2
walker2d medium replay
87.6 1.4
88.8
87.6 0.8
89.2
1: 1.1
90.0
10.0::1:0
1.0
89.7
+:
89.8
1 0.9
ant medium replay
99.9 1.1
101.9
= 1.1 192.1
1.9
103.0
11.00:10 1.0
102.
1: 0.8
102.4
0.9
1026
aalfcheetah-medium
4.2
4.2
47.3 022
47
0.2
0.2
0.2
0.2
hopper-medium
87.3
86.2
96.2222
844
26
84.3
3.2
83.5
28
3.67
84.5
3.5
mallee2d-medium
88.7
86.1
86.1
=
86.2
0.
04
0.5
86.
ant- ant-medium
104.2 1.2
104.2
0.5
1048 + 0.5
104.7
0,5
104.8
+
0.6
104.6
0.5
104.4 0.7
Average over datasets
88.0 1.7
89.8 4 1.8
90.7 = 1.3
91.8 +
1.2
91.6
1.4
91.7
1.7
91.7 = 1.4
23

---

Published as a conference paper at ICLR 2024
Table 25: Best score for CQL and CQL+MDP tre-training with different number of pre- pre-training updates.
Besl Score
COL
1K updates
10K updates
40K updates
100K updates
500K updates
1M updates
halfcbbcrall-modinn--tte
48.2
38.3
71.3
81.432
3.1
85,6
1..6
8.84
hopper-meee medium--eppet
110.8 +: 1.0 110.7
1.6
111.1
: 0.7
111.3 0.5
111.3 = 111
111.4 = 0.8
111.5
0.5
medium expert
ant ant-medium-exppe
127.8 2: 2.0
130.5 = 1.0
129.7
131.5
4 0.8
132.1 1.1
1327 0.
132.1 ::00
halfchootal-medinngg meplu
45.7 0.4
45.7
45.9
45.8
45.7
4
hopper- medium- replay
99.5
100.3
1.4
1.6
99.6
100.1
101.2 2
100.8 1.1
malkkr2d-medium-ree
87.6 1.4
88.4
89.1 1.4
19.7211
89.2
89.4 1.2
aut-medium-replay
102.5
halfcheetub medium
4..9.1.0.2
47.3 0.2
47.3
47.3 0.2
47.4 0.2
47.3 0.1
47.2 0.2
hopper medium
4..3
walker2d-medion
0.5
86.2
0.6
86.3
06.3003
86.0 05
862 0.6
86.3 0.5
863 0.5
ant-medium
104.2
104.7
104.7 1044
104.7
104.7 0.6
105.0
Average
11
24

---

Published as a conference paper at ICLR 2024
C.2 CQL FINE-TUNING CURVES FOR INDIVIDUAL DATASET
Figure S and Figure 6 show the normalized test performance and the combined loss respectively for each dataset during the fine-tuning process. Each curve is averaged over 20 seeds. We shift all CQL+MDP and CQL+IID curves to the right to offset the synthetic pre-training for 100K updates.
50
COL
COL+MOP
40
Normalzzd Score
COL
Nomalizee
Normalized
20:
30
20
20
10
10
CQL+MDD
CQL+MDP
CQL+IID
CQL+IID
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Number of Updates
Number of Updates
Number Updates
(a) HalfCheetah medium- expert
(b) HalfCheetah medium
(c) HalfCheetah medium-replay
100
Norall zed Score
80
so.
80
90-
70
60
60
CQL
30:
CQL
50
40
CQL
20
COL+MDP
20
CQL+MDP
MOL+MDP
COLIII
CQL+IID
IID
20
CQL+ID
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Number Updates
Number Updates
Number
(d) Hopper medium-expert
(c) Hopper medium
(f) Hopper medium-replay
100
80
80
Normalized Score
Honmalized Score
80
800
60
60
40
40
40
COL
COL
CQL
20
20
CQL
CQL+ID
CQU+IID
CQU+II
0.6
0.8
0.8
Updates
Updates
Updates
(g) Walker medium-expert
(h) Walker medium
(i) Walker medium-replay
120
100
Normaiizdd Score
Normalized Score
100
ao
60
60
70
60
CQL
CQL
CQL CQL+MDP
CQL+MDP
CQL+ MDP
20
CQL+IID
20
Number of Updates
Number Updates
Number of Updates
(i) Ant medium-cse expert
(k) Ant medium
()) Ant medium-replay
Figure 5: Fine-tuning performance curves for CQL baseline, CQL with synthetic MDP pre-training. and CQL with synthetic IID pre-training, on each individual dataset.
25

---

Published as a conference paper at ICLR 2024
250
12.5-
COL
140
COL COL-MDP
CDL Training Loss
Loss
120
200
7.5-
COL+
CQL+II
5.0
COL Treining
150
2.5
60
100
COL+ MDP
COL
-2.5
40
50
COL+IID
-5.0
20
-7.5
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Number of Updates
Number of Updates
Number of Updates
(a) HalfCheetah medium-expert
(b) HalfCheetah medium
(c) HalfCheetah medium-replay
COL
CQL
80
COL
C9L Training Loss
25
COL+MDP
CQL+IID
17.5
CQL- MDP
CQL
CQL Training
70
CQL-MDP
IID
12.5
60
10.0
50
155
7.5-
5.0
40
0.0
0.2
0.4
0.6 Updates
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Number
Number of Upciates
Number Updates
(d) Hopper medium-export expert
Hopper medium
(f) Hopper medium- replayy
20
CQL
CQL
100
CQL
CQL Training Loss
15
COL+MDP
40
CQL-MDP
CQL Training Loss
90-
CQL-MDP
CQL+IDD
CQL+II
CQL+IID
10
CQL Training
80
20
20
100
60
50
-10
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.6
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Number Updates
Number Updates
Number Updates
(g) Walker medium-expert
(h) Walker medium
Walker medium-replay
80
COL Training Loss
400
COL+MDP
COL
CQL Training Loss
60
CQL
CQL-MDP
CQL Training Loss
120
CQL+MDP
CQL
COL+IID
CQL+IID
100
CQL+IID
300
80
200
20
60
100
40
-20
20
0.0
0.2
1.0
0.0
Updates
Updates
Number Updates
(i) Ant predium-expers
(k) Ant medium
(1) Ant medium-replay
Figure 6: Combined loss (Q loss and conservative loss) for CQL, CQL with synthetic MDP pre- training, and CQL with synthetic IID pre-training, on each individual dataset.
26

---

Published as a conference paper at ICLR 2024
D ADDITIONAL FIGURES
Figure illustrates how temperature values affect the transition distribution:
0.5
0014
0.030
0.4
00012
2.0000
02
0.04
20006
0.005
8.000
9.0
80
20
68
50
100
0.0
20
69
80
100
0.00
20
05
00
J00
0.000
40
80
100
3.000
20
00
60
80
100
(a) r = 0.001
(b) 7 = 0.01
(c) T = 0.1
(d) (d)=1
(e) T 10
Figure How different temperature values affect the transition distributions for the synthetic MC. The x-axis shows different states in the state space, and the y-axis shows the probability of transitioning into that state when the current state fixed. The states on the x-axis are sorted from lowest probability to highest probability. Under low temperature, the transition probabilities are concentrated on just a few states. Under higher temperature, the distribution nearly uniform.
27

---

Published as a conference paper at ICLR 2024
E DT WITH MORE FINE-TUNING UPDATES
Table 26 shows what happens when the DT baseline trained with more fine-tuning updates. The results show that more updates can indeed further improve the DT baseline. The DT baseline with 80K more updates (DT+80K more) is able to achieve a similar or even stronger performance than DT+Wiki. However, the DT baseline with 80K additional updates is still significantly weaker than DT+Synthetic. Note that DT+80K more updates uses a total of 180K updates, while our DT+Syythetic scheme uses a total of 120K updates (20K pre-train and 100K finetune), and Sy+Sttthttic is still better. These results further support our finding that (1) Wiki pretraining does not have special benefit, and (2) synthetic pre-training can significantly outperform the baseline, even with fewer total updates.
Figure presents the learning curves where the x-axis indicates the total number of updates, pre- training included. Here we train all three variants longer, to a total of 180K updates. Note that for the two pre-training schemes, the curve is shifted to the right to account for the number of pre-training updates. The curves are shown for the different datasets separately in Figure 9 Figure showss that when trained for more fine-tuning updates, all variants obtain slightly better performance, while Sy+thhhtii achieves the best performance quite consistently for different values of the total number of updates.
Table 26: Performance for DT, DT pre-trained with Wikipedia data, and DT pre-trained with synthetic data. We also show results for DT with 20K and with 80K more fine+tuning updates. DT+Synthetic has the same number of total updates as DT+20K more. DT+Wiki has the same number of total updates as DT+80K more. With the same number of total updates, DT+Synthetic provides the best performance.
Averago Four
DT
DT+20K more
DT+80K
DT+Wiki
DT-Synthetic
expert
44.1
44.2
112:20 2.6
47.1 5.5
43.9 2.7
49,5
hopper medium expert
73.6
15.5
87.6
1
10.5
92.4
1 7.4
94.0 1 8.9
99.6
walker2d- med-idiimmmmmmmmm expert
106.5
+: 1.4
104.
2.9
106.9
+ 2.0
102.7 6.4
107.4 0.8
ant- medium-cxperrt balfcheetah-meddine
102.3 5.1 5.1
110.9
- 68
116.1
5.7
113.9
10.5
117.9
8.7
0.1
4.22
42.5
+.2
hopper medium
56.4 2.1
60.6 1: 3.4
60.3 1 2.5
$8.4 3.3
60.2
2.1
2alker2d-medium
71.0 1.7
70.1 3.2
69.0 4.6
70,8
71.5
ant medium
90.3 3.7
8.6
89.7 3.5
88.5 4.2
87.8
halfcheetah mediumm replay
37.6 1.1
37.3 1.4
38.1
111
39.1
39.3
penger-meduum-epppe
43.5
47.5 128
46.4
10.2
514
13.6
61.8
13.9
malker2d-mxdiun-rpllg
55.4 8.9
50.6
55.8 + 9.1
55.2 7.7
56.8
5.1
78.1 5.3
Average (All Settings)
67.2 4.0
69.1 5.1
70.7 1 4.6
69.9 5.5
73.6 4.9
DI
Normalized Score
75
DT Training Loss
DT+ Wiki
DT Synthetic
70-
65
10-1
60
DT
DT Wiki
55
Sy+Shtteitic
4x13
Total Number of Updates
Total Number of Updates
(a) Learning curves for the fine-tuning stage. (b) Loss curves for the fine-tuning stage.
Figure 8: Performance and loss curves, averaged over 12 datasets for DT, DT+Wik DT+tStthetic. The DT baseline is fine-tuned for 180K updates. To account for the pre- pre-training updates, we offset the curve for DT+Synthetic to the right by 20K updates and fine-tune for 160K updates, and we offest the curve for DT+Wiki to the right by 80K updates and fine-tune for 100K updates. For the same number of total updates, SS+Stthhtti performs significantly better.
28

---

Published as a conference paper at ICLR 2024
62
DT
100
Normalzee Score
Sy+Synthetc
90-
Hormalzzd
58
Normaizzed
80
56
70
54-
52
30
50
50
DT+Sytthetic
Dy+Sttttete
Tocal Updates
Updates
Total Number Upcates
(a) Hopper-meding
(b) Hopper-medium-replyy
(c) Hopper-mediumeeper
110
60
70-
100
Nomalized Score
100
65
Normalized
40
80
55-
WTi
ST+Sythhetic
20
D7-Sytthetk
Syntttee
Toal Number of Updates
Totall Numbor of Updatos
Total Number of Updates
(d) Walker-medium
(e) Walkeer-medium--eplay
(1) Walker-meddiu--expeet
42.5
40
55.0
42.0
Normalized Score
52.5
Cf Synthetic
STt Wilk
41.5
Normalized
34
50.0
47.5
41.0
32
40.5
DT+
DT
30
DT-
42.5
SymShtet
28
Synthetticc
40.0
0
Total Number Updates
Total Number or Updates
Total Number or Upcates
(g) HalfCheetah-mmedimm
(h) HallChheetaa-mmccuuu--rrpaaaa
(i) HallCheetth-mmdduu--oprrr
92
125
90
Nornaltred Scare
g 120
$ 115
88
Nomaaize
86
80
Normalbe &
110
105
25
82
WIM
95
SymSttteetic
SynSyttett
Sy+Synteetc
80
0.0
Number
Number Updates
Number Updates
Ant-medium
Ant-modium-replay
Ant-medium-expert
Figure 9: Performance curves for individual datasets for DT, DT+ and DT+SSnthetic. The DT baseline is fine-tuned for 180K updates. To account for the pre- prainiain updates, we offset the curve for DT+ Synthhttc to the right by 20K updates and fine-tune for 160K updates, and we offset the curve for DT+ Wiki the right by 80K updates and fine-tune for 100K updates.
29

---

Published as a conference paper at ICLR 2024
F OTHER DT PRE-TRAINING STRATEGIES
In Table 27 we study two alternative synthetic data generation schemes inspired by He et al. 2023). namely, Identity Operation and Token Mapping. For Identity Operation, the model is simply trained to predict the current state. For Token Mapping, the model is trained to predict a fixed one-to-one mapping from each state (token) in the state space (vocabulary) to state (token) in the target state space (vocabulary). We use similar vocabulary size as in et al. (202)). As before, the DT baseline fine-tuned for 100K updates, and DT+ Wiki is pre-trained for 80K updates and fine-tuned for 100K updates. DT+Synthetic, DT+Identity, and DT++Mapping are all pre-trained with 20K updates and fine-tuned for 100K updates. The results show that these alternative schemes also lead to improved performance over the DT baseline; however, our proposed synthetic scheme provides the strongest performance boost.
Table 27: Final performance for DT, DT pre-trained with Wikipedia data, DT pre- pre-ttrined with Markov Chain synthetic data of default parameters, and DT pre-trained with two additional synthetic datasets: Identity Operation and Token Mapping.
Average Last Four
DT
DT+Wiki
DT-SSythetic
DT+Identity
DT+Mapping
halfcheetal medrum-expert
44.9
43.9 1.77
4.
4.12
49,6
hopper medium expert
81.0 1: 11.8
94.0 1 8.9
99.6 1 6.5
88,9
11.0
86.9 1 21.2
walker2d medium expert
105.0
0.1
9
ant-medium--on expprtt
107.0 8.7
113.9 10.5
117.9
112.1
11.
120.4 = 29
halfcheetah medium
42.4 = 0.5
42.61:02
42.5 0.2
42.7 0.2
42.5 1: 0.1
hopper-mediom walker2d-medium
5822 3.2
3.3
2.1
4.92
2.
70.4 = 2.9
70.8 3.0
71.5
69.9
71.8 1.8
ant
hulfcheetah medium replay
37.5 1.3
39.1 1.6
19.3
38.8
39.9 0.6
nopper-medium-reping
10.6
514
11.6
walker2d-medee milliucd-mmeiu
49.2 - 10.1
7.7
$6.8
4..5
$1.9
ant-medium-replee
80.9 = 3.9
78.1
2.7
8..6
87.4
Average (All Setlings)
5.7
30

---

Published as a conference paper at ICLR 2024
G IMPACT OF MC PARAMETERS FOR DIFFERENT AMOUNTS OF FINE-TUNING UPDATES
75.0
72.5
70.0
70.0
67.5
FT
67.5
67.5
2
65.0
DT
65.0
DT+N
62.3
60.0
60.0
57.5
510000
51000
57.5
2-9C
S.C
57.5
110.0
55.0
55.0
55.0
6-
Number finetune Updates
Number
finetune Updates
Number of firetune Updates
(a) State ablations.
(b) MC step ablations.
(c) Temperature Ablations.
Figure 10: Ablations for how the MC parameters (state, MC steps, and temperature) impact perfor- mance for different amounts of fine-tuning updates, averaged over 12 datasets. In the temperature ablations, "IID" stands for pre-training with synthetic IID data generated uniformly.
72
dermalized Score
Marmalliedd Scare
72
termalized
70-
72
DT+
0T
DT-Wid
510
60
TO.1
5100
1-KC
11.0
88
51000
66
GA
04
64
6
Ainetune Data Ratio
Finetune Data Ratio
Finetune Ratto
(a) State ablations.
(b) MC step ablations.
Temperature Ablations.
Figure 11: Ablations for how the MC parameters (state, MC steps, and temperature) impact perfor- mance for different amounts of fine-tuning data, averaged over 12 datasets. "Finetune Data Ratio" means the portion of the data with respect to the original dataset. For example, "Finetune Data Ratio" of 0.1 means that the models fine-tuned with 10% of the original datasets.
31

---

Published as a conference paper at ICLR 2024
H ANALYSIS OF HOW PRE-TRAINING EFFECTS DT WEIGHTS AND FEATURES
In this section, we provide an empirical analysis to shed light on how the different pre-training schemes affect the weights and features in the Decision Transformer. We consider the weights and features at three stages: (1) the randomly initialized weights; (2) the weights after pre-training, and (3) the weights after fine-tuning.
Figure 12 shows in blue the weight cosine-similarities between the pre-trained and fine-tuned weights (PT vs. FT), and shows in orange the weight cosine-similarities between the randomly initialized (before training) and fine-tuned weights (RI vs. FT). Figure (2(b) is analogous but for features instead weights. To obtain the features, we pass portion of each offline RL dataset through a frozen model to extract the feature vectors before the prediction head. The comparison is made across all datasets for the DT baseline, DT with Wiki pre-training. DT with default synthetic pre-tra vre-training, DT with random IID data pre-training, DT with Identity Operation pre- pre-training, and DT with Mapping pre- pre-training. From Figure 12
we observe the following:
Weight
Comparison
Feature (Symlog) Cosine Similarity Comparison
1.0
PT vs. FT
Fealure Cosine Similarity
0.33
0:66
0:73
0.56
Weight Cosine Siriil
0.81
0.82
0.76
0.77
0.7
0.72
0.70
0.69
0.65
0.59
0.58
10002
0,0050
0.0029
-000
RI vs.
0T
DT Wild
Symtheetiic
DT+ID
DT ldertity
DT Mapping
DT Wiki
DT+ Synthetc
DT++10
DT -danniyy
DT Mapping
Pretraining Schemes
Pretraining Schemes
(a) Weight cosine similarity comparison
(b) Feature cosine similarity comparison
Figure 12: Weight and Feature cosine similarity comparisons among the DT baseline and the various pre-training schemes.
1. From Figure 12al
that the cosine- sosine-similarity between the initial weights and the weights after fine-tuning (RI vs. FT) for all the pre-training schemes are lower compared that of the DT baseline (fine-tuning without pre-training at all). This suggests that pre- training together with fine-tuning alters the angle of the weights more than when doing fine-tuning alone. This phenomenon suggests that pre-training is able move the weight vector to a new region which is more beneficial for downstream RL tasks.
2. From Figure 12a the cosine-similarities of the weights before and after fine-tuning (PT vs. FT) for the pre-training schemes are all higher than that of the DT baseline. This suggests that during the fine-tuning stage of pre-training scheme, the weights are changed less. However, we observe that the similarities in the pre- pre-training schemes are inversely proportional to their performance. (ynthhttt has the best performance while being the least similar, while Wiki has the worst performance while being the most similar). This suggests that during the fine-tuning stage, encouraging a bigger movement in the weights is beneficial, and that our synthetic pre- pre-training scheme positions the weights to allow for such movement.
3. Similar to the weight comparison, from Figure 12b
also find that the cosine cosine-similaritiss between the features from randomly initialized models and those after fine-tuning (RI vs. FT) for all the pre-training schemes are much lower compared to that of the DT baseline (by three orders of magnitude), suggesting that there a bigger change in the features from pre-trained and then fine-tuning than when doing fine-tuning alone. Such movement of the feature vectors might indicate better learning of the feature representations.
4. We see from Figure 126 that the features before and after fine-tuning (PT vs. FT) for the synthetic pre-training schemes are more similar than those of DT+Wiik (0.33). This suggests
32

---

Published as a conference paper at ICLR 2024
that for the Wiki pre- tre-training scheme, the features need to be altered more due to the domain gap between language and RL, potentially hindering its performance.
33

---

Published as a conference paper at ICLR 2024
I CQL WITH MORE FINE-TUNING UPDATES
Recall that for our CQL pre-training experiments in the main body of the paper, we pre-trained for 100K updates. Table 28 shows the performance of the CQL baseline when is trained with additional 100K fine-tuning updates, so that the total (pre-training plus fine-tuning) number of updates is the same across all schemes. We see that the additional 100K fine-tuning updates JCQL+100K more) does not improve the final performance of the CQL baseline. Consequently, CQL pre- preitrained with synthetic data still significantly outperforms the baseline.
Figure 1 presents the learning curves where the x-axis indicates the total number of updates, pre- training included. All three variants are trained with 1.1M updates total. Note that for the two pre-training schemes, the curve is shifted to the right to account for the number of pre-training updates. The curves are shown for the different datasets separately in Figure [4
Table 28: Performance for CQL, CQL pre-trained with MDP synthetic data, and CQL pre-trained with IID data. We also show results for CQL with 100K more fine-tuning updates. CQL+MDP and CQL+IID have the same number of total updates as CQL+ more. With the same number of total updates, CQL+MDP provides the best performance.
Averuge Last Four salfchietal-modium-eee
CQL
CQL+ (QL+
COL+MDP
CQL+ID
37.3 4.9
36.7 6.9
65.3 8.0 63.2
hopper medium- expert
71.4
walker2d medium expert
106.0 1: 5.0
110.2 0.3
109.9 1.4
109.8 1.3
ant medium-eeepppppe -ipptt
114.6 +:
111.7 7.3
128.5 3.6
12
halfcheetah medium replay
46.7 11 0.2
46.4 1: 0.4
1.2
46.7
1: 0.3
pnpper-ediuum-epphe
95.6 1.1
95.3 + 1.5
94.0
92.4
3.8
walker2d- 2d-eein--mii
79.3 6.0
80.0
0.9
2.4
ant aut-mediuu--play
95.5 2.7
99.4
101.5 4.2
101.5
11 4.1
halfcheelah mediumi
hopper- medium
68.0 +: 2.4
68.4
67.7 3.7
64.9
2.6
walker2d-medium
80.8 - 1.5
82.4
83.8
834 0.7
medium
1: 2.7
100.6
101.5
46
104,0
1.4
Average over datassets
78.6 12: 4.6
76.7 4.1
85.3 3 3.6
83.6 = 39
COL
120
COL
Nomalized Score
CQL CQL Training Loss
100
80
60
40
lotal Number of Updates
Intal Number of Updates
(a) Learning curves the mne-tunng stage.
(b) Loss curves for the fine-tuning stage.
Figure 13: Performance and loss curves, averaged over 12 datasets for CQL, CQL+MDP, and CQL+IID. The CQL baseline fine-tuned for 1.1M updates. To account for the pre-training updates, we offset the curve for CQL+MDP and CQL+IID to the right by 100K updates and fine-tune them for 1M updates. For the same number total updates, Synthhnthetic performs the best.
34

---

Published as a conference paper at ICLR 2024
Narmalized Score
Normalized Score
CDL -D
Number
Updates
(a) Hopper-medium
(b) Hloper--edium--relae
(c) Hopper-medium
medium-exxpeee expert
Normalized aniiededd Score
Normalized Score
104
6 6 6 Total Number of Updates
6
Total Number Updates
Total Number Updates
(d) Walker2d-medium
(c) Walker2d medium-replay
Walker2d medium-exxert
Normalized Score
Normalized Score
004-80
Total Number of Updates
Total Number Updates
Total Number Updates
(g)
Halfcheetahh-mddimm
(h) Halfcheetah-mediuu-rrplyy
Halfchectah-medduu--pppe
Narmelized Score
Nomolized Score
Normalized Score
COK+MCT
Total Number of Updates
Total Number of Updates
Total Nurbber of Updates
(j) Ant-medium
(k) Aut-medium-replay
(I) Ant-medium-expert
Figure 14: Performance curves of individual datasets for CQL, CQL+MDR, and CQL+IID. The CQL
baseline is fine-tuned for 1.1M updates. To account for the pre-training updates, offset the curve for CQL+MDP and CQL+IID to the right by 100K updates and fine- tune them for 1M updates.
35

---

Published as a conference paper at ICLR 2024
J OTHER CQL PRETRAINING STRATEGIES
In Table 29 we again study the two alternative synthetic data generation schemes inspired byHHe He et al. (2023). similar to what did in Appendix F for DT. For Identity Operation, the model simply trained to predict the concatenated vector of the current state and current action. For Token Mapping, the model is trained to predict fixed one-to-one mapping from each state-action pair in the source space to a state-action pair in the target space. As before, the CQL baseline is fine-tuned for IM updates. CQL+MDP, CQL pre-trained with Identity Operation (CQL+Identity), and CQL pre-trained with Token Mapping (CQL+Mappingg are all pre-trained for 100K updates and fine-tuned for 1M updates. The results show that these alternative schemes also lead to improved performance over the CQL baseline; however, proposed synthetic scheme provides a significantly higher performance boost.
Figure presents the learning curves where the x-axis indicates the total number of updates, pre- training included. All variants are trained with 1M updates in total (pre-traingg training and fine-tuning) and for the three pre-training schemes, the curves shifted to the right to account for the number of pre-training updates. With fewer fine-tuning updates, the result shows that CQL pre-trained with either of the two additional synthetic data does surpass the performance of the CQL baseline, while our MDP synthetic pre-training scheme still significantly boosts the performance.
Final performance for CQL, CQL pre-trained with MDP synthetic data, and CQL pre- trained with additional synthetic datasets: Identity Operation (CQL+Identity) and Token Mapping (CQL+Mapping).
Table 29:
Average Last Four
CQL
CQL+MDP
CQL+Identity
CQL+Mapping
4.
6.03
32. 4.7
53.5
hopper-medium-con
71.4 25.0
94.7 147=144
80.7 = 14.4
20.9 64.4 20.9
walker2d meddmmm-mxpeee
106.0 1: 5.0
109.9
109.1 1
110.3 = 1.0
ant-mediun-- expett
114.6
3.6
6.
117.8 26
halfcheetah medd---eeddilpppaa
46.7 0.2 0.2
4..5
4.
45.9
hopper replay
95.6
walker medium-replla
6.0
81.3
82.0 2.
79.6
ant-medium--eplay
9.
4.2
97
9.
halfchectah-meee medeuu
48.4 0.1
48.7
18.7-002
4.1
48.3
hopper-mediam walker2d-medium
68.0 2.4
6.
71.3
68.8
068 2.7
80.8 L55
83.2
ant ant-medium
2.7
101.5 + 4.6
99.0 4.1
102.4
3.8
Average over datascts
78.6 4.6
85.3
1944
80.9
Normalized Score
80
60
40
CQL
CQL+ MDP
20
CQL+ IOL+idenity
CQL+ Mapping
0.0
0.2
0.6
Total Number of Updates
Figure 15: Performance curves, averaged over 12 datasets for CQL, CQL pre-trained with default MDP synthetic pre-training, and CQL areitrained with two additional synthetic datasets. The CQL baseline is fine-tuned for 1M updates. To account for the pre-training updates, offset the curve for all three synthetic pre-trained CQL to the right by 100K updates and fine-tune them for 900K updates.
36

---

Published as a conference paper at ICLR 2024
K IMPACT OF MDP PARAMETERS FOR DIFFERENT AMOUNTS OF FINE-TUNING UPDATES
80
80
Norm Normalize aeleleeeed Score
30
70-
70
Normaized
60
60
60
13.1
52.00
50
40
40
40
67
Number of Finetune Updates
Number of Finetune Updates
Number of Finctune Updates
(a) State ablations.
(b) Temperature ablations.
(c)
Pre-training Updates Abla-
tions.
Figure 16: Ablations for how the MDP data parameters (state, temperature, and number of pre-training updates) affect the performance with different fine-tuning updates, averaged over 12 datasets. the temperature ablations, "IID" stands for pre- gre-training with synthetic IID data generated uniformly.
83
Normalloed Score
78-
Normllced
73
Normlline
68
63
63
$1.0
Sw
588
58
53
525.000
53
6
6
6
67
Finetune Data Ratio
Finetune Data Ratio
Finetune Data Ratio
(a) State ablations.
(b) Temperature ablations.
(c) Pre-training Updates Abla- tions.
Figure 17: Ablations for how the MDP data parameters (state, temperature, and the number of pre-training updates) affect the performance with different amounts of fine-tuning data, averaged over 12 datasets. The x-axis 'Finetune Data Ratio" means the portion of the data used to fine-tune the model, with respect to the whole offline RI. data.
37


Reviews:
[{"review_id": "dB9dt84aHw", "reviewer": "Reviewer_dSfS", "summary": "The manuscript presents a unique approach to pre-training for offline reinforcement learning, utilizing synthetic data generated by a Markov Chain in lieu of traditional real-world language resources. The core premise is that this synthetic data can achieve comparable results to real-world data in downstream task performance, which is a significant assertion in the field of offline RL.", "questions": "1. Could you provide a more detailed justification for the methodological choices, specifically the use of a Markov Chain for data synthesis?\n\n2. Are there illustrative examples of synthetic data that could be shared to better understand its characteristics and how it compares to real-world data?\n\n**After Rebuttal**\nThank you for the response. I have raised my score from 5 to 6 and my confidence is 2.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clarity of Presentation: The paper is well-structured, making it accessible even to those who may not be deeply versed in the domain. The significance of the research question is conveyed effectively, which facilitates a quick grasp of the paper's importance.\n\n2. Innovation in Data Construction: The methodology employed for the generation of synthetic data is both novel and straightforward, potentially offering a simpler alternative to more complex data generation strategies.", "weaknesses": "1. Methodological Justification: The rationale behind the adoption of a Markov Chain for synthetic data generation requires further elaboration. While the introduction suggests that understanding the underlying question is crucial for enhancing pre-training in deep reinforcement learning (DRL), the link between this understanding and the proposed method is not convincingly established.\n\n2. Need More Deep Analysis: The paper primarily demonstrates the efficacy of the proposed method without a robust analysis. It is advisable that the authors consider incorporating analysis akin to those found in the literature regarding Synthetic Data utilization in Transformer models (see Synthetic Pre-Training Tasks for Neural Machine Translation (ACL 2023) and its related works). This could potentially refine the proposed method and offer deeper insights through a more comprehensive analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YdIKAupbpj", "reviewer": "Reviewer_WUkY", "summary": "This paper explored pre-training Transformer (DT) with synthetic data. They found that pre-training with synthetic IID data can match the performance gains from pre-training on large-scale language data. \n\nThe authors also apply the pre-training methods into the conservative Q-learning (CQL) framework. Experimental results show that pre-training with IID data and Markov decision process data can improve the performance of CQL.", "questions": "How much do different synthetic data construction methods affect the results?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposes a simple yet effective pre-training method with synthetic data for Decision Transformer.\n\n2. Results demonstrate that the proposed pre-training method with CQL can achieve significant improvements.", "weaknesses": "1. The experiments lack comparison for some pre-trained DT models, such as Future-conditioned Unsupervised Pretraining for Decision Transformer(https://proceedings.mlr.press/v202/xie23b/xie23b.pdf).\n\n2. It is more convincing to evaluate the proposed methods on more tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fdVhCcV58X", "reviewer": "Reviewer_jxkz", "summary": "Reid et al. (2022) demonstrated that pre-training a Decision Transformer (DT) on a Wikipedia corpus can substantially improve its performance on downstream Deep Reinforcement Learning (DRL) tasks. This paper explores whether a synthetic pre-training corpus can act as a substitute for the Wikipedia corpus. The main finding is that synthetic data generated using a one-step Markov Chain with a state space of 100 states and 75% fewer updates outperforms Wikipedia for pre-training. Additionally, the performance is relatively unaffected by the order of the Markov Chain or the size of the state space. A softmax temperature of 1.0 yields the best results. IID data performs marginally worse than Markov Chain samples, but still outperforms both no pre-training and pre-training with Wikipedia. For conservative Q-learning (CQL), synthetic pre-training data generated using a Markov Decision Process (MDP) leads to significant improvements over no pre-training. Similar to Decision Transformer, the best performance for CQL is achieved using a state space with a size of 1000, a softmax temperature of 1, and 100k updates. IID data performs slightly worse than MDP data, but still outperforms no pre-training.", "questions": "* How does the optimal configuration of synthetic data vary as a function of the number of fine-tuning updates?\n* What is the performance of the baseline (DT) if it is run for an additional 80k updates on Wikipedia or 20k updates on the proposed synthetic data? This would provide us a side-by-side comparison of the 3 models when run for the same number of updates.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* Presents empirical evidence that contradicts the prevailing belief that language data is essential for pre-training models for offline Deep Reinforcement Learning.\n\n* Demonstrates the benefits of synthetic pre-training data for both transformer and Q-learning based approaches to offline DRL.\n\n* Reports ablation studies to investigate the influence of various parameters such as the size of the state space, temperature, and order of the Markov Chain.", "weaknesses": "* The paper does not investigate the impact of the number of updates during fine-tuning, which is kept constant at 100k for DT and 1M for CQL. It would be useful to understand the relationship between the parameters of the synthetic data and the number of updates in fine-tuning.  This has the practical implication that fine-tuning data is typically task-specific and its availability may be severely limited. Alternatively, computational constraints may limit the fine-tuning budget.\n* The paper does not report the performance of the baseline (DT/CQL) if it is run for x more updates (where x is 80K for Wikipedia and 20k for the proposed synthetic data).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)