Bootstrapped Transformer for Offline Reinforcement Learning
Authors: Kerong Wang
Hanye Zhao
Xufang Luo
Kan Ren
Weinan Zhang
Dongsheng Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
Offline Reinforcement Learning
Sequence Modeling
Sequence Generation
Bootstrapping

Abstract: Offline reinforcement learning (RL) aims at learning policies from previously collected static trajectory data without interacting with the real environment. Recent works provide a novel perspective by viewing offline RL as a generic sequence generation problem, adopting sequence models such as Transformer architecture to model distributions over trajectories and repurposing beam search as a planning algorithm. However, the training datasets utilized in general offline RL tasks are quite limited and often suffering from insufficient distribution coverage, which could me harmful to training sequence generation models yet has not drawn enough attention in the previous works. In this paper, we propose a novel algorithm named Bootstrapped Transformer, which incorporates the idea of bootstrapping and leverages the learned model to self-generate more offline data to further boost the training of sequence model. We conduct extensive experiments on two offline RL benchmarks and demonstrate that our model can largely remedy the limitations of the existing offline RL training and beat other strong baseline methods. We also analyze the generated pseudo data and the revealed characteristics may shed some light on offline RL training.

Bootstrapped Transformer for Offline Reinforcement Learning
Kerong Wang"
Hanye Zhao Shanghai Jiao Tong University
Xufang Luo Microsoft Research Asia
Shanghai Jiao Tong University
Kan Ren Microsoft Research Asia
Weinan Zhang Shanghai Jiao Tong University
Dongsheng Li Microsoft Research Asia
Abstract
Offline reinforcement learning (RL) aims at learning policies from previously collected static trajectory data without interacting with the real environment. Recent works provide novel perspective by viewing offline RL as generic sequence generation problem, adopting sequence models such as Transformer architecture model distributions over trajectories, and repurposing beam search as a planning algorithm. However, the training datasets utilized in general offline RL tasks are quite limited and often suffer from insufficient distribution coverage, which could be harmful training sequence generation In this paper, propose novel algorithm named Bootstrapped Transformer, which incorporates the idea bootstrapping and leverages the learned model to self-generate more offline data further boost the sequence model training. We conduct extensive experiments on offline RL benchmarks and demonstrate that the existing offline RL training limitations and beat other strong baseline methods. We also analyze the generated pseudo data and the revealed characteristics may shed some light on offline RL training. The codes and supplementary materials are available Inttps: //t:::///ee //seqml, wit.tgitthu...... wiw.obt.oooooorrrroo
model can largely remedy
Introduction
requires environments and interactions to train policies. However, such requirement obstructs RL from being applied to many real-world problems where simulation environments are unavailable or interactions are incredibly costly. Offline RL tries to get rid of the requirement and learn policies from previously collected offline dataset without interacting with the real environment. This setting enables more practical RL training in the real world and thus attracts lots of attention
Online reinforcement learning (RL) 42
33 g 50
27
Unlike conventional approaches that focus on learning value functions 41 or computing policy gradients (34), recent works (32 3 view the offline RL problem from novel perspective, where it is considered a generic sequence generation problem. Specifically, these works utilize mainstream sequence modeling neural network, i.e., Transformer [47], model the joint distribution of the offline decision-making trajectory data, after discretizing each term vector within the transitions (states, rewards, actions), in supervised learning (SL) manner. In such sequence modeling framework, trajectory consisting of states, actions, and rewards is viewed as a stream tokens, and the learned Transformer can generate tokens in autoregressive way and subsequently be utilized for execution in the real environments. This framework unifies multiple components in offline RL in one big
'The work was conducted during the internship of Kerong Wang and Hanye Zhao at Microsoft Research. i CCorespondence to Kan Ren, contact: kan.ren @ microsoft.com
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

sequence model, such as estimation of behavior policy and modeling for predictive dynamics 32] thus achieving superior performance.
However, there exist two major data limitations in offline RL tasks, which might be harmful to training sequence generation models yet have not been well addressed in the previous works. First, the offline data coverage is limited. The training data in offline RL tasks could be collected by arbitrary policies like human experts or random policies, which cannot guarantee sufficient coverage of the whole state and action spaces [ 36]. The second issue lies in the amount of training data. Specifically, the largest dataset in the widely-used offline RL benchmark D4RL [9] contains only about 4,000 trajectories, which is quite limited especially considering those in natural language sequence modeling such as WikiText [31] which contains about 8,000000 sentence sequences. Such limited datasets may deteriorate the sequence generation model, since the common belief in the literature is that training sequence generation models from scratch often requires large amount of data with rich information [24). Therefore, handling limited datasets should be carefully considered in the sequence modeling methods in offline RL tasks.
Some previous works try to tackle the above data limitations via data augmentation to expand the data coverage, either through image augmentation 2 52 5T such as cropping, or by applying perturbations on the state inputs 226 43]. Nevertheless, these works can neither be applicable for other types of observations except image-based input, nor ensure to be consistent with the underlying Markov decision process (MDP). Another way is to train an additional model learning environment dynamics [8] 20 30 53 Yet it requires extra efforts, and the goals of model learning and policy optimization have not been aligned, which may derive suboptimal solutions [54] Moreover, all these methods have not considered the advanced sequence modeling framework.
In this paper, we try to relieve these problems under the sequence modeling paradigm for offline RL. Specifically, we consider using the learned model itself generate data and remedy the limitations in training datasets. We further propose a novel algorithm, Bootstrapped Transformer (BooT), implement the above idea. Generally, BooT uses Transformer to generate data and then uses the generated sequence data of high confidence to further train the model for bootstrapping. To make our study comprehensive. we investigate two generation schemes. The first one performs autoregressive generation, and the other uses the teacher-forcing strategy 48).. We also test two bootstrapping schemes. One trains the model further on the generated data and discards them once they have been used; the other uses the generated data repeatedly during the later training process.
In this way, the above issues in previous works can be addressed simultaneously. Specifically, the sequence generation model unifies predictive dynamics models and policies [32]; thus, we do not need train extra environment models for generating data, and such generation-based method can also be applied to proprioceptive observations (e.g., positions & velocities information). Moreover, since the sequence model learns the distribution of the dataset and models the sequence generation probability, the data generated by the learned model itself will be more consistent with the original offline data [29]. Thus, we expect higher consistency of the pseudo trajectory data generated by the model than those of adding random noises, which has been empirically verified in our experiment.
We further conduct comprehensive experiments multiple offline RL benchmarks, and BooT achieves significant performance improvements. For Gym domain in D4RL benchmark, BooT even reaches much higher average score than one of the baselines with a pre-trained 80 times larger model (39. which strongly demonstrates the superiority of BooT. We also provide an in-depth analysis of data generated by BooT with both qualitative and quantitative results. Visualization demonstrates that the generated data act as an expansion of the original training dataset. Besides, we quantitatively measure the difference between the original and generated trajectories. and empirically conclude that the generated pseudo trajectories from BooT generally lie in a neighborhood of the corresponding original trajectories while keeping consistent with the underlying MDP RL tasks, providing strong support for our motivation.
2 Related Work
Offline Reinforcement Learning. Offline RL algorithms learn policies from a static dataset without interacting with the real environment (27. Deploying off-policy RL algorithms directly into the offline setting suffers from the distributional shift problem, which could cause a significant performance drop, revealed in [T]] To address this issue, model-free algorithms try to constrain the policy
2

---

closed to the behavior policy [1T [0] or penalize the values of out-of- df--offbbiiiitiiinn state-action pairs
23 22. On the other side, model-based algorithms [53] 20 simulate the real environment and generate more data for policy training. All these methods tend to require the policy to be pessimistic, while our algorithm does not involve this additional constraint. Besides, our algorithm built upon the sequence generation framework, which is different from the above methods.
Sequence Modeling. Sequence modeling is a long-developed and popular area, and various methods have been proposed in natural language processing [16] 44], speech recognition [6], and information system [40]. Recently, Transformer [47] has illustrated powerful model performance and has been widely used in sequence modeling and generation tasks [5]. Some recent papers adapt Transformer into RL by viewing RL as a sequence modeling problem. Decision Transformer (DT) [3] learns the distribution of trajectories, and only predicts actions by feeding target rewards and previous states, rewards, and actions. Trajectory Transformer (TT) (32) further brings out the capability of the sequence model by predicting states, actions, and rewards and employing beam search. We leverage the paradigm of TT and provide a generation- based data augmentation methodology to improve the data coverage and further improve the performance. Recent works also try use pre-trained Transformers on other domains and finetune on offline RL [39 or other decision-making [17 [88] tasks. We focus on the data augmentation method, which is different and compatible with the pre-training techniques.
Data Augmentation (DA). DA a common approach promoting data richness and enhancing modeling efficacy in supervised learning scenarios. They adopt data augmentation either through pseudo labeling [49], input perturbation [4) or generative model [15] to produce additional data. Although DA has not been widely used in RL tasks, recent works have shown the promising performance of DA in RL methods. Some of them target the data inefficiency issue. Specifically, incorporates common perturbation- based approaches, while DrQ [11]. DrQ-v2 [52), and DrAC 37 also introduce regularization terms with respect to the augmented data in actor and critic. Some other works 14, [3 try to achieve high stability for online RL via data augmentation. For offline RL, S4RL 43 provides various raw-state- based DA methods and takes similar regularization to DrQ. Different from the previous works, we augment the offline data distribution through sequence generation by the learned Transformer model to remedy the limitations in training datasets for better facilitating offline policy learning in this paper.
RAD 22
3 Preliminary
3.1 Online and Offfine Reinforcement Learning
Generally, RL models the sequential decision making problem Markov Decision Process M= A, 7), where the state space and A is the action space. Given state s, s E S and
action E A. T(''ss a): S x Ax x S + [0, 1] is the transition probability and r(s, a): S x A + R
defines the reward function. 7 E (0, 1] is the discount factor. The policy æ: S x A [0, 1] takes action at state with probability x ())... At time step t [1, T), the accumulative discounted reward in the future, named reward-to-go
R₂ = Ei= -tre. The goal of online RL is to find a /Et=1 (r a:)] by learning from the
policy π that maximizes J = (a,-at(1s))) transitions (s, a, s') through interacting with the environment in an online manner 45
Offline RL, instead of interacting with the real environment, makes of fixed dataset D collected by behavior policy Th. to learn a policy æ that maximizes J for subsequent execution in the real environment, Here, Ty can either be single policy mixture of policies and is inaccessible. We assume that the collected data is trajectory -iiee i.e., D = [ where T = {(s,, a, Ti, {
3.2 Offline Reinforcement Learning as Sequence Modeling
In this paper, we take the offline RL problem as a sequence modeling task, following the previous work 32) We introduce the details of the sequence modeling paradigm in this subsection. After that, we illustrate our data augmentation and bootstrapping strategy to remedy the limitations in training datasets and further improve the learning performance in Sec.
3

---

Algorithm Training Procedure of Boo'T'
Input: The original training set D Parameter: Number of epochs E, bootstrap epoch threshold k, generation percentage 7%% and generation length T".
Note: Either BooT-o BooT-r is executed.
1: for = .,....,.......... E do
2: Initialize the set of generated trajectories G = 0 Eq.
3: for each batch of trajectories T D
4: Train the model with T
5: if > then # (perform bootstrapping)
6:
Generate K new trajectories g' {T'}] from as Eq. 5 and Eq. (6
7:
Select 17% K] trajectories with the highest confidence calculated as Eq. (9)
+ arg ( (Eress Crr) (BooT-o) Train the model with T' g as Eq. 9: (BooT-r) G Gug' end if
8:
10:
11:
end for
12: (BooT--) D + DUg
end for
First, we treat each input trajectory as a sequence and add reward-to-go R, = =I Et'=i 'r after reward rz auxiliary information at each timestep I, which acts future heuristics for further execution planning,
Taux (81, a1 T1, Ri...., 8r, GT, TT; RT)
Second, discretization approach is adopted to transform the continuous states and actions into discrete token spaces, each of which contains N and tokens, respectively. This turns Taux into sequence of discrete tokens with total length of (N + M 2)T as
diss (..., s S
St oz aZ. ....,........................... at Tr, R.....)
Without causing confusion, will keep using indicate discretized augmented trajectory Tdis in the following.
Finally, we model t with the Trajectory Transformer (TT) 32 from the perspective of sequence modeling. regards each trajectory as a sentence and is trained with the standard facherr-orring procedure. Let 0 be the parameters of TT and Pe be its induced conditional probabilities. Define
log Po(((TTe)) Σ log Po( Tc))) + Σ
log Pe (a/la,) st, Tx))
(3)
j=1
+ log Po(r, a, 8t, Txt) - log (o a 84, Bt, Tc)),
the training objective to maximize
T
L(T) = Σ
log PT(T.(Te)).
(4)
t=1
where The is the trajectory from timestep 1 to (t 1), s dimension to 1) of state 8t, and the similar for a²"
As for model inference, we adopt beam search as the planning algorithm to maximize cumulative discounted reward plus reward-to-go estimates, similar to 332
4 Bootstrapped Transformer
In this section, we describe our Bootstrapped Transformer (BooT) algorithm, utilizing self-generated trajectories as auxiliary data further train the model, which is the general idea of bootstrapping. Bootstrapping for supervised learning, such pseudo labeling [19] and self- self-training [49], has been

---

Trajectory Transformer
Truncatod Original Trajectory
R₂
Autoregressive Generation
Trajectory Transformer
R
Trjectory Transformer
fracher-forcing Generation
Concatenate
(M)
Truncated Original Trajectory
Predicted Tokens
Illustration of trajectory generation. The generated trajectory is produced by concatenating the truncated original trajectory and the newly generated sequence hy either of the two different generation schemes from the truncated position, described in Sec.
Figure 1:
widely used. Apart from those using pseudo label generation, in the RL domain, we directly utilize the learned sequence model to generate novel trajectory data, which has been an expansion of the original offline data (as shown in Sec. (5.3), and we further augment the original data distribution using the generated data to better approximate the real distribution following the underlying MDP. The self-generated trajectories are fed as additional training data further boost sequence modeling and decision making.
Overall, we divide BooT into two main parts, (i) generating trajectories with the learned sequence model and (ii) using the generated trajectories to augment the original offline dataset for bootstrapping the sequence model learning. The overall training algorithm of BooT has been described in Algorithm
4.1 Trajectory Generation
In this section, we demonstrate the details of trajectory generation. For an original trajectory T E D in the original dataset, we resample the last T' timesteps (T" T) as
RT) ~ Po (TST-
(5)
where TST-7" represents the original trajectory truncated at timestep (T (T) Then, we concatenate the generated token sequence to the truncated original trajectory and obtain a new trajectory
TST- TTT-T
where denotes concatenation operation.
We investigate two different trajectory generation schemes below, both of which are demonstrated in Figure We briefly compare these two methods in both efficiency and generation quality here, and perform detailed comparison experiments in Sec, 553
Autoregressive generation produce each predicted token on by sampling from the probability conditioned on the previously generated tokens as
Tn Pa (yr Den TST-T")
Here we denote the th generated token as yn and the sequence of all the previously generated tokens as Den- The autoregressive generation scheme requires recurrent predictions based on the previously generated tokens.
5

---

2. Teacher-forcing generation adopts similar procedure as teacher- forcing training by condition ing on the original token ground truth from the previous timesteps
In Pe (yn V <n TST-T")
(8)
Here, Snn denotes tokens from the original trajectory with position previous to in. Teacher- forcing generation only relies ground truth tokens.
Due to error accumulation in the sequential generation process, autoregressive generation tends to produce trajectories that deviate more from the original data distribution than teacher-foreing generation. As has been found in Sec. 5.3] autoregressive generation could expand the coverage of the training data more effectively and might derive better learning efficacy. Compared to autoregressive generation, teacher-forcing requires only one forward pass and thus consumes less time. However, since seacher-forcing generation is not able to expand dataset coverage as effectively as autoregressive generation, might lead to slightly less improvement in performance.
4.2 Training on the Augmented Dataset
After generating new trajectories, we use these trajectories to train our sequence model in addition to the original trajectory data.
In BooT,, we conduct two different bootstrapping schemes: Bootstrap-once, denoted as BooT- o, and Bootstrap-repeat, denoted BooT--. As shown in Algorithm I both methods will first train the sequence model over the original offline trajectories and then utilize it to generate novel trajectories. After that, BooT-o will train the model again the generated trajectories and discard them immediately once they have been used. In contrast, BooT-r will append the generated trajectories into the dataset D, and the generated trajectories will be used in every epoch after they are appended to the dataset.
In order to prevent accumulating learning bias caused by training with inaccurately generated data, we choose a part of the trajectories with the highest confidence scores in each batch according to the generation percentage 17%. Here, we define confidence as the average log probability of all the generated tokens as
T
<(T) T'NN + M 2)
Σ T
log Po(T:(T=())
where log Po (T(TT(T<c) is defined in Eq. 3 Intuitively, since the confidence score of trajectory is defined by its prediction probability that models the sequence generation mechanism, by filtering the generated trajectories with their confidence score, we can keep only the generated trajectory data of a high quality to further help the sequence model training through feeding these quality data back for bootstrapping.
Moreover, stabilize bootstrapped training, we leverage the idea of curriculum learning described in 2 but adopt a more straightforward method here. The sequence model is first trained over the original offline dataset and then gradually learns from the self- seff-generated trajectories. To achieve this, only use the generated trajectories to train our model after training & epochs over the original trajectories (without generated data). Though could use more sophisticated scheduled sampling strategies such as (2]. they may incorporate additional hyperparameters, resulting in complex parameter tuning. turns out that our simple proposed version has shown promising effectiveness in the experiments.
5 Experiments
In this section, we mainly (i) evaluate the performance BooT and (ii) investigate the characteristies of the generated trajectories of BooT. We first compare the evaluation results of our algorithm to other offline RL algorithms, and conduct a comparison among different bootstrapping and sequence generation schemes. Then study the characteristics of generated data with both qualitative and quantitative analysis illustrate more insights of our method. Finally, we introduce the ablation study on hyperparameters and analyze their influence BooT.

---

Table 1: Results D4RL dataset. BooT performs best 5 out of datasets compared to other baselines and obtained the highest average score 9 datasets. For results (Reproduce, Re-train, +S4RL (All), +S4RL (Last)) and BooT, we report the mean and standard deviation corresponding to 15 random seeds (5 training seeds for independently trained Transformers and 3 evaluation seeds for each model).
MBUP
CUL
Boo'I
Dutaset
Envicomment
Originul
GPT2
Original
Reprodicce
Re-trala
+S4KL
+S4RL (Las))
Medium Expert
Hairtheenuh
105.9
91.8
N1.
87.54
213
Medium Expers
55.1
105.4
197.6
130.9
10K.0
374
192.3
Molliun-Fpprt
Walker2D
105.1
105.9
102.2
90
110.4
Medium
HaltCheetdh
44.6
44.0
06.1
44
460
50.6
Maaiion
Herpper
07.1-90
61.9
70.2-
Malium
2D
71.1
82.9
Mallimn-Roplay
HalrChaatth
4.3
96
46.5
Maaliun- Replun
Hapqper
96.4--111
80:8
92.9
Maliun--Rpha
00.3
02.3
Average
47.8
70.1
69.4
5.1 Evaluation Settings
Benchmark and Compared Baselines. We evaluate our BooT algorithm on the dataset of continu- ous control tasks from the D4RL offline dataset [9]. We first perform experiments Gym domain, which includes three environments (halfcheetah, hopper, walker2d) with three levels (medium, medium-replay, medium-expert) for each environment. We compare BooT to the following various baselines: (1) behavior cloning (BC), a pure imitation learning method; (2) model-based offline plan- ning (MBOP ()) current strong baseline with model-based offline RL algorithm; (3) conservative Q-learnng (CQL (33). a current strong baseline of model-free offline RL algorithm; (4) Decision Transformer (DT Original (3) with its variant using GPT2 pre-traine trairddiddd model (DT GPT2, (39)), and (5) Trajectory Transformer (TT (2). two recent works applying Transformer to offine RL; and multiple variants derived from our proposed method BooT for ablation studies. In addition, we perform experiments on Adroit domain (38 in D4RL. dataset, including four environments (pen, hammer, door, relocate) with also three levels (human, cloned, expert) for each environment to better compare our method with TT and CQL baselines.
We report the normalized scores according to the protocol in [9] where normalized score of O corresponds to the average return of random policies and 100 corresponds to that of a domain-specific expert. The performance of BC is taken from 33, and the performance of CQL is taken from [22). The results of MBOP, DT Original, DT GPT2, and TT are reported from their original papers.
Besides, we reproduce the TT algorithm, namely TT (Reproduce). We also perform an additional experiment with a slightly modified setting as TT (Re-train): after the 1 original training epoch, for each training batch, we randomly choose 11/ K] original trajectories and train the model for the second time. This setting is to ensure the numbers of training steps are the same for TT and BooT.
To compare with other offline data augmentation methods, we perform two additional experiments using TT with S4RL (43 techniques, which adds random noise the input trajectories as augmented data. Specifically, 7T+S4RL (All) adds random noise to all states in trajectory, while 77RS4RL (Last) adds random noise to the last T" timesteps of a trajectory. The former setting resembles adding random noise to raw states in S4RL. while the latter is to compare with BooT where generated trajectories only differ from the original trajectories at the last T' timestep.
For a fair comparison, we adopt the same training hyperparameters with TT in all the above experi- ments. We also use the same model structure and model size TT for all BooT experiments.
Finally, we compare the performances of different bootstrapping schemes for BooT, using BooT-o and BooT-r, with autoregressive and teacher-forcing trajectories, respectively. All results in this section correspond to 15 random seeds, including 5 training seeds for independently trained Transformer models and 3 evaluation seeds for each model, if not explicitly mentioned.
5.2 Detailed Results
In this section, we first present the overall result of BooT compared with other baselines in Table [ and then show the detailed results of BooT with different bootstrapping schemes in Table 2 As is shown in Table I BooT achieves improvements from TT and performs on par or better than other
7

---

Table 2: Comparison between different trajectory generation and bootstrap schemes Gym domain. BooT-o and BooT-r at header refer to bootstrapping schemes, and Autoregressive (AR) and Teacher-forcing (TF) refer to trajectory generation schemes.
Dataset
Environment
BooT-o, AR
BooT-r; AR
B-o TF
Boo'T-r TF
Med-Expert
HallCheetah
94.0 1 10
94.5 09
93.6 13
95.2 1:
Med-Expert
Hopper
102.3 1: 194
88.7 298
96.1 215
94.6 1 24.5
Med-Expert
Walker2d
110.4
111.1 10
111.
111.3
Medium
HallCheetah
$0.6 1 0s
51.4 1.00
50.1 1:1
51.3 1.077
Medium
Hopper
70.2 4: 18.1
71.2
71.1
70.4
Medium
Walker2d
82.9 117
84.0 2:3
85.7 #:31
84.6 2:6
Med-Replay
HalfChectah
46.5
46.4
45.6 2.1
46,6 1.3
Med-Replay
Hopper
92.9 +: 132
91.3 +: 178
89.1 +: 26.3
96.8 +: 11.9
Med-Replay
Walker2d
87.6 a: 99
73.7 a: 25.9
77.1 1 24.6
77.2 1: 24.1
Average
81.9
79.1
79.9
80.9
offline RL algorithms. Compared our reproduced results of TT, BooT obtains improvements by 16.8% in average score, and achieves better performance on all 9 datasets.
Comparison among TT (Reproduce), TT (Re-train), and BooT suggest the improvement in the performance of BooT does not result from the extra training steps of the algorithm. Fur- thermore, we observe that directly adding Gaussian noise the training data does not help improve the per- BooT from
Table 3: Comparison between TT and CQL baselines and variants for BooT on Adroit domain. We list the mean and standard deviation corresponding taken from the original paper and they did not provide performances on expert datasets.
15 random seeds. Note that results of CQL are
Dataset
Environment
COL
TT
BooT-o, TF
BooT-r, TF
Expert
Pen
72.0 a 62.
72.3 644
79.7 = 0000
Expert
1
0.7 1:0
1:0
formance effectively the results of TTRSL experiments. We argue that data generated by the model itself are more consistent with the underlying MDP than by adding noise; thus, BooT obtains larger per- formance improvements, which also echos to our motivation.
Expert
Door
94.1 233
93.3 328
106.5 3.44
Expert
Relocate
10.3 a: 186
5.3 194
10.0 =: 119
Human
Pen
55.8
36.4 = 66.0
28.4
54.3 02
Human
Hammer
2.1
0.8 #6
0.7 =:
0.7 0.7
Human
Door
9.1
0.1 #:01
0.1 =:00
0.6 1:
Human Relocate
0.4
0,0
0.0
0.0
Cloned
Pen
40.3
11.4 4:4.5
31,5
31.0 68.
Cloned
Hammer
5.7
0,5 0:05
0.3 #:02
0.5 0:06
Cloned
Door
3.5
-0.0
-0.0 1:
Cloned Relocate
-0.1
-0.1 2:00
-0.0 2:01
-0.0 1:01
Experiment results in Table 3 indicate that BooT can also help improve the performance on more complicated tasks in Adroit domain, where BooT achieves better performance than TT baseline with an average improvement of 17.9%. Results show that seacher-forcing generation has already obtained noticeable improvement compared to the basclines, and autoregressive generation consumes too much time in Adroit domain as the number of dimensions of the trajectories is too large. As a result, we only perform experiments with teacher-foreing generation on Adroit domain.
Average
14.6
20.1
19.4
23.7
5.3 Further Analysis of Bootstrapping
Origine Dat
Cleeenttd Dats
Conarte
-50
-50
-50
-50
-50
-25
25
50
-25
25
Original Daraset
Teacher-foriing Generation
Autoragrassive Generntio
Augmented Dataset
Figure 2: Distribution of (s, a, R) transitions visualized with t-SNE in one example environment. Blue points correspond to (s, r, R) transitions from the betcchetah-meddiue dataset, red points correspond to generated data with feacher-forcing trajectory generation, and yellow points correspond to autoregressive trajectory generation. Generated data are drawn from the last training epuch. Best viewed in color.
8

---

To better understand how BooT helps improve the performance, we perform numerical analysis on the generated trajectories in Table I the distance from the generated trajec- tories of BooT to their corresponding original offline trajectories, denoted as Dataset, and the distance from the generated trajectories to real trajecto- ries from the environment in the eval- uation phase, denoted Environment, compared to those from TT+S4RL method. As investigating generation results but not training schemes here, we calculate the distance using only BooT-o algorithm, which is sufficient for analysis. RMSE stands for the Root Mean Squared Error value of the trajectory difference, and MMD stands for the Maximum Mean Discrepancy of two trajectory sets. More details can be found in Appendix D.1.
Table 4: Comparison of different distance metrics generated trajectories and original trajectories. list the mean value on datascts.
We calculate
Metric
TT S4RL
BooT-o, AR
BooT-o, TF
Avg. performance
71.1
81.9
79.9
RMSE
16.2
8.9
Dutuset
MMD (x (x100")
8.7
26.2
9.8
Environment
RMSE MMD 10
20.0
14.8
14.7
33.1
7.6
12.2
In Table the results of Dataset show that BooT produces trajectories with smaller RMSE compared to adding random noise, which demonstrates that generated trajectories generally lie in neighborhood of the corresponding original offline trajectories in Euclidean space. Also, trajectories generated by BooT have larger MMD compared with the baseline, showing that generated data are different from original offline trajectories w.r.t. the data distribution. This indicates that BooT expands the original offline data, and the visualization in Figure 2 also verifies this claim. But is such expansion reasonable? The answer is yes and is supported by Environment results, which show that the distance between the generated trajectories and real trajectories from the evaluation environment is relatively smaller compared with the baseline, demonstrating that the generated trajectories are more consistent with the real distribution of the underlying MDP in RL tasks. Thus, BooT provides more reasonable expansion to the original offline data, and subsequently improves the performance of the model.
Furthermore, in Figure 2 we visualize the distribution of (i) transitions (8, a, R) in the original data, (ii) data from teacher-forcing generation, and (iii) data from autoregressive generation in BooT. We excerpt the last T" transitions from these trajectories, reduce them to 2-dimension via t-SNE [46] altogether, and plot them separately. We also provide more visualization in Appendix D.2.
It is clearly illustrated that a large portion of data generated by the teacher-forcing method overlaps with the support of the original dataset, while those generated autoregressively lie out of the original data distribution. The overall results demonstrate that generated trajectories from BooT expand the data coverage while still keeping consistency with the underlying MDP of the corresponding RL task, thus resulting in better performance compared to the baselines. This also suggests why BooT does not require extra mechanisms such as conservatism to work well.
5.4
Ablation on Hyperparameters
80
80
Average Score
78
78
76
Avcrage
76
74
74
72
BooT-o, AR
BooT-o, IF
BooT-o, AR
BooT-o, TF BooT-r,
70
RoaT-r, AR
BoaT-r,
72
022
0.4
0.6
0.8
(BoeT-o) 256
4%
6%
8%
10%
(BooT--1)0.22
0.4%
0.6%
0.8%
1.0%
9%
Figure 3: Ablation experiment results on bootstrap epoch threshold shown in Algorithm T
Figure 4: Ablation experiment results on genera- tion percentage 77% as shown in Algorithm] |
When start to perform bootstrapping? We performed an ablation study on generation threshold k. which determines the threshold to start performing bootstrapping. For & = 0.2, BooT will perform bootstrapping after the model trained without generating sequences for 20% of training epochs. As is shown in Figure 3 the performance of BooT will first increase and then decrease as k increases. It reasonable that bootstrapping starts too early, the model will generate sequences that differ from the training dataset, and accumulate more error in the following training procedure. If bootstrapping starts too late, the model will learn little from bootstrapping as the model may have converged. Thus, an appropriate value of k could be selected.

---

How many new trajectories to generate in bootstrapping? We conduct an ablation study on generation percentage 17%, which is used control the ratio of the generated trajectories to the original offline trajectories. As shown in Figure the performance of Boo'T first increases and then decreases as v7% increases. For a small 17% value, number of total training steps on generated data might be too small to effectively improve the performance of the model. However, large 17% value may involve too much generated data with a low confidence score. As explained in Sec. 4.2 this could cause accumulating learning bias and results in performance drop.
How long should new trajectories be generated? We perform an additional ablation study on generation length T', which is used to control the length of generated sequences. As shown in Figure 5 the performance BooT with TF generation generally decreases T increases, and most results are better than the baselines. Furthermore, the performance of BooT-- is relatively robust to the changes TV. As the number of tokens per timestep differs among different environments, difficult to determine the most suitable 7' for all environments. Besides, T 1 performs the best in experiments, so we think is sufficient to simply set T for most cases.
80-
Boo
TT retrained w loss-weighted (luqpst) data
Avenge Score
78
Average Soore
[T retrained with loss weighod (lowest) data
76
TT (Original)
74
(Original)
70
TT (Reprodacc)
72-
TT (Re train)
(Reproduce)
BooT-o, TF
BooT-r, TF
3
25
4%
6%
8%
10%
77%
Figure 5: Ablation results on generation length T" with TF generation, where original trajectory length T
Figure 6: TT-retrain results with the lowest largest 17% loss in each batch. We report the mean results corresponding 15 random seeds.
5.5 Experiments of TT Retrained with Loss-weighted Data
We implement additional TT baseline with loss-weighted retraining, to experiment whether the generated data are better for bootstrapping. In the experiment, we add a simple loss-weighted re-train component for TT-retrain, choosing the data with the lowest largest 17% loss in each batch to re-train the model, to better compare with TT and TT-retrain baselines. The results are shown in Figure 6 For supplementary, the numerical results are provided in Appendix C.4.
The results have illustrated that using leighweighted re-train does help to improve the performance of the model, but BooT still performs much better than both schemes of loss-weighted re- re-train. Compared to simple random re-training scheme, re- re-training the model with data corresponding to the lowest loss helps improve the performance of the model, while using data corresponding to the largest loss has less improvement or even possibly degrades the performance.
6 Conclusion and Future Work
In this paper, we consider the data coverage issues encountered in offline RI scenarios and propose bootstrapping algorithm with sequence modeling paradigm to self-generate novel trajectory data and in turn feed them back to boost sequence model training. We adopt two sequence generation schemes, and each of them generates trajectories with unique characteristics compared to the original offline data distribution. Compared to the other strong baselines, the experiments over the public offline RL benchmarks have illustrated the great effectiveness our proposed method, and the analysis also provides some insightful evidence about how it works.
One drawback of our method is the relatively larger training time consumption compared to other methods because of online pseudo training data generation. In the future, we plan to improve the efficiency by incorporating more advanced techniques such as non-autoregressive sequence generation 12 35), improve the training efficiency.
10

---

References
[1] Arthur Argenson and Gabriel Dulac-Arnold. Model-based offline planning. In ICLR, 2021.
[2] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Newr/PS, 2015.
[3] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In NeurlPS, 2021.
[4] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
[5] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
[6] Linhao Dong, Shuang Xu, and Bo Xu. Speech-transformer: a no-recurrence sequence-to- sequence model for speech recognition. In ICASSP. 2018.
[7] Linxi Fan, Guanzhi Wang, De-An Huang, Zhiding Yu, Li Fei-Fei, Yuke Zhu, and Animashree Anandkumar. Secant: Self-expert cloning for zero-shot generalization of visual policies. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1088-3099, PMLR, 18 18-24 Jul 2021.
[8] Yuchen Fang, Kan Ren, Weiqing Liu, Dong Zhou, Weinan Zhang, Jiang Bian, Yong Yu, and Tie- Yan Liu. Universal trading for order execution with oracle policy distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 107-115, 2021.
[9] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072199 2020.
[10] Scott Fujimoto and Shixiang Gu. A minimalist approach to offline reinforcement learning, In NeurlPS, 2021.
[11] Scott Fujimoto, David Meger, and Doina Precup. Off- deep reinforcement learning without exploration. In ICML, 2019.
[12] Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non- autoregressive neural machine translation. arXiv preprint arXiv:1711.0228 2017.
[13] Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. In Advances in Neural Information Processing Systems, volume 34, pages 3680-3693. Curran Associates, Inc., 2021.
[14] Nicklas Hansen and Xiaolong Wang. Generalization reinforcement learning by soft data augmentation, 2020.
[15] Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie- Yan Liu, and Wei- Ying Ma. Dual learning for machine translation. NeurlPS, 2016.
[16] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.
[17] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXiv:2201.07207, 2022.
[18] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, volume 32, 2019.
[19] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 2021.
11

---

[20] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: model-based offline reinforcement learning. NeurIPS, 2020.
[21] Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofrr Nachum. Offline reinforcement learning with fisher divergence critic regularization. In ICML, 2021.
[22] Ilya Kostrikov, Ashvin Nair, and Sergey Levine Offline reinforcement learning with implicit q-leamnig. arXiv preprint arXiv:2110.06169) 2021.
[23] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In NeurlPS, 2020.
[24] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In ICLR, 2019.
[25] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive unsupervised repre- sentations for reinforcement learning. In Proceedings of the 37th International Conference Machine Learning, volume 1 119, pages 5639-5657 PMLR, 13-18 Jul 2020.
[26] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto. Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. In NeurIPS, 2020.
[27] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0166 2020.
[28] Shuang Li, Xavier Puig, Yilun Du, Clinton Wang, Ekin Akyurek, Antonio Torralba, Jacob Andreas, and Igor Mordatch. Pre-trained language models for interactive decision-making. arXiv preprint arXiv:2202.01777/ 2022.
[29] Zhiwei Liu, Ziwei Fan, Yu Wang, and Philip S Yu. Augmenting sequential recommendation with pscudo-prior items via reversely pre-training transformer. In Proceedings of the 44th international ACM SIGIR conference Research and development in information retrieval, pages (608-1612) 2021.
[30] Chaochao Lu, Biwei Huang, Ke Wang, José Miguel Hernández-Lobato. Kun Zhang, and Bernhard Schölkopf. Sample-effciient reinforcement learning via counterfactual bassd data augmentation, 2020.
[31] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In ICLR, 2017.
[32] Janner Michael, Li Qiyang, and Levine Sergey. Offline reinforcement learning as big sequence modeling problem. In NewrlPS, 2021.
[33] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 2015.
[34] Doina Precup. Eligibility traces for off- off-policy policy evaluation. Computer Science Department Faculty Publication Series, page 80, 2000.
[35] Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang, Lin Qiu, Weinan Zhang, Yong Yu, and Lei Li. Glancing transformer for non-autoregressive neural machine translation. arXiv preprint arXiv:2008.09905. 2020.
[36] Rongjun Qin, Songyi Gao, Xingyuan Zhang, Zhen Xu. Shengkai Huang. Zewen Li, Weinan Zhang, and Yang Yu. Neorl: A near real-world benchmark for offline reinforcement learning. arXiv preprint 102.:2202.007/11 14, 2021.
[37] Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Auto- matic data augmentation for generalization reinforcement learning. Advances Neural Information Processing Systems, volume 34, pages 5402-5415. Curran Associates, Inc., 2021.
12

---

[38] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforee- ment Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018.
[39] Machel Reid, Yutaro Yamada, and Shixiang Shane Gu. Can wikipedia help offline reinforcement learning? arXiv preprint wrXi::221.112122 2022.
[40] Kan Ren, Jiarui Qin, Yuchen Fang, Weinan Zhang, Lei Zheng, Weijie Bian, Guorui Zhou, Jian Xu, Yong Yu, Xiaoqiang Zhu, et al. Lifelong sequential modeling with personalized memorization for user response prediction. In SIGIR, 2019.
[41] Martin Riedmiller. Neural fitted q iteration-tirst experiences with data efficient neural reinforcement learning method. In European conference on machine learning, pages 317-328. Springer, 2005.
[42] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 2017.
[43] Samarth Sinha, Ajay Mandlekar, and Animesh Garg. S4RL: surprisingly simple self superviserviion for offline reinforcement learning in robotics. In CoRL, 2021.
[44] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NeurlPS, 2014.
[45] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[46] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[48] Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1989.
[49] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In CVPR, 2020.
[50] Guan Yang, Minghuan Liu, Weijun Hong, Weinan Zhang, Fei Fang, Guangjun Zeng, and Yue Lin. Perfectdou: Dominating doudizhu with perfect information distillation. arXiv preprint arXiv:2203.164006 2022.
[51] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data autteneeeeetedd reinforcement learning. arXiv preprint 2177v22107.09645 2021.
[52] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: deep reinforcement learning from pixels. In ICLR, 2021.
Regularizing
[53] Tianhe Yu, Garrett Thomas, Lantao Yu. Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: model-based offline policy optimization. In NeurIPS, 2020.
[54] Weinan Zhang, Zhengyu Yang. Jian Shen, Minghuan Liu, Yimin Huang, Xing Zhang, Ruiming Tang, and Zhenguo Li. Learning to build high-fidelity and robust environment models. In Joint European Conference on Machine Learning and Knowledge Discovery in Databuses, pages 104-121. Springer, 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We described in Sec. 6
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] We provide reproducible code for our work in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] They are specified in Appendix A.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Our experiment results are provided with mean and std. values.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] They are included in Appendix A.2.
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] They are specified and cited in Sec.5.11
(b) Did you mention the license of the assets? [Yes] They are specified in Appendix B.
(c) Did you include any new assets either the supplemental material or as a URL? [Yes]
We provide reproducible code for work in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "Vobo9qBR-3A", "reviewer": "Reviewer_i5fj", "summary": "This paper extends recent sequence-modeling approaches to RL with model-based data augmentation, with the end result looking like a Dyna-inspired [[Sutton 1991](https://dl.acm.org/doi/10.1145/122344.122377)] variant of the Trajectory Transformer. A few design decisions, such as the sampling strategy (autoregressive versus teacher-forced) and how often to reuse the model-generated data, are investigated. The algorithm is evaluated on the locomotion and adroit domains of the D4RL offline RL benchmark.", "questions": "1.  _\"However, since teacher-forcing generation is not able to expand dataset coverage as effectively as autoregressive generation, it will lead to less improvement in performance.\"_\n\nThis explanation is what I would have expected, but does it actually pan out in the results? Table 2 shows teacher forcing (TF) underperforming autoregressive (AR) generation with “-once” sampling, but (TF) outperforming (AR) generation with “-repeat” sampling. Overall, though, it’s all a bit of a wash, with the performance differences being on the order of 2%. I do appreciate the inclusion of these variants in the results section; I am just not sure that they support the claim that TF generation _“lead[s] to less improvement.”_\n\n2. On that note, are the authors surprised that the gap between AR and TF is so small, given the qualitative results in Figure 2 and distance metrics in Table 4?\n3. How does Figure 2 differ if, instead of plotting all of the original data in blue, you only plot the last $T$ trajectories? Since these occur at the end of trajectories, you might expect these to be disjoint from the rest of the original data even if they are not resampled by a model.\n\nCopy-editing:\nL334: _\"How much new trajectories\"_ $\\rightarrow$ \"How many new trajectories\"\n\n", "limitations": "As discussed in the “Strengths and Weaknesses” section, the main weakness is that BooT is a somewhat straightforward extension of existing work in model-based RL. However, the paper does a thorough investigation of this extension and does not overclaim its contribution.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper considers a relatively straightforward combination of existing algorithms, but does so thoroughly. It is careful in its empirical evaluations, with baseline variants including (a) TT with extra training to isolate the effects of BooT’s extra gradient steps and (b) an S4RL-modified version of TT to compare the model-generated data to other forms of data augmentation. The paper is also careful not to overclaim its novelty, and combined with its thoroughness, certainly executes on the premise it describes in the abstract and introduction.\n\nThe section on “Further Analysis of Bootstrapping” hints at a question I have myself wondered about: when a model is trained on the same underlying data as a policy but can then generate more data that further improves the policy, what about the model-generated data caused that improvement? The answer is somewhat heuristic and qualitative here, but this question has underrecognized subtlety, so I appreciated an attempt at addressing it empirically here. I do think that the introduction might have overpromised a bit here, saying that the _“generated pseudo trajectories… [keep] consistent with the underlying MDP”_. Most would interpret that as meaning “consistent with the initial state distribution and transition function”, but the investigation the authors conducted doesn’t actually take into account dynamical feasibility but inter-state distances. That being said, this is mostly a quibble about terminology, and any attempt at answering this question at all is a good start.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QoAVQzU37ap", "reviewer": "Reviewer_9mfT", "summary": "In order to tackle the problem that offline RL data is often small in size and lacks in coverage, this paper proposes to treat RL as a sequence modeling task and augment the training data by selecting the high confidence generated samples from the sequence model that is going through training. \n\nThe proposed method is evaluated on a subset of the D4RL continuous control tasks and is shown to outperform several baselines (including transformer-based and model-based ones as well as CQL and BC). for a bit over half of the tasks.\n", "questions": "* The method seems to be more stable than TT judging from table 1. Any idea why?\n* I wonder if the authors have done experiments where they use this only as an extra source of data and train CQL (or other offline-RL methods) with the extra data?\n* For teacher-forcing generation (l180-183), Is that only for one token, or one timestep, or something else? Either way it sounds quite limiting and the sampling method is yet another hparam that needs to be evaluated.\n* Did you perform experiments on other datasets in D4RL gym? E.g. the “random” and “expert” datasets.", "limitations": "There are quite a few additional limitations not mentioned in the paper (l349-350): 1. The extra two hyperparameters introduced k and η require finetuning, which depends on availability to the environment or a good OPE method. 2. As mentioned in l37-39, for other tasks in general, it is unclear whether the dataset available is sufficient to train a BooT, unless we try it, which will incur extra training time and cost, as mentioned in l349-350.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n* The algorithm's main body is simple and orthogonal to other offline-RL works. The only constraint is to use a sequence model. \n* Interesting analysis of the data distribution generated by teacher-forcing vs auto-regressive.\n* Clear description of the method.\n\nWeakness:\n* It is unclear whether the gain of BooT comes from 1. Extra data 2. Different architecture (pretrained gpt2 vs not) 3. Some inherent property in the sequence model as opposed to other world models that may only predict the observation and the reward.\n* It is unclear from the paper whether bootstrapping is novel beyond supervised learning (e.g. in RL)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KaMWJ_s_PFF", "reviewer": "Reviewer_hqzJ", "summary": "This paper proposes Bootstrapped Transformer, which uses bootstrapping to self-generate more offline data to improve sequence model training of Trajectory Transformer in offline RL setting. The authors study autogenerative generation and teacher forcing for trajectory generation in bootstrapping along with data filtering based on confidence score (prediction probability) and simple curriculum learning. They demonstrate better performance comparing to other offline RL baselines in D4RL dataset, specifically on Gym-Mujoco and Adroit. They also conduct analysis on bootstrapped data from the perspective of coverage/diversity and alignment with MDP, and an ablation to provide simple guideline of bootstrapping.", "questions": "Please check weakness in the previous section. Specifically, my major concerns lie in\n- Some straightforward potential improvement/alternative of baselines.\n- How does the proposed method perform under non-expert demonstration given bootstrapping greatly relies on the \"correctness\" of the original model?\n- Further discussion and empirical evidence for autoregressive generation.", "limitations": "Apart from large training time due to online pseudo training data generation mentioned by the authors, it would be nice to see more details or even failure mode resulting from instability of bootstrapping. The authors mentioned some of them in ablation as well as techniques to remedy such issue, but if any more, it would be very helpful to see all other failure strategies that the authors have tried.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is well-written and easy to follow.\n- The proposed method is well-motivated and the problem being attacked is important in using sequence modeling in offline RL.\n- The main results are in general quite good and support the major contribution of this work.\n\nWeakness\n- TT-reproduce is somewhat very different from TT-original; actually, it's worse in general. Thus I wonder if it's possible that TT-original with retraining can achieve better results.\n- Why S4RL used in the experiment only considers Gaussian noise but not also other augmentation techniques proposed in the original paper like dropout, mixup, etc.?\n- Regarding TT-retrain baseline, additional to uniformly sample from the original dataset for additional training, what about a simple weighted sampling based on loss or data likelihood?\n- The author claims in line 184-190 that autoregressive generation is better than teacher forcing due to its more diverse generation. In spite of the seemingly persuasive explanation, the marginal difference in Table 2 plus no BooT AR in Adroit experiment (Table 3; though the author mentions it is too time-consuming, I wonder if there is an alternative to provide more concrete justification) don't well justify the statement.\n- From Figure 2, AR and TF seem to be complementary to each other (with TF covers sparser regions in the original data distribution). I wonder combining these two bootstrapping techniques (with simple methods like randomly choosing one approach to generate data) will provide further improvement.\n- How does the proposed method perform under non-expert demonstration? Specifically, bootstrapping largely depends on the \"correctness\" of the original model (i.e., TT here). I wonder if we gradually mix less-performing data to the original training set (e.g., replacing some portion of dataset with \"random\" dataset in D4RL), how will this negatively affect bootstrapping?\n- While autoregressive generation can produce larger variation, how does the accumulated error affect bootstrapping? Autoregressive generation with larger generation length (T') produces data with larger deviation from the original dataset but at the same time suffers from larger accumulated error. Thus, T' may be an interesting hyperparameter for ablation study to see how to strike a good balance between the tradeoff.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept