Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes Bootstrapped Transformer, which uses bootstrapping to self-generate more offline data to improve sequence model training of Trajectory Transformer in offline RL setting. The authors study autogenerative generation and teacher forcing for trajectory generation in bootstrapping along with data filtering based on confidence score (prediction probability) and simple curriculum learning. They demonstrate better performance comparing to other offline RL baselines in D4RL dataset, specifically on Gym-Mujoco and Adroit. They also conduct analysis on bootstrapped data from the perspective of coverage/diversity and alignment with MDP, and an ablation to provide simple guideline of bootstrapping.
strengths_and_weaknesses: Strengths
- The paper is well-written and easy to follow.
- The proposed method is well-motivated and the problem being attacked is important in using sequence modeling in offline RL.
- The main results are in general quite good and support the major contribution of this work.

Weakness
- TT-reproduce is somewhat very different from TT-original; actually, it's worse in general. Thus I wonder if it's possible that TT-original with retraining can achieve better results.
- Why S4RL used in the experiment only considers Gaussian noise but not also other augmentation techniques proposed in the original paper like dropout, mixup, etc.?
- Regarding TT-retrain baseline, additional to uniformly sample from the original dataset for additional training, what about a simple weighted sampling based on loss or data likelihood?
- The author claims in line 184-190 that autoregressive generation is better than teacher forcing due to its more diverse generation. In spite of the seemingly persuasive explanation, the marginal difference in Table 2 plus no BooT AR in Adroit experiment (Table 3; though the author mentions it is too time-consuming, I wonder if there is an alternative to provide more concrete justification) don't well justify the statement.
- From Figure 2, AR and TF seem to be complementary to each other (with TF covers sparser regions in the original data distribution). I wonder combining these two bootstrapping techniques (with simple methods like randomly choosing one approach to generate data) will provide further improvement.
- How does the proposed method perform under non-expert demonstration? Specifically, bootstrapping largely depends on the "correctness" of the original model (i.e., TT here). I wonder if we gradually mix less-performing data to the original training set (e.g., replacing some portion of dataset with "random" dataset in D4RL), how will this negatively affect bootstrapping?
- While autoregressive generation can produce larger variation, how does the accumulated error affect bootstrapping? Autoregressive generation with larger generation length (T') produces data with larger deviation from the original dataset but at the same time suffers from larger accumulated error. Thus, T' may be an interesting hyperparameter for ablation study to see how to strike a good balance between the tradeoff.
questions: Please check weakness in the previous section. Specifically, my major concerns lie in
- Some straightforward potential improvement/alternative of baselines.
- How does the proposed method perform under non-expert demonstration given bootstrapping greatly relies on the "correctness" of the original model?
- Further discussion and empirical evidence for autoregressive generation.
limitations: Apart from large training time due to online pseudo training data generation mentioned by the authors, it would be nice to see more details or even failure mode resulting from instability of bootstrapping. The authors mentioned some of them in ablation as well as techniques to remedy such issue, but if any more, it would be very helpful to see all other failure strategies that the authors have tried.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In order to tackle the problem that offline RL data is often small in size and lacks in coverage, this paper proposes to treat RL as a sequence modeling task and augment the training data by selecting the high confidence generated samples from the sequence model that is going through training. 

The proposed method is evaluated on a subset of the D4RL continuous control tasks and is shown to outperform several baselines (including transformer-based and model-based ones as well as CQL and BC). for a bit over half of the tasks.

strengths_and_weaknesses: 
Strengths:
* The algorithm's main body is simple and orthogonal to other offline-RL works. The only constraint is to use a sequence model. 
* Interesting analysis of the data distribution generated by teacher-forcing vs auto-regressive.
* Clear description of the method.

Weakness:
* It is unclear whether the gain of BooT comes from 1. Extra data 2. Different architecture (pretrained gpt2 vs not) 3. Some inherent property in the sequence model as opposed to other world models that may only predict the observation and the reward.
* It is unclear from the paper whether bootstrapping is novel beyond supervised learning (e.g. in RL)
questions: * The method seems to be more stable than TT judging from table 1. Any idea why?
* I wonder if the authors have done experiments where they use this only as an extra source of data and train CQL (or other offline-RL methods) with the extra data?
* For teacher-forcing generation (l180-183), Is that only for one token, or one timestep, or something else? Either way it sounds quite limiting and the sampling method is yet another hparam that needs to be evaluated.
* Did you perform experiments on other datasets in D4RL gym? E.g. the “random” and “expert” datasets.
limitations: There are quite a few additional limitations not mentioned in the paper (l349-350): 1. The extra two hyperparameters introduced k and η require finetuning, which depends on availability to the environment or a good OPE method. 2. As mentioned in l37-39, for other tasks in general, it is unclear whether the dataset available is sufficient to train a BooT, unless we try it, which will incur extra training time and cost, as mentioned in l349-350.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper extends recent sequence-modeling approaches to RL with model-based data augmentation, with the end result looking like a Dyna-inspired [[Sutton 1991](https://dl.acm.org/doi/10.1145/122344.122377)] variant of the Trajectory Transformer. A few design decisions, such as the sampling strategy (autoregressive versus teacher-forced) and how often to reuse the model-generated data, are investigated. The algorithm is evaluated on the locomotion and adroit domains of the D4RL offline RL benchmark.
strengths_and_weaknesses: This paper considers a relatively straightforward combination of existing algorithms, but does so thoroughly. It is careful in its empirical evaluations, with baseline variants including (a) TT with extra training to isolate the effects of BooT’s extra gradient steps and (b) an S4RL-modified version of TT to compare the model-generated data to other forms of data augmentation. The paper is also careful not to overclaim its novelty, and combined with its thoroughness, certainly executes on the premise it describes in the abstract and introduction.

The section on “Further Analysis of Bootstrapping” hints at a question I have myself wondered about: when a model is trained on the same underlying data as a policy but can then generate more data that further improves the policy, what about the model-generated data caused that improvement? The answer is somewhat heuristic and qualitative here, but this question has underrecognized subtlety, so I appreciated an attempt at addressing it empirically here. I do think that the introduction might have overpromised a bit here, saying that the _“generated pseudo trajectories… [keep] consistent with the underlying MDP”_. Most would interpret that as meaning “consistent with the initial state distribution and transition function”, but the investigation the authors conducted doesn’t actually take into account dynamical feasibility but inter-state distances. That being said, this is mostly a quibble about terminology, and any attempt at answering this question at all is a good start.

questions: 1.  _"However, since teacher-forcing generation is not able to expand dataset coverage as effectively as autoregressive generation, it will lead to less improvement in performance."_

This explanation is what I would have expected, but does it actually pan out in the results? Table 2 shows teacher forcing (TF) underperforming autoregressive (AR) generation with “-once” sampling, but (TF) outperforming (AR) generation with “-repeat” sampling. Overall, though, it’s all a bit of a wash, with the performance differences being on the order of 2%. I do appreciate the inclusion of these variants in the results section; I am just not sure that they support the claim that TF generation _“lead[s] to less improvement.”_

2. On that note, are the authors surprised that the gap between AR and TF is so small, given the qualitative results in Figure 2 and distance metrics in Table 4?
3. How does Figure 2 differ if, instead of plotting all of the original data in blue, you only plot the last $T$ trajectories? Since these occur at the end of trajectories, you might expect these to be disjoint from the rest of the original data even if they are not resampled by a model.

Copy-editing:
L334: _"How much new trajectories"_ $\rightarrow$ "How many new trajectories"


limitations: As discussed in the “Strengths and Weaknesses” section, the main weakness is that BooT is a somewhat straightforward extension of existing work in model-based RL. However, the paper does a thorough investigation of this extension and does not overclaim its contribution.
review_title: null

Meta-review: metareview: ## Summary
Offline RL (RL) algorithms aim to learn policies without interacting with an environment purely from the state and actions covered in the offline datasets. However, in real-world datasets, the coverage can be insufficient to learn good policies. Thus it is an important research direction to improve the sample efficiency of those methods. This paper approaches offline RL from a sequence modeling perspective. The paper adopts a variant of trajectory transformers for data generation, and they investigate two of the main design decisions in those models:
* Sampling methods (autoregressive vs. teacher-forcing based).
* Reuse of model-generated data

The authors validate their idea on two D4RL tasks:
* adroit
* locomotion

## Decision

Overall the paper is well-written and easy to understand. The results and experiments are thorough, and the paper goes for more depth in the experiments rather than breadth. The ideas in the paper are not novel, but the paper does not overclaim its contributions, and results are interesting. As a result, I think both NeurIPS and the broader offline RL communities would benefit from the findings of this paper. I am nominating this paper for acceptance.

The reviewers were very positive about this paper during the rebuttal and discussion paper. They all agreed that the paper is valuable and interesting contribution to the community. The main criticism of this paper that came up during the discussion period was that the idea is just a straightforward combination of the existing techniques. However, the idea presented in the paper is coherent, reasonable, and executed well. 

The authors provided a very detailed rebuttal with clarifications to the points that reviewers raised. As a result of the rebuttal, some of the reviewers increased their scores. I would recommend that the authors incorporate some of those clarifications into the camera-ready paper version. Some of those are:

* In response to reviewer 9mfT’s question on the results with CQL on additional data generated by Boot is very interesting. I think the authors should include it in the camera-ready version of the paper. 
* Additional experiments on other datasets from the D4RL gym environment as asked by reviewer 9mfT.
* The experiments asked by the reviewer *hqzJ*.

confidence: Certain
Decision: Accept