Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed using value bootstrapping to augment the return data for learning decision transformers or RvS (RL via supervised learning) in general. This approach, which performs explicit bootstrapping / stitching, allows mapping optimal actions to optimal return-to-gos that do not appear in the actual experience and thus achieve better performance. 
strength_and_weaknesses: Strength:
* Very interesting and novel proposal that addresses the critical bottleneck of decision transformers (DT) or RvS of not being able to infer the optimal return by bootstrapping. I am not aware of previous works that successfully combine DT/RvS and TD-bootstrapping.
* The empirical results on D4RL are very impressive, especially on AntMaze, where an optimal policy must stitch together behaviors from several suboptimal demonstrations, from almost zero to reasonable performance.

Weakness
* How quantile regression was perform and how bias was added into sampling (specifically the implementation related to Dabney et al. (quantile regression) and Lee et al. (multi-game decision transformers)) was not clearly described. I suppose these could be very important to the stability of offline TD-learning. I would like to see more details and potentially more sensitivity analysis on these. 
* Many details are missing: Hyper-parameters related to the above point are missing. Network architecture is also missing. Although the reader can refer to code from previous work, it's better to make the paper self-contained and improve reproducibility.
clarity,_quality,_novelty_and_reproducibility: I believe the contribution is novel and writing quality is good. As pointed out in "weakness", I think there are rooms for improvements for clarity and reproducibility. I would also suggest open-sourcing the code to improve reproducibility. 
summary_of_the_review: I lean to acceptance. However, I would like to see if the authors can improve on clarity of implementation details, provide more insight from sensitivity / ablation studies, and increase reproducibility.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a data augmentation method for return-conditioned supervised learning that takes into account temporal compositionality. The proposed method shows higher performance compared to vanilla baselines.
strength_and_weaknesses: Strengths
- The paper deals with an important problem of return-conditioned supervised learning methods, which is that they cannot stitch trajectories to find optimal behavior.
- The proposed return augmentation method does seems to improve performance on various benchmarks.

Weaknesses
- The source code is not provided, which makes the credibility of the work questionable.
- As the authors note, a major advantage of the return-conditioned SL methods is that they do not require value bootstrapping and thus does not fall into the infamous 'deadly triad'. However, adopting the return augmentation with iterative value updates can reintroduce this bootstrapping problem.
- The return conditioning procedure is not explained in detail even though the algorithm's performance depends heavily on it (Table 2). How is the value model V_\phi used to sample "high, but plausible returns for the current episode"? Also, Appendix B.2 notes that the targeted value generated from the value model is increased by some \Delta. How is this \Delta tuned? Overall, how much does each design choice affect the performance?
- The performance of the proposed method is highly sensitive to N, the number of bootstrapping iterations, especially on Antmaze.
- The D4RL benchmark results are averaged over only 3 seeds. Adding at least 1~2 more seeds would be more credible.

Questions
- What does the numbers on Figure 3 (b) mean? Does it refer to the expected value (from V_\phi) ?

############## Post-rebuttal comment ##############

I appreciate the authors for updating the manuscript, but there still are some remaining concerns.

First, the hyperparameter tuning procedure is still somewhat ambiguous. The tuning process (search range, selection method, ...) for \Delta is not included in the paper. Also, the authors mention that "each method has one set of hyperparameters shared across all tasks" on the updated Table 1, but this is wrong since the proposed method tunes \Delta per task.

Second, I agree that the work can be interpreted as unifying value-based methods and supervised-RL, but given the empirical results, I think the proposed method fails to achieve the best of both worlds (the proposed method outperforms value-based baselines on only Antmaze). I encourage the authors to improve the method so that the cons from both approaches are minimized while the pros are maximized.
clarity,_quality,_novelty_and_reproducibility: Mentioned above.
summary_of_the_review: The paper provides a data augmentation method for return-based supervised learning in offline RL. While finding an appropriate data augmentation technique for return-based supervised learning (and in general, offline RL) is important, the proposed method seems to have some drawbacks.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The submission introduces SuperB, a new approach for offline RL following the RL via Supervised Learning (RvS) framework. SuperB addresses a shortcoming of RvS methods, which fail to stitch together trajectories that were executed as parts of non-optimal trajectories in the data, but which would result in an optimal (or high-performing) trajectory if combined into a single trajectory. The approach is simple conceptually: train a return model to estimate the distribution of returns from any given state, and use that to compute potentially higher returns for states in the data set that would be attained if a different trajectory had been followed thereafter, thereby augmenting the data with better trajectories. The authors show theoretically a necessity result (that n-steps returns are necessary for the data set to contain the optimal return if the data-generating policy is Markovian) and a coverage result (that n-step returns enable generating up to exponentially many unseen trajectories). Empirically, SuperB performs competitively to value-based RL and outperforms RvS methods in settings where the data-generating policy is non-Markovian. 

strength_and_weaknesses: ########### Strengths ###########
1. The paper is clearly written and easy to follow. The motivation is specified clearly from the beginning and repeated/clarified throughout the paper
2. The proposed SuperB approach is simple conceptually, and is a reasonable solution to handle the problem of trajectory stitching from offline data, as demonstrated theoretically and empirically
3. The idea of using a return model to stitch trajectories, instead of attempting to explicitly find trajectories in the data that could be stitched, is clever

########### Weaknesses ###########
1. The theoretical results might be somewhat overstated, and they appear to fall out by construction. They do provide some grounding to validate the proposed solution, so I do think they should be kept in the paper, if perhaps rephrased and downplayed
2. The resulting approach is not particularly strong compared to baselines, especially value-based approaches. While state-of-the-art performance shouldn't be the focus of publications, it is important to discuss and analyze where these ideas might take us in the future if pursued further
3. Some details about the experimental setting are missing and would be necessary for ensuring a fair comparison

clarity,_quality,_novelty_and_reproducibility: The paper is very clearly written, and the results are correct and somewhat interesting. The baseline results mostly come from published results, but the authors should release their code to enable reproducibility of their own results. The proposed SuperB approach is surprisingly simple, especially as compared to current value-based offline RL methods, which require complex combinations of Q-learning-or-Actor-Critic-style approaches with distributional reasoning. To me, this is the biggest selling point of the paper: the simplicity of the approach makes it easy to understand, implement, and potentially build upon. 

I have multiple concerns about the paper, which make me be mostly on the fence about recommending acceptance. I hope the authors take the time to engage in the discussion and improve their manuscript during the discussion period.

My primary concern regards the motivation for the proposed solution. The authors mention that the main advantages of RvS are "simple training objective, robustness to hyperparameters, and strong performance". However, they also state that value-based RL tends to have higher performance (as demonstrated also in the expeirments of Section 3), and it is unclear in practice whether SuperB in particular is robust to hyperparameters (more on that later). With this, I wonder exactly what the need for another RvS method is, especially as the shortcoming that this paper addresses (RvS's inability to stitch suboptimal trajectories into optimal trajectories) is already handled "for free" by value-based methods. Since we are not gaining improved performance nor (potentially) robustness to hyperparameters, then what is the advantage of RvS with respect to value-based methods? Or, if there are none so far, what are potential advantages in the future of methods that follow a line of work starting from SuperB? Is it only the method's simplicity? And if so, at what cost does it come? Is there a potential path towards methods with the performance levels of value-based offline RL that maintains the simplicity of SuperB?

I also think perhaps the theoretical results are somewhat overstated. 
- The authors explicitly state in Section 2.5 that they "show formally how SuperB is necessary to learn optimal policies for some sets of offline RL problems." Similar language is used throughout the paper when referring to Proposition 1. However, what proposition 1 shows is that n-step bootstrapping is necessary for constructing a data set that contains the optimal returns given a state-action pair. This is a substantially weaker result. First, SuperB is not the only conceivable way to incorporate the n-step returns; indeed, the authors themselves state that they made several design choices along the way, which necessarily implies that there are alternatives. Second, not containing the optimal return in the data does not imply that it is not possible to learn optimal policies. Broadly speaking, there are surely other algorithms that could achieve the "temporal compositionality" that SuperB enables (trivially, as the authors mention, value-based methods satisfy that criterion). I think what the authors most likely mean is that temporal compositionality (generally speaking) is needed for RvS learners to achieve optimal performance (in the context of non-Markovian policies, which is clearly stated). This seems to be true almost by definition. That's not to say that the result isn't useful, just that it should be qualified accordingly throughout the paper. 
- The second theoretical results seems a bit less useful. First, the result seems to not require a lot of insight: if trajectories overlap, stitching them together leads to more possible trajectories. But more importantly, the simplified chain MDP used to develop this result restricts the amount of insight that we gain from the result. Sure, in the limit, we might obtain an exponential advantage. But, as the authors mention, in the opposite end of the spectrum, we might get no advantage if there is no overlap between trajectories. How are we to interpret what exists in-between? A far more interesting result would consider a more general MDP, and provide results in terms of some intrinsic property of the MDP and data-generating policy to capture some measure of trajectory overlap, and state the gains of trajectory stitching in this setting. Is such a result feasible, or are we doomed to only know what happens at the extreme ends of the space of MDPs?

In terms of the experiments, I have one subtantial concern: how is \Delta, mentioned in the final sentence of the appendices, tuned? The authors do not mention this anywhere, and I think it is critical towards assessing the fairness of the empirical evaluation. How does the authors' hyperparameter tuning compare to the baselines'? Are the authors evaluating the resulting policy online in the environment or using some form of offline performance estimation? If it is the former, I encourage the authors to look at e.g. [1]. Moreover, as the authors stated that robustness to hyperparameters is one of the key advantages of RvS, I would like to see some discussion of how the need to tune \Delta relates to the claimed robustness. Say, if we chose not to tune it, but to set it manually, how would we go about doing that, and how would that affect the obtained results? Similarly, the results seem to show a substantial sensitivity to the number of iterations (N) hyperparameter. How would one go about choosing N more generally? 

Also in the experiments, one ablation that I think would have been nice to include is a bootstrapped version of the value function. The authors made the choice to use Monte Carlo samples to train the value function, and there's some reasonable motivation behind that choice. But, was that choice validated empirically? Could we swap in a Q-learning method, or perhaps even a full-fledged off-line RL method, in place of the simple Monte Carlo value estimate? Would we gain anything from such a combination?

############## Additional feedback ##############

The following points are provided as feedback to hopefully help better shape the submitted manuscript, but did not impact my recommendation in a major way.

I would recommend against the use of the term "temporal compositionality", which is commonly used to talk about combining skills with some higher level policy guiding how skills are composed. While closely related, these two are not the same. The term "stitching" seems more appropriate. 

Intro
- The intro is very clearly written and gives a good understanding of where the paper is going

Sec 2.2
- I wonder if the reason BCQ and CQL beat BC is truly related to the ability to compose behaviors. BC doesn't train a sequence model (usually, at least), and thus is also able to compose trajectories if they intersect in the data set. Of course, it will just learn to replicate the most commonly observed a given s, so if that corresponds to a poor behavior, then that will be the chosen a. But, this seems to be an issue due to ignoring the reward during training, and not necessarily of compositionality of trajectories.

Sec 2.4
- How is the original data augmented with returns? It would seem that s,a pairs from earlier in the trajectory would have higher returns. Or are these somehow normalized?

Sec 3
- Some results don't quite seem to validate some of the claims in the paper. For example, CQL is worse than all RvS methods on AntMaze, which is supposed to require temporal compositionality.
- What is the point of the reward transformation ablation? It's not really a design choice of SuperB, and it already follows standard practice from prior work.
- Could the authors better clarify exactly how V is used to sample trajectories for evaluation, and what the ablated version in Table 2 does instead? The former should be included in Section 2, as it is an important part of the algorithm itself, which wasn't mentioned until Section 3.5, and only somewhat clarified in the appendices. 

Sec 4
- [2] may be a relevant cite for the final paragraph of Sec 4, though it certainly addresses a different problem setting than SuperB

Typos/style/grammar/layout
- Sec 3.5, "Dynamic target return selection": necessarily need -> need
- Appendix B.2: neurial -> neural

[1] Le Paine et al. "Hyperparameter Selection for Offline Reinforcement Learning". 2020

[2] Yu et al. "Conservative Data Sharing for Multi-Task Offline Reinforcement Learning". 2021

summary_of_the_review: Overall, I think this submission is a good contribution and I'm leaning towards recommending its acceptance. My main concerns relate to the motivation for the proposed approach, the fairness of the experimental evaluation, and the somewhat overstated theoretical results. I would like to see the authors' response to these and would be glad to revise my recommendation accordingly.

############# Update after rebuttal #############

I am updating my score from 6 (marginally above threshold) to 5 (marginally below threshold) per the discussion with the authors below.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors observed that RvS, their main baseline method, does not do well on Antmaze when not given privileged information such as the goal location. They hypothesized that by modifying the reward to do bootstrapping, similar to other methods based on temporal difference learning(e.g. CQL, BCQ), that will allow the RvS methods to do better. 
They observed no significant benefit on halfcheetah, hopper, and walker2d tasks (apart from the random data split which benefits more from trajectory stitching), but they outperformed the previous SOTA in 3 of the 6 mazes in AntMaze.

strength_and_weaknesses: 
The paper has a clear motivation and diagnosis for the problem they are trying to tackle, a sound logic for their proposed solution (SuperB) accompanied by a proof. They showed that their proposed method improves over the baseline RvS method (the one without privileged goal info) by a significant margin. 

The main weakness is where else this approach can be used. As shown in Table 1, it does not result in SOTA performance unless it is clear from the dataset and problem setup that an agent will benefit from stitching together multiple trajectories.  It is also assumed that different trajectories will visit similar states, allowing the bootstrapping to be effective. SuperB is designed specifically for those assumptions and works well on AntMaze medium and umaze. It is less clear why the method works less well on antmaze large compared to IQL.
The authors are aware of the issues of their proposed method and as mentioned in 3.4, “there is no universally optimal number of iterations”, meaning that one needs to either use domain knowledge (like in RvS-G) or do large scale hyperparameter sweeping to find the optimal setup. 
Nevertheless, the method looks like a strict improvement over the baseline RvS on the D4RL dataset, and assuming reasonably that the state representation is reasonably good, there does not seem to be a strong downside to swapping out RvS rewards for the SuperB rewards. 


clarity,_quality,_novelty_and_reproducibility: 
The description of their experiment looks clear to me and their result sounds reasonable. In terms of novelty, I would argue that this work is an incremental improvement over the existing RvS method as opposed to a fundamental change in ways of thinking or in the SOTA numbers. Even though there are no clear downsides over RvS, in order for this method to really outperform the baseline, one needs domain knowledge about the task to tune the number of bootstrap iterations. 

summary_of_the_review: This paper proposes SuperB, an incremental improvement over RvS where they introduced bootstrapping to the target reward. The method is shown to work well where it is known that the offline data contains multiple imperfect trajectories that can be combined together to form a better one. I recommended a rating of 6 due to their sound experimental results and due to the limited novelty, but also due to the paper’s thorough discussion of the method’s limitations and future works.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper addresses a problem of data sparsity in offline reinforcement learning, which results in typically low performance in reward ranges that have not been observed in the training data. The authors proposes an approach called SuperB, which augments data by combining intersecting trajectories. Empirical validation is performed on D4RL.

Reviewers found the paper well written, with an important and well-motivated approach that addresses a key limitation of RL as Supervised learning (RvS) approaches. The approach makes "clever" use of a return model and achieves encouraging results, especially on the AntMaze domain.

Initial reviews noted a number of concerns, some of which have been addressed in the rebuttal phase. More clarity was needed on the description of components of the approach (e.g., how quantile regression was applied) and for the paper to be self-contained. Some felt that the provided theoretical results were over-stated. Reviewers raised questions about the sensitivity to hyperparameter N (the number of bootstrap iterations), and raised the need to better understand limitations of the proposed approach compared to value-based approaches (e.g., relatively lower empirical performance on MuJoCo tasks).

Authors replied in detail, updating the paper with clarifications, providing access to the source code, and adding experimental details to the appendix. They addressed concerns about hyperparameter sensitivity by moving a corresponding table to the appendix, and arguing that the table gave a misleading impression. 

Some of the reviewers concerns were addressed in the rebuttal, but some of the concerns remained. In particular, the hyperparameter tuning procedure was still deemed ambiguous, and the choice of moving sensitivity analysis of the effects of N was not considered an appropriate choice. Reviewers argued that claims about the robustness of the approach to hyperparameters should be removed given the lack of evidence. Additional questions focused on a more detailed understanding of the claim that the method provided a "best of both worlds" approach between value-based and RvS methods, both  conceptually and empirically. An error analysis of SuperB on an environment where it achieves relatively lower performance has potential to generate valuable insights.
justification_for_why_not_higher_score: The paper presents an interesting and novel approach, but important remaining limitations are noted by reviewers. The procedure for hyperparameter tuning remains unclear, and authors should remove the claim of robustness to hyperparameters. The method performs relatively lower on some of the MuJoCo tasks, and an error analysis would provide valuable additonal insights.
Decision: Reject