Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a model-based data augmentation method (CABI) for offline RL, which uses the agreement of forward and backward models to assess the validity of transitions. It proceeds as follows:
1. Train forward and backward dynamics models, and forward and backward rollout policies based on CVAEs
2. Use models and rollout policies to generate transitions
3. Apply the backward model to transitions which were generated from the forward model, and vice-versa. Sort the transitions by the deviation of the predicted state from the original state, and keep only the transitions with relatively low deviation.

After generating these transitions, any model-free offline RL algorithm can be applied to the augmented dataset. Experiments on the D4RL benchmark demonstrate that the data augmentation is beneficial, improving over the base offline RL algorithm, in almost every case.
strengths_and_weaknesses: Strengths
* The proposed double check is a novel idea, to my knowledge.
* The paper is clearly written.
* Experimental evaluation is thorough and demonstrates the performance benefits of CABI across a variety of datasets. The ablation study also shows the benefit of the double check compared to ensemble variance as an uncertainty quantifier.

Weaknesses
* Hyperparameter tuning: the appendix states that “The best ratio η strongly depends on the dataset and may need to be tuned manually.” One must also tune the rollout horizon.
* The improvement over the base offline RL algorithm, while consistent across tasks, is not large in most cases. This may be because the generated samples are very similar to the original samples (since only in-support actions are tried).

questions: * “We resort to sorting the transitions in a mini-batch by the state deviation from small to large and keep the top k% of them.” I’m guessing this is a typo and you meant the bottom k%? (Since you want small deviations)
* What is the exact formula used to compute ensemble variance? This should be included in the paper or appendix.
limitations: Yes, they mention that computational cost is a limitation of the work.
I agree that there is not significant risk of negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a new offline model-based RL method that trains a bidirectional model for model rollouts and filters out model-generated states given by the forward model that cannot recover the previous real state from the backward model. By doing so, the method can effectively get rid of inaccurate model-generated states and mitigate the distributional shift problem in offline model-based RL. Through extensive experiments, the method is able to outperform prior offline RL methods on various tasks in D4RL.
strengths_and_weaknesses: **Pros:**

1. The method appears to be novel and addresses an important problem in offline RL, i.e. how to incorporate augmented data without introducing much distributional shift.

2. The results of the method on challenging domains such as Adroit are clearly better than the prior methods listed.

3. The paper is clearly written and easy to understand.

**Cons:**

1. The bidirectional models seem to assume that the action is reversible, which I think might not always hold in all MDPs. While the empirical results seem to suggest that incorporating such rollouts double-checked by the bidirectional models is helpful, there are definitely cases where the previous state is not possible to be inferred from the next state and the action, e.g. robotic manipulation situations that are contact-rich. I wonder if the authors can show the methods can also work in those situations (both empirically and theoretically).

2. The empirical evaluations are not that thorough. I believe one important baseline that is missing is S4RL, which also handles data augmentation in offline RL and achieves impressive results in adroit (even better results than the proposed method after skimming through their results table). The authors also didn't compare to MOReL, which is another important model-based offline RL method. The authors should also include evaluations in other domains such as antmaze and kitchen from D4RL.

3. It is a bit strange that the numbers of the baselines in the paper are different from/worse than ones in the original papers, e.g. CQL etc. I think it is important to show the original numbers to ensure fair comparisons.
questions: See cons in the above section.
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a method of augmenting an offline dataset for use with model-free offline methods by generating synthetic rollouts that are “double-checked” - i.e., validated by a bidirectional transition model. The paper powerfully demonstrates the principle on a toy environment, and the proposed enhancement leads to a significant performance gain for BCQ on Adroit but a more minor gain for TD3+BC on MuJoCo.

strengths_and_weaknesses: Strengths:
- Clearly articulated method for validating the trajectories generated by model rollouts, strong illustrative toy example
- Enables model rollouts to be integrated with standard offline model-free RL algorithms with strong performance on the Adroit environments
- Extensive ablations against different data generation procedures and action-augmentation schemes

Weaknesses:
- Limited gain on the MuJoCo environments which authors attribute to the BC term in TD3+BC, would it be possible to see a similar strong gain with an alternative such as CQL?

Minor:
- Line 116: perhaps too strong a claim, certain offline model-free algorithms exhibit strong performance (IQL, TD3+BC) without imagined rollouts
- Line 146: training for a fixed number of epochs could lead to over/underfitting, is there a validation dataset?
- The number of seeds should be listed in table captions.
- Evaluation protocol for determining offline performance should be described (e.g. policy taken at the end of training?)

questions: - Could standard offline model-based algorithms be used with the improved rollouts? Would there be a performance gain by only filtering out inaccurate forward rollouts in an algorithm like MOPO?
- Are there some states that are consistently discarded due to model inaccuracies? What proportion of states would be rejected with a 1-step rollout and double check?
- How was the ratio of real data to imagined trajectories tuned?

limitations: The authors discuss the limitations of their work well. Two points are:

- The authors address the limited gain on the MuJoCo-based environments, but could demonstrate the method is still useful by selecting a different algorithm to augment.
- [1, 2] show how inaccurate rollouts that are well penalized can still be useful for model-based training, and this point would be good to discuss.

[1] MOPO: Model-based Offline Policy Optimization Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, Tengyu Ma

[2] Revisiting Design Choices in Offline Model-Based Reinforcement Learning. Cong Lu, Philip J. Ball, Jack Parker-Holder, Michael A. Osborne, Stephen J. Roberts.
review_title: null

Meta-review: metareview: **Strengths**: The paper introduces a new and interesting idea of "double checking" with bi-directional models, and thoroughly evaluates the idea on a variety of offline RL datasets and through multiple ablations. 

**Weaknesses**: The main weaknesses seem to be that (1) some of the performance improvements are small, and (2) like prior methods, the method is heavily reliant on tuning a hyperparameter based on the quality of the dataset. It also is somewhat strange that the paper uses v0 datasets from D4RL, since the more recent versions have fixed bugs in the datasets. Otherwise, the author response did a good job at discussing and addressing the other reviewer concerns.

The reviewers and AC agree that the strengths outweigh the weaknesses, and would make a valuable addition to NeurIPS.
confidence: Certain
Decision: Accept