Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents a new algorithm (ROMI) for offline RL that is based on data augmentation using a learned reverse imagination model. This reverse model takes in a state and action, and attempts to infer the previous state that occurred before the query state. Concretely, it models $P(s, r | s', a)$. Data is collected using the learned reverse model and reverse rollout policy, and is added to the offline dataset, for data augmentation. Finally, a conservative model-free algorithm like BCQ or CQL is used for policy improvement using the augmented dataset. Results presented on D4RL benchmark tasks suggest that the developed algorithm achieves good (near SOTA) performance on a majority of tasks.
main_review: **Strengths:** Interesting idea of using a reverse model, which to my knowledge is novel. Empirical results are also quite strong, although there are a few issues as I note below. 

**Limitations and Open Questions**
- My biggest concern with the paper is there is no clear justification or reason for why a reverse model should work better than a forward model. In particular, taking a particular trajectory snippet, if learning $P(s, r | s', a)$ is possible, I fail to see why predicting $P(s', r | s, a)$ is any more difficult. Could it just be implementation details or is there something more fundamental? 
- Experiment results don't compare with MOReL and COMBO, which to my knowledge are the SOTA model-based algorithms. I think a comparison with them are warranted, since they have been public for a long time now.
- In particular, I see a lot of parallels between COMBO and ROMI, thereby necessitating detailed comparisons between them. Just like ROMI, COMBO also uses model rollouts to augment datasets for a conservative model-free algorithm (CQL in this case). The main differences between them are: (1) ROMI uses a reverse model while COMBO uses a forward model (2) ROMI performs the augmentation rollout using a fixed policy while COMBO data augmentation is adaptive with rollouts being performed using the current policy. Both these components require clear ablations to understand the relative gains -- which among (1) and (2) contribute most.
- The states from which FOMI and ROMI start the rollouts could also be quite important -- how did you make that choice for FOMI vs ROMI? 
- Figure 1b and its description in lines 123-130 is a bit misleading and uses a straw-man argument. A naive forward forward rollout with an arbitrary policy might indeed collide with the wall, but the same is true of reverse rollouts with arbitrary policies too. Additionally, conservative model-based methods (eg MOReL, MOPO) are explicitly setup to avoid going into regions where data support is unavailable, and thus in principle would flag colliding into the wall as a bad event.

Overall, while the work has some innovative ideas, it is unclear why the method works. Given the number of open questions and limitations in evaluations, I am unable to recommend acceptance at this stage.
limitations_and_societal_impact: There are some limitations and open questions, which I have outlined in my main review. The paper does not have a discussions or limitation section -- I would recommend the authors add such a section.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a method for offline RL based on reverse dynamics models. That is, rather than sampling trajectories by predicting $s’$ given $(s,a)$, they predict $s$ given $(s’,a)$. As a result, all the $s’$ in the augmented dataset are “real” states, reducing the effects of harmful extrapolation.
Once the reverse transitions have been generated (by combining the reverse dynamics model with a learned reverse policy based on CVAE), a policy can be trained on the augmented dataset using a model-free offline RL algorithm such as BCQ or CQL. This framework is referred to as Reverse Offline Model-based Imagination (ROMI). A practical instantiation of ROMI yields strong results on the D4RL benchmark.
The authors also show that (on one of the environments), ROMI leads to trajectories which are closer to the offline dataset than if forward models were used instead. This provides additional justification for the use of reverse models.
main_review: The algorithmic contribution is essentially a way to do data augmentation, so the novelty is entirely in the way they perform that generation. This paper contains one useful core insight, which is that generating trajectories in reverse order leads to more conservative behaviors. A conjecture for why this happens, which is verbally alluded to in the text but not written out mathematically, is that it results in better target Q values $Q(s’,a’)$ for fitting the Bellman equation, because $s’$ is always in-distribution. Whereas for forward MB methods such as MOPO, the target Q value is evaluated on some state which may be out of distribution, harming the learning.
The authors have shown that forward models tend to lead the policy farther from the dataset (Fig. 3j), and it is natural to believe that this may lead to worse Q value estimation. If the authors agree with this hypothesis, perhaps they can have evaluate the learned Q functions under FOMI and ROMI compared to a Monte Carlo estimate to see which is more accurate.

The strong empirical results, backed up by the plot of the trajectory discrepancy for FOMI and ROMI, makes a good case for the utility of reverse models. On that note, I would also be interested to see the trajectory discrepancy measure for the other environments as well, to see if the demonstrated trend holds more generally.

The writing of the paper is fairly clear, and I appreciate that the authors have put a lot of effort into creating nice figures to illustrate their idea (Fig. 1), as well as to show the case study (Fig. 3).
limitations_and_societal_impact: They briefly touch on limitations, but not on societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a model based approach for offline RL that utilizes reverse models instead of forward models to augment the dataset for use with an offline model free RL algorithm. They demonstrate that using reverse models is able to produce more conservative behaviors than forward models, which may extrapolate overaggressively, while leading
to more successful behaviors than running model free offline RL algorithms on only the original dataset. They demonstrate improved performance over previous baselines in the D4RL benchmark, particularly on maze navigation tasks.
main_review: The paper is well written, with the motivation for using reverse model rollouts over forward model rollouts well explained and informative figures illustrating the potential failure modes of different methods. Overall, I find the idea of using reverse rollouts to limit optimistic extrapolation in models to be both interesting and novel, and the performance of the proposed algorithms on the maze environemnts impressive. I would however also like to see ablations on how the rollout length (which seems to be set as ah yperparameter to either 5 or 1 depending on the task) affects the algorithm. Can the reverse models and action samplers generate reasonable trajectories over longer horizons, and does limiting the rollout length in the maze environments limit the algorithm's ability to stitch together good trajectories?

Regarding baselines: Given the strong performance of the model free CQL on the Gym Mujoco tasks and that the proposed algorithm also uses CQL as the underlying model free algorithm, it would be good to add comparisons to COMBO (which is cited in the related work, but not evaluated here), which integrates forward model rollouts with CQL and demonstrated improved performance, to see if reverse imagination offers much benefit outside of the maze navigation tasks.

Minor notation suggestion: The authors train a generative model over actions which they denote as $\hat G_{\theta}(s)$. It would be clearer to denote it as $\hat G_{\theta}(a \vert s')$ to make it explicit at first glance that it gives a distribution over actions instead of states.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes an offline model-based RL approach (ROMI) for data augmentation based on generating "reverse" rollouts that can reach a given a target goal state using an inverse dynamics model and a reverse policy, which are trained using maximum likelihood estimation and conditional VAE respectively. This approach can be easily combined with existing model-free offline RL methods and leads to state-of-the-art results on a subset of D4RL benchmark tasks.
main_review: **Originality**: The work seems apply the idea of learning reverse dynamics model and policy to generate rollouts for augmenting the offline dataset, which seems quite neat. This seems to more conservative than forward-model based approaches for which the authors present some evidence in the experimental section. 

*Question*: Could the authors please elaborate on why the proposed approach is expected to be more conservative? For example, the reverse model could imagine trajectories reaching the target state from a failure state such as crossing through a wall as shown in Figure 3(g) (right). 

Related work is discussed adequately, a possible connection to goal-conditioned approaches could also be briefly mentioned. 

**Quality**: The submission is technically sound with well supported claims by experimental results with the strengths of the work being discussed adequately.  

**Clarity**: The submission is clearly written and well organized. There are minor typos but they don't hurt the clarity of the paper. One possible suggestion is to move the ablation of the reverse policy in the appendix to the main paper by making the size of the Figure 3 smaller. 

Also, why aren't standard deviations not reported for BC/MF/MB in Table 1 as well as that of BCQL in Table 3 (Some of the std deviations seem to pretty large and should be reported if possible. 

Some typos:
- Table 2 Caption: MoJoCo --> MuJoCo
- Line 305: bais --> bias

**Significance**: The results seem to be important and I believe others (especially those working on model-based offline RL) are likely to build on the ideas proposed in this work. The results on harder AntMaze tasks on D4RL (where existing model-based RL methods fail) as well as the ablation studies in the main paper and the appendix show the merit of the proposed idea.
limitations_and_societal_impact: A major limitations of this work which is not discussed is the reliance on the reconstruction based objectives to learn the reverse model and the policy which may not scale well to real-world offline RL settings with rich observation spaces like images. Example benchmark domains with high-dimensional visual observation spaces include the Atari 2600 games [`1], DM locomotion tasks in RL Unplugged [2], and image-based locomotion and manipulation tasks [2]. A detailed discussion for this limitation as well as possible remedies or experiments on such domains should be included in the paper.

References:

[1] Agarwal, R., Schuurmans, D., & Norouzi, M. (2020, November). An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning (pp. 104-114). PMLR.

[2] Gulcehre, Caglar, et al. "RL Unplugged: A Collection of Benchmarks for Offline Reinforcement Learning." Advances in Neural Information Processing Systems 33 (2020).

[3] Rafailov, Rafael, et al. "Offline Reinforcement Learning from Images with Latent Space Models." arXiv preprint arXiv:2012.11547 (2020).
review_title: null

Meta-review: This paper presents an offline RL method based on reverse models. The contributions are original, and and while further analysis would be helpful in rigorously understanding how reverse models are helping, the empirical analysis convincingly shows that the improvements do not simply come from better model fitting. The empirical results are quite strong, including on challenging offline RL benchmarks. I recommend acceptance.

Please see the reviews and other comments on this page for further feedback when preparing the final version of the paper!
Decision: Accept (Poster)