Offline Reinforcement Learning with Reverse Model-based Imagination

Abstract: In offline reinforcement learning (offline RL), one of the main challenges is to deal with the distributional shift between the learning policy and the given dataset. To address this problem,  recent offline RL methods attempt to introduce conservatism bias to encourage learning in high-confidence areas. Model-free approaches directly encode such bias into policy or value function learning using conservative regularizations or special network structures, but their constrained policy search limits the generalization beyond the offline dataset. Model-based approaches learn forward dynamics models with conservatism quantifications and then generate imaginary trajectories to extend the offline datasets. However, due to limited samples in offline datasets, conservatism quantifications often suffer from overgeneralization in out-of-support regions. The unreliable conservative measures will mislead forward model-based imaginations to undesired areas, leading to overaggressive behaviors. To encourage more conservatism, we propose a novel model-based offline RL framework, called Reverse Offline Model-based Imagination (ROMI). We learn a reverse dynamics model in conjunction with a novel reverse policy,  which can generate rollouts leading to the target goal states within the offline dataset. These reverse imaginations provide informed data augmentation for model-free policy learning and enable conservative generalization beyond the offline dataset. ROMI can effectively combine with off-the-shelf model-free algorithms to enable model-based generalization with proper conservatism. Empirical results show that our method can generate more conservative behaviors and achieve state-of-the-art performance on offline RL benchmark tasks.


Reviews and author responses:

 We thank the reviewer very much for the inspiring feedback! We really appreciate your comments and will incorporate empirical evidence of the rebuttal into the next revision.

 I thank the authors for providing additional empirical evidence in support of the idea that reverse models lead to more conservatism, as well as for comparisons to MOReL and COMBO. I am a bit concerned by the high standard deviation of the average trajectory discrepancy on several of the environments, but the overall trend for FOMI vs. ROMI is clear.

After reading the other reviews, I am still in favor of acceptance despite the lack of theoretical justification (which can be explored more in future work) because of the empirical results.

 Thanks for your inspiring comments. We provide clarification to your questions as below. We appreciate it if you have any further questions or comments.

**Q1**: Is that it results in better target Q values for fitting the Bellman equation, because $Q(s', a')$ is always in-distribution. 

**A1**: This hypothesis inspires deep discussions about why ROMI works. With reverse model-based imagination, ROMI combines off-the-shelf model-free algorithms (e.g., BCQ or CQL) onto the augmented data support. Such learning process has two modules: (i) data support $D=\\{(s, a, r, s')\\}$ and (ii) Q function. Model-free offline RL methods learn policies constrained into the dataset to maximize the expected future returns (such as Q value). What we would emphasize is that data support is quite important and ROMI's augmented data support, $D_{\text{ROMI}}$, is more conservative than that of FOMI, $D_{\text{FOMI}}$. For example, in *maze2d-umaze*, $D_{\text{FOMI}}$ will contain imagined action sequences towards the walls (Fig. 3h), whereas augmented action sequences of $D_{\text{ROMI}}$ will guide the agent to the given datasets (Fig. 3g). In this case, model-free offline RL algorithms can exclude the undesired candidate actions (e.g., actions towards the walls) during ROMI's policy learning.

More specifically, BCQ has learned a conditional VAE to represent the candidate actions from the given dataset. We train BCQ on $D_{\text{ROMI}}$ and $D_{\text{FOMI}}$, respectively, and report their *Average Trajectory Discrepancy* (Def. 1) for randomly sampled candidate actions as follows. As actions are randomly sampled regardless of Q value, the result could represent how conservative the augmented dataset is.

| maze2d        | ROMI (ours)     | FOMI            |
| ------------- | --------------- | --------------- |
| umaze-sparse  | 0.051$\pm$0.051 | 0.392$\pm$0.177 |
| umaze-dense   | 0.046$\pm$0.032 | 0.116$\pm$0.054 |
| medium-sparse | 0.085$\pm$0.058 | 0.276$\pm$0.175 |
| medium-dense  | 0.052$\pm$0.030 | 0.193$\pm$0.132 |
| large-sparse  | 0.146$\pm$0.109 | 0.278$\pm$0.183 |
| large-dense   | 0.244$\pm$0.152 | 0.196$\pm$0.122 |

The result shows that ROMI's candidate actions incline to be around the given offline dataset, while FOMI tends to lead the policy farther from the dataset and does harm on the learning, which is a key reason for ROMI to enable model-based generalization with proper conservatism. 

**Q2**: I would also be interested to see the trajectory discrepancy measure for the other environments as well, to see if the demonstrated trend holds more generally.

**A2**: We add more results on the *Average Trajectory Discrepancy* (Def. 1) to demonstrate the generalizability of ROMI. The following table shows the comparison of ROMI-BCQ, BCQ, FOMI-BCQ and MOPO on six tasks of the *maze2d* environment. Overall, the *Average Trajectory Discrepancy* of ROMI-BCQ and BCQ are lower than that of FOMI-BCQ and MOPO, implying reverse imagination is more conservative than forward imagination in general. Note that in some cases the *Average Trajectory Discrepancy* of ROMI-BCQ is higher than that of BCQ, because the reverse imagination could augment the dataset and induce more effective actions beyond the support of the dataset.

| maze2d        | ROMI-BCQ (ours) | BCQ             | FOMI-BCQ        | MOPO            |
| ------------- | --------------- | --------------- | --------------- | --------------- |
| umaze-sparse  | 0.042$\pm$0.019 | 0.076$\pm$0.047 | 0.427$\pm$0.151 | 0.607$\pm$0.374 |
| umaze-dense   | 0.075$\pm$0.044 | 0.065$\pm$0.037 | 0.480$\pm$0.185 | 0.561$\pm$0.266 |
| medium-sparse | 0.187$\pm$0.116 | 0.102$\pm$0.070 | 0.276$\pm$0.148 | 0.247$\pm$0.027 |
| medium-dense  | 0.205$\pm$0.134 | 0.088$\pm$0.038 | 0.276$\pm$0.123 | 0.300$\pm$0.421 |
| large-sparse  | 0.344$\pm$0.271 | 0.165$\pm$0.098 | 0.272$\pm$0.168 | 0.464$\pm$0.140 |
| large-dense   | 0.205$\pm$0.026 | 0.164$\pm$0.100 | 0.318$\pm$0.332 | 0.204$\pm$0.091 |


 We thank the reviewer very much for the evaluation and comments. We will incorporate the empirical results of the rebuttal into the next revision.

 The authors have addressed my primary concerns about empirical comparisons and the effect of the horizon. After reading the other reviews, I am still in favor of accepting the paper and will keep my score as is.

 We thank the reviewer for the feedback. We would like to present a simple example to illustrate why reverse rollouts are more conservative in offline RL.

Assume a state $s_\text{in}$ is in the offline dataset and has an existing trajectory leading to the goal (denoted by $s_\text{goal}$) within the dataset. Considered below are two five-step trajectories from forward imagination and reverse imagination, which visit the same state sequence, but in the opposite direction (hence the actions are not the same). Let the forward imaginary trajectory be from $s_\text{in}$ to $s_5$, where $s_5$ is the end state of the trajectory. Similarly, let the reverse imaginary trajectory be from $s_5$ to $s_\text{in}$. Note that the rollout orders of these two trajectories are the same, starting at $s_\text{in}$ and ending at $s_5$. We will formalize these two imaginary trajectories as follows:

- Forward rollout: $\langle s_\text{in}, a^f_1, s_1, a^f_2, s_2, a^f_3, s_3, a^f_4, s_4, a^f_5, s_5\rangle$,
- Reverse rollout: $\langle s_5, a^r_5, s_4, a^r_4, s_3, a^r_3, s_2, a^r_2, s_1, a^r_1, s_\text{in}\rangle$,

where $s_{1:5}$ denotes the imaginary state sequence, $a^f_{1:5}$ denotes the action sequence for forward imagination, and $a^r_{1:5}$ denotes the action sequence for reverse imagination.

During the training process, the reverse rollout will expand the trajectory from $\langle s_\text{in}, \dots, s_\text{goal}\rangle$ to $\langle s_5, \dots, s_\text{in}, \dots, s_\text{goal}\rangle$, i.e., $s_{1:5}$ can now reach the goal. Through the reverse imagination of $s_{1:5}$, the reverse rollout can benefit the policy learning in this task. For the forward rollout, there are three cases for the state $s_5$:

1. Consider $s_5$ is outside the dataset and the value of $s_5$ (i.e., $Q(s_5, \cdot)$) is overestimated, which is a common challenge for function approximation in offline RL [1]. The forward rollout may mislead the learning policy from $s_\text{in}$ to $s_5$ and harm the learning performance. In contrast, the reverse rollout does not have such negative impact. It is because that $s_\text{in}$ does not have an action sequence to reach $s_5$ in the reversed data augmentation. 
2. If $s_5$ is outside the dataset and the value of $s_5$ is not overestimated, the forward rollout does not affect the policy learning. It is because that the execution policy will not go to the state $s_5$ with a lower $Q(s_5, \cdot)$. In contrast, as discussed above, the reverse rollout would benefit the learning in this case due to its effective trajectory expansion with the opposite direction of imagination.
3. Consider $s_5$ is inside the dataset. When $s_5$ has a trajectory leading to the goal in the dataset, we believe the forward rollout can improve the learning by selecting the better of the two successful trajectories, $\langle s_\text{in}, \dots, s_\text{goal}\rangle$ and $\langle s_\text{in}, \dots, s_5, \dots, s_\text{goal}\rangle$. In the reversed data augmentation, there is only one successful trajectory from the state $s_\text{in}$, i.e., $\langle s_\text{in}, \dots, s_\text{goal}\rangle$, but this could naturally be compensated by generating a reverse rollout from state $s_5$. Note that, in this case, both forward and reverse methods may not need to deal with the conservatism issue, because $s_5$ is in the original offline dataset.

In summary, we would like to highlight that reverse imagination is more conservative for imagined states that are outside the dataset and may even have overly estimated values. This explains why it works well in offline RL.

[1] Fujimoto, S., Meger, D., & Precup, D. (2019). Off-policy deep reinforcement learning without exploration. 36th International Conference on Machine Learning, ICML 2019, 2019-June, 3599–3609.

 I thank the authors for the reply, but unfortunately, I still do not see a precise answer to my question. I will first state my observation/question again, and then share my thoughts to perhaps ground the discussion.

---

The authors repeatedly state that forward rollouts go from dataset to outside, while reverse rollouts go from outside to the dataset. The authors do not provide a clear reason for why this is the case, and simply assert it. I disagree with this statement and premise, as I outline below.

In A1, the argument the authors use is that forward rollouts go from the dataset to outside **not because of the forward model itself, but rather due to the rollout procedure.** Specifically, as the policy is updated over the course of training, the conditional action distribution deviates from the dataset, eventually leading to new states and compounding errors in the rollouts. I would assert that the same is equivalently true for the reverse model, when identical policies are used for reverse rollouts. 

**Q: Specifically, if we use a different policy than the data collecting policy (or equivalently a policy different from $G_\theta$ described in lines 145-163 of submission), why should we expect rollouts from a reverse model to be conservative?**

----

My understanding from the author response is that the conservative behavior is unrelated to forward vs reverse models themselves, but rather entirely a reflection of the rollout procedure. If the data augmentation in the forward direction was performed using the data collecting policy or the appropriate counterpart of $G_\theta$ (lines 145-163), I fail to see why reverse models should be any better. Can the authors comment?

 Thanks for your comments. We provide clarification to your questions and concerns as below. If you have any additional questions or comments, please post them and we would be happy to have further discussions.

**Q1**: If the forward rollout is sufficiently short horizon, it is from the dataset to dataset, since errors are unlikely to compound in short horizons.

**A1**: The forward rollout is still from the dataset to the outside even with a sufficiently short horizon. It is because that the rollout policies (e.g., SAC of MOPO) will lead the agent to deviate from the offline dataset, but not because of the large compounding errors. We agree that both forward and reverse model errors are mild within a proper rollout length. However, the rollout policies of these methods will generate diverse imaginary actions (rather than states) beyond the offline dataset (e.g., randomly initialized SAC of MOPO), which is the motivation for the generalization of model-based offline RL methods [1]. As the rollout length increases, the deviation range of forward imaginary trajectories will correspondingly expand to the outside. Although the reverse imagination has a similar expansion range, its learned actions are from the outside towards the dataset and provide informed data augmentation leading to the target goal states (i.e., the offline dataset). We elaborate on the impact of forward and reverse augmented data support on policy learning with different rollout lengths. Please refer to Q3 for detailed discussions.

**Q2**: Why is the reverse model less susceptible to long-horizon compounding errors compared to forward model?

**A2**: ROMI is not set up to mitigate long-horizon compounding errors. Literature on model-based reinforcement learning [2] has pointed out that long rollouts tend to suffer from large compounding error, and hence to be unreliable for policy optimization. Instead, shorter rollouts (e.g., one step or five steps) are sufficient to provide pronounced benefits for sample efficiency [3]. In this work, we also focus on the scenario of short rollouts, and we choose the proper rollout length based on the model error, as discussed in Appendix A.

**Q3**: Can the authors expand on precisely why reverse models should be more conservative than forward models. I see no advantage/disadvantage in one over the other for generating synthetic rollouts for augmentation.

**A3**: The synthetic rollouts generated by the reverse models introduce an informed data augmentation towards the offline datasets, while the augmentation of the forward models has an opposite direction, i.e., which is facing the outside. This difference makes the policy learning using the reverse model closer to the offline dataset than the learning with the forward model. 

Specifically, ROMI combines reverse model-based imagination with off-the-shelf model-free algorithms (e.g., BCQ or CQL) using the offline dataset. Such learning process contains two components: (i) data support $D=\\\{(s, a, r, s')\\\}$ and (ii) Q function. Model-free offline RL methods learn policies constrained onto the dataset to maximize the expected future returns (such as Q value). We would emphasize that data support is quite important for ROMI and its augmented dataset, $D_{\text{ROMI}}$, is more conservative than that of FOMI (i.e., the counterpart of a forward model), $D_{\text{FOMI}}$. As discussed in Q1, the reverse model can imagine trajectories from the outside to the dataset, while the imaginary trajectories of the forward model are from the dataset to the outside. In this case, during the policy learning of ROMI, model-free offline RL algorithms can exclude the candidate actions towards the outside, but FOMI cannot. This exclusion of undesired actions makes ROMI's policy learn on a more confident support and enables model-based generalization with proper conservatism. 

To demonstrate the advantage of ROMI empirically, BCQ (i.e., a model-free offline RL method) has learned a conditional VAE to represent the candidate actions from the given dataset. We train BCQ on $D_{\text{ROMI}}$ and $D_{\text{FOMI}}$ in *maze2d-umaze*, respectively, and report their *Average Trajectory Discrepancy* (Def. 1) for randomly sampled candidate actions as follows. As actions are randomly sampled regardless of Q value, the results could represent how close the augmented data support is to the given dataset. In addition, to investigate the forward and reverse model in different rollout lengths, we also conduct an ablation study by varying this hyperparameter.

| Discrepancy   | ROMI (ours)    | FOMI           |
| ------- | -------------- | -------------- |
| len=1   | 0.128$\pm$0.076 | 0.138$\pm$0.101 |
| len=5\* | 0.051$\pm$0.051 | 0.392$\pm$0.177 |
| len=10  | 0.059$\pm$0.016 | 0.304$\pm$0.189 |

\*: the hyperparameter used in our paper.

The overall result shows that, with ROMI, BCQ tends to sample candidate actions around the given offline dataset, while FOMI tends to lead the policy farther from the dataset. We find that the *discrepancy* of ROMI shrinks when increasing the rollout length from 1 to 5 or 10, and the *discrepancy* of FOMI increases in the longer horizon. As the rollout length becomes longer, training policy with forward model may visit more outer area, while reverse model-based imagination will constrain the policy close to the offline dataset. We also illustrate the learning performance of ROMI and FOMI as below. The empirical experiments show that ROMI performs well with varying rollout lengths, especially in 5 steps, whereas FOMI cannot solve this problem in several rollout lengths. Thus, reverse model is critical for ROMI, which induces conservatism bias into data augmentation, hence generalizes beyond the offline dataset.

| Score   | ROMI (ours)    | FOMI           |
| ------- | -------------- | -------------- |
| len=1   | 82.0$\pm$31.5  | -6.8$\pm$1.1   |
| len=5\* | 139.5$\pm$3.6  | 8.1$\pm$15.5   |
| len=10  | 134.4$\pm$17.2 | 0.0$\pm$3.2    |

\*: the hyperparameter used in our paper.

[1]: Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Neurips, 2020.

[2]: Gu, Shixiang, et al. "Continuous deep q-learning with model-based acceleration." International conference on machine learning. PMLR, 2016.

[3]: Janner, Michael, et al. "When to Trust Your Model: Model-Based Policy Optimization." Neurips, 2019.

 Thank you authors for the overall response. I wanted to expand and discuss Q1 and Q3 further. It would be best to go back to my original framing of the questions.

----

*My biggest concern with the paper is there is no clear justification or reason for why a reverse model should work better than a forward model. In particular, taking a particular trajectory snippet, if learning $P(s, r | s', a)$ is possible, I fail to see why predicting $P(s', r | s, a)$ is any more difficult.*

In the answer to this, the authors seem to suggest that both models have the same 1-step error, but reverse models are more conservative than forward models. Can the authors expand on precisely *why* reverse models should be more conservative than forward models. I am not looking for empirical evidence that they are more conservative, but rather a principle or explanation for why we should expect them to be.

----

From author response A1: *In the case of Figure 1b, both forward model and reverse model could rollout to the states in the wall. However, the directions of these imagined trajectories are opposite, in which the forward rollout trajectory is "from the dataset to the wall", and the reverse is "from the wall to the dataset".*

I do not agree with this explanation or intuition. If the forward rollout is sufficiently short horizon, it is from the dataset to dataset, since errors are unlikely to compound in short horizons. With longer rollouts, we can get extraneous trajectories like collisions, wall penetrations, or even NaNs. The exact same holds for reverse rollouts too. For short rollouts, we might remain in distribution (or go from wall to data distribution if reverse of such trajectory snippets are part of the dataset). However, when we perform long horizon simulation, we have the exact same compounding error problem leading to extraneous trajectories.

So in summary, my question is why is the reverse model less susceptible to long-horizon compounding errors compared to forward model? If the two are comparable, I see no advantage/disadvantage in one over the other for generating synthetic rollouts for augmentation.


 **Q4**: Experiment results don't compare with MOReL and COMBO, which to my knowledge are the SOTA model-based algorithms.

**A4**: We will add the performance of MOReL and COMBO in the main paper for the revision, following the implementations in two popular repositories - [MOReL](https://github.com/SwapnilPande/MOReL) and [COMBO](https://github.com/takuseno/d3rlpy). The results are reported below. Because of limited time, we evaluated them on the *maze2d* and *antmaze* environments, and used results reported in their original papers for the *gym* environment.

The results show that ROMI performs equally well or better than COMBO on the *maze2d* and *gym* environments, and has significant improvement on the harder *antmaze* environment. One possible explanation is that harder tasks need more conservatism. Compared to COMBO's implicit regularization on Q value, ROMI's reverse imagination is a more direct and effective way to be conservative. MOReL is less efficient compared to ROMI and COMBO, which is probably caused by the inaccurate estimation of model uncertainty.

| maze2d        | ROMI (ours)    | MOReL         | COMBO          |
| ------------- | -------------- | ------------- | -------------- |
| umaze-sparse  | 139.5$\pm$3.6  | -14.4$\pm$0.5 | 76.4$\pm$19.2  |
| medium-sparse | 82.4$\pm$15.2  | -4.8$\pm$0.2  | 68.5$\pm$83.6  |
| large-sparse  | 83.1$\pm$22.1  | -2.3$\pm$0.2  | 14.1$\pm$10.7  |
| umaze-dense   | 98.3$\pm$2.5   | 36.9$\pm$15.7 | 86.2$\pm$1.4   |
| medium-dense  | 102.6$\pm$32.4 | -1.9$\pm$20.1 | 84.2$\pm$9.5   |
| large-dense   | 124.0$\pm$1.3  | -3.2$\pm$9.8  | 100.2$\pm$24.1 |

| antmaze        | ROMI (ours)   | MOReL | COMBO         |
| -------------- | ------------- | ----- | ------------- |
| umaze-fixed    | 68.7$\pm$2.7  | 0     | 80.3$\pm$18.5 |
| medium-play    | 35.3$\pm$1.3  | 0     | 0             |
| large-play     | 20.2$\pm$14.8 | 0     | 0             |
| umaze-diverse  | 61.2$\pm$3.3  | 0     | 57.3$\pm$33.6 |
| medium-diverse | 27.3$\pm$3.9  | 0     | 0             |
| large-diverse  | 41.2$\pm$4.2  | 0     | 0             |

| Dataset Type  | gym         | ROMI (ours)   | MOReL | COMBO |
| ------------- | ----------- | ------------- | ----- | ----- |
| random        | walker2d    | 7.5$\pm$20.0  | 37.3  | 7.0   |
| random        | hopper      | 30.2$\pm$4.4  | 53.6  | 17.9  |
| random        | halfcheetah | 24.5$\pm$0.7  | 25.6  | 38.8  |
| medium        | walker2d    | 84.3$\pm$1.1  | 77.8  | 75.5  |
| medium        | hopper      | 72.3$\pm$17.5 | 95.4  | 94.9  |
| medium        | halfcheetah | 49.1$\pm$0.8  | 42.1  | 54.2  |
| medium-replay | walker2d    | 109.7$\pm$9.8 | 49.8  | 56.0  |
| medium-replay | hopper      | 98.1$\pm$2.6  | 93.6  | 73.1  |
| medium-replay | halfcheetah | 47.0$\pm$0.7  | 40.2  | 55.1  |
| medium-expert | walker2d    | 109.7$\pm$5.3 | 95.6  | 96.1  |
| medium-expert | hopper      | 111.4$\pm$5.6 | 108.7 | 111.1 |
| medium-expert | halfcheetah | 86.8$\pm$19.7 | 95.6  | 90.0  |

**Q5**: Just like ROMI, COMBO also uses model rollouts to augment datasets for a conservative model-free algorithm (CQL in this case). Both these components require clear ablations to understand the relative gains -- which among (1) and (2) contribute most.

**A5**: Since COMBO does not learn a reverse rollout policy, we could not directly apply reverse imagination on it. Thus, we conduct two ablation stuides by using ROMI, FOMI and COMBO, to investigate the effect of these two components ((1) and (2) discussed by the reviewer) as follows.

(1) As mentioned in Section 4.3, to determine the effect of reverse and forward model, we compare the performance of ROMI and FOMI (list below or Table 3 in the paper). The results show that ROMI generally outperforms FOMI, which indicates that reverse imagination is more effective than forward imagination. 

(2) To investigate different rollout policies in model-based RL learning, i.e., an adaptive policy vs. a fixed policy, we compare the performance of FOMI and COMBO as follows. The results show that COMBO performs better on the *maze2d* environment and FOMI outperforms COMBO on the *antmaze* setting, which implies that rollout with current learning policy might help with the training process.

The ablation studies show that reverse model enables safe generalization and two rollout policies (i.e., adaptive or fixed) hold great promise for different settings. Therefore, we believe that both (1) and (2) contribute to the improvement of ROMI and COMBO. Combining reverse imagination with adaptive rollout policy will be an interesting future direction for model-based offline RL.

| maze2d        | ROMI (ours)    | FOMI          | COMBO          |
| ------------- | -------------- | ------------- | -------------- |
| umaze-sparse  | 139.5$\pm$3.6  | 8.1$\pm$15.5  | 76.4$\pm$19.2  |
| medium-sparse | 82.4$\pm$15.2  | 93.6$\pm$41.3 | 68.5$\pm$83.6  |
| large-sparse  | 83.1$\pm$22.1  | -2.5$\pm$0.0  | 14.1$\pm$10.7  |
| umaze-dense   | 98.3$\pm$2.5   | 30.7$\pm$0.9  | 86.2$\pm$1.4   |
| medium-dense  | 102.6$\pm$32.4 | 64.7$\pm$37.0 | 84.2$\pm$9.5   |
| large-dense   | 124.0$\pm$1.3  | -0.7$\pm$7.1  | 100.2$\pm$24.1 |

| antmaze        | ROMI (ours)   | FOMI         | COMBO         |
| -------------- | ------------- | ------------ | ------------- |
| umaze-fixed    | 68.7$\pm$2.7  | 79.5$\pm$2.5 | 80.3$\pm$18.5  |
| medium-play    | 35.3$\pm$1.3  | 26.2$\pm$5.5 | 0             |
| large-play     | 20.2$\pm$14.8 | 12.0$\pm$3.3 | 0             |
| umaze-diverse  | 61.2$\pm$3.3  | 66.8$\pm$3.5 | 57.3$\pm$33.6 |
| medium-diverse | 27.3$\pm$3.9  | 12.3$\pm$2.1 | 0             |
| large-diverse  | 41.2$\pm$4.2  | 17.8$\pm$2.1 | 0             |

**Q6**: The states from which FOMI and ROMI start the rollouts could also be quite important -- how did you make that choice for FOMI vs ROMI?

**A6**: Both FOMI and ROMI start the rollouts from the states randomly sampled from the offline dataset. Uniform sampling is a general method for most datasets.

**Q7**: There are some limitations and open questions, which I have outlined in my main review. The paper does not have a discussions or limitation section.

**A7**: We will expand the limitation and future work in the conclusion as a separate section for a detailed discussion. As mentioned in the paper, it could be challenging for ROMI to learn with the dataset without any good trajectory (e.g., reaching goal states or with high rewards). One potential solution is to combine reverse and forward models to generate bolder imaginations. Additionally, as reviewer 3cj7 suggested, on the domains with complex visual input, learning a neural-based model could be hard for both online and offline RL. In the paper, we have reported the one-step model error for forward model and reverse model in Appendix A, in which ROMI's reverse model has similar accuracy to the forward model, thus we may trust the reverse model as well as the traditional forward model. Even so, it is still challenging for reverse model to be accurate enough to support long horizon rollout in the visual-input settings. One potential solution is to embed the reverse model and policy in the latent space, which is proved to be scalable in high dimensional visual observation space [1]. Theoretical understanding of the reverse imagination in offline RL is also an interesting future direction.

[1] Rafailov, Rafael, et al. "Offline Reinforcement Learning from Images with Latent Space Models." arXiv preprint arXiv:2012.11547 (2020).

 Thanks for your constructive comments. We provide clarification to your questions and concerns as below. We appreciate it if you have any further questions or comments.

**Q1**: My biggest concern with the paper is there is no clear justification or reason for why a reverse model should work better than a forward model. Figure 1b and its description in lines 123-130 is a bit misleading and uses a straw-man argument. A naive forward rollout with an arbitrary policy might indeed collide with the wall, but the same is true of reverse rollouts with arbitrary policies too.

**A1**: There is a misunderstanding on why reverse imagination is more conservative in offline RL. In the case of Figure 1b, both forward model and reverse model could rollout to the states in the wall. However, the directions of these imagined trajectories are opposite, in which the forward rollout trajectory is "from the dataset to the wall", and the reverse is "from the wall to the dataset". Therefore, forward imagination may further mislead the agent to an undesired area (e.g., colliding with the wall), whereas reverse imagination would guide the agent back to the dataset from outside. ROMI can generate rollouts leading to the target goal states within the distribution of the given offline dataset. In this vein, ROMI conducts a more *conservative* model-based imagination and provides informed data augmentation.

**Q2**: Additionally, conservative model-based methods (e.g., MOReL, MOPO) are explicitly setup to avoid going into regions where data support is unavailable, and thus in principle would flag colliding into the wall as a bad event. 

**A2**: We claim that ROMI goes one step further than related works in the conservative model-based imagination. Current model-based offline RL methods (e.g., MOReL, MOPO) often use conservatism (e.g., model-uncertainty) quantifications to ensure the confidence of model-based rollouts. In this paper, we find that these quantifications may suffer from overgeneralization due to the limited dataset, especially in out-of-support regions, which could lead to overaggressive behaviors. In contrast, ROMI reverses the direction of model-based imagination and avoids the conservative measures to encourage more conservatism. More concretely, this paper has provided a case study in *maze2d-umaze* (see Sec. 4.4) to illustrate that model-uncertainty quantification of MOPO cannot flag colliding into the wall as a bad event (Fig. 3f) and ROMI's reverse imaginations exclude the imagined action sequences of colliding into the wall (Fig. 3g).

**Q3**: If learning $P(s,r\vert s',a)$ is possible, I fail to see why predicting $P(s',r\vert s,a)$ is any more difficult. Could it just be implementation details or is there something more fundamental?

**A3**: ROMI has fundamental advantages, because reverse imagination naturally induces conservatism and informed data augmentation as discussed in Q1. In fact, learning $P(s,r\vert s',a)$ is as hard as learning $P(s',r\vert s,a)$. We have reported the one-step model error for forward model and reverse model in Appendix A, in which ROMI's reverse model has similar accuracy to the forward model. Empirical results (Table 1 in Appendix) show that we can trust the reverse model as the traditional forward model in our experimental settings.

 Thank you very much for your evaluation and comments. We will incorporate relevant discussions and experiments of the rebuttal into the next revision.

 Thanks for your insightful comments. We provide clarification to your questions as below. We appreciate it if you have any further questions or comments.

**Q1**: Could the authors please elaborate on why the proposed approach is expected to be more conservative?

**A1**: ROMI combines reverse model-based imagination with off-the-shelf model-free algorithms (e.g., BCQ or CQL) for policy search around the offline dataset. This learning process contains two components: (i) data support $D=\\{(s, a, r, s')\\}$ and (ii) Q function. Model-free offline RL methods learn policies constrained onto the dataset to maximize the expected future returns (such as Q value). We would emphasize that data support is quite important for ROMI and its augmented dataset, $D_{\text{ROMI}}$, is more conservative than that of FOMI, $D_{\text{FOMI}}$. For example, in *maze2d-umaze*, $D_{\text{FOMI}}$ will contain imagined action sequences towards the walls (Fig. 3h), whereas augmented action sequences of $D_{\text{ROMI}}$ will guide the agent toward the given datasets (Fig. 3g). In this case, model-free offline RL algorithms can exclude undesired candidate actions (e.g., actions towards the walls) during ROMI's policy learning. 

More specifically, BCQ has learned a conditional VAE to represent the candidate actions from the given dataset. We train BCQ on $D_{\text{ROMI}}$ and $D_{\text{FOMI}}$, respectively, and report their *Average Trajectory Discrepancy* (Def. 1) for randomly sampled candidate actions as follows. As actions are randomly sampled regardless of Q value, the results could represent how conservative the augmented dataset is.

| maze2d        | ROMI  (ours)    | FOMI            |
| ------------- | --------------- | --------------- |
| umaze-sparse  | 0.051$\pm$0.051 | 0.392$\pm$0.177 |
| umaze-dense   | 0.046$\pm$0.032 | 0.116$\pm$0.054 |
| medium-sparse | 0.085$\pm$0.058 | 0.276$\pm$0.175 |
| medium-dense  | 0.052$\pm$0.030 | 0.193$\pm$0.132 |
| large-sparse  | 0.146$\pm$0.109 | 0.278$\pm$0.183 |
| large-dense   | 0.244$\pm$0.152 | 0.196$\pm$0.122 |

The result shows that ROMI's candidate actions incline to be around the given offline dataset and FOMI tends to lead the policy farther from the dataset and harm the learning, which is a key reason for ROMI to enable model-based generalization with proper conservatism. 

**Q2**: For example, the reverse model could imagine trajectories reaching the target state from a failure state such as crossing through a wall as shown in Figure 3(g) (right).

**A2**: When the reverse model imagines trajectories reaching the target state from a failure state, it does not harm the learning. Because the agent should not be initialized in a failure state (e.g., starting from a wall), and as discussed in Q1 and Fig. 3g, model-free offline RL methods (e.g., BCQ) would not lead the policy farther from the dataset but instead guide agents towards the dataset.

**Q3**: Also, why aren't standard deviations not reported for BC/MF/MB in Table 1 as well as that of BCQL in Table 3 (Some of the std deviations seem to pretty large and should be reported if possible.

**A3**: In certain tasks, our implementations of some baselines could not achieve the results reported in their original papers, so we directly use their reported results instead, which does not include the standard deviation. We report the standard deviation of BCQ in Table 3 as follows. We will conduct a more thorough evaluation in the revision.

| maze2d        | ROMI-BCQ (ours) | BCQ           |
| ------------- | --------------- | ------------- |
| umaze-sparse  | 139.5$\pm$3.6   | 41.1$\pm$7.6  |
| medium-sparse | 82.4$\pm$15.2   | 9.7$\pm$14.2  |
| large-sparse  | 83.1$\pm$22.1   | 38.3$\pm$10.4 |
| umaze-dense   | 98.3$\pm$2.5    | 37.0$\pm$5.3  |
| medium-dense  | 102.6$\pm$32.4  | 37.9$\pm$4.5  |
| large-dense   | 124.0$\pm$1.3   | 79.8$\pm$12.2 |

| antmaze        | ROMI-BCQ (ours) | BCQ           |
| -------------- | --------------- | ------------- |
| umaze-fixed    | 68.7$\pm$2.7    | 75.3$\pm$13.7 |
| medium-play    | 35.3$\pm$1.3    | 0             |
| large-play     | 20.2$\pm$14.8   | 0             |
| umaze-diverse  | 61.2$\pm$3.3    | 49.3$\pm$9.9  |
| medium-diverse | 27.3$\pm$3.9    | 0             |
| large-diverse  | 41.2$\pm$4.2    | 0             |

**Q4**: Typos and a detailed discussion for this limitation as well as possible remedies or experiments on such domains should be included in the paper.

**A4**: Thanks for spotting the typos. We will fix them in the revision. On the domains with complex visual input, learning a neural-based model could be hard for both online and offline RL. In the paper, we have reported the one-step model error for forward model and reverse model in Appendix A, in which ROMI's reverse model has similar accuracy to the forward model, thus we may trust the reverse model as well as the traditional forward model. Even so, it is still challenging for reverse model to be accurate enough to support long horizon rollouts in the visual-input settings. One potential solution is to embed the reverse model and policy in the latent space, which is proved to be scalable in high-dimensional visual observation space [1]. We will add a detailed discussion for this limitation in the revision.

[1] Rafailov, Rafael, et al. "Offline Reinforcement Learning from Images with Latent Space Models." arXiv preprint arXiv:2012.11547 (2020).

 Thanks for your thoughtful comments. We provide clarification to your questions as below. We appreciate it if you have any further questions or comments.

**Q1**: Can the reverse models and action samplers generate reasonable trajectories over longer horizons, and does limiting the rollout length in the maze environments limit the algorithm's ability to stitch together good trajectories?

**A1**: The rollout length of the reverse models largely depends on the accuracy of the reverse model. In some simpler environments such as *maze2d*, the model could generate reasonable trajectories over longer horizons; however in more complex environments such as *gym*, the model could accumulate large error after multi-step rollout, so that we lower the rollout length. In Appendix A, we include the one-step model error for different environments. As the Table 1 in Appendix A reported, the model could not predict accurately in *walker2d* and *halfcheetah*, so that we set the rollout length to one for these two environments. 

To investigate the effect of the rollout length more clearly, we conduct an ablation study by varying this hyperparameter on the *maze2d* and *antmaze* environments. The results are listed below. We find that on *maze2d*, ROMI performs equally well or even better when increasing the rollout length to 20, because the model is accurate enough for longer horizon imagination. However, increasing ROMI's rollout length has a negative impact on *antmaze*, where model cannot predict future states accurately after multi-step rollout.

| maze2d        | rollout (len=1) | rollout (len=5)\* | rollout (len=10) | rollout (len=20) |
| ------------- | --------------- | ----------------- | ---------------- | ---------------- |
| umaze-sparse  | 82.0$\pm$31.5   | 139.5$\pm$3.6     | 134.4$\pm$17.2   | 135.6$\pm$7.3    |
| medium-sparse | 70.4$\pm$28.6   | 82.4$\pm$15.2     | 82.9$\pm$6.2     | 104.5$\pm$54.7   |
| large-sparse  | 120.3$\pm$18.3  | 83.1$\pm$22.1     | 116.2$\pm$49.3   | 76.8$\pm$17.2    |
| umaze-dense   | 68.0$\pm$11.5   | 98.3$\pm$2.5      | 113.6$\pm$7.0    | 99.2$\pm$10.3    |
| medium-dense  | 88.5$\pm$20.9   | 102.6$\pm$32.4    | 93.2$\pm$16.3    | 83.0$\pm$10.1    |
| large-dense   | 120.2$\pm$4.8   | 124.0$\pm$1.3     | 113.6$\pm$17.9   | 97.5$\pm$40.7    |

| antmaze        | rollout (len=1) | rollout (len=5)\* | rollout (len=7) | rollout (len=10) |
| -------------- | --------------- | ----------------- | --------------- | ---------------- |
| umaze-fixed    | 69.7$\pm$6.5    | 68.7$\pm$2.7      | 44.2$\pm$29.6   | 54.8$\pm$25.4    |
| medium-play    | 36.8$\pm$6,8    | 35.3$\pm$1.3      | 33.5$\pm$14.9   | 28.0$\pm$10.4    |
| large-play     | 37.5$\pm$6.1    | 20.2$\pm$14.8     | 7.3$\pm$12.7    | 0                |
| umaze-diverse  | 12.8$\pm$19.7   | 61.2$\pm$3.3      | 58.3$\pm$18.5   | 42.8$\pm$14.3    |
| medium-diverse | 25.3$\pm$4.9    | 27.3$\pm$3.9      | 18.2$\pm$9.3    | 13.5$\pm$4.6     |
| large-diverse  | 31.2$\pm$2.4    | 41.2$\pm$4.2      | 32.8$\pm$5.1    | 25.0$\pm$5.8     |

\*: the hyperparameter used in our paper.

**Q2**: It would be good to add comparisons to COMBO (which is cited in the related work, but not evaluated here).

**A2**: We will add the performance of COMBO in the main paper for the revision. We also add another baseline MOReL as reviewer GXD3 suggested. We follow the implementations of two popular repositories -  [MOReL](https://github.com/SwapnilPande/MOReL) and [COMBO](https://github.com/takuseno/d3rlpy). The results are reported below. Because of limited time, we evaluated them on the *maze2d* and *antmaze* environments, and used results reported in their original papers for the *gym* environment.

The results show that ROMI performs equally well or better than COMBO on the *maze2d* and *gym* environments, and has significant improvement on the harder *antmaze* environment. One possible explanation is that harder tasks need more conservatism. Compared to COMBO's implicit regularization on Q value, ROMI's reverse imagination is a more direct and effective way to be conservative. MOReL is less efficient compared to ROMI and COMBO, which is probably caused by the inaccurate estimation of model uncertainty.

| maze2d        | ROMI (ours)    | MOReL         | COMBO          |
| ------------- | -------------- | ------------- | -------------- |
| umaze-sparse  | 139.5$\pm$3.6  | -14.4$\pm$0.5 | 76.4$\pm$19.2  |
| medium-sparse | 82.4$\pm$15.2  | -4.8$\pm$0.2  | 68.5$\pm$83.6  |
| large-sparse  | 83.1$\pm$22.1  | -2.3$\pm$0.2  | 14.1$\pm$10.7  |
| umaze-dense   | 98.3$\pm$2.5   | 36.9$\pm$15.7 | 86.2$\pm$1.4   |
| medium-dense  | 102.6$\pm$32.4 | -1.9$\pm$20.1 | 84.2$\pm$9.5   |
| large-dense   | 124.0$\pm$1.3  | -3.2$\pm$9.8  | 100.2$\pm$24.1 |

| antmaze        | ROMI (ours)   | MOReL | COMBO         |
| -------------- | ------------- | ----- | ------------- |
| umaze-fixed    | 68.7$\pm$2.7  | 0     | 80.3$\pm$18.5 |
| medium-play    | 35.3$\pm$1.3  | 0     | 0             |
| large-play     | 20.2$\pm$14.8 | 0     | 0             |
| umaze-diverse  | 61.2$\pm$3.3  | 0     | 57.3$\pm$33.6 |
| medium-diverse | 27.3$\pm$3.9  | 0     | 0             |
| large-diverse  | 41.2$\pm$4.2  | 0     | 0             |

| Dataset Type  | gym         | ROMI (ours)   | MOReL | COMBO |
| ------------- | ----------- | ------------- | ----- | ----- |
| random        | walker2d    | 7.5$\pm$20.0  | 37.3  | 7.0   |
| random        | hopper      | 30.2$\pm$4.4  | 53.6  | 17.9  |
| random        | halfcheetah | 24.5$\pm$0.7  | 25.6  | 38.8  |
| medium        | walker2d    | 84.3$\pm$1.1  | 77.8  | 75.5  |
| medium        | hopper      | 72.3$\pm$17.5 | 95.4  | 94.9  |
| medium        | halfcheetah | 49.1$\pm$0.8  | 42.1  | 54.2  |
| medium-replay | walker2d    | 109.7$\pm$9.8 | 49.8  | 56.0  |
| medium-replay | hopper      | 98.1$\pm$2.6  | 93.6  | 73.1  |
| medium-replay | halfcheetah | 47.0$\pm$0.7  | 40.2  | 55.1  |
| medium-expert | walker2d    | 109.7$\pm$5.3 | 95.6  | 96.1  |
| medium-expert | hopper      | 111.4$\pm$5.6 | 108.7 | 111.1 |
| medium-expert | halfcheetah | 86.8$\pm$19.7 | 95.6  | 90.0  |

**Q3**: Minor notation suggestion.

**A3**: Thanks for the suggestion on the notation of the generative model. We will make it clearer in the revision.

 After reading other reviews and response from the authors, I am still in favor of acceptance and decided to keep my score as is. I recommend the authors to include the clarifications they provided here in the camera ready. 

 We thank all the reviewers for the constructive feedback and for deeming our work to deliver a "novel, neat and interesting insight" (Reviewer GXD3, 23C4, bryw, 3cj7), to be "clearly written, well organized" (Reviewer 23C4, bryw, 3cj7) and with "informative figures" (Reviewer 23C4, bryw) while presenting "strong empirical results" (Reviewer GXD3, 23C4, 3cj7).

In summary, our response includes the following aspects:

1. Clarification on the motivation and mechanism for reverse imagination.
2. Additional comparison with two SOTA model-based offline algorithms - MOReL and COMBO.
3. Additional results on the *Average Trajectory Discrepancy* (Def. 1) for other environments.
4. Ablation study on the rollout length.

We hope our response could address the reviewers' concerns. If you have any other questions, please post them and we are happy to have further discussions.

This paper presents a new algorithm (ROMI) for offline RL that is based on data augmentation using a learned reverse imagination model. This reverse model takes in a state and action, and attempts to infer the previous state that occurred before the query state. Concretely, it models $P(s, r | s', a)$. Data is collected using the learned reverse model and reverse rollout policy, and is added to the offline dataset, for data augmentation. Finally, a conservative model-free algorithm like BCQ or CQL is used for policy improvement using the augmented dataset. Results presented on D4RL benchmark tasks suggest that the developed algorithm achieves good (near SOTA) performance on a majority of tasks.  **Strengths:** Interesting idea of using a reverse model, which to my knowledge is novel. Empirical results are also quite strong, although there are a few issues as I note below. 

**Limitations and Open Questions**
- My biggest concern with the paper is there is no clear justification or reason for why a reverse model should work better than a forward model. In particular, taking a particular trajectory snippet, if learning $P(s, r | s', a)$ is possible, I fail to see why predicting $P(s', r | s, a)$ is any more difficult. Could it just be implementation details or is there something more fundamental? 
- Experiment results don't compare with MOReL and COMBO, which to my knowledge are the SOTA model-based algorithms. I think a comparison with them are warranted, since they have been public for a long time now.
- In particular, I see a lot of parallels between COMBO and ROMI, thereby necessitating detailed comparisons between them. Just like ROMI, COMBO also uses model rollouts to augment datasets for a conservative model-free algorithm (CQL in this case). The main differences between them are: (1) ROMI uses a reverse model while COMBO uses a forward model (2) ROMI performs the augmentation rollout using a fixed policy while COMBO data augmentation is adaptive with rollouts being performed using the current policy. Both these components require clear ablations to understand the relative gains -- which among (1) and (2) contribute most.
- The states from which FOMI and ROMI start the rollouts could also be quite important -- how did you make that choice for FOMI vs ROMI? 
- Figure 1b and its description in lines 123-130 is a bit misleading and uses a straw-man argument. A naive forward forward rollout with an arbitrary policy might indeed collide with the wall, but the same is true of reverse rollouts with arbitrary policies too. Additionally, conservative model-based methods (eg MOReL, MOPO) are explicitly setup to avoid going into regions where data support is unavailable, and thus in principle would flag colliding into the wall as a bad event.

Overall, while the work has some innovative ideas, it is unclear why the method works. Given the number of open questions and limitations in evaluations, I am unable to recommend acceptance at this stage. There are some limitations and open questions, which I have outlined in my main review. The paper does not have a discussions or limitation section -- I would recommend the authors add such a section.

The paper proposes a method for offline RL based on reverse dynamics models. That is, rather than sampling trajectories by predicting $s’$ given $(s,a)$, they predict $s$ given $(s’,a)$. As a result, all the $s’$ in the augmented dataset are “real” states, reducing the effects of harmful extrapolation.
Once the reverse transitions have been generated (by combining the reverse dynamics model with a learned reverse policy based on CVAE), a policy can be trained on the augmented dataset using a model-free offline RL algorithm such as BCQ or CQL. This framework is referred to as Reverse Offline Model-based Imagination (ROMI). A practical instantiation of ROMI yields strong results on the D4RL benchmark.
The authors also show that (on one of the environments), ROMI leads to trajectories which are closer to the offline dataset than if forward models were used instead. This provides additional justification for the use of reverse models.  The algorithmic contribution is essentially a way to do data augmentation, so the novelty is entirely in the way they perform that generation. This paper contains one useful core insight, which is that generating trajectories in reverse order leads to more conservative behaviors. A conjecture for why this happens, which is verbally alluded to in the text but not written out mathematically, is that it results in better target Q values $Q(s’,a’)$ for fitting the Bellman equation, because $s’$ is always in-distribution. Whereas for forward MB methods such as MOPO, the target Q value is evaluated on some state which may be out of distribution, harming the learning.
The authors have shown that forward models tend to lead the policy farther from the dataset (Fig. 3j), and it is natural to believe that this may lead to worse Q value estimation. If the authors agree with this hypothesis, perhaps they can have evaluate the learned Q functions under FOMI and ROMI compared to a Monte Carlo estimate to see which is more accurate.

The strong empirical results, backed up by the plot of the trajectory discrepancy for FOMI and ROMI, makes a good case for the utility of reverse models. On that note, I would also be interested to see the trajectory discrepancy measure for the other environments as well, to see if the demonstrated trend holds more generally.

The writing of the paper is fairly clear, and I appreciate that the authors have put a lot of effort into creating nice figures to illustrate their idea (Fig. 1), as well as to show the case study (Fig. 3). They briefly touch on limitations, but not on societal impact.

The paper proposes a model based approach for offline RL that utilizes reverse models instead of forward models to augment the dataset for use with an offline model free RL algorithm. They demonstrate that using reverse models is able to produce more conservative behaviors than forward models, which may extrapolate overaggressively, while leading
to more successful behaviors than running model free offline RL algorithms on only the original dataset. They demonstrate improved performance over previous baselines in the D4RL benchmark, particularly on maze navigation tasks.  The paper is well written, with the motivation for using reverse model rollouts over forward model rollouts well explained and informative figures illustrating the potential failure modes of different methods. Overall, I find the idea of using reverse rollouts to limit optimistic extrapolation in models to be both interesting and novel, and the performance of the proposed algorithms on the maze environemnts impressive. I would however also like to see ablations on how the rollout length (which seems to be set as ah yperparameter to either 5 or 1 depending on the task) affects the algorithm. Can the reverse models and action samplers generate reasonable trajectories over longer horizons, and does limiting the rollout length in the maze environments limit the algorithm's ability to stitch together good trajectories?

Regarding baselines: Given the strong performance of the model free CQL on the Gym Mujoco tasks and that the proposed algorithm also uses CQL as the underlying model free algorithm, it would be good to add comparisons to COMBO (which is cited in the related work, but not evaluated here), which integrates forward model rollouts with CQL and demonstrated improved performance, to see if reverse imagination offers much benefit outside of the maze navigation tasks.

Minor notation suggestion: The authors train a generative model over actions which they denote as $\hat G_{\theta}(s)$. It would be clearer to denote it as $\hat G_{\theta}(a \vert s')$ to make it explicit at first glance that it gives a distribution over actions instead of states. Yes

The paper proposes an offline model-based RL approach (ROMI) for data augmentation based on generating "reverse" rollouts that can reach a given a target goal state using an inverse dynamics model and a reverse policy, which are trained using maximum likelihood estimation and conditional VAE respectively. This approach can be easily combined with existing model-free offline RL methods and leads to state-of-the-art results on a subset of D4RL benchmark tasks.  **Originality**: The work seems apply the idea of learning reverse dynamics model and policy to generate rollouts for augmenting the offline dataset, which seems quite neat. This seems to more conservative than forward-model based approaches for which the authors present some evidence in the experimental section. 

*Question*: Could the authors please elaborate on why the proposed approach is expected to be more conservative? For example, the reverse model could imagine trajectories reaching the target state from a failure state such as crossing through a wall as shown in Figure 3(g) (right). 

Related work is discussed adequately, a possible connection to goal-conditioned approaches could also be briefly mentioned. 

**Quality**: The submission is technically sound with well supported claims by experimental results with the strengths of the work being discussed adequately.  

**Clarity**: The submission is clearly written and well organized. There are minor typos but they don't hurt the clarity of the paper. One possible suggestion is to move the ablation of the reverse policy in the appendix to the main paper by making the size of the Figure 3 smaller. 

Also, why aren't standard deviations not reported for BC/MF/MB in Table 1 as well as that of BCQL in Table 3 (Some of the std deviations seem to pretty large and should be reported if possible. 

Some typos:
- Table 2 Caption: MoJoCo --> MuJoCo
- Line 305: bais --> bias

**Significance**: The results seem to be important and I believe others (especially those working on model-based offline RL) are likely to build on the ideas proposed in this work. The results on harder AntMaze tasks on D4RL (where existing model-based RL methods fail) as well as the ablation studies in the main paper and the appendix show the merit of the proposed idea. A major limitations of this work which is not discussed is the reliance on the reconstruction based objectives to learn the reverse model and the policy which may not scale well to real-world offline RL settings with rich observation spaces like images. Example benchmark domains with high-dimensional visual observation spaces include the Atari 2600 games [`1], DM locomotion tasks in RL Unplugged [2], and image-based locomotion and manipulation tasks [2]. A detailed discussion for this limitation as well as possible remedies or experiments on such domains should be included in the paper.

References:

[1] Agarwal, R., Schuurmans, D., & Norouzi, M. (2020, November). An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning (pp. 104-114). PMLR.

[2] Gulcehre, Caglar, et al. "RL Unplugged: A Collection of Benchmarks for Offline Reinforcement Learning." Advances in Neural Information Processing Systems 33 (2020).

[3] Rafailov, Rafael, et al. "Offline Reinforcement Learning from Images with Latent Space Models." arXiv preprint arXiv:2012.11547 (2020).

Meta-review: This paper presents an offline RL method based on reverse models. The contributions are original, and and while further analysis would be helpful in rigorously understanding how reverse models are helping, the empirical analysis convincingly shows that the improvements do not simply come from better model fitting. The empirical results are quite strong, including on challenging offline RL benchmarks. I recommend acceptance.

Please see the reviews and other comments on this page for further feedback when preparing the final version of the paper!

Decision: accept