Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper suggests image synthesis method to improve data efficiency in offline RL.  In offline RL, we can only use stored data, but it is too small for improving performance. Thus, many researches try to learn model dynamics, and explore new states. For image based RL, they need observations as pixels. Thus, authors generate image data from new states with recurrent generative model from input state and previous observed image. 
strengths_and_weaknesses: Strength
To generate data in offline RL, they try to use multi-modal generative model for enough information. 
S2P largely improves the performance in DMControl tasks. 

Weakness
Not enough baseline, there are no comparison with other augmentation. 

questions: 1. Standard deviation of scores in tables. 
2. In table1, they mix dataset with random and expert dataset. Then, in some tasks, mixed dataset improve performance rather than expert dataset. It means that expert dataset cannot show diverse state and random policy would be Thus, S2P can be not needed, and we can improve performance with simple augmentations or exploration with random policy or small perturbed policy.  
3. How much do you need data to train S2P model? 
4. How can you evaluate the quality of synthesized images?  
limitations: It is hard to check the quality of synthesized images without a proper metric for evaluation. 
Now, I cannot assure that this generative model is needed instead of other data augmentation methods without comparison. 
If S2P does not improve performance significantly, it is not used because of the load of training for generative model.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes S2P: a method for state-conditioned image synthesis for offline image-based RL. The authors motivate their method by the fact that offline RL suffers from distribution shifts in the low data regime, and augmenting datasets using a learned dynamics model has previously been shown to be effective for mitigating this distribution shift. The proposed method is evaluated on 6 continuous control tasks from DMControl by training a number of offline RL methods on the original dataset as well as an augmented dataset and comparing the difference in downstream performance.
strengths_and_weaknesses: Strengths:
- Simple and intuitive extension of a state-based RL method to image-based RL.
- The paper is generally easy to follow and the contributions are clearly presented.
- Experimental evaluation demonstrates that synthetic transitions are valuable across a variety of algorithms, tasks, and data distributions. The comparison to conventional image augmentations in appendix is appreciated.

Weaknesses:
- The assumption that datasets consist of pairs of images and ground-truth states is obviously a very strong assumption that will not hold in a lot applications. This is not a problem in itself, but the authors don't seem to ever state it explicitly in the paper. I'd like to see this made absolutely unambiguous to readers.
- Given the above assumption of (I,s) pairs, there are a number of conceptually simpler methods that may or may not suffice. For example, one could simply learn an I -> s predictor using supervised learning and then apply one of the existing state-based offline RL algorithms. A slightly more sophisticated method could learn the I -> s predictor, augment the dataset using a state-based dynamics model, and then learn a state-based policy on the augmented dataset, and several other such approaches come to mind. In light of this, I'd like the authors to justify their more complex approach, ideally backed by data although that may not be feasible during the rebuttal itself.
- It is unclear if it is indeed the case, but it appears that Dreamer does not leverage state information at all in the comparison, although it is assumed available. A more fair comparison would be to also condition the RSSM of Dreamer on state. Judging by the results in appendix, it appears that data quantity is not a big factor in the poor performance of Dreamer, so such an experiment would help clarify whether the poor results indeed are due to the objective or lack of access to privileged state information.

**Post-rebuttal:** The author response addresses my main concerns and I am willing to raise my rating with the expectation that authors include reviewer suggestions in a future revision.
questions: I believe that my greatest concerns are already made explicit in the "weaknesses" section above. Specifically, I'd like the authors to

- Explicitly state key assumptions made + limitations of the method
- Justify why simpler approaches are not sufficient given the assumption of (I,s) pairs in the dataset
- Clarify whether Dreamer leverages the available state information in the comparison, and if not, what the justification for that is
limitations: The authors do *not* discuss potential negative societal impact of their work and do *not* explicitly discuss limitations of their method despite claiming so in the checklist. See my previous comments on limitations. I do not anticipate ethical issues.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work extends the distribution of the data generating policy in offline RL by first learning a generative model (S2P) which synthesizes raw pixels given state information and a previous frame. This generative model is dynamics-consistent and employs multiple modalities which is a new contribution compared to the single modality models of previous works. They show that their data augmentation strategy improves SOTA algorithms like CQL, IQL and an offline version of SLAC. They conduct several ablations showing the value of different components they introduce such as position encoding, multimodal affine transformations, state only affine transformations etc. They also show that even on unseen tasks with unseen image distribution, just having the state distributions allows S2P to generate reliable images to train an offline RL algorithm. 

strengths_and_weaknesses: This paper proposes a state-to-pixel generative model that can be used to generate dynamics-consistent transition augmentations to any offline RL algorithms. The paper is clearly written, describes the contributions and how it differs from other similar works in the area. The paper also conducts extensive experiments with three different algorithms on the DMControl datasets. The results are solid and the ablations show the values of each of the algorithmic pieces that were added. I believe that the work is still significant and demonstrates that augmenting data with dynamics-consistency is significantly useful for any offline RL learning algorithm. Thus it is a useful contribution to the offline RL research.

The paper has conducted experiments limited to only DMControl with reliable state transitions. It would make the model claims stronger if one were to see if the same data augmentation strategy could be applied to other image observation environments and partially observable environments. This model relies on the fact that the state representations are reliable and learns to generate new transitions via “exploring” in the state space and generating corresponding image observations. Not all problems provide reliable state transitions and representations and this work has not fully explored cases where state representations/dynamics are intrinsically less reliable and noisy.

questions: * I wanted to confirm if the baselines without and with S2P are trained with exactly the same number of transitions. If so, please clarify in the paper. Also, if the total frames are fixed, it will be good to know how the performance varies as a fraction of the mixing ratios. 
* There are a few very minor corrections in the paper that can be fixed in the final version if the authors do another pass over the paper  – eg. (“only single the previous image” in Sec 4.3). Section 3.2 claims that “naively optimizing a learned MDP is prone to failure”, but misses appropriate citations. 
* There is an interesting ablation where you zero out the image but keep the state intact in Figure 5c. Was the other ablation done ? i.e. to mask the state and only use the image to generate the next image. I understand that you present the Dreamer baseline for this case, but it would be nice to have the ablation on the same model to be more precise about the gains.
* You have shown interesting ablations with SAT, MAT and positional encoding in Figure 5. Since you introduce three components of the loss, it will also be interesting to see what value each component adds in terms of generalization. 

limitations: The authors have not addressed any specific limitations of the work. Please refer to the earlier replies for potential areas of improvement.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focuses on an offline reinforcement learning problem. A generative model S2P is proposed to generate image transition data by virtually exploring the state space while keeping the dynamics consistency. In detail, a multi-modal affine transformation module is proposed to utilize both state and previous image data to generate modulation parameters. The proposed method is evaluated on the DMControl benchmark, and thorough empirical results are given from aspects including offline policy learning, model-based method, i.e. Dreamer comparison, ablation study, and generalization performance.
strengths_and_weaknesses: **Originality**

Although the model-based generation is commonly-used in reinforcement learning (RL) methods, the image-based offline RL problem is novel, and the authors proposed the first image-based offline data generation method. But the core module in S2P, i.e. the multimodal affine transformation (MAT) is not novel, which has been investigated in previous works [1], [2].

**Quality**

The authors conduct detailed experiments to validate the proposed method S2P, and the results sound. It shows significant performance improvement in 6 image-based DMControl tasks. The authors also explore the influence of different data generation policies and give thorough visualization to help understand the image generation capability and the importance of each module in S2P. The overall empirical results are convincing. But there are still a few problems with the importance of multimodal input and other baselines, which are listed in the following Questions section.

**Clarity**

The paper is well-written and easy to understand. The motivation is clear. Sufficient related literature is included. The experiment configuration details are given in the appendix which helps the further implementation.

**Significance**

The underlying offline RL problem is important both in academic and industrial communities. The authors try to tackle an interesting image-based data generation problem. Although the core module MAT has been investigated in previous computer-vision community, the proposed S2P is still essential since the overall idea is simple and the performance improvement is significant. This encourages the RL community to explore more complex offline RL problems, i.e. with image input.

[1] Li B, Qi X, Lukasiewicz T, et al. Manigan: Text-guided image manipulation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 7880-7889.

[2] Park T, Liu M Y, Wang T C, et al. Semantic image synthesis with spatially-adaptive normalization[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 2337-2346.
questions: 1. Why is high-quality and accurate synthesis image generation important than data generation in latent space? Given the generated synthesis image, the offline agent should also to learn how to encode the image, which may cause additional learning difficulty.
2. What is the offline learning performance if only using state-based affine transformation instead of multimodal affine transformation, i.e. SAT vs. MAT?
3. What is the performance comparison with other offline reinforcement learning method, e.g. COMBO [1], ROMI [2] in which a learned dynamics model are used for data generation?
4. Why use the previous step image instead of the previous step state? By concatenating the current step and previous step state, the features change can also bring velocity which is the mean reason to introduce the image as in line  117:126.
5. Why VGG encoder pre-trained on Imagenet can be used for RL tasks, especially DMControl image observation feature encoding in Eq. 5 since the two domains differ a lot?
6. Is The baselines’ encoder different from S2P as mentioned in line 230:231? How do keep comparison fair?
7. Why introducing more generated image data cannot bring distinct performance improvement as shown in Table 1?

[1] Yu T, Kumar A, Rafailov R, et al. Combo: Conservative offline model-based policy optimization[J]. Advances in neural information processing systems, 2021, 34: 28954-28967.

[2] Wang J, Li W, Jiang H, et al. Offline reinforcement learning with reverse model-based imagination[J]. Advances in Neural Information Processing Systems, 2021, 34: 29420-29432.


Update after rebuttal

All questions have been addressed.
limitations: This submission is not related to a distinct negative social impact.
review_title: null

Meta-review: metareview: ## Summary
The performance of the offline RL methods can be limited by the amount of coverage in the dataset. Most real-world problems have limited coverage in the offline RL datasets and the sample efficiency of the pixel-based offline RL methods in general is often poor. Thus, it is an important direction of research to improve the sample-efficiency those continuous control pixel-based offline RL algorithms. This paper proposes a method called S2P which generates pixel based observations from the states by using a generative model. The paper shows improved results on offline DeepMind control datasets.

## Decision

The paper in general is well-written and clear. The idea is simple and seems to be effective compared to other data augmentation approaches. The reviewers were in general positive about this paper. I think the NeurIPS and offline RL community *would benefit from the findings of this paper*.  However, I think a few clarifications in the final version of the paper would make the contributions of this paper more clear. 

1. I found the improvements shown in the paper very encouraging. However, I found the choice of the dataset odd and confusing. In particular, I am curious why the authors did not decide to use the standards datasets published in RL Unplugged benchmark for offline RL. I think the authors should justify why they did not use those datasets in the camera-ready version of the paper, provide results on those datasets, and release the datasets that they used in this paper (perhaps contacting the RL Unplugged authors to see if it is possible to release them under RL Unplugged benchmark.) As it stands out, this paper only compares against baselines that the authors themselves implemented in the paper. Nevertheless, running experiments on the RL Unplugged would enable us to be able to compare S2P against other published offline RL baselines.
2. Authors should include the standard deviations in the camera-ready version of the paper as request by *reviewer i7hg*.
3. In general, I think the authors did a good job during the rebuttal and many of the reviewers raised their scores as a result of additional results and experiments that the authors have provided. The authors should include those results in camera-ready version of the paper including the clarifications about the questions that the reviewers asked in the rebuttal.
4. Currently the links and the references in the supplementary material are all broken. The authors should fix those in the camera-ready version of the paper.

**With the above points addressed in the camera-ready version of the paper, I think this paper would be ready for publication.**

confidence: Certain
Decision: Accept