Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes to use cycle consistency for self-supervision and shows that this approach effectively increases RL data efficiency on discrete and continuous control benchmarks. The ablation study provides the useful intuition that cycle consistency helps mainly with regularizing the visual encoder and less so with training the forward dynamic model. 
main_review: 
Strength
- The paper is clearly written. The experimental results are relatively strong. 
- Cycle consistency is intuitively a simple auxiliary tasks that could strengthen learning signals and it is nice that the paper shows empirical supports for this argument. 
- Appendix B.2 is very helpful for understanding how the runtime overhead. 

Weakness

- The author pinpoints the problem that the encoder is less likely to be trained on transition pairs that are under-explored by the current policy. However why training an extra backward model, with the same amount of real-world data, would address the problem? In order to make this claim, one needs to see to what extent do these virtual trajectories lie in the latent space of meaningful states. Is it possible that the forward and backward models are exploiting meaningless latent space or the current policy, such that the virtual trajectories become degenerated solutions to the auxiliary task? 

Minor
- Line 345, RDM (I assume it's reversed dynamics model) is inconsistent with BDM used in the rest of the paper. 

limitations_and_societal_impact: Yes, the authors addressed the limitations and potential societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In order to increase sample efficiency of image-based Reinforcement Learning (RL), the authors propose PlayVirtual. They leverage the auxiliary task of learning a backwards and forwards dynamics model as well as as cycle-consistency loss between the two models. They use the model to generate virtual trajectories which they use to train the RL algorithm. The authors evaluate the method on DMC and Atari tasks as well as providing interesting ablations about the usefulness of each aspect of the method. 



main_review: Strengths: This paper, to my knowledge, presents a novel method in image-based RL. Cycle consistency is known to be useful in computer vision, and leveraging it and recent work in self-supervised representation learning is very interesting. The method tackles a problem of significance and the authors have explained different aspects of PlayVirtual well. The ablations and experiments are quite informative. The figures and the paper are clear and it is overall well written. 



Weaknesses: 

While the experiments are detailed and comparisons are thorough, I have a few concerns about the  exact reported results. It would be good to see exactly how much the virtual data helped the method. For example other methods both learn an RL policy and a model, and use synthetic data as well, such as [1], Dreamer, SLAC etc. An ablation over the number of synthetic trajectories would be useful. Additionally, do the authors have intuitions as to why the Atari experiments don't seem to show too much of a benefit over the other methods? 

In its current form, it is unclear how the method exactly differs from SPR, and the authors should clarify that. Why does SPR underperform PlayVirtual? Additionally, it would be good if the authors provided more intuition as to why a cycle consistency loss is good. 



[1] When to Trust Your Model: Model-Based Policy Optimization. Janner et al., 2019. 
limitations_and_societal_impact: The authors have disucssed limitations but not broader soceital impact. Image-based RL, especially in robotics settings has concerns such as safety etc which can definitely be discussed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper aims to improve data efficiency and state representation of RL by introducing cycle consistency objective and augmenting training with virtual trajectories generated by jointly learned dynamics models.

In addition to the (forward) dynamics model to predict future states like SPR, this work also learns a backward dynamics model to predict the previous states given current state and previous action. A cycle consistency loss is introduced to enforce consistencies between forward and backward state transitions on a given sequence of actions. All models are trained jointly to optimize the model-free RL loss (Rainbow and SAC), prediction loss over the observed next / previous states, and cycle consistency loss between forward and backward trajectories. The learned dynamics models are auxility tasks that improve state representation of the encoder, but are not directly used in policy learning (like model-based RL) otherwise.

The method outperforms SPR and other baselines on Atari and DeepMind Control Suite benchmark.
main_review: The proposed method looks good and well motivated overall. Introduction of backward dynamics model, cycle consistency loss, and virtual trajectory augmentation are interesting contributions over existing work like SPR, that is shown to be effective in improving state representation and data inefficiencies.

The method is focused on using learned dynamics models and data augmentation to learn better state representation, but not on improving the policy learning (RL loss). The cycle consistency loss provides learning signals that enhances the encoder representation without ground truth supervision, which is a nice way to improve data efficiency. One question I have is
whether the cycle consistency loss is trained on both virtual trajectories and real trajectories?
IIUC, real trajectories will only update the encoder but not DM and BDM, while virtual trajectories can update all of those. It would be nice to have a summary of what components are being updated by each loss for different examples. Related to that, can you clarify what Baseline + BDM in the ablation study consists of?

The authors evaluated their method on both discrete and continuous control tasks, with ablation studies for backward dynamics model, prediction steps, and the impact of cycle consistency loss on encoder and dynamics model. Empirical results on both Atari and DeepMind Control Suite benchmark are impressive, outperforming the baselines and existing SOTA. The evaluation is mainly on the quality of the final policy, but not on the state representation or dynamics models, which are the main contributions of this work. Some form of comparison of the dynamics model with SPR will be interesting, but not required.
limitations_and_societal_impact: There is no clear limitations or negative societal impact.
review_title: null

Meta-review: The paper presents a new state representation learning algorithm for RL by introducing cycle consistency objective and augmenting training with virtual trajectories generated by jointly learned dynamics models. The idea seems novel and sound. Most reviewers find the paper clearly written with a novel idea, and the experimental results are strong enough on Atari and Deepmind suite to support the hypothesis of this work. 

Learning state representation and world models for RL has been a popular area, this work manages to also leverage ideas from SPR, and also add a  cycle consistency loss to enforce consistencies between forward and backward state transitions on a given sequence of actions, which is a novel idea.

One question that I have is the possibility to use the learned model not only for regularizing representation learning but also for model-based control, since right now the RL part is still based on model-free methods. But on the overall, this is still an interesting paper for the NeurIPS community.
Decision: Accept (Poster)