Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers state representation learning problem in deep RL. It exploits the cycle-consistency supervision and develops a “learning via retracing” approach. Such supervision signals are generally inherent in existing data and does not need additional interaction with environment, which leads to better sample efficiency. Learning from predictive supervision from temporally forward and backward directions reveals information from both the future and past to the target state, leading to more accurate latent state inference. In particular, the paper proposes the Cycle-Consistency World Model (CCWM) along with practical considerations (e.g., adaptive truncation to remove irreversible states), under the model-based framework based on generative dynamics modeling (CCWM).
main_review: Strength: The cycle-consistency constraints could provide extra supervision signal (beyond forward predictive loss) for learning state representations and transition models, without additional interaction with environment. It also improves the representation learning performance (including the zero-shot transfer capability and long-range predictions) by obtaining better latent state inference. 

Weakness: The concept of cycle-consistency constraints have also been considered in the PlayVirtual work under model-free setting. Therefore, more thorough discussion about the novel contribution relative to PlayVirtual should be included.

Comments:
- The authors may need to discuss how frequent such cycle-consistency constraints appear in practical applications. For example, it is more common in navigation-type applications? Listing more practical application settings in this regard and discuss them would help justify how widely applicable the proposed approach is.
- Why the continuity detection on action-value function could detect the irreversible states? It seems that there is not much discussion on this point in Section 3.3. It is crucial to clarify it.
- VirtualPlay should also be included as a baseline in the experiment section in order to show which approach best exploits cycle-consistency.

summary_of_the_review: The paper proposes CCWM, a learning via retracing method, based on cycle-consistency constraint. It leads to better sample efficiency and final performance by learning better state representations. Need more clarification about its novel contribution relative to a recent work (VirtualPlay), which considers cycle-consistency learning in model-free RL setting.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper investigates a self-supervised approach for learning the state representation in RL tasks. In addition to the predictive (reconstruction) supervision in the forward direction, the authors include a “retraced” transitions for representation/model learning, by enforcing the cycle-consistency constraint between the original and retraced states. The authors claim that this facilitates stronger representation learning and improve upon the sample efﬁciency of learning. As it is not always possible to find such a cycle consistency between two states and for counteracting the negative impacts brought by the “irreversible” transitions, the authors add a novel adaptive “truncation” mechanism.
main_review: - The motivation for the cycle consistency is not really discussed. Why would enforcing cycle consistency improve representation learning?
- In equation 7: can you explain the choice of the KL distance between two rewards? How is the 2-wassertstein distance estimated from data?
- The "model-free instantiation of learning via retracing" (appendix C) is not clear. In particular, the paper states that there is a graphical illustration "shown in Figure" but there is no reference to any figure.
- In Figure 3, can you clarify why some of the baselines are straight lines? Can you provide more details than this explanation: 'We report the asymptotic scores for the model-free algorithms due to the large gap in sample efﬁciency comparing to the model-based methods."
- Section 3.3: The prediction of the irreversibility of a transition is not very clear. How is a "sudden change in the value function" a good predictor of the irreversibility of the transition? 
- In the conclusion, it is written "We empirically show that CCWM yields improved performance over state-of-the-art model-based and model-free methods on a number of challenging continuous control benchmarks, in terms of both the sample efﬁciency and the asymptotic performance." However, CCWM is only compared to dreamer and yields to comparable results in most cases? (the other baselines are only given as "asymptotic scores" and they have a direct access to the state?)

Minor comments:
- The paper formalizes an observation space, O that is "high-dimensional, due to either redundant information or simply because only visual inputs are available". Can you clarify whether that is the POMDP setting? If it is not the POMDP setting, it might be worth clarifying as this is the usual denomination used in POMDP.
- The paper is based on the open source implementation of Hafner et al. Is there an open source implementation of the modifications presented in this paper?
summary_of_the_review: The motivation for the contribution is not very clear and there are a few unclear elements in the formalization (e.g. sudden change in the value function for the prediction of the irreversibility) .
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a self-supervised approach for learning the state representation of RL tasks. The main contribution apart from prior works is to involve additional 'retracing' trajectory samples in representation training, which are trained by minimizing the propensity between retraced samples and forward posterior samples. They also proposed an intuitive way to mitigate the irreversibility in RL dynamics.
main_review: The strengths of the paper involve solid experiments and well-structured writing. However, I think some details are not illustrated clearly, so that I have the following questions:
1. Is the representation learning part jointly trained with RL algorithms or pretrained? 
2. In the $L_{retrace}$, why retracing samples $\check{z}$ are from the variational predictive distribution but forward samples $\tilde{z}$ are from the variational posterior distribution? 
3. How the “reversed” action policy $p_{\eta} $ is learned? by $L_{retrace}$?
4. The proposed method for dealing with “irreversibility” is too intuitive. I think it might fail in heavily irreversible environments. Why the proposed truncation method is not involved in your full algorithm in the appendix?
5.  As you mentioned CCWM could be combined with any existing RL algorithms, why not try one or two to show that the new representation learning loss could actually improve the performance. I think the performance of CCWM is comparable with Dreamer given the huge additional computational complexity. 

summary_of_the_review: The proposed representation learning method for RL is interesting, which involves addtional retracing samples for training. However, more explanations are needed to make it clear. I expect to see more elaborations.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes "learning via retracing" as an approach to learn state representations, through matching a trajectory both in the forward and in the backward direction. This paper then introduces Cycle-Consistency World Model (CCWM) which is a model-based RL algorithm which learns through retracing. This method is sample efficient and provides good state representations for predicting the future and generalization. Furthermore, it proposes a value-based approach to identify "irreversible" transitions.

main_review: Pros:

- Interesting technique to retrieve more supervisory signals from the data in RL
- Mathematically sound approach, with clear figure 2 to visually understand.
- Extensive experiments to evaluate the idea and interesting ablations and investigations described.
- Extensive details of algorithm and implementation + code is available.
 
##########################################################################

Cons:

- A few inconsistencies in the paper that make me think that there were extra parts that have been cut out to reach the page limit without reformatting:
    - Section 5.3: Details of modifications to the environment are said to be found in Appendix A, but Appendix A describes implementation details and not the environment.
    - Appendix C, last sentence. Model-free instantiation is shown in Figure ?? Figure doesn't exist.

##########################################################################

Questions during rebuttal period:

- Address/clarify cons above.
- How do you select the predefined threshold to find the sudden changes in the value function for your adaptive scheduling?
- I can imagine from Table 1, that the components that you change in the cheetah-run environment are related to Reward, Mass, Friction, Stiffness. I don't understand how changing the reward would affect evaluating an episode, as it doesn't change the dynamic behavior of the physics simulator and so the actual trained model shouldn't have any problem in evaluating.
- In Section 5.4, you write that the curve for hopper-stand in Figure 3b for CCWM lacks the adaptive truncation of irreversible states. Does this mean that all plots in Figure 3b for CCWM are trained on all transitions, with no truncation? Please clarify, as I imagined that the experimental evaluation of your method included both contributions (CCWM+adaptive truncation).
- In algorithm 1: I don't understand in the input, why in ${O^n_{t_n:t_n+K}, a^n_{t_n:t_n+K}}$ the $t$ has a subscript $n$, which should represent the n-th sample in the batch. Also, I think there is a typo and the subscript should be to $t+T$ and not $t+K$.
- In figure 8 (from Appendix F), you explain how CCWM yields less accurate predictions for the irrecoverable states, but to me the first row (true trajectory for Walker), doesn't look like an irrecoverable trajectory.

##########################################################################

Typos/formatting:
- Section 3.3, first paragraph: "Here we propose an approaches" -> "Here we propose an approach"
- None of the citations in Appendix A are in brackets anymore.
- Appendix A: "outputting the means the Gaussian distributions" -> "outputting the means of the Gaussian distributions"
- Appendix D, second paragraph: "empirical evaluation" -> "empirically evaluate"
summary_of_the_review: Overall, I am for accepting. I like the idea and I think it's a novel approach for state representations in RL. My major concern is about the clarity of the paper, as written above. Hopefully the authors can address my concern in the rebuttal period.
review_title: null

Meta-review: This paper proposes augmenting standard forward prediction techniques used for representation learning with backward prediction as well, termed "learning via retracing". The paper implements this idea in a Cycle-Consistency World Model (CCWM) and demonstrates that CCWM improves performance of a Dreamer agent across a number of Control Suite tasks. The paper also proposes a way to detect "irreversible" transitions and exclude them from the backwards prediction step.

This paper generated mixed opinions, and the reviewers did not come to a consensus on whether it should be accepted or rejected. In particular, Reviewer VSAG maintained it should be accepted, while Reviewer NEVM maintained it should be rejected. The other reviewers did not reply; I thought the authors' responses to their questions were reasonable so I assume their concerns were addressed (additionally, I don't believe a comparison to PlayVirtual is a justifiable request per the [reviewing guidelines](https://iclr.cc/Conferences/2022/ReviewerGuide), as it is concurrent work).

The reviewers generally agreed that the cycle-consistency idea proposed by the paper is interesting, well-motivated, and borne out by the experimental results. I agree with these points. The main weakness of the paper, brought up by multiple reviewers, was the justification/motivation for the method for irreversibility detection. The authors clarified in the rebuttal that the motivation comes from the idea that temporally-adjacent states with very different values will tend to be far apart in representation space. While I believe that is true, it's not at all clear to me that this necessarily *entails* irreversible transitions between those states. That said, my feeling is that this is approach is (1) not the main contribution of the paper and (2) empirically seems to work, based on the experiments, even if the motivation is unclear/unjustified. Therefore, I do not think the concern about irreversibility detection is grounds on its own for rejection.

Overall, I find this is a sensible approach to better representation learning in MBRL. I recommend acceptance as a poster.
Decision: Accept (Poster)