Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper shows a latent-planning RL model  to
learn control policies directly from pixels. To learn a better
representation it uses recurrent model contrastive learning approach
which enhances the representation learning performance of single
frame based contrastive methods.
This was tested robotic control suites with challenging distracting
backgrounds.

The main contribution of the work is the addition of recurrence to the
model and the extensive testing and explanation on the reason it tends
to work better.
main_review: ## Strong points

The paper is well written and easy to follow. I particularly
liked the more straight-to-the-point and honest approach on writing.
The authors could have easily try to mask their similarity with some
other methods in the literature but they were upfront with that.
This made the understanding of the paper way easier and also
easier to understand the main contributions.

There is a substantial comparison with relevant models in the literature that already
deal with the DCS. The results obtained are really impressive
and are a new state-of-the-art  on interactive environments
with distractive backgrounds.

The ablations specifically the ones concerning the usability of the recurrent
contrastive method are very useful. Specially the intuition on the negative
samples per mini batch.
Further, the appendix provides even more ablations, making very
clear that a recurrent state has a major impact on the general performance.


The visualizations are also good standard to be used on this field which performs
control directly from pixels. They also show that their model is capable of directly
detecting the object of interest.

##  Weak points

The proposed model is indeed very similar to Dreamer[1]. The authors address specifically
one point from dreamer: the contrastive learning strategy, which failed to produce better
results as reported by [1]. However, I do believe that this closer look into this
specific point can be useful for the community and insightful in general.

## Questions

I would be curious to find more references on the idea that hard negatives needs to be found
on the mini batch in order to contrastive learning to have a good performance. 

I wonder how the performance of this method would be in a more open task than mujoco
style robotic control. Tasks with a different initial state or with other elements in the scene 
that the agent need to interact might have a negative impact on this method results.


[1] Hafner, Danijar, et al. "Dream to control: Learning behaviors by latent imagination." arXiv preprint arXiv:1912.01603 (2019).
summary_of_the_review: I think this is a very useful and well written paper.
Even though the scope is small, the results are convincing
and it shows a very clear way on how to effectively use contrastive
representation learning methods  while learning to control directly from
pixels.


## After Rebuttal

After reading the other reviewers comments and the rebuttal 
I see that i missed some literature that needed further comparison.
I think it is a good and well written paper but I would lean for rejection 
given this new data.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a recurrent state-space model that learns robust representations for robotic control. The proposed method builds on top of prior works on world-models which learn a latent dynamics model of the agent which can be used for planning and action selection. Different from prior work such as Dreamer and SLAC which rely on pixel-based observation reconstruction, this paper highlights that a simpler contrastive loss for the next-observation prediction achieves better results **if** a recurrent state-space model is used for the latent space. Results are presented on the Distracting Control Suite benchmark and show strong improvements over prior approaches.
main_review: # Strengths
* The clarity of the paper is good and the approach has been described well.
* The experiments have been well designed and indicate strong improvements in the model robustness (particularly Figure 2 and Table 3). Table 2 was helpful to understand the key design choices and their impact on final performance. 
* The mask visualizations in Figure 5 provide good clarity on how the robustness is achieved by the proposed method.
* The ablation studies in the main paper and supplementary clearly highlight the benefits of individual components of the method.


# Weaknesses

## Novelty
* The novelty of the approach isn't quite clear. In Sec. 1 (page 2), the authors highlight that one of the key findings is that "contrastive learning can in fact lead to surprisingly strong robustness to severe distractions, provided that a recurrent state-space model is used". This finding itself is valuable in my opinion, but the method "CoRe" does not seem novel to me. Relative to Dreamer and SLAC, the key novelty appears to be the contrastive loss term from Eqn (1) which predicts future observations instead of auto-encoding. However, the idea of using a recurrent model to predict encodings of future observations (i.e., not an auto-encoder) has been studied in CPC [1], action-conditioned CPC (aka CPC|A) [2], Predictions of Bootstrapped Latents (aka PBL) [3], etc. CPC does not use action conditioning, but uses contrastive learning. CPC|A uses action conditioning + contrastive learning (very similar to CoRE). PBL uses action conditioning + reconstruction (similar to "Recon" baseline, but has an additional loss to predict state conditioned on observation representation). 
* Given the above, the methodological novelty of CoRe is not clear to me (particularly, relative to CPC|A). Is there something different about the contrastive loss in CoRe (relative to CPC|A) without which the performance degrades severely? Or is the novelty only in the observation that recurrent models are needed for contrastive loss to work well? Note, while some recent work like DBC does not use recurrent models, prior approaches already use them with contrastive loss. 


## Intuition behind why recurrent state models are needed for contrastive learning
*  [End of section 1] The authors suggest that when recurrent models are used along with contrastive learning, the smoothness of the state-space ensures the presence of informative hard negatives in the same mini-batch of training. This isn't clear to me. Positives and negatives are obtained from the next-step real observation's encodings (just a single frame) and does not use the recurrent model. How is the smoothness of the state-space related to hard negatives? While the performance degrades without the recurrent model in Figure 3 (top), isn't this more likely due to a poorer state representation caused by the lower capacity (no RNN) and lack of information aggregation over time? This would affect all models, not just the CoRe.


## Missing baselines / related works
A few related methods have not been compared with or discussed. 

* Augmented Temporal Contrast (aka ATC) [4] does not use recurrence, but uses future observation prediction and has been shown to achieve good improvements over CURL. 
* CPC|A [2] uses recurrence, future observation prediction (contrastive), and action conditioning. This is very similar to the proposed method and should be compared with.
* PBL [3] uses recurrence, future observation prediction (reconstruction), and action conditioning. This method introduces an additional P(state | observation) term that results in bootstrapped learning (might improve over the Recon baseline).


## Other concerns

* Why is CoRe worse than PSE on 4 / 6 tasks for DAVIS (2 videos) in Table 1? Why is CoRe better with 60 videos?
* Why are only 2 baselines used in Figure 4? Can the authors include the complete set? Do we observe similar trends as Figure 2?
* Are the findings from Figure 5 specific to CoRe? Or do other methods (like PSE) also learn similar masking functions? 



[1] CPC - Oord, Aaron van den, Yazhe Li, and Oriol Vinyals. "Representation learning with contrastive predictive coding." arXiv preprint arXiv:1807.03748 (2018).  
[2] CPC|A - Guo, Zhaohan Daniel, et al. "Neural predictive belief representations." arXiv preprint arXiv:1811.06407 (2018).  
[3] PBL -  Guo, Zhaohan Daniel, et al. "Bootstrap latent-predictive representations for multitask reinforcement learning." International Conference on Machine Learning. PMLR, 2020.
[4] ATC - Stooke, Adam, et al. "Decoupling representation learning from reinforcement learning." International Conference on Machine  Learning. PMLR, 2021. 
summary_of_the_review: I am concerned about the lack of clear novelty in the paper, and other experimental issues highlighted in "Weaknesses". I will update my rating based on the author's responses.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper has presented CoRe, Contrastive Recurrent state space model, for model-based robust model-based reinforcement learning for robotic control. Standard reconstruction-based state space models are less robust in the unstructured real-world scenarios because of the high-frequency details. Instead, CoRe learns the state space model with contrastive learning, which greatly improves robustness. In addition to this, a policy is being learned with SAC. Experiments on distracting control suites and several robotic control tasks demonstrate the better robustness of CoRe.
main_review: Weaknesses:

The major issue is that the proposed idea and the experiment setup is not novel. They highly overlap with a prior CoRL 2020 paper, Contrastive Variational Reinforcement Learning for Complex Observations (CVRL) [1], which, however, has not been cited in the submitted manuscript. They overlap in the following aspects:

1/ The idea is the same. CVRL also extends RSSM using contrastive learning and aims to improve the robustness of the learned model against real-world observations with high frequency noise. Both of them use InfoNCE for contrastive learning and the same RSSM structure.  In addition, both CVRL and the proposed method use the policy loss from dreamer for learning the policy network. The resulting equations and loss functions of the two algorithms are almost the same.

2/ CVRL also experimented on natural Mujoco games, which introduces moving backgrounds into the standard dm-control suites. This is exactly the same with the distracted dm-control suites used in the submitted manuscript.

Beyond what has been discussed in the paper, CVRL has mathematically shown that by replacing the generative observation likelihood with a contrastive objective, we can lower bound the original ELBO.

There are some other weaknesses, but I believe the issues discussed above are sufficient to make it a clear rejection.
summary_of_the_review: The paper is highly similar to a prior work as mentioned above, and as a result, the contribution of the paper is very limited. I would vote for a rejection.
review_title: null

Meta-review: Meta Review of Robust Robotic Control from Pixels using Contrastive Recurrent State-Space Models

This work investigates a recurrent latent space planning model for robotic control from pixels, but unlike some previous work such as Dreamer and RNN+VAE-based World Models, they use a simpler contrastive loss for next-observation prediction. They presented results on the DM-control suite (from pixels) with distracting background settings. All reviewers (including myself) agree that this is a well-written paper, with clear explanation of their approach. The main weaknesses of the approach are on the experimental side (see review responses to author’s rebuttal by skrV and cjX3). Another recommendation from me is to strengthen the related work section to clearly position the work to previous work - there is clear novelty in this work, but this should be done to avoid confusion. The positive sign is that in the discussion phase, even the very critical cjX3, had increased their score and acknowledged the novelty from previous related work. In the current state, I cannot recommend acceptance, but I’m confident that with more compelling experiments recommended by the reviewers, and better positioning of the paper to previous work, I believe that this paper will surely be accepted at a future ML conference or journal. I’m looking forward to seeing a revised version of this paper for publication in the future.
Decision: Reject