Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Problem Setup: The paper proposes a mutual information objective to learn a latent representation which
can be used for planning. The paper note that most of the existing model based RL methods learn a 
model of the world via reconstruction objective, which requires to predict each and every detail of the visual 
input, and hence can be detrimental in case of noisy inputs or in the presence of distractors. 

Proposed idea: In order to tackle this problem, the paper proposes a mutual information objective to maximize 
the mutual information between the latent codes at distinct time steps.  In order to capture the history of the past, 
the authors utilize a recurrent model (from Dreamer Model) to encode information about the history of the trajectory.
The paper also uses 2 different objectives in order to prevent the representation from collapsing. The paper proposes
to use a mutual information objective between the observation and the encoding of the observation (as in Dreamer), 
as well as consistency objective in the latent space (already used before). Essentially the underlying idea behind the proposed method is not new per se, but as far as I know this is the first paper, which has shown to make it work on DeepMind Control tasks.

Experiments: The authors compare the proposed method to DREAMER model on 6 DeepMind Control (DMC) tasks. 
The authors also evaluate the robustness of the proposed method by evaluating the capability of the proposed method 
in dealing with complicated backgrounds (given the scenario, when the entity of interest occupies a small region in the input).

Clarity: The paper is clearly written. 

References: Their are bunch of references that could be cited. Shaping belief paper [1] also uses a CPC style objective for learning a model of the environment. [2] also learns a model of the environment by predicting only the relevant information by constructing a temporal information bottleneck but still within the framework of maximum likelihood prediction. [3] also uses a mutual information based objective and without any explicit reconstruction.

- [1] Shaping Belief States with Generative Environment Models for RL https://arxiv.org/abs/1906.09237
- [2] Learning dynamics model in reinforcement learning by incorporating the long term future https://arxiv.org/abs/1903.01599
- [3] Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction
 https://arxiv.org/abs/2007.14535

Scalability: It would be interesting to see how the proposed method evaluates on more challenging tasks just as on atari or on continuous control tasks such as "box" stacking which requires some relational reasoning. Since the underlying idea has been tried in some other context and in this work the contribution is to make it work for deep RL problems, it becomes important to evaluate on more challenging problems and tasks. 

======

After Rebuttal: I have read the rebuttal, as well as reviews by other reviewers. I keep my original score. Hope to see a better version of the paper soon.


review_title: PREDICTIVE CODING FOR PLANNING IN LATENT SPACE

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
The paper introduces a new method for model based RL that learns a dynamical latent representation from pixel data (images) using a maximum mutual information criterion together with a predictability loss. The core idea is that maximizing mutual information between states bias the encoder toward learning predictable (and therefore potentially task relevant) latent features while discarding unpredictable features.

Relevance:
The paper addresses the very relevant problem of learning representations usable in a control task.

Originality:
The core novelty of the paper is to combine the mutual information approach introduced with PC3 in the context of control theory with the "dream to control" approach for model-based reinforcement learning in pixel space.

Scientific quality:
- The proposed approach is in general well motivated. However. I am not convinced by the emphasis that the authors put in the proposition that the maximum mutual information loss helps to learn task relevant features. While it is true that the proposed approach is biased towards temporally predictable features, most distinctive features both in the real wold and in most games and simulations have as much temporal predictability as the task relevant ones. For example, the video backgrounds in the experiments are completely task irrelevant and at the same time highly predictable. In general, the encoder cannot trulely promote task-relevant features without having access to the reward structure.
- The experiment section offers a decently wide range of experiments. However, the authors should include more baselines, possibly including other model-based methods such as [1] and model-free methods such as some variant of DQNs.

Pros:
- Very relevant research area
-Rather original combination of methods
-Clear and well-written paper

Cons:
- The main claim that the method is biased towards learning task-relevant features is questionable. 
-  The experiments should contain more baselines including other model based approaches and some model free approach.

References:
[1] Hafner, Danijar, et al. "Learning latent dynamics for planning from pixels." International Conference on Machine Learning. PMLR, 2019.


review_title: Interesting work using information theory for model-based RL

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

This paper proposes an information-theoretic framework for learning a world model that encodes task-relevant information of the world. It shows that the learned encoder and dynamics model can be used to train the policy and fitting the value function to agent to perform comparibly well to Dreamer on standard tasks and outperform them when there are distractions in the scene. The paper also provides a theoretical anylisis of the task-relevant information in the encoding.

Pros:

This paper provides a clean framework that learns the embeddings that contain task-relevant information.

I appreciate the theoretical formulation to show that the optimal embedding will containis sufficient to train the policy on. I believe this theoretical analysis is novel.
 
The experiments provide an insightful ablation and show that the proposed contrastive model is less agnostic to the background distraction.

Cons:

My main conern is regarding the novelty. This paper misses a few recent relevant work [1, 2, 3]. These work also use contrastive losses to learn a latent world model and outperforms Dreamer on DMControl tasks and natural background. I'd like to see the comparison against [1,2] or clarify how NMPC is different and/or better in some ways. 
How do you think NMPC performs against this model-free version with similar task-relevant objective [4]? 

In the orignal paper dreamer also proposes a contrastive loss for training the latent model although it is not temporal. Is the ablation NMPC-No-Rcr in figure 2 comparable to that or different? Also, how important is having a recurrent dynamics model as supposed to a feedforward dynamics model?

How senstive the algorithm is to the hyperparameter combinations? The loss seems to have a few terms to be tuned. It would be helpful if the ablation can show the importance of other components such as Non-Markovian consistency and dynamics smoothing.

Conclusion:

Overall, the paper is well written and easy to understand. It proposes a simple framework with experimental and theoretical support. The main downsight lies in its novelty as a few other works that aim to tackle this issue in a similar way. If this can be addressed, I think it is a really good paper.

References:

[1] Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction (https://arxiv.org/abs/2007.14535)

[2] Contrastive Variational Model-Based Reinforcement Learning for Complex Observations (https://arxiv.org/abs/2008.02430)

[3] Learning Predictive Representations for Deformable Objects Using Contrastive Estimation (https://arxiv.org/abs/2003.05436)

[4] Learning Invariant Representations for Reinforcement Learning without Reconstruction (https://openreview.net/pdf?id=-2FCwDKRREu_
review_title: Simple and clean framework; Novelty concern

final_score: rating: 5: Marginally below acceptance threshold
review_content: ---- Summary ----

The paper proposes a method for visual model-based reinforcement learning that relies on contrastive learning to learn the predictive model. Building on Hafner’20, the paper replaces the image reconstruction objective with a noise contrastive estimation (NCE) objective for the latent dynamics model, an NCE objective between the images and representations, and an additional maximum likelihood objective for the latent dynamics. It is shown that the method is competitive to Hafner’20 on the DM Control benchmark, and outperforms Hafner’20 on DM Control tasks with natural images used as background. The paper also theoretically analyzes one of the used objectives, arguing that it may lead to discarding irrelevant information.

---- Decision ----

The paper tackles a relevant and important problem of building predictive models that do not rely on image reconstruction, and proposes a promising method to this end. However, there are several technical weaknesses of the presented approach, and further important baselines and ablations are omitted. Due to lacking experimental evaluation, I lean toward rejecting the paper, but would be happy to update my score if the experimental evaluation were improved.

---- Strengths ----

The paper investigates a relevant and promising direction. The experiments show the effectiveness of the proposed method in simple continuous control scenarios, further supported by some theoretical analysis.

---- Weaknesses ----

The major weakness of the paper is a lack of representative baselines. The paper cites several similar papers, including Ding’20, and Shu’20, which the model is called “an extension of”. How is the proposed method better than these prior methods? Further, the central claim of the paper is the importance of non-markovian predictive coding, also claimed to be the main technical novelty (although this was already used in the foundational paper by van den Oord). It is never evaluated whether the non-markovian part improves performance. The contrastive learning version of Dreamer should be added to the plots as well.

A harder to fix issue is that the proposed method is rather unprincipled. Three different competing objectives are used (e.g. maximum likelihood of the latent dynamics reduces I(z;z’) while the other objectives increase it), and the interactions between them are not discussed. The paper presents only intuitive justification for the three objectives. This issue also causes the need to tune balance terms between different losses. For instance, the method requires tuning of the weight on the reward prediction, and requires different weights for different environments (Table 1 in the appendix), while in Dreamer this weight is always set to 1 and no hyperparameters are environment-specific. The weight for the recurrent CPC objective $\lambda_1$ does not seem to be specified in the paper.

---- Additional comments ----

Eq 1 - none of the symbols in the equation are defined. 

### ---- Update ----

The authors' response does not satisfactorily address my concerns. My main concern is that the paper does not properly evaluate alternative choices, even though a large literature on contrastive learning exists.  While in the revision one baseline was added to the appendix, the main experiment still only contains a comparison to Dreamer and ablations. Further, it appears that the proposed method fails completely when the non-markovian part is removed. This is rather concerning since learning markovian latent dynamics is important and also possible with other methods (e.g. PlaNet-RNN). As far as I can tell, the paper does not discuss this issue and does not explain why learning non-markovian dynamics is crucial.  

Overall, the paper proposes an interesting method but fails to provide any insight into how the method compares with possible alternatives. I, therefore, maintain my borderline score. 
review_title: Review 2: promising method, evaluation is lacking

Meta-review: This paper presents Non-Markovian Predictive Coding (NPMC), a method for learning state representations in visual RL domains that can be used for planning. This work builds on recent work on PC3 (Shu et al. 2020) and PlaNet (Hafner et al. 2020). Concretely, NPMC replaces the image reconstruction objective in PlaNet with a noise contrastive estimation (NCE) objective for the latent dynamics model, an NCE objective between the images and representations, and an additional maximum likelihood objective for the latent dynamics.

Reviewers were in agreement that this paper tackles an important problem and appreciated the writing quality, the experiments that demonstrate effectiveness of NPMC in continuous control scenarios, and accompanying theoretical analysis. However, reviewers were on balance in consensus that this paper needs another iteration before it can appear. Aside from discussion of related work, the main weakness noted by reviewers is that the manuscript in its current form makes it clear how NPMC differs from closely related methods from a technical point of view, but does not make it sufficiently clear to what extent non-Markovian predictive coding leads to improved planner performance. In particular, the paper lacks detailed comparisons to baselines, and reviewers were not sufficiently convinced by experiments that were added to the appendix after discussion.

The authors indicate that their contribution is that NPMC extends PC3 to RL tasks. The metareviewer appreciates that experimental comparisons can require creative thinking when baselines are not directly applicable to the tasks of interest, but would nonetheless like to encourage the authors to consider how they can improve their experiments.
Decision: Reject