Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper built upon recent advancement in NLP and CV to introduce an auxiliary task in RL. More specifically, the authors proposed to reconstruct latents from masked observations, with the goal of improving representations used in RL. A few experiments across both discrete and continuous control to demonstrate the promise of the proposed methods.
strengths_and_weaknesses: Strength:
1. Using the idea of auxiliary tasks to boost the RL performance has been explored and this paper belongs to this line of research. The idea from the proposed method looks reasonable and interesting.

2. The authors designed different types of experiments to justify the promise of the proposed method, and the results generally look promising.

3. This paper is mostly well written and organized.


Weakness:
1. One issue is that the current version lacks the explanation and investigation of why the proposed method should outperform the previous algorithms combining representation learning with RL, e.g., CURL. We can see some promise based on experimental results, but in-depth exploration on why MLR works is integral to make the claims convincing.

2. The numbers and plots in the experiments section definitely show that MLR achieves competitive performance. From Table 1, however, it seems that MLR has very similar performance when compared against PlayVirtual and CURL with 500k steps, which may suggest that the representations from MLR may not be sufficiently distinguishable from previous work. 

3. My another concern regarding the experiments corresponds to Atari games. In Table 1 of the appendix, the authors showed that MLR can outperform Baseline, which denotes Rainbow. The issue is that the scores here are much lower when compared with those in the Rainbow paper. Just pick a simple one: For the game Pong, the reported score here is -3.1 while it's 19 in the Rainbow paper. Consequently, it's not convincing to me that the auxiliary task can help the RL task on Atari games.

4. A minor point regarding the background in Section 3.1: POMDP distinguishes state and observation in its formulation, but the text there lacks the discussion on states.
questions: 1. The authors stated in L122 that MLR ``...is the theoretically applicable to different RL algorithms''. Is there any theory behind the proposed algorithm for the application in RL?

2. Do the features used for RL come from the output of online encoder, i.e., \tilde{s}?
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a mask-based latent reconstruction/prediction auxiliary task for visual RL. The proposed auxiliary task takes as input a sequence of observations, applies a uniformly sampled spatiotemporal mask to the sequence, and then task the network with predicting the latent representation of the unmasked sequence encoded by an EMA of the online encoder. The proposed method, MLR, is evaluated on DMControl and Atari tasks, and a number of ablation studies uncover the key design choices of the proposed framework.
strengths_and_weaknesses: Strengths:
- Conceptually simple, clearly presented, and ablations are insightful.
- Experimental results are encouraging, the provided rliable metrics are appreciated, and code is provided.

Weaknesses:
- DMControl evaluation. I understand that the particular choice of DMControl tasks used in the evaluation is convenient due to the number of available baseline results. However, performance on these tasks have largely been saturated by prior work and are thus not a good indicator of algorithmic improvements (e.g., does a median performance of 914 vs 929.5 vs 935 vs 907 vs 948 @ 500k steps really matter? I understand that 100k steps may be more informative than 500k but point still stands, and the improvement of curves for these tasks in Figure 2 of appendix don't appear significant to me, presumably for the same reason). I strongly suggest the authors evaluate on more challenging tasks / benchmarks in future revisions, although I realize that this may not be feasible within the strict time frame of a rebuttal.
- Related work. I would like to see more discussion of RL methods that use BYOL-style auxiliary objectives and data augmentation/masking in the related work section, given the strong similarity to such methods. I will be happy to provide references, but am under the impression that the authors are familiar with the literature given some of these works (e.g. SPR) are cited and/or briefly mentioned in the experiment section.

**Post-rebuttal:** In light of the author response I am willing to raise my score by 1. I expect the authors to include reviewer suggestions in a future revision.
questions: I believe I have already made my biggest concerns clear in the "weaknesses" section above. However, I do have a few questions for the authors that should be addressable during the rebuttal:
- It is not clear to me after reading the paper + appendix exactly which data augmentation is used and where. Can you elaborate on this?
- I'd also like the authors to clarify on how the "Baseline" SAC implementation differs from that of other published works. Appendix describes the architecture to be based on CURL; would "Baseline" be equivalent to CURL w/o the auxiliary loss? Is data augmentation applied?
- How do the listed wall-times for the proposed method compare to (1) the "Baseline" implementation without the auxiliary loss, and (2) prior work such as CURL, DrQ, SPR, or PlayVirtual?
limitations: I believe the authors provide a satisfactory discussion of potential negative societal impact of their work, but discussion of limitations is minimal. I'd like to see more explicit discussion of when the proposed auxiliary loss can be expected to be useful, and when it may fail (e.g., which key assumptions are made).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces mask based reconstruction (MLR) for RL with image inputs. MLR is an auxiliary objective optimized together with the policy learning objective. The idea is to reconstruct masked pixels in the latent state instead of the input space where state representations are inferred from original unmasked frames. The paper proposes to mask portions on both spatial and temporal dimensions in the observation sequence. There is an “online encoder” for masked observations and a momentum-encoder for original images. Furthermore, there is a Transformer-based latent decoder that also takes action embeddings as input. MLR uses a projection + prediction head to get the final prediction result, and cosine similarity to compute the distance between masked and original sequence. The paper presents several ablations about the masking strategies, depth of decoder, similarity losses, etc. Results on DMC-100K and Atari-100K show that MLR outperforms leading algorithms like SPR, PlayVirtual.  

strengths_and_weaknesses: ### Strengths
- Paper is well written and easy to follow.
- Results and ablation studies are thorough and demonstrates the efficacy of each component.
- Interesting direction to apply mask-based reconstructions in latent space as it is challenging to learn good latent representations while solving RL tasks.

### Weaknesses
- Novelty is limited as Masked based reconstruction are already known to work well in Vision tasks.
- Limitations of the proposed method are not discussed.

questions: 1. The method was not compared with EfficientZero which is an important baseline. How does MLR compare with EfficientZero?
2. The Appendix talks about the training time and details. How they compare with leading methods like SPR? 
3. There are no discussions on scenarios where the mask-based reconstruction is helping. For example, are improvements on games like MsPacman where the background is not moving or games with moving viewpoints like Crazy Climber? This could also help understand the limitations of the proposed method. 
limitations: The paper does not highlight the limitations of MLR. The paper should talk about potential bottlenecks when scaling MLR to more complex tasks.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper tackles the field of state representation learning in visual RL. They propose Mask-based Latent Reconstruction (MLR) for learning state representations based on masking sequences of input images. This practice is inspired by the computer vision and natural language processing communities. The approach is different than the ones used traditionally in these fields by reconstructing the missing content in the latent space instead of the original one. The authors provide an extensive evaluation to demonstrate the improvement in performance their method achieves to SOTA methods. Their evaluation includes discrete and continuous control environments as well as ablations of the different design decisions introduced in their approach.
strengths_and_weaknesses: Strengths:
========

- The idea of using masked observations for state representation learning is to my knowledge novel in RL
- The paper is well situated with respect to previous related work
- The paper is mostly (with few exceptions) well-written and easy to follow (clear)
- The proposed method is simple to implement
- The evaluation is extensive: it includes multiple baselines, different kinds of environments (discrete and continuous action spaces), and ablations of the different components of the method
- The results show that the proposed method mostly outperforms the baselines and also leads to a low reward variance. 

Weaknesses:
==========

- In section 3.2, paragraph iii) (seconding) should be rewritten and made clearer. Multiple details in this paragraph are ambiguous and hard to follow. The first sentence in this paragraph is itself very confusing and could be interpreted in many ways (what is meant by "visible ones" and "implicit way"). The motivation for using the actions is missing or at least not convincing. Understanding relative positional embedding requires reading [43], which makes the paper not self-contained. Additionally, it's unclear what the plus in equation 2 means, is it an actual addition or a concatenation? it can't really be an addition since the two vectors have different dimensions, so is it a concatenation or something else? please specify.
- The proposed approach claims to reconstruct the masked content in the latent space. This is done using the cosine loss based on a target that is obtained using the original images and a moving average of the online encoder and projection head. It is unclear to me what is the meaning of such a loss, why would we want to push the representation to be similar to another one produced by a moving average over its previous weights but using the original images? I would have been more convinced if the method simply relied on reconstructing the original pixels as done in the ablation. Let me rephrase, I'm not saying that the method doesn't make sense, I'm just saying that the intuition behind the state representation learning method is unclear (at least to me). I'm willing to change my rating if this aspect is made clearer and of course convincing.

Minor Problems:
============

- the paper repeatedly mentions that the presented approach reconstructs the masked content in the latent space, which is a false statement. The method attempts to reconstruct the information available in the masked content in the latent space.
- some typos and grammar problems here and there. I would recommend a non-author to go over the paper + some grammar check software. e.g. in line 262 neural --> natural. 
questions: - If I understood correctly, "baseline" in your tables refers to either SAC or rainbow. In table 1, that should be SAC. Can you explain how SAC is outperforming other methods which add on top of it such as SAC-AE and CURL? these methods were built with the motivation that SAC on pixels isn't good enough. Can you explain how these results relate to the original ones from those papers?
- this brings me to the next question: how do you ensure a fair comparison in your evaluation? do all the methods use the same SAC implementation, same encoder, decoder architectures, and hyperparameters, etc...? Please elaborate on the standardization of experiments.
- Related to what I mentioned in the weakness section, can you elaborate on the intuition behind your choice of targets?  to me, this seems like a random choice. In other words, what are you trying to enforce on the state representation? what kind of property? and how does your method allow for this property to emerge?  For instance, when using autoencoders, the property is compressing, with contrastive methods, it's somehow clustering... Please give a clear explanation here and not a high-level explanation.
- Related to the previous comment, if MRL is not integrated with policy learning and used instead for pretaining only, would it actually help? I know that one of the motivations of the approach is that the two objectives are simultaneously and jointly trained, but such a test would be important to validate the importance of the learned state representation. Can you include such an experiment in the final version? for the rebuttal it doesn't have to be over all the environments...(whatever can be managed).
limitations: The paper lacks an evaluation of the obtained state representations. An evaluation could be something like the regression accuracy (or a correlation of the two variables) of the original states (provided by simulation) based on the representation. The pertaining experiment I mentioned in my last question would also qualify as a (qualitative) evaluation of the obtained representations.
review_title: null

Meta-review: metareview: Unanimous accept (scoring 5675, with confidence, from reviewers who have published in similar areas)

All four reviewers all agree that the work is novel (using such masked-based representatives/auxiliary losses in an RL setting in the _latent_ space opposed to the original RGB space) which doesn't sound like an obvious idea a-priori, but (all reviewers agree) the authors back this up with extensive experiments and ablations of their own method in Atari and DMC. The authors were very responsive with reviewers, resulting in 3 reviewers increasing their scores each by one. All reviewers agree this paper is easy to follow, and most of their stated weaknesses/confusions have now been addressed or clarified.

My view of this work is that the main contribution is a latent reconstruction loss, as an additional objective to whatever the RL task objective is. And this is useful for using data augmentation / self-supervised learning in RL tasks where representations that can do pixel-reconstructions aren't what's really required (and they show this experimentally) e.g. distractions exist in images, so they focus on the lossy latents instead. This seems more novel and distinct to simply pre-training some representation using contrastive / masked methods common in the literature. And then in their (extensive) DMC+Atari experiments they augment by removing (pixel) spatial-time cubes in video sequences (given correlations nearby etc) and force the latent structure to capture whatever still needs capturing, perhaps a Q-function input for SAC, trained jointly. So I agree with the other reviewers that this work seems interesting and novel where the NeurIPS community would benefit from reading + understanding it
confidence: Less certain
Decision: Accept