Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper improves the DreamerV2 model-based RL framework, by replacing the the discrete world model encoder (looks like a VQ-VAE style encoder with recurrency? but I' m not sure) to a MAE-like encoder (The main difference is that it's masking the feature instead of the origin image). Experiments show that the proposed method achieves much better performance tha the original DreamerV2.
strengths_and_weaknesses: - Strength: It achieves much better result than the baseline method in several tasks in the settings that the author propose.

- Weakness: Experiment settings seem to have a huge gap between this submission and the original DreamerV2 paper. To be more specific, in the dreamer V2 paper (https://arxiv.org/pdf/2010.02193.pdf), it reports result after 4e7 steps for Humanoid Walk (2010.02193.pdf Appendix Fig A.2). Yet in this paper, there seems to be no more than 3e6 steps executed in any of its setups. As we all know, the training process of RL agents are unstable and the may suffer from zero-reward at the beginning of its training process, it would be important to wait until the training process to converge. Furthermore, in Figure 3 of this submission, Hand Insert / Shelf Place task increase concern about this issue, where we notice the DreamerV2 reward begin to increase after ~2e6/~3e6 steps, which is quite similar to 2010.02193.pdf Appendix Fig A.2.

I'm also quite confused why the paper does not report any Atari Game or Humanoid task in the DM Control Suite, which are reported by the original DreamerV2 paper. 

Since the core contribution of the submission is additional (i.e. change the encoder), experiments details and solidity are important.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: See the Weakness section. My main concern about this paper is the experiment part that, as has been mentioned in the Weakness section. Since the core contribution of the submission is additional (i.e. change the encoder), experiments details and solidity are important. Since the experiment setups may be not aligned with the DreamerV2 paper, and the paper does not report any Atari Game or Humanoid task in the DM Control Suite, which are reported by the original DreamerV2 paper, I am not very certain if the experiment part of this paper is convincing enough.

That being said, I myself do not have actual experience training model-based RL agents, so I rate the confidence (Reviewer Expertise) to be 2. If there are any severly technically wrong part of my review please the authors and the ACs point them out.
issues: - Concerns about experiment details mentioned in the Weakness section.
- There are some obvious mistakes in the figures. For example, I noticed that unit of success rate is "%" yet the scale is from 0.0 - 1.0. Obviously the author means 0.0 - 100.0.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a method for decoupling visual representation and latent dynamics learning for control. It does so by combining  recent approaches for visual representation learning (ViT) and latent dynamics learning (Dreamer) for control such that they are decoupled. This decoupling allows for learning a better prediction model as demonstrated through experiments in a diverse set of environments. 
strengths_and_weaknesses: Strengths:
- The paper is well written and well structured 
- has a number of interesting experiments that show the proposed methods superiority to the chosen baseline Dreamer V2 as well as an extensive ablation study
- Learning the reward alongside latent features in ViT is an interesting way of encoding task relevant information in visual representation architecture.

Weaknesses
- Missing relevant works
- Only one baseline that does not decouple visual representation and dynamics learning 
- Will the auxilliary reward prediction in the visual representation learning not limit the generalizability of the learned visual representor to only one RL task?
- Minor issues such as missing acronym expansion
- Lack of hardware experiments

Questions/Doubts:
- In Figure 6.a, is the plot for Pixel patch (MAE) produced with reward learning?
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper combines two recent approaches for visual representation learning and latent dynamics learning for control such that they are decoupled. While this proposal is not entirely novel, this paper demonstrates the advantage of this decoupling empirically with several experiments in different simulation environments. There is one novel idea (in my opinion) - the auxilliary reward learning for encoding task relevant information. However, this might limit the learned visual representation's generalizability to different tasks.
issues: - Missing relevant works specifically with respect to decoupled visual representation learning and dynamics learning:
	- [1] uses an special bottlenecked autoencoder that generates relevant 2D keypoints on the image for learning visual representation separately from learning a dynamics model on top of these features.  [2, 3] demonstrate this method's effectiveness on hardware including arm robots [3]. This method can be considered an additional baseline.
	- Another relevant work in this regard includes  [4] which learns Dense Descriptions as visual representation  and learns a dynamics model from these representations separately. This method requires RGBD images, however.
- Minor:
	- On Page 3, CLS is not defined.

References :

[1] Minderer, M., Sun, C., Villegas, R., Cole, F., Murphy, K. P., & Lee, H. (2019). Unsupervised learning of object structure and dynamics from videos. Advances in Neural Information Processing Systems, 32.

[2] Lambeta, M., Chou, P. W., Tian, S., Yang, B., Maloon, B., Most, V. R., ... & Calandra, R. (2020). Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters, 5(3), 3838-3845.

[3] Das, N., Bechtle, S., Davchev, T., Jayaraman, D., Rai, A., & Meier, F. (2020). Model-based inverse reinforcement learning from visual demonstrations. CoRL 2020.

[4] Manuelli, L., Li, Y., Florence, P., & Tedrake, R. (2020). Keypoints into the future: Self-supervised correspondence in model-based reinforcement learning. arXiv preprint arXiv:2009.05085.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents Masked World Models (MVW), a model-based reinforcement learning framework for visual control.

There are a few unique designs of MVM compared to existing approaches:
- MVM decouples visual representation learning from dynamics learning, this is different from the existing Dreamer framework that uses a single model to learn both visual representations and dynamics together.
- MVM uses a self-supervised Visual Transformer for representation learning, and it performs masking on convolutional features, instead of masking on pixels.
- MVM also uses an auxiliary reward prediction to encode task-relevant information.

Experiments on synthetic datasets show that MVW significantly outperforms the state of the art.
strengths_and_weaknesses: Strengths:
- It seems to me that, MVM incorporates a few design changes (mentioned in the summary part of my review) to the Dreamer approach and showed that these design changes lead to significant performance gain.
- The paper reads very well. Although I am not in this field, I was able to quickly grasp the related work and the unique contributions of this paper.

Weaknesses:
- Although the paper is well written, the contributions seem incremental -- it is adding small changes and tweaks to existing approaches.
- Real-world robotic experiments would be more convincing.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, interesting ideas and good results. I vote for weak accept.
issues: I am curious does decoupling the visual learning from the dynamics learning make it "slower" to train the approach, compared to having a single network doing both tasks?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper considers model-based RL with visual input. The key idea is to decouple visual representation learning and dynamics learning. For visual representation learning, inspired by the recent masked auto-encoder (MAE), the paper trains an autoencoder with convolutional layers and vision transformers (ViT) to reconstruct pixels given masked convolutional features. To encode task-specific information, the paper introduces an auxiliary reward prediction objective for the autoencoder. For dynamics learning, the paper adopts the DreamerV2 approach to learn a latent dynamics model that operates on visual representations from the autoencoder. The paper presents performance results on a variety of visual robotic tasks from Meta-world and RLBench.
strengths_and_weaknesses: Strengths

The paper proposes MWM consisting of an autoencoder with convolutional layers and vision transformers (ViT) to reconstruct pixels given masked convolutional features,  an auxiliary reward prediction objective for the autoencoder, a latent dynamics model that operates on visual representations from the autoencoder. 

It shows that self-supervised ViT trained to reconstruct visual observations with convolutional feature masking can be effective for visual model-based RL. 

Weaknesses

How MWM differs from DreamerV2 is not clearly stated. The main difference seems to be a redesign of the representation model and image decoder. The two models of MWM use ViT encoder and decoder with conv feature masking. 

There are several questions regarding the comparison with DreamerV2. What codebase does the paper use? Since DreamerV2 only evaluated on Atari games, does the paper make any changes to DreamerV2? The poor performance might be due to no adaptation of model architecture and hyperparameters. How does MWM compare with DreamerV2 on Atari games?

The most relevant paper is cited in appendix,
T. Xiao, I. Radosavovic, T. Darrell, and J. Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06173, 2022.

which uses features from a masked auto-encoder. The paper should compare MWM with this paper.




robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: The paper introduces masked auto-encoder features and conv feature masking to DreamerV2. This redesign of DreamerV2 inspired by vision transformer and masked auto-encoder seems to be very appealing and promising since the feature learning can potentially be much better. However, the comparison with DreamerV2 lacks details. Comparing DreamerV2 designed for Atari games on Meta-world and RLBench with no change does not seem to be convincing. One way to address this concern is to compare MWM directly with DreamerV2 on Atari games.

Furthermore, the Xiao et al. paper has applied masked auto-encoder to motion control. The paper should also consider comparing with the this most related paper cited in appendix.

===
Several of my concerns are clarified during rebuttal. I upgrade my rating to weak accept from weak reject.
issues: It seems that predicting reward is very important to achieve high performance. The paper should provide more information on what features  are enabled by predicting reward. How does this finding change with different environment and tasks? Since the paper evaluates only in a simulator, can reward be effectively predicted for real world data? 

It seems that conv feature masking is a type of dropout. Instead of conv feature masking, if dropout is used, will it achieve the same performance results?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper successfully achieved visual representation learning for world models in visual model-based RL with good accuracy using appropriate transformer models and masking processes.
As a result, the proposed method achieved higher success rates than conventional methods in variety of robot tasks.
strengths_and_weaknesses: Strengths:
- The proposed method was validated in a wide variety of ways and demonstrated sufficient experimental value.
- The intention of the model design was clearly explained qualitatively.

Weaknesses:
- Although the proposal is a decoupling of representation learning and dynamics learning, the reward function is included in representation learning, which makes the visual representation task-relevant and lacks versatility.
- Although the masking method was changed, its benefit was only shown qualitatively and experimentally, so I cannot find mathematical explanations.
- The analysis of why the learning curve was better than the baseline was insufficient, since only the learning curve was included in this paper.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The performance of the proposed method was experimentally verified by a large number of simulations, including ablation studies.
On the other hand, the proposed method was not mathematically convincing.
Too much emphasis was placed on exceeding the baseline in terms of the control performance, so further analysis may be needed to understand why the proposed method was better.
issues: I think, for the versatility, it would be better to learn visual representation and dynamics together because they are task-irrevant (and ultimately, environment-irrevant) components, and the reward prediction should be decoupled from them.
It seems to me that some other task-irrelevant information should be utilized to prioritize observations, as discussed in Limitation.

The addition of the reward prediction finally led to exceed the baseline, so I feel that it would be more important than masking and decoupling, which are in the title of this paper.

I know the page limitation is strict, but the authors putted too much information in Appendix, causing bad readability.
At least, in my opinion, the optimization problem for the proposed method should be inserted in the main text, rather than the definition of Dreamer.
review_title: null

Meta-review: metareview: This paper introduces an model-based RL approach that decouples visual representation learning and dynamics learning using a ViT. Results show that both this decoupling and the replacement encoder result in improved performance on a range of simulated benchmarking suites and tasks (RLBench, metaworld, dm_control). In the initial review process reviewers noted that the paper was

* Well written, interesting idea and a unique model design
* Good performance demonstrated in simulation and well structured ablations

but expressed concerns about 
* Some missing literature, and questions about the strength/ suitability of the baselines (eg. no comparison with other decoupled models, baseline tuning)
* No experiments on real hardware
* Potential limitations of the approach re versatility due to the inclusion of reward in the representation learning task, which learns task conditioned dynamics models

The authors rebuttal included some additional baselines, argued that there is evidence that similar models do transfer to real world settings, and added additional results to address concerns about overfitting to tasks by conditioning on reward (reward signal can be task agnostic or aid in representation learning). Reviewers were generally positive about this work post rebuttal, so I am recommending acceptance. Although there is a strong focus on real robot experiments at this years CoRL, I believe the extensive number of experiments on multiple benchmarks makes up for the missing real robot experiments to some extent. 

I encourage the authors to include the additional experiments in the supplementary material included with the camera ready paper, along with additional clarifications and discussion points raised during the review process.


confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)