Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper designs a multi-view masked autoencoder for learning features for visual control. The motivation is that better visual representation leads to better control, which can be obtained from multi-view pre-training using the proposed framework. Results on RLBench have validated the effectiveness of the proposed framework and each introduced components. 
strength_and_weaknesses: **Strength**

- This paper first verified the effectiveness of using multi-view data for pre-training. Moreover, the multi-view pre-training also helps the single-view control tasks. 
- The proposed framework MV-MAE is conceptually simple yet works great. It out-performs the single-view baseline and alternative self-supervised methods baseline. Each sub-modules also contribute to the final performance. 
- Experiments are solid and the results are promising.
- Writing is easy to follow. Related work is properly discussed.

**Weakness**

- To me, the problem setting isn't challenging enough to test transfer learning. The pre-training and the testing scene are all RLBench and almost the same (same background, robot, etc). Compared to the normal setting in CV tasks, the training-testing gap isn't significant enough. It's encouraging that the visual pre-training still help in this scenario, but it could be more convincing if such representations can be tested under different setting. 
- Even though the masked auto-encoder (MAE) models outperfoms the Masked Siamese Network (MSN) and Time Contrastive Network (TCN), I don't get the motivation behind it. What's the issue that multi-view MAE addressed? What's the scenario that MSN/TCN fail to handle but MAR is able to? What's the difference between the learned features? Can we somehow visualize it and see the difference clearly? 
- The technical contribution is somewhat limited as 1) video MAE is proposed in Feichtenhofer et al., 2022, and 2) the benefit of using multi-view data for control has been studied before (related work last paragraph). This work is a bit incremental considering prior work MWM (Seo et al. 2022a). 

clarity,_quality,_novelty_and_reproducibility: **Clarity and Quality**

The proposed framework MV-MAE is technically sound. It's an multi-view extension of prior work MWM (Seo et al. 2022a). Even though simple, MV-MAE works well on RLBench tasks. Each sub-module is properly ablated. Overall, the paper is clean and easy to understand.

**Novelty**

As I stated in the weakness section, the proposed MV-MAE is a bit incremental considering prior work in visual pre-training (MAE, video-MAE) and in pre-training for control (MWM). Also, there are existing work studying the benefit of multi-view data.

**Reproducibility**

The source code has been submitted. Even though I didn't run it and try to reproduce the results in the paper, the code reads reasonable. 
summary_of_the_review: This paper presents a framework for using multi-view visual pre-training for robot control. Even though sound and promising, the technical contribution and the experimental evaluations are both limited.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents Multi-View Masked Autoencoder (MV-MAE) for multi-view representation learning and show its benefits in visual manipulation tasks. Experiments are conducted on RLBench demonstrating that MV-MAE (with MWM) significantly outperforms baselines (MWM, MWM+MSN, TCN) consistently across various setups. Ablation experiments provide enough details to understand the effects of different design choices. Written presentation is clear and easy to read & understand. 
strength_and_weaknesses: # Strength
- The idea of multi-view representation learning with MAE and applied to visual control tasks is novel and interesting.
- Experimental results are strong and superior to baselines. The set of selected baselines fully cover most cases of comparisons(to understand the differences). Ablation experiments also fully cover most of design choices.
- Written presentation is great (high clarity and easy to follow).

# Weakness
- Experiments are done on only one benchmarks.
- On the downstream application of visual control tasks, only one model, aka MWM, is used with MV-MAE.
clarity,_quality,_novelty_and_reproducibility: * The proposed approach is novel.
* The experiments are solid and the author(s) provide code (assume to be reproducible).
* The paper is written in high clarity and easy to read.
summary_of_the_review: As mentioned above, the proposed approach is novel and experimental results solid, I would recommend an acceptance for this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Motivated by the fact that data diversity plays a key role in representation learning, this paper proposes "Multi-View Masked Autoencoder" to leverage the cross-view and cross-frame information to learn the visual representation for control tasks. Experiments validate that the proposed multi-view representation learning framework outperforms both single-view and other multi-view representation learning baselines. 
strength_and_weaknesses: __Strengths:__
- The paper writing is in general clear and easy to follow. 
- The performance gain is significant compared with single-view and other multi-view baselines. Ablation experiments are also on point and validate various design choices
-  Detailed appendix about reproducibility.


__Major Weaknesses:__
- "Method" part is argurably the weakest part of the paper, as this work pretty much follows the _Masked World Models for Visual Control_ paper and extend it into a cross-view setting without major changes. It applies MAE into the multi-view scenario and validates its performance in this setting.
- The message this work delivers is well-known already. It is obvious that learning from cross-view cameras can boost the single-view results.   
- Some minor issues regarding the paper writing, typo and citations. (See below)

__Minor Issues:__
- First paragraph of introduction. starting off the paper with multi-modal learning is inappropriate. The paper is using camera only and it's not comparing itself with nor improving upon multi-modal approaches. Quoting multi-view camera papers makes more sense in backing up your argument in diversity, as opposed to quoting multi-modal papers.
- Do not include Google Brain in the citation of "Time-contrastive networks: Self-supervised learning from video" 
- Typo. Figure 1 caption, the autoencoder reconstruct"s" all frames at the same time
clarity,_quality,_novelty_and_reproducibility: __Clarity__: The paper is clearly written and easy to follow.

__Quality__: The quality of writing and visualization are both good.

__Novelty__: The technical novelty is marginal. The paper uses existing architecture and applies it into a new setting. 

__Reproducibility__: The paper seems to provide enough details to reproduce the results. But I did not try it myself.

summary_of_the_review: The paper is well written and it has its significance in experimental results as the improvement over baselines is noticable and the ablation study is well executed. But the technical novelty is limited as it directly applies existing Visual Control MAE model into the cross-view scenario. The message about the importance of using multi-view information is delivered by many other vision work already so it's not new as well. Given the great results and the marginal novelty, the paper straddles right on the acceptance threshold, leaning a bit towards rejection due to the lack of new messages.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper explores multi-view representation learning and introduces the Multi-View Masked Autoencoder (MV-MAE) framework. A video autoencoder is updated to function with multi-view input video streams. The learned representations are shown to be useful for the downstream task of visual control by training reinforcement learning agents. Evaluation is conducted on visual manipulation tasks from RLBench. Results show improvement over baselines. 
strength_and_weaknesses: Strengths

- Interesting problem area.

- Well written and easy to follow.

- Evaluated versus baselines, including an ablation study.

Weaknesses

- Lack of technical novelty

The proposed MV-MAE is an agglomeration of recent techniques. The underlying method is MAE [1], but using input in the style of [2] (convolutional features, instead of raw pixels). A learnable parameter reflecting the camera viewpoint / frame timestep is added to each token in the style of [3]. The rest follows a standard video masked autoencoder [4]. Though a view masking strategy is proposed, it is ultimately not compared against any alternative strategies. The visual control portion of the paper uses representations from MV-MAE as input to an off-the-shelf RL algorithm from [2] with evaluation from benchmark suite RLBench.

[1] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. "Masked world models for visual control." Conference on Robot Learning, 2022a.

[3] Geng, Xinyang, et al. "Multimodal Masked Autoencoders Learn Transferable Representations." International Conference on Machine Learning: Workshop on Pre-Training, 2022.

[4] Feichtenhofer, Christoph, et al. "Masked Autoencoders As Spatiotemporal Learners." arXiv preprint arXiv:2205.09113 (2022).

- Lack of detail

Most of the pieces of the proposed approach are from pre-existing works, therefore no detail is given in the manuscript outside of the reference. For example, what's the format of the learnable parameter representing camera ID / frame ID and how are they integrated? Additionally, what are the optimization details and how does the iterative training process work?

- Limited evaluation

Though the evaluation is fairly extensive, the experiments don't give insight into specific design choices that are crucial for MV-MAE to "work". For example, what is the impact of the view masking strategy? 
clarity,_quality,_novelty_and_reproducibility: - Easy to follow. Well written.

- Likely reproducible.

- The work appears to be original.
summary_of_the_review: The topic area, multi-view representation learning, is interesting.  Empirically, the proposed method (MV-MAE) shows improvement over baselines on the downstream task of visual control. However in terms of technical novelty, the representation learning component is a slightly tweaked masked autoencoder. The visual control portion exists just for evaluating the learned representations, and uses an off-the-shelf RL algorithm from a recent paper. The manuscript is of high quality but very little space is allocated to describing the proposed method (essentially just Section 3.1). For me, the paper reduces to applying masked autoencoders to a pre-existing visual control task. The results are compelling, but novelty is lacking, which makes this a tough paper to review. 

Ultimately, I feel this paper falls on the "marginally below the acceptance threshold" side of things. For me, I would want the paper to allocate more space to method description: highlighting important details that make it work (e.g., is it the specific choice of view masking strategy?) and adding experiments that reflect why this is the right way to structure things in the multi-view case.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary of the paper: This paper considers learning a good visual representation for downstream visual manipulation tasks. The key idea is to extend the masked autoencoder to be capable of leveraging multi-view input video streams. Experiments on RLBench show the effectiveness of the proposed method for training a standard reinforcement learning agent, outperforming both single-view and other multi-view representation learning baselines. Also, the proposed method is flexible for training reinforcement learning agents with different numbers of cameras during representation learning and behavior learning. Some ablation studies are performed to investigate the effect of different design choices.

Strengths: The reviewers recognize the major strengths of the paper as 1) a simple yet effective approach; 2) a good set of experiments with strong results and informative ablations for a specific task/setup.

Weaknesses and missing in the submission: The rebuttal has addressed some of the reviewers’ concerns. But there is a general consensus on the weaknesses and limitations of the paper among reviewers, as summarized below:

1) Lack of technical novelty is the major weakness. The proposed method can be viewed as a combination of existing works: extending video masked autoencoders (Feichtenhofer et al., 2022) to multi-view and following standard visual control framework (Seo et al. 2022a) for downstream task evaluation. As mentioned by the reviewers, except this combination, the specific technical contribution of the work is limited and also not well highlighted in the current submission. In particular, as pointed out by the reviewers, why components of the proposed method lead to the correct way for extending video masked autoencoders to the multi-view setting and why the proposed method outperforms baselines - there is a lack of in-depth discussion and empirical validation in the submission.

2) Reviewers are also concerned with the setting of the paper, and argue that more challenging settings should be considered, such as addressing the domain shift between pre-training and testing, conducting additional experiments on other benchmarks, and investigating other reinforcement learning algorithms. The authors followed the reviewers’ suggestion. Some results are provided by the end of the discussion stage (e.g., on the Meta-World benchmark), while other results are pending (making the proposed method work for another reinforcement learning algorithm.
justification_for_why_not_higher_score: While the authors improved the submission and made some clarification in the discussion stage, the reviewers were still not convinced and were firm with their initial assessment. By the end of the discussion stage, three out of four reviewers kept their ratings as marginally below the acceptance threshold. The other reviewer was slightly positive about the novelty, but mentioned the pending experiment and low confidence regarding if the current setup is challenging enough.

Given the reviewers’ comments and recommendation, the area chairs did not find sufficient cause to overturn the reviewers' consensus. The area chairs think the paper has great potential and implications for practical applications. By addressing the remaining concerns (e.g., technical novelty and more comprehensive evaluation), the paper would improve a lot in the next cycle. 

justification_for_why_not_lower_score: N/A.
Decision: Reject