Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The article proposes a new model for pretraining 3D vision downstream tasks with Masked Image Modelling. The authors focus on improving the performance for monocular 3D vision downstream tasks such as depth estimation.  Even though the article has appropriately presented the intended ideas, the following comments can be considered to improve the article.
strengths_and_weaknesses: Abstract
1.	It is suggested to include the highlight of the results obtained using the proposed model. And, more detailed explanation of the constraints of the existing methods could be reduced as it is repeated in introduction as well


1.	In table 1, the last two rows show the impact of this normalization on the Croco model pretraining. But in line 217, it is mentioned as first two rows 
2.	Figure 5 and 6 could have the Y -axis label mentioned on the graphs to get a clear visibility 


1.	List of acronyms can be displayed in the appendix.
2.	Punctuation operators like comma, spacing in equations, math notations, and between lines should be thoroughly checked for correctness.

questions: Introduction
1.	Line 40, the word “instead” could be removed as it does not add meaning to the introduction of the proposed model
2.	Alignment of Figure 2 and 3 could be changed in a closer view with the in-text citation and the corresponding abbreviation of “Croco” could be given before describing both figure 2 and 3 as this term is used in image labelling’s.
3.	Line 52 , as per the understanding and explanation given about  the proposed model, it “first(masked) image” to be mentioned instead of “first(target) image”
Related works
1.	Outcome (research gaps) of related works or summary of related works can be added at the end of the related works section.
Cross-view Completion pretraining
1.	In subsection, Details on the decoder in line 144, more detailed explanation on the self-attention and multi-head attention block working could be given. As they contribute majorly in prediction of the masked area on the image.
2.	In line 154, the CatBlock proceeds to multi-head self-attention and MLP layer but it is not clearly shown in Figure 4 
3.	What is the difference between multi-head cross-attention and multi-head self-attention. It would be better if more explanation is given

limitations: It will be good if the difference between downstream tasks and downstream classifiers be mentioned in self-supervised setting.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a novel unsupervised representation learning task that utilizes pairs of images showing the same scene from different viewpoints, unlike the existing solution that uses a single image itself for pre-training. The proposed cross-view completion pretext task helps the networks to focus on other image, as well as visible content of input, which in turn improve the pre-training ability. Experiments shows that the proposed pretext task has shown the superiority performance compared to existing methods.
strengths_and_weaknesses: + Using the cross-view completion as a novel pretext task is very interesting and makes sense. 
+ The state-of-the-art performance was attained, even though the methodology is very simple (it can be easily implemented similarly to MIM).
+ Two novel attention blocks in the decoder, which are tailored to cross-completion tasks, are interesting and makes. 
+ The paper is well written and easy to follow.
questions: - For the pre-training dataset, the authors used "synthetic" image pairs of 3D indoor scenes driven from 3D models, such as ScanNet or Replica, which makes sense. But, we also has many "real" image pairs of 3D scenes, such DTU [CVPR'14]. It would be interesting if the performance with such real cross-view dataset is evaluated. 
- Similarly to above, idea of using "another" images for Masked Image Modeling (MIM)-like pretext task is very interesting, which definitely reduce the ambiguity of existing MIM-like pre-training. Then we may consider other interesting setting in a manner that we may use a synthetic warping to generate "synthetic" cross-view image, similarly to geometric matching [rocco, CVPR'17]. Such a framework can also be applied to ImageNet benchmark as well. In the current version, Habitat dataset is only used for pre-training for most experiments. It would be very interesting if additional comments or experiments are done regarding this.
- The constraint of "co-visibility greater than 50%" seems heuristic. The additional experiments as varying the such co-visibility ratio, e.g., 10% or 90%, would be very interesting. 
- One more interesting thing is that in MAE [CVPR'22], masking ratio of 75% was an optimal in some cases. But in this paper, 90% masking is considered as optimal, mainly based on the analysis in Fig. 5. Is there any dataset dependency of decoder dependency for this? It would be very interesting and important experiments and analysis.
- What happens if we just use a pure decoder? or decoder from MAE [CVPR'22]?
- Computational complexity would be compared.

Minor comments:
- Masking schedule will be also very interesting issue, e.g., dynamic masking ratio?
- Cross-view vs. Multi-view? clarify this difference. In my opinion, "multi-view" is also fine.
limitations: The authors mentioned the limitations of this paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents CroCo, a new self-supervised learning approach for pretraining transformers for low-level and 3D computer vision tasks. The design of the approach is similar to the recent Masked Autoencoder (MAE), but uses two images corresponding to two views of the same scene (with some overlap) rather than single images. The main novelty is conditioning the prediction of the masked patches of an image on another complete view of the scene, in addition to the visible patches of the image and their spatial encodings. This allows the model to use the information in this second view to predict the masked patches, if it can successfully reason about the spatial relationships between the two views.

The proposed method is evaluated on an array of tasks, including standard ImageNet classification, depth estimation on NYUv2, 8 geometric tasks from Taskonomy, and camera pose estimation. On the tasks that require geometric reasoning, CroCo consistently outperforms MAE.
strengths_and_weaknesses: ## Originality

### Strengths 
This paper follows a long stream of self-supervised learning works targeting image classification and dense prediction tasks like detection and segmentation. This work builds on the MAE idea of predicting masked patches of an image, but conditions this prediction on additional view of the scene. This is very good idea for self-supervised pre-training of representations for downstream tasks that involve reasoning about the geometric properties of images, because it makes sense that in order to use the information in this additional view, the representation has to encode information about the spatial/geometric properties of each view.

### Weaknesses
There is some related work that is highly relevant but not properly cited. These are recent works that tackle self-supervised learning from RGBD data. The authors do not need to compare with these methods, but it would be beneficial to include them in the related work.

El Banani, Mohamed, and Justin Johnson. "Bootstrap your own correspondences." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

Liu, Yunze, et al. "P4contrast: Contrastive learning with pairs of point-pixel pairs for rgb-d scene understanding." arXiv preprint arXiv:2012.13089 (2020).

Hou, Ji, et al. "Pri3d: Can 3d priors help 2d representation learning?." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

## Quality

### Strengths
1. _Motivation and technical implementation of main idea_: As mentioned in the originality section, condition on a complete 2nd view for prediction of masked patches is well motivated. The paper proposes two different decoder modules to combine the information from the masked and 2nd view, CrossBlock and CatBlock, and empirically evaluates the performance/computational cost trade offs between each module (Table 1). The paper presents high quality ablations of masking ratio and target normalization, as well as showing downstream performance as a function of amount of pretraining, which ensures that baseline methods have been trained enough.
2. _Evaluation_: The datasets used for evaluation as well as the baselines are well chosen, with comparisons being made with MAE, DINO, MultiMAE trained on ImageNet and MAE trained on Habitat (Habitat is the main training set for CroCo). Using both the semantic and geometric tasks confirms that masked patch prediction conditioned on an additional scene view is a significantly better choice for pretraining geometric tasks than ImageNet.
3. _Performance_: The margin of performance improvement over the standard MAE is significant 

### Weaknesses
1. _Comparison relative to fully supervised methods_: While the authors provide comparisons with supervised baselines for camera pose estimation, it would be highly beneficial to include such results to serve as a reference for the other tasks as well. 
2. _Current reliance simulators like Habitat to generate multiple views_: The current training setup requires access to a scene data simulator like Habitat to generate pairs of overlapping views. This potentially limits the applicability of CroCo to more complex, non-indoor scene domains.

## Clarity

### Strengths
The paper is very well written and easy to understand. The architecture diagrams, visualizations and supplementary video are high quality and help with understanding the paper. The plots and tables are clear.

Minor weakness: It might be useful to label the rows of Figure 2 on the figure itself rather than the caption.

## Significance

### Strengths
This work clearly demonstrates the utility of using an additional scene view for self-supervised training via masked patch prediction. Further, the empirical analysis shows that existing masked  patch prediction methods are less suitable for spatial/geometric tasks.
questions: 1. What are ways in which the limitation of requiring a simulator like Habitat (which in itself is "expensive" as it requires labor intensive scans of indoor 3D scenes) can be addressed? How does this impact the downstream applications that this model is suitable for?
limitations: The authors have discussed some limitations of their work throughout the draft. The paper itself doesn't have high potential for negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes to extend MAE to cross-view completion as a pretext task. The encoder takes in a masked input and unmasked reference view, and the decoder use cross-view attention to reconstruct the input. The paper shows that it improves 3D related monocular tasks comapred to MAE / DINO that do not do cross-view pretraining. The paper also shows comparable results on optical flow, pose estimation, etc.

strengths_and_weaknesses: Strength
+ The paper proposes an interesting idea that extend MAE to mult-view for 3D downstreaming tasks that goes beyonds semantic to more geometric task. 
+ The ablations are solid and support their design. 
+ The empirical results seem to demonstrate the effectiveness of their method on 3D monocular tasks. e.g. Figure 6, Table 2. 

Weakness
1. The method assumes >50% overlaps between the reference view and the target view. I wonder if they have studied how the area affects the performance. 
2. The model is trained with a set of synthetic data (specifically indoor scans in this paper), which may limits its application and scalability.
3. I do not fully understand why the masked ratio matters a lot even with cross-attention since the query will have a global “field of view” from the reference image and no direct context from its own. Can the authors comment on that? 

questions: See weaknesses 1-3
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: Combining masked image modeling with cross-view completion, the paper develops a self-supervised pretext task appropriate for learning visual representations for downstream 3D tasks.  After the author response and discussion period, all four reviewers give positive ratings.  The pretext task design is novel, well-motivated for 3D representation learning, and shown to be experimentally effective on downstream 3D vision tasks.
confidence: Certain
Decision: Accept