Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to integrate depth signal into self-supervised learning methods as an additional signal in 2 ways. The first method simply concatenates the depth channel onto the RGB image and trains the SSL methods with RGBD input. The second method uses the single-view view synthesis methods to generate new views and use them as augmentations. Experiments show significant improvements over the baseline methods especially in corruption datasets, ImageNet-C and ImageNet-3DCC.
strength_and_weaknesses: o	Strengths
	The paper is well written.
	Experimental results seem promising as the results in both 2 methods outperform the baseline results.
	Ablation experiments are well conducted although some experiments are missing.
o	Weaknesses
	There is no computational complexity information about adding the depth information in both methods. For the first method, concatenating a depth channel to the input, depth maps are pre-extracted before the training and used in augmentations, only in Random Crop and Horizontal flip transforms. In other types of transforms, depth map is assumed to be the same and as stated in the paper, it has a stronger correspondence.
Furthermore, only one of the two images is augmented (the other is the original image) rather than both augmented images in the originally proposed methods, SimSiam [1], BYOL [2] and SwAV [3]. Whether there is a specific reason to this choice is not described.
	Experiments are detailed except for the SwAV experiments in Table 1, ImagiNette, (+3D views are not presented.) and Table 2, ImageNet-100D, (there is no SwAV experiment.)
	This idea lacks of novelty. This paper can be considered as well performing components for each task, which are Depth Prediction Transformer, SwAV and AdaMPI but no new model is proposed.
	This is a nice application/engineering paper but ICLR is not the right venue for this paper.
	At least in ImageNette dataset, computing the depth map for each corresponding augmentation and training the methods as originally proposed should be presented. Also, SwAV experiments in Table 1 and 2 should be completed.

[1] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750–15758, 2021.
[2] Jean-Bastien Grill, Florian Strub, Florent Altche, Corentin Tallec, Pierre Richemond, Elena ´ Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271–21284, 2020.
[3] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912–9924, 2020.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. 
The paper combines the preexisting well performing methods. 
Its code is not shared, therefore, it is not reproducible.
summary_of_the_review: As stated above, although this is a nice application/engineering paper, ICLR is not the right venue for this paper. It’s more suitable for a computer vision conference.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to incorporate depth in the commonly used contrastive learning based self supervised learning frameworks. The depth map is generated using a state of the art RGB to depth predictor, namely DPT (Rangftl et al. 2021). The depth is simply appended to the input RGB image for the purpose of performing SSL procedure. Another contribution by the authors is in the use of AdaMPI (Han et al. 2022) to generate multiple views as an additional augmentation method. The overall  SSL procedure remains the same as the baseline method.

strength_and_weaknesses: Strengths:
- The paper is easy to follow and clearly written.
- The idea of incorporating the depth maps is interesting and to the best of my knowledge, it is the first attempt to use depth maps in the SSL framework.
- The proposed augmentation method and depth appended yields consistently better results on ImageNet100 and ImageNette dataset. 

Weaknesses:
- The technical novelty in the proposed method is quite limited. The main contribution of the authors is to integrate existing approaches such as AdaMPI (Han et al. 2022) and DPT (Rangftl et al. 2021) in existing SSL procedures.
- As it’s an empirical paper, one would expect large scale experiments to test the proposed idea. The SSL experiments are performed on the subset of ImageNet dataset with 10 and 100 classes. To validate the idea, full Imagenet experiments will be needed. The main advantage of SSL methods is to learn on a large dataset of unlabeled images. 
- The experiments are all performed on downstream tasks of classification. SSL literature also focuses on downstream tasks such as object detection and segmentation.
- In Table 2, adding 3D views results in worse performance. Can the authors explain their understanding why 3D views have negative effects on the SSL procedure? Why do some methods perform better at KNN based evaluation and some at Linear evaluation?
- In case of training with 3D views, are the novel views generated like depth maps beforehand or during training? Is there any computational overhead with the proposed method?


clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is easy to understand and clearly written
Quality and Novelty: Technical novelty is limited and empirical performance in some cases is good.
Reproducibility: I would suggest the authors to release the code during the review process.
summary_of_the_review: Since the technical novelty is limited and the main contribution is to combine two papers in terms of augmentation in existing SSL procedure, I think the paper is not good enough for publication in the conference. For an empirical contribution, I would expect authors to validate their idea with further analysis on large scale datasets.

Post-rebuttal:
Thanks to authors for the response to my comments. After reading the authors response and other reviewers concerns, I decide to stick to the original rating.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a novel self-supervised learning method for 2D image classification. Instead of augmenting image in 2D space, the authors first use RGB-to-Depth to estimate the depth channel for the given input image. Then a new view of the image is synthesized from the RGB-D tensor. The synthetic views are then used in a contrastive learning framework. The authors validate their method on several small  scale datasets, including a subset of ImageNet.
strength_and_weaknesses: [Strength]

* Using RGB-to-Depth to get 3D presentation of image and then synthesizing a new view from 3D space is a novel idea in SSL.
* The proposed method improves the classification accuracy significantly on the benchmarked dataset. 
* The idea is very simple and straightforward but very effective.

[Weakness]

* The novelty of this work is more or less incremental in nature. The authors combine two techniques from two deep learning domain. That is saying, this work itself dose not contribute any novel algorithm, except for the algorithm combining.

* Lack large-scale experiments as previous SSL works. ImageNet-100 is clearly too small for a serious study especially in SSL researches. Suggest to report results on ImageNet-1k and ImageNet-21k if possible.

* Possible information leak when using pre-trained RGB-to-Depth models. If the pre-trained models are trained on large-scale labeled dataset, using these pre-trained models is not fair then. This is because SSL aims to reduce the number of manual labels in deep learning. If this method requires a pre-trained model learned on an even larger-scale dataset, it simply makes no sense.

clarity,_quality,_novelty_and_reproducibility: * The paper is well written and easy to follow.
* Experiments are well presented with sufficient ablation studies.
summary_of_the_review: The proposed method is motivated from a simple but interesting idea which mimics binocular vision system in self-supervised learning. The method shows promising improvement on small-scale image datasets but lacks large-scale experiments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to leverage the 3D information of natural images in contrastive learning. Specifically, the authors use the depth signal to generate multiple views from different camera positions, so that they can produce the 3D augmentation for contrastive learning. The idea is simple and natural, and validated on different popular baselines such as BYOL and SimSiam. Experiments on various styles of Imagenet demonstrate the effectiveness of the proposed methods.
strength_and_weaknesses: **Strengths:**

(1). The idea is novel and interesting. I feel this is a very natural extension from the regular 2D image contrastive learning. This technique seems also very general and can be used in different CL backbones.

(2). The paper is well-written and easy to follow. I can understand the main idea by just reading the abstract.

(3). Experiments are strong. The improvements are not marginal.


**Weaknesses:**

(1). My main concern is that the monocular depth estimation usually needs a lot of human annotations (in pixels). In this case, when we really want to use the technique proposed in this paper, we will have to put lots of effort into annotation. More importantly, as we know that the effectiveness of a pretrained monocular depth estimation model may hevalily depend on the parameters/setting of the camera, we can not say whether the depth estimation is accurate for different datasets.

(2). Details should be further clarified. It is unclear how the model performs the estimation and how it generates the new views via modifying the camera position. These are important technical details, but they are missing.

(3). Since the idea is rather straightforward, I think the authors should consider releasing their code (maybe on some anonymous URL).
clarity,_quality,_novelty_and_reproducibility: Overall, this paper is well-written even though some critical technical details are not discussed. The idea is interesting and useful.
summary_of_the_review: I would like to vote for a "borderline accept" due to the above weaknesses I mentioned before. I would consider increasing my score if they are well solved.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: This paper improves existing contrastive-based self-supervised learning (SSL) methods by integrating additional depth signals. Two strategies are investigated: (1) A prior RGB-to-depth prediction model DPT (Rangftl et al. 2021) is used to generate depth maps, and the depth channel is then concatenated onto the RGB channels; (2) images from new viewpoints are synthesized via a prior single-view synthesis method AdaMPI (Han et al. 2022), and then used as augmentations. Once the augmentation is generated, a typical SSL procedure follows. Experiments show improvements over baselines without using depth signals in small-scale 2D image classification experiments (e.g., subsets of ImageNet). During the discussion phase, some large-scale experimental results on ImageNet-1k are provided.

Strengths and Weaknesses: The reviewers generally acknowledge that it is an interesting direction to explore how to leverage 3D information or depth signals to improve self-supervised learning, but they have concerns about the novelty of specific methods in the paper. In particular, the techniques used in this paper for obtaining depth signals directly come from prior work on RGB-to-depth prediction DPT (Rangftl et al. 2021) and novel-view synthesis AdaMPI (Han et al. 2022), leading to incremental contribution of algorithm combination.

In addition, the experiments conducted in the original paper are small-scale. While the reviewers appreciate the additional large-scale experiments provided in the discussion phase, they feel that those experiments are preliminary. Also, the improvement in the linear probe setting seems not significant. More systematic large-scale evaluation is needed to validate the proposed approach.

Since additional pre-trained models are used to obtain depth signals, the extra needed supervision and computational cost from these models are not clearly discussed in the original paper. The authors clarify these in the discussion phase. Following this, it is important to clearly state the supervision and data needed to train the monocular depth estimation models, and why they do not break the standard SSL setting. And also, a comprehensive study is needed regarding how precise the obtained depth information should be or how the SSL performance would change with respect to the accuracy of depth information. There is some ablation in the paper, but a more in-depth study would be helpful to understand the behavior of the proposed approach.

The experiments are conducted on 2D image classification. The reviewers point out that SSL literature also focuses on downstream tasks such as object detection and segmentation. Following this comment, it would be interesting to investigate downstream tasks that rely on depth information, which may further validate the effectiveness of the proposed approach.

Therefore, the paper is not ready for this ICLR. I encourage the authors to continue this line of work for future submission.
justification_for_why_not_higher_score: The current recommendation is based on the weaknesses summarized above: in particular, limited technical novelty and lack of systematic large-scale evaluation.
justification_for_why_not_lower_score: N/A.
Decision: Reject