Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed a new self-supervised MVS method called ElasticMVS. Instead of relying on local photometric consistency as in existing methods, the proposed method utilizes geometric correlations parsed from images as guidance for pixel-wise multi-view correspondences. The paper first proposed a novel elastic part representation that encodes spatial proximity and surface connections, which is trained by a contrastive learning-based strategy. The paper then proposed a self-supervised MVS framework namely ElasticMVS that estimates per-view depth using a part-aware propagation and evaluation scheme. Experiments show that ElasticMVS archives significant performance gain over both the supervised and self-supervised approaches.
strengths_and_weaknesses: Strengths:
-  Addressing the limitation of local photometric consistency is an important topic for self-supervised MVS. The insights and motivation of the proposed method are good.
- The paper presents extensive and sufficient comparisons with baselines, and provides detailed and informative ablation studies.
- The proposed method works very well and achieves state-of-the-art performance, supported by various qualititive and quantitative results. 

Weaknesses:

I don't have major concerns about this paper but I have a few questions to ask for clarification.
- It is not clear to me why it is necessary to learn the part representation $z_p$. $z_p$ is learned primarily using geometric cues such as the 3D point-to-plane geometric distance, which is also available during inference. Why can't we design some simpler similarity functions directly based on the supervision signal of $z_p$ which is the 3d distances between p and q? For propagation, the candidates for pixel p can just be its nearby pixels that satisfy line 113-117. What is the benefit of learning the part representation compared to directly using geometric based heuristics? 
- The quality of the elastic part representation relies on initial depth map. How robust is the elastic part representation to the noise and error in the initial depth map? Is the method still able to handle textureless regions if the initial depth map for these regions are already erroneous? 
- To me, the abstract is a bit overclaiming. In line 6 it says the method explores "geometric correlations such as surface connectedness and occlusion boundaries". This sets my expectation high but I found it does not really match the proposed method. The method does not reason about surface connectedness or occlusion boundaries explicitly, but rather implicitly. In my opinion the key part of the method  is clustering points based on geometric proximity (both 3D point-to-plane and in full 3D), and use it to guide patchmatch propagation and evaluation. It is still a neat idea but the description in the abstract can be a little misleading. I think the sentence in line 6 can be replaced with a more proper description.
- It is possible that two spatially far apart pixels can be coplanar and thus have a small 3D point-to-plane distance, but that is not desirable. What part of the method suppresses this kind of points? 
questions: - What exactly is $\beta$ in equation (7)? It is not mentioned in the implementation details. 

Minor:
Line 94: "sucn" -> "such" 
limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduced elastic part representation into self-supervised MVS framework to achieve more accurate dense correspondences. The pixel-wise part representation is trained by a contrastive learning after a pre-training process. The proposed elastic part representation mainly identifies pixels of similar appearance for depth propagation. The proposed approach is evaluated on the DTU and T&T benchmark datasets and achieves promising results. 
strengths_and_weaknesses: # Strengths
1)	This paper focuses on self-supervised learning for MVS. It adopts the contrast learning to identify pixels belonging to a connected surface or not based on learned features and guide the depth propagation
2)	It achieves promising results on two benchmark datasets.
3)	Comprehensive ablation studies are presented in the paper.

# Weaknesses.
1)	The figure 1 cannot accurately reflect the proposed algorithm. It is a multiple stage framework. Especially, the proposed framework relies Gipuma without part representation to obtain an initial depth which will be used to define the positive and negative set for contrastive learning for the start of the algorithm. It is missing from Fig.1.
2)	Some arguments in the paper are kind of misleading. Line 29-31 argues about the pixel-wise photometric regularisation is sensitive to textureless regions or illumination changes. It is not clear how the proposed elastic parts representation could overcome this. My understanding is the contrastive learning is based on the positive and negative samples obtained from an unsupervised learning framework which is already affected by the textureless and illumination changes. No depth information can be obtained from this initialisation. It is questionable the proposed representation can fix wrong depth estimation for the textureless regions. Line 32-35, It is not clear why these are related to the reconstruction.
3)	Depth estimation is obtained by traditional optimisation rather than learning based on the description between line 166-171.

questions: 1)	How was the initialisation of the depth map obtained for contrastive learning? Is this process repeated once an improved depth map is obtained?
2)	How were the depth hypotheses chosen? Are they related to the initialisation without the pixel-wise part representation learning?
3)	Please also clarify how the optimisation of the energy function in line 171 is solved. The reviewer interprets that no network learning is involved in the process listed in Figure.2.
4)	Does the runtime reported in Table 5 include the calculation of feature distance, confidence map and the normal map?

limitations: 1)	The method still cannot estimate accurate depth for textureless regions even with better propagation region estimation.
2)	It is potentially time consuming because it requires iterative optimisation, calculation of the feature distance, confidence map, and solve the optimisation problem. Surprisingly, the reported run time is reasonable.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposed an interesting idea to improve the pixel-wise multi-view correspondence matching using geometric correlations. 
An elastic part representation is proposed to encode the surface connectedness. Specifically, the input image is passed through a convolutional neural network to get the per-pixel elastic part representation. Then based on the estimated depth and normal map, the authors proposed self-supervised contrastive learning to encourage the part representation to be similar on the same physical surface. 

strengths_and_weaknesses: Strengths:

The authors presented a novel elastic part representation and provided an ablation study to prove the effectiveness.

A thorough experiment is conducted and the proposed method is compared to a large set of state-of-the-art methods, notable improvements are observed compared to other self-supervised methods.

Weakness:

Although contributions are presented in the paper, the core idea of the work is very close to PatchmatchNet (Wang et al. 2021a) and the resulting depth estimation relies on the initial depth estimation from Gipuma (Galliani et al. 2015). Especially the key idea of "Propagation" proposed in this paper is very similar to the "Adaptive Propagation" proposed in the PatchmatchNet (Wang et al. 2021a), more discussion on the differences in-between are expected.

Confusing symbols. $z_p$ in Sections 3.1 and 3.2 is referred to as "per-pixel elastic part representation" and "feature on pixel local p". I assume these two terms mean the thing?

The second paragraph of Section 3.1 is better fitted in the Network Training section and Equations (1) and (6) are the same (almost). Not sure why the same equation is written twice in the text.

The authors talked about the two loss functions used in the training but didn't provide what is the final loss function. I assume it is a weighted sum of these two loss functions?

In the Experiment Section, the values reported for Gipuma (Galliani et al. 2015) on the DTU dataset seem different from the ones reported in the original paper. Also, since the reconstruction results of COLMAP on T&T dataset are provided in Figure 4, it seems strange to not provide the quantitative evaluation in Table 2.

Nitpicking:

Abbreviations used without explanations, e.g. NCC

The supplementary document, Section C.3, first sentence:
"The depth map predicted in each iteration is not always unreliable ...", shouldn't it be "not always reliable"?
questions: 1) Please clarify why the idea of "propagation" is very similar to the "adaptive propagation" in PatchmatchNet while the latter is not suitable cited. Also, discussions on the difference between these two should be discussed.

2) Please clarify the usage of $z_p$ in the text.

3) What is the final form of the loss function used to train the CNN. 
limitations: Well discussed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a self-supervised MVS framework that learns the representation and estimates depth following a part-aware propagation and evaluation scheme. To achieve this, this paper utilizes learns part-aware representation and optimizes a part-level consistency loss. The proposed method consists of part representation learning and part-aware matching. The part representation encodes geometric details to predict a piecewise-smooth depth map. The part-aware matching conducts the depth hypotheses propagation following the idea of patchmatch. 
strengths_and_weaknesses: Strength
1. The proposed method achieves state-of-the-art performance. The proposed method even outperforms some of the supervised methods. 
2. In some datasets, the proposed methods show dramatic performance improvements w.r.t. the state-of-the-art methods in both supervised and self-supervised methods.
2. The proposed method does not require a fixed number of labels for segmentations and a ground-truth segmentation map.
3. The proposed method is geometrically reasonable and is quite novel.
4. The visualization of the elastic part representation demonstrates the effectiveness of the proposed method.

Weakness
1. The evaluation metric and unit are missing in table 3. Because DTU and T&T use different evaluation metrics, table 3 can be confused without the evaluation metrics.
2. The running time is not competitive compared to cost-volume-based methods.
3. The proposed method has many hyperparameters for the threshold, such as \epsilon, \xi, \eta, so I think it is quite vulnerable to the parameters. It would be better to show how robust the proposed method is w.r.t the hyperparameters.
questions: 1. How robust is the proposed method w.r.t. the threshold parameters? 
2. Does the authors use the same threshold values for all datasets? or find the best parameters for each dataset?
3. Generally, how many seed points are obtained? (In figure 11 on supplementary materials, six pixels are provided.) Generally, six points are enough to cover all the images?
limitations: Yes
review_title: null

Meta-review: metareview: All the reviewers acknowledged the strength of the paper: self-supervised learning for MVS using contrastive learning to help correspondence based on learned features, SOTA results are obtained on DTU and T&T benchmarks, and the evaluations/ablation studies are well presented. The reviewers also shared weaknesses: the elastic part representation may not deal with textureless regions, the core idea of “propagation” in the paper is very close to PatchmatchNet and too many parameters for threshold. The reviewers engaged in the rebuttal and discussion phase, and they all decided to keep their ratings. In general, the proposed part representation is interesting and leads to promising results. Please address the issues of how the learned representation can fix the missing depth regions caused by texturelessness as pointed out by both reviewer pJ8f and HBPd.
confidence: Less certain
Decision: Accept