ElasticMVS: Learning elastic part representation for self-supervised multi-view stereopsis
Authors: Jinzhi Zhang
Ruofan Tang
Zheng Cao
Jing Xiao
Ruqi Huang
LU FANG
Venue: NeurIPS 2022 Accept (2022)

Abstract: Self-supervised multi-view stereopsis (MVS) attracts increasing attention for learning dense surface predictions from only a set of images without onerous ground-truth 3D training data for supervision. However, existing methods highly rely on the local photometric consistency, which fails to identify accurately dense correspondence in broad textureless and reflectance areas.In this paper, we show that geometric proximity such as surface connectedness and occlusion boundaries implicitly inferred from images could serve as reliable guidance for pixel-wise multi-view correspondences. With this insight, we present a novel elastic part representation which encodes physically-connected part segmentations with elastically-varying scales, shapes and boundaries. Meanwhile, a self-supervised MVS framework namely ElasticMVS is proposed to learn the representation and estimate per-view depth following a part-aware propagation and evaluation scheme. Specifically, the pixel-wise part representation is trained by a contrastive learning-based strategy, which increases the representation compactness in geometrically concentrated areas and contrasts otherwise. ElasticMVS iteratively optimizes a part-level consistency loss and a surface smoothness loss, based on a set of depth hypotheses propagated from the geometrically concentrated parts. Extensive evaluations convey the superiority of ElasticMVS in the reconstruction completeness and accuracy, as well as the efficiency and scalability. Particularly, for the challenging large-scale reconstruction benchmark, ElasticMVS demonstrates significant performance gain over both the supervised and self-supervised approaches.  

ElasticMVS: Learning elastic part representation for self-supervised multi-view stereopsis
Jinzhi Zhang Ruofan Tang 1.3 Zheng Cao', Jing Xiaoo, Ruqi Huang and Lu Fang !
Department of Electronic Engineering, Tsinghua University
TTinggua Shenzhen International Graduate School
3 Dept. of Automation, Tsinghua University, BBirenTech Research, SPingan Group
*zj 2:j2220mails. edu. en
Abstract
Self-supervised multi-view stereopsis (MVS) attracts increasing attention for Iearn- ing dense surface predictions from only set of images without onerous ground- truth 3D training data for supervision. However, existing methods highly rely on the local photometric consistency, which fail to identify accurately dense correspon- dence in broad textureless or reflectant areas. In this paper, we show that geometric proximity such as surface connectedness and occlusion boundaries implicitly ferred from images could serve reliable guidance for pixel-wise multi-view correspondences. With this insight, we present a novel elastic part representation, which encodes scales, shapes and boundaries. Meanwhile, a self-supervised MVS framework namely ElasticMVS is proposed to learn the representation and estimate per- viewview depth following a part-aware propagation and evaluation scheme. Specifically, the pixel-wise part representation is trained by contrastive learning-basee strategy, which increases the representation compactness in geometrically concentrated areas and contrasts otherwise. ElasticMVS iteratively optimizes part-level consistency loss and a surface smoothness loss, based on a set of depth hypotheses propa- gated from the geometrically concentrated parts. Extensive evaluations convey the superiority of ElasticM in the reconstruction completeness and accuracy, as well the efficiency and scalability. Particularly, for the challenging large-scale reconstruction benchmark, ElasticMVS demonstrates significant performance gain over both the supervised and celf-supervised approaches. Code avaliable httis://thu-luvisionn.iii
part segmentations with elastically-varyin
Introduction
Given a set of images with known camera parameters, multi-view stereopsis (MVS) aims to re- construct the dense and accurate geometry of the scene. Existing learning-based approaches (47). typically being cast as an end-to-end depth regression task, outperform the traditional geometry-based ones (4). However, their generalization ability in open environments is severely hindered by the availability of the onerous laser-scanned 3D training data. Self-supervised MVS (2 [1] lifts such limitation by leveraging the multi-view photometric consistency in place of the supervisory signals. Unfortunately, the pixel-level photometric regularization is susceptible to textureless patterns and illumination variations, leading to incomplete and inaccurate reconstructions, especially in outdoor environments.
*Coreesooding Author: Tugghuung@s.ttigghnua.cu... fanglu @isii@@ua.inin.n..rnn
luvision.net
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Erput images
Elastic part representation
Laiest spuce
space
Contrastive leamiig
Training intialize
Depth Giguna
Backprup
Chnfideace
Part-aware
patchmatch
Reference
Part-aware propagation
view
Part-aware evaluation
Depth map
Figure I: An overview of ElasticMVS, self-supervised MVS framework to learn the representation and estimate per-view depth. The elastic part representation each image is trained by a contrastive Garming-based strategy, which increases the representation compactness in geometrically concentrated areas and contrasts otherwise. Benefiting from the elastic part representation, the inherent geometric correlations are effectively exploited by patchmatch-basee strategy. Without the ground-truth 3D information, positive and negative samples for training are based on the depth map from the traditional geometry-basee stereopsis (4].
In this paper, tackle the above challenges with novel image parsing paradigm. Instead of treating images as projections of 3D scenes, we show that the inherent geometric correlations, such surface connectedness, smoothness and boundaries, could be inferred implicitly from images and serve as a reliable guidance for pixel-wise multi-view correspondence estimation. The mainstream solutions of feature-based regularizations introduce either hand-crafted structures such lines and planes from superpixels [32, 4 30), or semantic features pre-trained from limited geometric- unrelated labels [4 17.. Different from those, we aim to build the correlation between the latent space of the representation and the scene space of the surface in self-supervised way. With this insight, we present novel elastic part representation which encodes physically-connecter part segmentations with elastically-varying scales, shapes and boundaries. Unlike the traditional segmentation networks with fixed number of labels, our pixel-wing part representation directly refects the surface correlation, i.e., the more similar the two representations are, the higher probability they belong to the same physical surface part. Meanwhile, self-supervised MVS framework namely ElasticMVS is proposed to learn the representation and estimate per-view depth iteratively following part-aware propagation and evaluation scheme. We believe that this framework provides neat solution to bridge the gap between the multi-view stereopsis and the point cloud reconstruction (3), which has the potential to be effectively exploited by many other MVS methods.
Benefiting from the clastic part representation, the inherent geometric correlations are effectively exploited by a patchmatch-bait estimated by selecting the best depth hypotheses adaptively propagated only from the geometrically concentrated part. Different from other adaptive propagation strategy [39 drl based only on pure
propagation strategy 13
Specifically, in each iteration, depth
image features without explicit geometric constraints, our part-aware propagation scheme directly relies on physically plausible indicators. Besides, the pixel-level photometric consistency loss enhanced by the constraints of feature-level correspondence and non-local piecewise surface smooth- ness, which provide part-aware matching cues to guide a more robust depth prediction in textureless and reflectant areas. To train the elastic part representation. contrastive learning baaning-bbased based strategy implemented by increasing the representation compactness in geometrically concentrated areas and contrasts otherwise. Without the ground-truth 3D information, these geometrically concentrated areas are identified by the depth map generated from the initialized ElasticMVS without the part representation, which owns the same workflow as the traditional geometry based stereopsis [34]. A noise-robust training loss is implemented to get rid of undesirable identifications.
Experiments on DDT'' and "Tanks and Temples (T&T)' datasets demonstrate that the proposed FlasticMVS significantly outperforms all state-of-the-art self-supervised learning-based methods. Remarkably, in the challenging large-scale reconstruction benchmark (T&T advanced), ElasticMVS

---

shows dramatic performance improvements w.r.t. both the supervised and self-supervised approaches. Besides, the systematic evaluations indicate that our elastic part representation superior modeling elastically-varied parts, while being robust and lightweight in training and inference.
2
Related work
2.1 Multi-view Stereopsis (MVS)
The major works in multi stereopsis focus on the view-wise depth map estimation [6 12 and then fuse together to form the point cloud 14] Among these methods, patchmatch- based depth optimization methods [ 34 44], initialized from the randomly sampled depth hypotheses, show great run-time and memory efficiency depth map estimation. Some learning- methods [39 26 introduce an iterative coarse-to-fine optimization in an end-to-end trainable architecture to improve each of the core steps in patchmatch. However, these algorithms suffer from incomplete and noisy depth predictions in textureless and reflectant areas. Several previous works in MVS solved the problem by introducing pre-defined global structural priors such as planes to resolve the ambiguity. For instance, TAPAMVS 322 segments the whole image into bunch of superpixels and assumes textureless areas piecewise planar. RANSAC is applied to estimate its corresponding plane parameters. Beyond that, bunch of works in point cloud reconstruction deal with reconstruction artifacts with outlier, noisy or missing data. For instance, implicit representation [2 is used to provide global surface smoothness prior given noisy data, and exterior visibility (2] is used detect and eliminate redundant data. However, neither the handcrafted plane structure nor the complex smoothness prior can provide sufficiently diverse and accurate surface part modeling for real-world scenarios.
Currently, learning methods based on 3D cost volume regularization demonstrate state-of-the-ee results in several benchmarks. These methods optimizes the 3D geometry by 3D CNN, unprojecting
from 2D images or features into the 3D volume [9 23 47 20 488 7 10 28 52). However, these supervised methods crucially rely ground-truth 3D data, which are hard to generalize in wild open environments. To realize self-supervised MVS without the onerous ground -ruth 3D training data, [T] 16
29 31 replace the supervision signal with the unsupervised rendering loss or cross-view feature consistency loss. However, reprojection under multiple views is highly sensitive environmental illumination, which is hard to generalize in highly variable scenarios.
24
51
43
2.2 Part Representation
Recently, learning representations from the image in self-supervised training strategy has gained growing popularity in 2D vision tasks, which use contrastive learning (42 g 9 6 to deal with the lack of ground-truth labels. They use InfoNCE (38) to generate unique embedding for each picture in a batch. [4] further leverages a dense projection head to generalize such embedding from image-level to patch-level, which is more suitable for performing dense prediction tasks such as assigns the equivariance constraint and the semantic consistency regularization on the learned part response representation (53] to predict part segmentations that are consistent across different instances or camera poses. 43 incorporates learned semantic features into multi-view stereopsis pipeline to boost the reconstruction ability. However, previous works only concentrate on generating semantic features and using it forward to perform down-stream tasks. Instead, we utilize the rough depth map as supervision to the semantic learning process, which forms a natural integration of photometric and geometric information to provide elastic part representations.
semantic segmentations. [1 41
Method
We estimate the depth maps of a given reference image and the corresponding M unstructured source images with known camera calibrations. The final point cloud reconstruction is obtained by fusing all of them. Since the geometry information gained from any photometric loss contains missing data and artifacts in some areas of the image (especially in textureless regions), a heuristic geometry regularization for depth prediction is unavailable in these regions. Therefore, the key innovation in our framework is novel elastic part representation, which encodes sufficient geometric details to guide a piecewise-smooth depth map prediction. We first formally define our part-aware representation in Sec. 3.1 Based upon the learned representation, introduce two improvements depth hypotheses
3

---

Part-aware correspondence
Fenture distance
Kandon pensetaation
Rrcioese now
Feature similanity
aegnin
Propagation
Depth mup
Blastic part representation Confidence map
Propegation candidates
Depth hypotheses
smoothness
ltratiiv refinement
Figure 2: Detailed architecture of part-aware patchmatch. Given the pixel-wise elastic part represen- tation from the ConvNet, depth hypotheses are sampled from high confidence regions which similar features. The optimal depth values in each iteration are chosen based on the combination of part-aware correspondence and part smoothness cost.
propagation and evaluation for part-aware patchmatch in Sec. 3.2 We present the self-supervised training scheme in Sec. 3.3 robustly train the representation with noisy initialization.
3.1 Elastic Part Representation
Suppose that (d, n) E #xxx4 are the underlying depth and normal map of the image x, we first define the geometry-awant part segmentation Sp related to each pixel location p E R which is geometricalty concentrated inside: Sp {q E (2)r úp);; Mq)] < <), where
[[(dp, mp); (dy. }g)} & ([g. dy] [p. dpl) npl is the 3D point-to-plans distance between two pixels.
Equipped with the part segmentation Sp for each pixel p, we propose learn per-pixel elastic part representation Zp TR. for each p TR22, such that Zp is compact in the geometric concentrated part Sp and contrast otherwise. Specifically, we use the Soft-Nearest Neighbor Loss 33 to formulate such:
Tpp -==-lop log Eptes, exp ((zp) //T)
Egep exp (2p. /T)
where 2n- E 2 is the per-pixel elastic part representation on pixel location p", (,)) and r the temperature.
1) is the dot product,
Finally, we use ConvNet, Fe(-), to model the generation of the elastic part representation, i.e., for reference image x E R x3, (x) e RdxxWD denotes the collection of per-pixel elastic part representations.
3.2 Part-aware Matching
After training the part representation (described in Sec. 3.3 module top of the seminal work of patchmatch [13]. That the final depth prediction is iteratively refined from the beginning of the random initialization. In particular, we incorporate our elastic part representation and propose the following hypotheses to neighbors with our learned elastic part representation; 2. Evaluation: we evaluate and choose the best hypotheses based on photometric consistency, feature-level correspondence, and geometrically smoothness. The resulting module is differentiable, and the architecture of part-aware patchmatch is illustrated in Fig.
we build our part-aware matching
novel improvements: 1. Propagation:
propagate
Propagation. The key idea of the part-aware propagation is to gather hypotheses from the nearby pixels of the same physical surface part. As illustrated in Fig. instead of propagating hypotheses naively from a static set of neighbors [3] our depth hypotheses are sampled from the nearby pixels which similar features of the learned elastic part representation.
Since the depth value is unreliable textureless and highly reflectant areas, confidence map [0, 1] xW: generated to pick the reliable candidates from noise, which is combination of

---

(a) Stundard progaaation
(b) Part-aware propagation
Figure 3: Visualization of all sampled locations for propagation (blue points) during 3 iterations of patchmatch given six pixels (red points).
the photometric consistency and the geometric consistency 34 Please refer to the supplementary material for more detailed explanation. For each pixel in the depth map be updated, K₂ patch hypotheses are propagated from the propagation candidates Tp. which is a set of nearby pixels whose feature by is close the feature 2p with high confidence Cgi
Tp T=
q
E
R2
Zp
According the property of elastic part representation in Eq. D the generated depth hypotheses from T, would stay close to the underlying surface.
Evaluation. Beyond the photometric consistency loss, we propose two novel losses for hypothesis evaluation Part-aware correspondence loss and Part smoothness loss.
Part-aware correspondence loss evaluates the feature similarity of the elastic part representation between the reference view and the warped source views. Given homogeneous coordinates and
[i]]
depth hypothesis d, we obtain the warped source feature "p:(d) of the i-th source view via differentiable
[o] [4]
bilinear interpolation. Therefore, given the features zp (d) of the elastic part representation in the reference and source view respectively, the part-aware correspondence depth d, and normal direction dp is given by:
Ma(dp. 2)) = Σ 10] D
(.) + as p(dp, Mpler x 111))
where p(-) is the bilateral weighted normalized cross correlation
34 implemented in the traditional
patchmatch algorithm.
Part smoothness loss aims to augment the local patchmatch evaluation with a non-local piecewise smoothness term corresponding to the same physical surface part. Inspired by the work of Locally Optimal Projection (LOP) [27] for surface reconstruction from point clouds, the part smoothness loss formulated using LI median loss, which is robust the outliers and gives a piecewise second-order surface approximation in each surface part (27).
More specifically, the depth map d E #xx1 is firstly transferred to the point in the scene space given the camera matrix M: fep- [ 1] M- dg }.
#x
Then we define our part smoothness loss as the sum of the Euclidean distance to the points from propagation candidates in Tp (Eq. 2
Mg (dp: Mep 22 = Σ Wy llep Eg]l,
TET,
where w. = cxp --cp - an ( (np, nuy) } is a weight correlated with the confidence cy and the normal similarity.
Putting the components together, as illustrated Fig. 2 the depth map initialization consists of Kp random values within the depth range of interest. For subsequent iterations, K, hypotheses are propagated based on the depth values of the former one. To deliver more diverse set of hypotheses than just using propagation, another K, hypotheses are sampled by adding small
5

---

random perturbation to the previous estimation. In each iteration, the optimal depth and normal are chosen as: (dp m argmin (M, nz + ag (dp ni |2)}. This function is not d;-";
directly solved using the first-order optimization mechanisms such as gradient descent, but rather approximated using a discrete sampling strategy, which has long been proposed in the field of stereo- matching (34). Empirically, 5 iterations are sufficient to obtain an accurate piecewise smoothness depth map.
3.3 Network Training
The elastic part representation 2, as described in Sec. 3.1 is compact in the area of smooth surface part. We now describe how train the ConvNet Fe using the self-supervised contrastive learning strategy without knowing the ground-truth depth and normal information.
The core of contrastive learning is to construct a set of positive and negative samples, in which the representation in the positive samples stay close to each other while the negative ones are far apart. Before training, depth map d and normal map of each reference image propagated and generated by the initialized ElasticMVS without the part representation, which is similar to the traditional patchmatch based algorithm, e.g., Gipuma [I].
When selecting positive and negative samples during the training stage, we get rid of the noise and eтror in the initial depth map using the confidence map. For each pixel p with high confidence
(cp Q), we construct set of pixel candidates Sp for positive samples, in which each of them is geometrically concentrated to p with high confidence:
Sp = fq E R2(T((dp. mp): mg)) S c, Cg 2 E}
(5)
Therefore, the dense contrastive loss for self-supervised training is defined as:
L = - Σ (0,25) log Eptes, exp ((Ep) 2pp 1/T) Egpp cxp (Epp TT)/T)
(6)
P
This loss function encourages representation compactness in parts with close surface distance and separation otherwise, i.c., the distance in the representation space naturally reflects the distance in the 3D scene space. Since this loss function only acts on pixels with above-the-threshold confidence, prevents the initial reconstruction artifacts from ruining the contrastive learning process. Due to the memory limitation, it is impossible to densely sample all positive and negative samples from the whole image in Eq. 6 Therefore, we randomly select N₂ points from all samples in Sp.. using probability distribution inversely proportional to the distance away from p.
To obtain spatially concentrated representation, we utilize a spatial concentration loss encourages all the pixel embeddings become isotropically isolated:
46 that
L = Σ Enes, exp (B (Ep, =q))
p
Eges, exp (B (zp. =g))
where 3 is constant parameter that controls the weight of the feature similarity between p and q. which prompts the weighted average of the sampled neighboring points to be close to the point Therefore, the pixel locations far away from the sampled point will be given low weights, leading to spatially concentrated representation. The concentration loss enables the spread of reasonable distance between each representation from high-confidence regions low ones, which enables robust training of the representation towards noisy and incomplete depth predictions. A more detailed analysis demonstrated in Table and Fig. 6
Our final loss function is a weighted sum contrastive loss and concentration loss, defined as
Listal = Le + Ys Le:
where 7 denotes the weight of concentration loss in the total loss function.
3.4 Implementation Details
The elastic part representation is trained with a Feature Pyramid Network, which shares the same structure in PatchmatchNet (39]. To train the network, use the SGD [5] with initial learning

---

Francis
Temple
HeI
Museum
Courtroom
(a) PatchmatchNet
(b) CasMVSNet
(C) COLMAP
VSSett
(c) IDACS
(1) ElasticM (S
Figure 4: Point cloud visualization in the T&T intermediate and advanced benchmarks, compared with three types of reconstruction methods: supervised methods (a)(b) 39 [5], self-supervised
methods (d)(e) [6 and one traditional geometry-based method (c) 34
rate of 1.0 10-4, The threshold for defining Sp is set to 0.5 in the DTU dataset and 0.01 in the T&T dataset. We fix the temperature number т = 0.5 and the confidence threshold 5 0.7 during training and propagation. We sample N. 8 points for each pixel. During the propagation,
select
for perturbation. We choose 1) = 0.1 in Eq 2 to find nearby pixels. During the reconstruction stage,
K₂ = 8 hypotheses for propagation and K,-
respectively a, 0.1 and ag 0.5 in
Eq. and [ For the loss functions, B = 0.5 in Eq. 7 and 7 = 0.005 in Eq. All the training and reconstruction conducted single NVIDIA GTX 3090 graphies card.
Experiment
4.1 Benchmarks
DTU dataset [1] is an MVS dataset consisting of different indoor objects scanned 49 camera positions under 7 different lighting conditions. We use the original image size (1600 1200) as input and follow the same evaluation pro- tocol M compute the accuracy and the completeness of the final reconstructions. The overall distance metric is defined as the mean of the two metrics.
Table 1: Evaluation of reconstruction quality by measuring the distance metric (lower is better) on the DTU dataset [
Method
Acc.(man) Comp, Overalltmm)
Furu
0.613
0.941
0.777
Tola
36
0.342
1.190
0.766
Gipuma
13
0.283
0.873
0.578
COLMAP
0.400
0.664
0.532
MVSNer
47
0.396
0.527
0.462
Quantitative results with three types of recon- struction methods are shown in Table [ in- cluding geometry-based, supervised and self- supervised MVS methods. Our method outper- forms all the self- eef-suuprviied methods includ- ing SurRF [51] and JDACS [43] in accuracy and overall distance metrics, and achieves the second-best performance in completeness. It remarkable that FlasticMVS competitive compared to toite-of-theaaad supervised methods,
SurfaceNot+
0.385
0.448
0.416
Point-MVSNet 7
0.342
0-441
0.376
CusMVSNel
0.325
0.385
0.355
UCS.NNet
0.338
0.349
0.344
PVA-MVSNet
0.379
0.336
0.357
D D*HC-RMVSNet 55
0.395
0.378
0.386
PatchmatchNet 39
0.427
0.277
0,352
Consistency [24
0.881
1.073
0.977
51
0.389
MVS I
0.760
0.515
0.637
M VSN [M
0.636
0.531
0.583
JDACS 43
0.398
0.318
0.358
FlasticMVS (ours)
0.374
0.325
0.349

---

e.g., our overall distance metric is the second best among all methods.
Tanks and Temples [25] dataset contains two benchmarks (intermediate and advanced) outdoor real-world scenes with complex geom- etry. We use the original image size (1920 1080) as the input. We evaluate the results with F-score (25,. the higher value of which indicates the better reconstruction. We summarize the quantitative results in Table [2] On the interme- diate benchmark, ElasticMVS achieves state- of-the-art performance among all supervised and self-superviser MVS methods. Especially, our method outperforms all the self-supervised MVS methods by significant margin. On the more challenging T&T advanced benchmark, which contains large-scale real-world scenes with more than 300 camera views for each scene, ElasticMSS even compares favourably with the supervised methods. The tremendous perfor- mance improvement in this benehmark demon- strates the effectiveness and robustness of the proposed ElasticMVS w.r. complex categories and geometries.
Table 2: Quantitative evaluation (F-score, higher better) of state-of-the-art learning-based meth- ods two benchmarks of the Tanks and Temples (T&T) dataset [25].
Method
Intermetialle
Auivance
MVSNer [1]
43,48
CasMVSNet
16
56.84
31,12
UCSNet [10
54,83
PVAMVSNel
49
54.46
SurfaceNet+
19
49.38
R R-MVSNei
48
$0.55
29.55
PoiNt-MVSNet
48.27
PatchmatchNet
39
53,15
32.31
Patchmatch--L
26
51.81
31.78
MVS*
37.21
M [6
37.67
SurRF [5]
54.36
45.48
COT 14
42,14
27.24
ElasticMVS (ours)
57.88
37.81
We also compare our reconstructions with various baselines qualitatively in Fig. 4 ElasticMVS produces much more complete predictions compared with COLMAP [34], while generating much less noise than existing self-supervised methods (M3VSNet [6 and JDACS 43]). Besides, we achieve even surpassing performances upon state-of-the-art supervised methods (CasMVSNet [15]
and PatchmatchNet [39]]
4.2 Ablation Study
In this section, we provide detailed ablation study and analysis of the key components of ElasticMVS, including the part-aware patchmatch and the elastic part representation.
Part-aware patchmatch. To investigate the effects of the key components in the part-aware patchmatch during inference, we conduct comparison with respect to the part-aware prop- agation and evaluation described in Sec. 3.2 Our part-aware patchmatch is compared with three variants: 1. Propagation without elastic Part Representation (w/o PR), which employs fixed sampled locations for propagation similar to Gipuma [13]: 2. Evaluation without Part- aware Correspondence (w/o PC), which only employs the photometric consistency in Eq. 3 3. Evaluation without Part Smoothness loss (w/o PS) in Eq. 4 which simply employs an for point cloud de-noising.
L1-median loss from all neighborhoods, similar to the LOP
As shown in Table 3 our part-aware propagation and evaluation modules both play important roles in boosting the overall performance across different datasets. Qualitative results are shown in Fig. [5] Im- plementations without the elastic part representation (a) and the part-aware correspondence (b) fail to re- cover the dense surface in textureless regions. Al-
Table 3: Ablation study of part-aware patch- match, without the help of part representation (w/o PR), part-aware Correspondenc (w/o PC) and part smoothness (w/o PS)
Methods
DTU (mm)
T&T (%)
w/o
0.455
40.05
w/o PC
0.430
43.82
though applying a naive surface de-noising and com- pletion method (c) helps fill in some of the holes in textureless areas, it still suffers from low accuracy and noisy predictions due to the lack of part aware- ness. Overall, the strong geometric coherence revealed by the learned elastic sampling strategy helps ElasticMVS gather more accurate and complete depth maps in both small complex surfaces and large flattened areas.
w/o PS
0.411
39,58
Ours
0.349
57.88
8

---

w/o PR
(b) w/o PC
(c) w/o PS
Ours
(c) Ground Truih
Figure 5: Reconstruction results of ablation study on part-aware patchmatch, without part representa-
tion (w/o PR). part-aware correspondence (w/o PC) and part smoothness (w/o PS) relatively.
Training loss of elastic part representation. Now we study how different training losses de- scribed in Sec. 3.3 affect the performance of the representation and further the depth map predic- tions. Our clastic part representation is compared with two variants, which are trained without confidence map (w/o Con.) in Eq. 6 and without spatial concentration loss (w/o Spa.) in Eq. We test on the DTU dataset. We first measure the compactness of the learned part representation as de- scribed in Eq. [ using the ground truth depth map. We also compare the predicted depth map error using these representations, which is evaluated by the LI distance from the predicted depth map to the ground- truth. As shown in Table 4 our elastic part representa- tion is the most compact one (with lowest contrastive loss).
Table 4: Ablation study of the representation concerning the training loss without the help of confidence map (w/o Con.) and spatial concentration (w/o Spa.).
Methods Contrastive loss Depth error(mm) w/o Con.
2.52
3.55
w/o Spa. 2.34
3.01
Ours
2.28
2.52
Visualizations the representation are demonstrated in Fig. [ shown by heat map of the represen- tation distance || zpll from a pixel location p to all other pixels 9 in the whole image. As expected, our elastic part representation successfully recognizes the part the same physical surface, resulting in proper surface boundary identifications with elastcaally-varying scales and shapes. Contrastingly, the representation trained without the confidence map (w/o Con.) fails to identify a complete surface part due to the noisy training samples. Meanwhile, the representations trained without spatial concentration loss (w/o Spa.) suffer from outliers far from the surface part, demonstrating the effectiveness of spatial regularization.
4.3 Discussion
Memory. In Table we compare the GPU memory consumption and the run-time dur- ing inference with two memory-efficient learning-based methods. We choose scan and fixed reference view downsam- pled to the scale of 640 800 in the DTU dataset this study. As shown in Table 5 ElasticMVS is more memory-efficien com- pared with two learning-based MVS back- bones. Notice that the memory and run- ning time are adaptively changed with the hypothesis number and the iteration num- ber, since propagation with more hypothe- will converge with fewer iterations, but take a larger memory consumption. Over- all, the experiment demonstrates the supe- riority memory efficiency of ElasticMVS.
05
155
3.5
12
12
(a) w/o Con.
(b) w/o Spa.
(c) Ours
Figure 6: Visualization of the elastic part representation for two pixels (black). Con. denotes confidence map, and Spa. denotes spatial concentration loss.
Impact of parameters. To better understand the effect of different parameters, we first evaluate the reconstruction quality w.n. different thresholds. As shown in T with lower e, the reconstruction accuracy grows higher while the completeness becomes lower. The reason is that (point-to-plane

---

Table 5: Memory and running time comparison with two learning-base MVS backbones.
No. of
GPU Mem.
Run-time
Methods
stages
(MB)
(S)
Cas
151
3489
0.12
PatchmatchNet 39
5
4439
0,08
BlasticMVS (8 hypotheses)
2679
0,48
ElasticMVS (16 hypotheses)
5785
0.37
Table 6: Impact of parameters during inference.
method
Acc. (mm)
Comp. (mm)
Overall (mm)
0.005, { = 0.7
0.368
0.354
0.362
0.02, { 0.7
0.380
0.301
0.341
0.04, { 0.7
0.396
0.331
0.364
= 0.01, { = 0.5
0.540
0.342
0.441
0.01, E = 0.6
0.405
0.317
0.361
0.01, 0.8
0.358
0.330
0.344
0.01, < 0.7
0.374
0.325
0.349
distance) controls the compactness of the surface. Smaller during inference leads to a more strict surface boundary with smaller regions.
On the other hand, with a higher confidence threshold ç. the accuracy and the overall performance of the prediction become higher. With loose threshold (E 0.5), the performance drops dramatically, Therefore, is better to set the confidence threshold at a high level.
Limitation and Future work. It takes a long time in ElasticMSSS to calculate the patch similarity for each pixel, since the total computational complexity to evaluate the photo consistency from M source images of size H x W with K hypotheses is O(M H W x K x 1²), where is the patch size. Empirically, = 11, which impossible to give a forward and backward pass from a single graphics card at once. As shown in Table the running time of ElasticMVS is not competitive compared with the nost-volume-bsee methods. One solution is to replace the x patch with the learned feature in Patchmatch. However, the accuracy of the Iearned photometric consistency in self-supervis stereopsis is still far from the traditional one. In future work, we plan to design lightweight network architecture represent not only the part correlation but also the pixel-wise wrapped image patches, jointly learned from supervupee learning- based strategy, which can meet the demand for more challenging tasks such real-time MVS and dynamic MVS.
5
Conclusion
In this paper, we present a novel elastic part representation that encodes physically connected part segmentations with clastically-varyin scales, shapes and boundaries. We build the correlation between the embedding space and the scene space to represent the surface connectedness and boundaries behind the image. self-supervised MVS framework. ElasticMVS, proposed to learn the representation and estimate per-view depth following part-aware patchmatch-han based algorithm. Benefiting from the inherent geometric correlations learned from the huge amount of images, ElasticMVS delivers sate-oofthhe-at performance in both reconstruction completeness and accuracy, and extraordinary generalization ability in the challenging large-scale scenes.
Acknowledgements
This work is supported in part by Natural Science Foundation of China (NSFC) under contract No. 62125106, 61 61860206003, 62088102 and 62171256, in part by Ministry of Science and Technology of China under contract No. 20212D0109901 and in part by Shenzhen Key Laboratory of next generation interactive media innovative technology, Funding No: ZDSYSS22210623990000000
10

---

References
[1] Henrik Aanæs, Rasmus Ramsbel Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl. 2016. Large-Scale Data for Multiple- View Stereopsis. International Journal of Computer Vision (2016). 1-16.
[2] Radhakrishha Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. 2012. SLIC superpixels compared to state-of-the thet superpixel methods. IEEE transactions on pattern analysis and machine intelligence 34, 11 (2012), 2274 2282.
[3] Matthew Berger, Andrea Tagliasacchi, Lee M. Seversky, Pierre Allicz, Joshua A. Levine, Andrei Sharf, and Cláudio T. Silva. 2014. State of the Art in Surface Reconstruction from Point Clouds. In Eurographics.
[4] András Bódis Szomorú, Hayko Riemenschncider, and Lue Van Gool. 2014, Fast, Approximate Piecewise- Planar Modeling Based on Sparse Computer Vision and Pattern Recognition (2014), 469-476.
and Superpixels. 2014 IEEE Conference on
[5] Léon Bottou. 2010. Large-Scale Machine Learning with Stochastic Gradient Descent. COMPSTAT:
[6] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. 2020. Un- supervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems 33 (2020), 9912-9924.
[7] Rui Chen, Songfang Han. Jing Xu, and Hao Su. 2019. Point-Based Vulw Stereo Network. In IEEE Conference on Computer Vision and Pattern Recognition.
[8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. International conference on machine learning. PMLR, 1597-1607
[9] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. 2020. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0422 (2020).
[10] Shuo Cheng, Zexiang Xu, Shilin Zhu, Zhuwen Li, Li Erran Li, Ravi Ramamoorthi, and Hao Su. 2020. Deep stereo using adaptive thin volume representation with uncertainty awareness. In Proceedings of the IEFF/CVF Conference Computer Vision and Puttern Recognition. 2524-2534.
[11] Yuchao Dai, Zhidong Zhu, Zhibo Rao, and Bo Li. 2019. Mvs2: Deep unsupervised multi-view stereo with multi-view symmetry. 2019 International Conference on 3D Vision (3DV). IEEE, 1-8.
[12] Yasutaka Furukawa and Jean Ponce. 2010. Accurate, dense, and robust multiview stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 8 (2010), 1376-136.
[13] Silvano Galliani, Katrin Lasinger, and Konrad Schindler. 2015. Massively parallel multiview stereopsis by surface normal diffusion. In IEEE International Conference on Computer Vision. 873-881.
[14] Michael Goesele. Noah Snavely, Brian Curless, Hugues Hoppe, and Steven Seitz. 2007. Multi- View Stereo for Community Photo Collections. 1-8 https: org/ 1109/ICCV 2007 4408933
[15] Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai. Feitong Tan, and Ping Tan. 2019. Cascade Cost Volume for Righ-hessottion Multi- View Stereo and Stereo Matching, arXiv preprint 17X.v:19/2.06378 (2019).
[16] Baichuan Huang, Can Huang, Yijia He, Jinghin Liu, and Xiao Liu. 2021. M3VSNet: Unsupervised Multi-metric Multi-view Stereo Network. ArXiv abs/2005.00363 (2021).
[17] Wci-Chih Hung, Varun Jampani, Sifei Liu, Pavlo Molchanov, Ming- Haug-Hsuan Yang, and Jan Kautz. 2019. Scops: Self-supervised co-part segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Patiern Recognition. 869-878.
[18] Mengqi Ji, Juergen Gall, Haitian Zheng, Yebin Liu, and Lu Fang. 2017. SurfaceNet: An End-to-End 3D Neural Network for Multiview Stereopsis. 2017 IEEE International Conference on Computer Vision (ICCV) (2017), 2326-2334.
[19] Mengqi Ji, Jinzhi Zhang, Qionghai Dai, and Lu Fang. 2020. SurfaceNet+: An End-to-end 3D Neural Network for Very Sparse Multi-view Stercopsis. JEEE Transactions on Pattern Analysis and Machine Intelligence (2020).
[20] Abhishek Kar, Christian Häne, and Jitendra Malik. 2017. Learning Multi View Stereo Machine.

---

[21] Sagi Katz. Ayellet Tal, and Ronen Basri. 2007. Direct visibility of point sets. ACM SIGGRAPH 2007 pupers. e4-ss.
[22] Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. 2006. Poisson surface reconstruction. In Proceedings of the fourth Eurographics symposium Geometry processing, Vol. 7.
[23] Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry. 2017. End-to-End Learning of Geometry and Context for Deep Sterco Regression. CoRR abs/1703. 00000004309900 (2017).
[24] Tejas Khot, Shubham Agrawal, Shubham Tulsiani, Christoph Mertz, Simon Lucey, and Martial Hebert. 2019. Learning unsupervised multi-view stercopsis via robust photometric consisteney. arXiv preprint TeXiv::905.005 (2019).
[25] Amo Knapitsch, Jaesik Park, Qian-Y Yi Zhou, and Vladlen Koltun. 2017. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics (ToG) 36, 4 (2017), 1-13.
[26] Jac Yong Lee, Joseph DeGol, Chuhang Zou, and Derek Hoiem. 2021. PatchMatch RL: Deep MVS with Pixelwise Depth, Normal, and Visibility. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (2021), 6138-6147.
[27] Yaron Lipman, Daniel Cohen-Or, David Levin, and Hillel Tal-Ezer. 2007. Parameterization-free projection for geometry reconstruction. In SIGGRAPH 2007.
[28] Leyao Liu, Tian Zheng, Yun-Jou Lin, Kai Ni, and Lu Fang. 2022. INS-Conv: Incremental Sparse Convolution for Online 3D Segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition. 18975-18984
[29] Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li. 2019. Learning to infer implicit surfaces without 3d supervision. In Advances in Neural Information Processing Systems. 8295-8306.
[30] Branislav Micusik and Jana Kosecka. 2010. Multi-view Superpixel Stereo in Urban Environments. International Journal of Computer Vision 89 (2010). 106-119.
[31] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. 2020. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
[32] Andrea Romanoni and Mattco Matteucci. 2019. TAPA-MVS: Textureless- Aware PAtchMatch Multi- -Vew Sterco. 2019 IEEE/CVF International Conference Computer Vision (ICCV) (2019), (0412-1042)
[33] Ruslan Salakhutdinov and Geoff Hinton. 2007. Learning nonlinear embedding by preserving class neighbourhood structure. In Artificial Intelligence and Statistics. PMLR, 412-419.
[34] Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Mare Pollefeys. 2016. Pixelwise view selection for unstructured multi-view stereo. In European Conference Computer Vision. Springer, 501-518.
[35] Keisuke Tateno, Federico Tombari, and Nassir Navab. 2015, Real-time and scalable incremental segmenta- tion on dense SLAM. 2015 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2015), 4463-4472.
[36] Engin Tola, Christoph Strecha, and Pascal Fua. 2012. Efficient large-scale multi-view stereo for ultra high-resolution image sets. Machine Vision and Applications (2012). 1-18.
[37] André Ückermann, Christof Elbrechter, Robert Haschke, and Helge J. Ritter. 2012. 3D scene segmentation for autonomous rohot grasping. 2012 IEFE/RSJ International Conference on Intelligent Robots and Systems (2012), 1734-17 1740.
[38] Aaron Van den Oord. Yazhe Li, and Oriol Vinyals. 2018 Representation learning with contrastive predictive coding, arXiv e-prints (2018), a8Xiv-1807
[39] Fangjinhua Wang. Silvano Galliani, Christoph Vogel. Pablo Speciale, and Mare Pollefeys. 2021. Patch- matchnet: Learned multi-view patchmatch stereo. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition. 14194-14203.
[40] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. 2021. Dense contrastive learning for sule-supe visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6024-3033.
12

---

[41] Yude Wang. Jie Zhang. Meina Kan, Shiguang Shan, and Xilin Chen. 2020. Self-supervised equivariant atlention mechanism for weakly supervised semantic segmentation. In Proceedings of the IELE/CVF Conference on Computer Vision and Pattern Recognition. 12275-12284.
[42] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. 2018. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3733-3742.
[43] Hongbin Xu, Zhipeng Zhou, Yu Qiao, Wenxiong Kang, and Qiuxia Wu. 2021. Self-supervised Multi-view Sterco via Effective Co-Segmentation and Augmentttntt.......... ArXiv abs/2104.05377 (2021).
[44] Qingshan Xu Wenbing Tao. 2019. Multi-scale geometric consistency guided multi-view stereo. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition.5883-5422 5483-5492.
[45] Jianfeng Yan, Zizhuang Wei, Hongwei Yi, Mingyu Ding, Runze Zhang, Yisong Chen, Guoping Wang. and Yu-Wing Tai. 2020. Dense Hybrid Recurrent Multi-view Stereo Net with Dynamic Consistency Checking. In European Conference Computer Vision. Springer, 674-689.
[46] Fengting Yang. Qian Sun, Hailin Jin, and Zihan Zhou. 2020. Superpixel Segmentation With Fully Convolutional Networks. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020), 13961-13970.
[47] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. 2018. unstructured multi-view stereo. In Proceedings of the European Conference Computer Vision (ECCV). 767-783.
MVSnet: Depth inference for
[48] Yao Yao, Zixin Luo, Shiwei Li, Tianwei Shen, Tian Fang, and Long Quan. 2019, Recurrent mvsnct for high-resolution multi-view stereo depth inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5525-5534.
[49] Hongwei Yi. Zizhuang Wei, Mingyu Ding, Runze Zhang, Yisong Chen, Guoping Wang, and Yu- Wing Tai. 2019. Pyramid Multi-view Stereo Net with Self-adaptive View Aggregation. arxiv preprint 19Xvv1112.00300 (2019).
[50] Christopher Zach. 2008. Fast and High Quality Fusion of Depth Maps. (01 2008).
[51] Jinzhi Zhang, Mengqi Ji. Guangyu Wang, Xue Zhiwei, Shengjin Wang, and Lu Fang. 2021. SurRF: Unsupervised Multi-view Stereopsis by Learning Surface Radiance Field. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).
[52] Jianing Zhang, Jinzhi Zhang, Shi Mao, Mengqi Ji, Guangyu Wang. Zequn Chen, Tian Zhang, Xiaoyun Yuan, Qionghai Dai, and Lu Fang. 2021. GigaMVS: A Benchmark for Ulrra-arge-scale Gigapixel-Ievel 3D Reconstruction. IEEE Transactions on Pattern Analysis & Machine Intelligence 01 (2021), 1-1.
[53] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. 2016. Learning deep features for discriminative localization. In Proceedings of the IEFE conference on computer vision and pattern recognition. 2921-29229 -2299.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec. 4.3
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
13

---

(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as URL)? [Yes] See in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See in Sec. 33.4amd in the supplemental material.
(c) Did you report error bars (c.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] See in the supplemental material.
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See in Sec. 3.4
4. If you are using existing assets (e.g., code, data, models) or curating/releasnng new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "wby08EFaTx5", "reviewer": "Reviewer_cy7P", "summary": "The paper proposed an interesting idea to improve the pixel-wise multi-view correspondence matching using geometric correlations. \nAn elastic part representation is proposed to encode the surface connectedness. Specifically, the input image is passed through a convolutional neural network to get the per-pixel elastic part representation. Then based on the estimated depth and normal map, the authors proposed self-supervised contrastive learning to encourage the part representation to be similar on the same physical surface. \n", "questions": "1) Please clarify why the idea of \"propagation\" is very similar to the \"adaptive propagation\" in PatchmatchNet while the latter is not suitable cited. Also, discussions on the difference between these two should be discussed.\n\n2) Please clarify the usage of $z_p$ in the text.\n\n3) What is the final form of the loss function used to train the CNN. ", "limitations": "Well discussed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe authors presented a novel elastic part representation and provided an ablation study to prove the effectiveness.\n\nA thorough experiment is conducted and the proposed method is compared to a large set of state-of-the-art methods, notable improvements are observed compared to other self-supervised methods.\n\nWeakness:\n\nAlthough contributions are presented in the paper, the core idea of the work is very close to PatchmatchNet (Wang et al. 2021a) and the resulting depth estimation relies on the initial depth estimation from Gipuma (Galliani et al. 2015). Especially the key idea of \"Propagation\" proposed in this paper is very similar to the \"Adaptive Propagation\" proposed in the PatchmatchNet (Wang et al. 2021a), more discussion on the differences in-between are expected.\n\nConfusing symbols. $z_p$ in Sections 3.1 and 3.2 is referred to as \"per-pixel elastic part representation\" and \"feature on pixel local p\". I assume these two terms mean the thing?\n\nThe second paragraph of Section 3.1 is better fitted in the Network Training section and Equations (1) and (6) are the same (almost). Not sure why the same equation is written twice in the text.\n\nThe authors talked about the two loss functions used in the training but didn't provide what is the final loss function. I assume it is a weighted sum of these two loss functions?\n\nIn the Experiment Section, the values reported for Gipuma (Galliani et al. 2015) on the DTU dataset seem different from the ones reported in the original paper. Also, since the reconstruction results of COLMAP on T&T dataset are provided in Figure 4, it seems strange to not provide the quantitative evaluation in Table 2.\n\nNitpicking:\n\nAbbreviations used without explanations, e.g. NCC\n\nThe supplementary document, Section C.3, first sentence:\n\"The depth map predicted in each iteration is not always unreliable ...\", shouldn't it be \"not always reliable\"?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QXPGDOiFp1K", "reviewer": "Reviewer_hSNX", "summary": "This paper proposes a self-supervised MVS framework that learns the representation and estimates depth following a part-aware propagation and evaluation scheme. To achieve this, this paper utilizes learns part-aware representation and optimizes a part-level consistency loss. The proposed method consists of part representation learning and part-aware matching. The part representation encodes geometric details to predict a piecewise-smooth depth map. The part-aware matching conducts the depth hypotheses propagation following the idea of patchmatch. ", "questions": "1. How robust is the proposed method w.r.t. the threshold parameters? \n2. Does the authors use the same threshold values for all datasets? or find the best parameters for each dataset?\n3. Generally, how many seed points are obtained? (In figure 11 on supplementary materials, six pixels are provided.) Generally, six points are enough to cover all the images?", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1. The proposed method achieves state-of-the-art performance. The proposed method even outperforms some of the supervised methods. \n2. In some datasets, the proposed methods show dramatic performance improvements w.r.t. the state-of-the-art methods in both supervised and self-supervised methods.\n2. The proposed method does not require a fixed number of labels for segmentations and a ground-truth segmentation map.\n3. The proposed method is geometrically reasonable and is quite novel.\n4. The visualization of the elastic part representation demonstrates the effectiveness of the proposed method.\n\nWeakness\n1. The evaluation metric and unit are missing in table 3. Because DTU and T&T use different evaluation metrics, table 3 can be confused without the evaluation metrics.\n2. The running time is not competitive compared to cost-volume-based methods.\n3. The proposed method has many hyperparameters for the threshold, such as \\epsilon, \\xi, \\eta, so I think it is quite vulnerable to the parameters. It would be better to show how robust the proposed method is w.r.t the hyperparameters.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OLgg5Frs1wp", "reviewer": "Reviewer_pJ8f", "summary": "This paper introduced elastic part representation into self-supervised MVS framework to achieve more accurate dense correspondences. The pixel-wise part representation is trained by a contrastive learning after a pre-training process. The proposed elastic part representation mainly identifies pixels of similar appearance for depth propagation. The proposed approach is evaluated on the DTU and T&T benchmark datasets and achieves promising results. ", "questions": "1)\tHow was the initialisation of the depth map obtained for contrastive learning? Is this process repeated once an improved depth map is obtained?\n2)\tHow were the depth hypotheses chosen? Are they related to the initialisation without the pixel-wise part representation learning?\n3)\tPlease also clarify how the optimisation of the energy function in line 171 is solved. The reviewer interprets that no network learning is involved in the process listed in Figure.2.\n4)\tDoes the runtime reported in Table 5 include the calculation of feature distance, confidence map and the normal map?\n", "limitations": "1)\tThe method still cannot estimate accurate depth for textureless regions even with better propagation region estimation.\n2)\tIt is potentially time consuming because it requires iterative optimisation, calculation of the feature distance, confidence map, and solve the optimisation problem. Surprisingly, the reported run time is reasonable. \n", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n1)\tThis paper focuses on self-supervised learning for MVS. It adopts the contrast learning to identify pixels belonging to a connected surface or not based on learned features and guide the depth propagation\n2)\tIt achieves promising results on two benchmark datasets.\n3)\tComprehensive ablation studies are presented in the paper.\n\n# Weaknesses.\n1)\tThe figure 1 cannot accurately reflect the proposed algorithm. It is a multiple stage framework. Especially, the proposed framework relies Gipuma without part representation to obtain an initial depth which will be used to define the positive and negative set for contrastive learning for the start of the algorithm. It is missing from Fig.1.\n2)\tSome arguments in the paper are kind of misleading. Line 29-31 argues about the pixel-wise photometric regularisation is sensitive to textureless regions or illumination changes. It is not clear how the proposed elastic parts representation could overcome this. My understanding is the contrastive learning is based on the positive and negative samples obtained from an unsupervised learning framework which is already affected by the textureless and illumination changes. No depth information can be obtained from this initialisation. It is questionable the proposed representation can fix wrong depth estimation for the textureless regions. Line 32-35, It is not clear why these are related to the reconstruction.\n3)\tDepth estimation is obtained by traditional optimisation rather than learning based on the description between line 166-171.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "CRH_dtSyXZi", "reviewer": "Reviewer_HBPd", "summary": "This paper proposed a new self-supervised MVS method called ElasticMVS. Instead of relying on local photometric consistency as in existing methods, the proposed method utilizes geometric correlations parsed from images as guidance for pixel-wise multi-view correspondences. The paper first proposed a novel elastic part representation that encodes spatial proximity and surface connections, which is trained by a contrastive learning-based strategy. The paper then proposed a self-supervised MVS framework namely ElasticMVS that estimates per-view depth using a part-aware propagation and evaluation scheme. Experiments show that ElasticMVS archives significant performance gain over both the supervised and self-supervised approaches.", "questions": "- What exactly is $\\beta$ in equation (7)? It is not mentioned in the implementation details. \n\nMinor:\nLine 94: \"sucn\" -> \"such\" ", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n-  Addressing the limitation of local photometric consistency is an important topic for self-supervised MVS. The insights and motivation of the proposed method are good.\n- The paper presents extensive and sufficient comparisons with baselines, and provides detailed and informative ablation studies.\n- The proposed method works very well and achieves state-of-the-art performance, supported by various qualititive and quantitative results. \n\nWeaknesses:\n\nI don't have major concerns about this paper but I have a few questions to ask for clarification.\n- It is not clear to me why it is necessary to learn the part representation $z_p$. $z_p$ is learned primarily using geometric cues such as the 3D point-to-plane geometric distance, which is also available during inference. Why can't we design some simpler similarity functions directly based on the supervision signal of $z_p$ which is the 3d distances between p and q? For propagation, the candidates for pixel p can just be its nearby pixels that satisfy line 113-117. What is the benefit of learning the part representation compared to directly using geometric based heuristics? \n- The quality of the elastic part representation relies on initial depth map. How robust is the elastic part representation to the noise and error in the initial depth map? Is the method still able to handle textureless regions if the initial depth map for these regions are already erroneous? \n- To me, the abstract is a bit overclaiming. In line 6 it says the method explores \"geometric correlations such as surface connectedness and occlusion boundaries\". This sets my expectation high but I found it does not really match the proposed method. The method does not reason about surface connectedness or occlusion boundaries explicitly, but rather implicitly. In my opinion the key part of the method  is clustering points based on geometric proximity (both 3D point-to-plane and in full 3D), and use it to guide patchmatch propagation and evaluation. It is still a neat idea but the description in the abstract can be a little misleading. I think the sentence in line 6 can be replaced with a more proper description.\n- It is possible that two spatially far apart pixels can be coplanar and thus have a small 3D point-to-plane distance, but that is not desirable. What part of the method suppresses this kind of points? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept