CURVATURE-GUIDED DYNAMIC SCALE NETWORKS FOR MULTI-VIEW  STEREO
Authors: Khang Truong Giang
Soohwan Song
Sungho Jo
Venue: ICLR 2022 Poster (2022)
Keywords: multi-view stereo
3D reconstruction
dynamic scale

Abstract: Multi-view stereo (MVS) is a crucial task for precise 3D reconstruction. Most recent studies tried to improve the performance of matching cost volume in MVS by introducing a skilled design to cost formulation or cost regularization. In this paper, we focus on learning robust feature extraction to enhance the performance of matching costs, without need of heavy computation in the other steps. In particular, we present a dynamic scale feature extraction network, namely, CDSFNet. It is composed of multiple novel convolution layers, each of which can select a proper patch scale for each pixel guided by the normal curvature of image surface. As a result, CDFSNet can estimate the optimal patch scales to learn discriminative features for accurate matching computation between reference and source images. By combining the robust extracted features with an appropriate cost formulation strategy, our final MVS architecture can estimate depth maps more precisely. Extensive experiments showed that the proposed method outperforms other state-of-the-art methods on complex outdoor scenes. It significantly improves the completeness of reconstructed models. Moreover, the method can process the high resolution with faster run-time and lower memory compared to the other MVS methods. 

Published as a conference paper at ICLR 2022
CURVATURE-GUIDED DYNAMIC SCALE NETWORKS FOR MULTI-VIEV STEREO
Khang Truong Giang" Soohwan Song
Sungho Jo
a School of Computing, KAIST, Daejeon, 34141, Republic of Korea
b Intelligent Robotics Research Division, ETRI, Daejeon 34129, Republic of Korea (khangtg, shjo @eaakkisstt wtckaastst....... kr, sochwansong@etr!! re.kr
ABSTRACT
Multi-view stereo (MVS) is a crucial task for precise 3D reconstruction. Most recent studies tried improve the performance of matching cost volume in MVS by designing aggregated 3D cost volumes and their regularization. This paper fo- cuses learning a robust feature extraction network to enhance the performance of matching costs without heavy computation in the other steps. In particular, we present dynamic scale feature extraction network, namely, CDSFNet. It is composed of multiple novel convolution layers, each of which can select a proper patch scale for each pixel guided by the normal curvature of the image surface. As a result, CDFSNet can estimate the optimal patch scales to learn discriminative features for accurate matching computation between reference and source images. By combining the robust extracted features with an appropriate cost formulation strategy, our resulting MVS architecture can estimate depth maps more precisely. Extensive experiments showed that the proposed method outperforms other meth- ods on complex outdoor scenes. It significantly improves the completeness of reconstructed models. As result, the method can process higher resolution in- puts within faster run-time and lower memory than other MVS methods.
INTRODUCTION
A challenging problem in multi-view stereo (MVS) is accurately estimating dense correspondences across collection of high-resolution images. Many MVS studies have tried to solve ill-posed
MVS problems such as matching ambiguity and high computational complexity. In such problems, learning-base -baed MVS methods (Chen et al.] (2099: Luo et al., 2019, Chen al.) 2020a [uu Tao
2020a Xue et al., 2019) usually outperform traditional methods (Galliani et al. 2015 Schonberger Frahm 2016). The learning-based methods are generally composed three steps, including feature extraction, cost volume formulation, and cost volume regularization. Most studies made an
effort to improve the performance of cost formulation (Luo et al.] 2019 Zhang et al. [2020] Xu &
Tao 2020c; Yi et al. 2020) cost regularization (Luo et al.) 2019 2020 Wang et al. 2021 The
visibility information (Zhang et al. 2020 Xu & Tao (202cc) or attention techniques (Yi et al. 2020
Luo et al. 2020) are considered for cost aggregation and feature matching, respectively. Further- more, some studies employed a hybrid 3D UNet structure (Luo et al. 2019:] Sormmnn et al.
2020)
for cost volume regularization. While these approaches significantly improve MVS performances,
their feature extraction networks have some drawbacks that degrade the quality of 3D reconstruction.
First, they observed restricted size of receptive fields from the fixed size of convolutional kernels. This leads to difficulty in learning robust pixel-level representation when an object's 's scale varies extremely in images. As result, the extracted features cause the low quality of matching costs, es- pecially when the difference of camera poses between reference and source images is large. Second, MVS networks are often trained on low-resolution images because of limited memory and restricted computation time. Therefore, the existing feature extraction methods utilizing fixed-scale feature representation on deep networks could not generalize to high-resolution images in prediction; the high-resolution image set.
MVS performance is downgraded
To address these issues, we propose a new feature extraction network that can be adapted to various object scales and image resolutions. We refer the proposed network as suraatureguuddes dynamic

---

Published as a conference paper at ICLR 2022
Comall
ktrge
SPEECC zmrg
sole mop al 354 258
seak sap 1152 864
Agen Sonoxx Aug
segul Bottos cap
(a) Illustration of scale selection for each pixel in reference image from proposed CDSConv
(b) CDSConv layer
Figure 1: The results extracted by the proposed CDSConv and overall pipeline of CDSConv
scale feature network (CDSFNet). The proposed network is composed of novel convolution lay- ers, curvature-guided dynamic scale convolution (CDSConv), which select a suitable patch scale for each pixel to learn robust representation. This selection is achieved by computing the normal curvature of the image surface at several candidate scales and then analyzing the computed outputs via a classification network. The pixel-level patch scales estimated from the proposed network are dynamic with respect to textures, scale of objects, and epipolar geometry. Therefore, it trains more discriminative features than existing networks (Gu et al. 2020 [Chnn et al. 2020 Yang et al. 2020) Zhang al., [2020] I Luo et al. (020)) for accurate matching cost computation. Fig. 1a illustrates the dynamic pixel-level patch scales estimated from CDSConv where the window size is ranged from 7 7xT 7 17 17. As shown in the figure, CDSConv predicts the adaptive patch scales where the scale is small in thin structure or texture-rich regions and large in the texture-less regions. Moreover, the scale is dynamic according to image resolution. CDSConv generally produces small scale values in a low-resolution image and large scale values in high-resolution image.
We also formulate a new MVS framework, CDS-MVSNet, which aims at improving the quality of MVS reconstruction while decreasing computation time and memory consumption. Similar to previ- ous works (Gu et al. 2020; Cheng et al. 2020 Yang et al. 2020 Wang et al., (02T), CDS-MVSNet composed of cascade network structure estimate high-resolution depth maps in coarse-to- fine manner. For each cascade stage, it formulates a 3D cost volume based on the output features of CDSFNet, which reduces the matching ambiguity by using proper pixel's scale. For instance, sider a 3D point on the scene that needs to be reconstructed, CDSFNet can capture the same context information in both reference and source views for this point to extract the matchable features. Fur- thermore, CDS-MVSNet applies visibility-based cost aggregation to improve the performance of stereo matching. The pixel-wise visibility is estimated from the curvature information that encodes the matching ability of extracted features implicitly. Therefore, CDS-MVSNet performs accurate stereo matching and cost aggregation even on low-resolution images. Finally, the proposed network can generate high quality 3D models only by processing half-resolution images. This approach can significantly reduce the computation time and memory consumption of MVS reconstruction.
The contribution of this paper is summarized as follows:
We present the CDSConv that learns dynamic scale features guided by the normal curvature of image surface. This operation is implemented by approximating surface normal curva- tures in several candidate scales and choosing proper scale via a classification network.
We propose new feature extraction network, CDSFNet, which is composed of multiple CDSConv layers for learning robust representation at pixel level. CDFSNet estimates the optimal pixel's scale to learn features. The scale is selected adaptively with respect to structures, textures, and epipolar constraints.
We present CDS-MVSNet for MVS depth estimation. CDS-MVSNet performs accurate stereo matching and cost aggregation by handling the ambiguity and visibility in the image matching process. It also significantly reduces the run-time and memory consumption by processing the half-resolution images while maintaining the reconstruction quality.
We verify the effectiveness of our method two benchmark datasets; DTU (Anass al. 2016) and Tanks & Temples (Knapitsch et al., 2017). The results demonstrate that the proposed feature learning method boosts the performance of MVS reconstruction.
2

---

Published as a conference paper at ICLR 2022
2 RELATED WORK
Multi-view Stereo. Traditional MVS studies can be divided into three categories: volumetric, point cloud-based, and depth map-based methods (Furukawa & Hermández] 2015). Comparatively, depth map-based methods are more concise and flexible, they estimate depth maps for all reference images and then perform depth fusion to achieving a 3D model (Galliani et al. 2015 Schönberger et al.) 2016 Xu & Tao, 2019). The popular MVS methods in this category are usually based on Patch- Match Stereo algorithm (Barnes et al. 2009). Recently, many studies have applied a learning-based approach, which has achieved significant performance improvements over traditional methods. Sev- eral volumetric methods (Kar et al., 2017 Ji et al. 2017) employed 3D CNNs to explicitly predict global 3D surfaces. Yao et al. (2018) proposed a depth-map- based method, MVSNet, which is an end-to-end trainable architecture with three steps. However, these methods still confronted the matching ambiguity problem and could not process high-resolution images due to limited memory. Many methods based on the three-step architecture of MVSNet are proposed to address these is- sues. Several studies (Gu et al.) 2020 Cheng et al.) 2020 Yang et al.) 2020 [aan et al., [021]] Zhang et al. 2020) applied a coarse-to-line Iramework which estimates the depths through multi- ple stages to reduce the computational complexity. They used the estimated depth of each stage to adjust the depth hypothesis range for cost formulation in the next stage. To enhance the matching cost volume, other methods introduced a novel strategy for cost volume formulation. [Luo et al. (2019) proposed a patch-wise feature matching instead of pixel-wise matching to compute the cost. Yi et al. (2020) and Luo et al. (2020) applied an attention technique to improve feature matching. Zhang et al. (2020); Xu & Tao (2020c) estimated visibility information to score the view weights for cost aggregation.
All these methods (Gu et al. 2020: Cheng et al. 2020; Yang et al. 2020 Wang et al. 2021 Zhang et al. 2020; Luo et al. 2020; Yu & Gao 2020) had trade-off between computation and perfor- mance; the methods with high performance usually require expensive computations. Also, they only applied standard CNNs with static scale representation for feature extraction. In this work, aim to learn dynamic scale representation for more accurate feature matching. We also estimate pixel-wise visibility information to remove noises and wrong matching pixels in cost aggregation. These goals are achieved by analyzing the normal curvature of the image surface.
Recently, Xu et al. (2020) proposed an MVS method that considers the normal curvature information for feature extraction. This method handles ambiguity in the patch- match stereo method (Barnes) al. 2009). It selects a patch scale for each pixel by thresholding the normal curvatures computed in several candidate scales. Although the method effectively improves the quality of patch matching, the use of its handcrafted features limits the overall MVS performance. contrast, our method does not require extracting handcrafted features and determining the threshold for scale selection. Furthermore, CDSConv operates not only on the color information of images but also on the high dimensional features. Multiple CDSConv operations can be stacked to form very deep architecture for robust feature extraction in MVS.
Multi-scale features. Due large-scale variations of objects and textures in complex scenes, multi- scale representations are adapted for accurate and effective dense pixel-level prediction in many computer vision tasks. For the MVS task, Feature Pyramid Network (FPN) (Lin et al.. 2017 Gu let al. 2020; Cheng et al.] 2020 Zhang et al., 2020) Luo et al. (020) is a common approach for multi-scale feature extraction. It is designed as the UNet-liked architecture (Ronneberger et al. 2015) to fuse features from different scales into a single one. However, the multi-scale of FPN is achieved at an image level through down-upsampling operations. It cannot capture the large-scale variation of objects at a pixel level. Several studies in image segmentation can learn lixel-level multi-scale features by using multiple kernels with different sizes (He et al.) 2019 Liu et al. 2016 Chen et al. 2017 Zhao et al. [017]_ However, these methods prefer large receptive field for segmentation tasks and usually ignore the details of objects essential for MVS tasks. In contrast to the aforementioned methods (Gu et al. 2020; Cheng al.) 2020 Zhang et al., 2020) Luo et al.) 2020 He et al.) 2019), our proposed CDSConv can select an optimal scale for each pixel to produce robust representation and enhance the matching uncertainty. Moreover, our feature extraction CDSFNet can cover denser scales by utilizing the power of a deep CNN.
Dynamic filtering. Instead of observing static receptive fields, several studies modified standard convolution to choose the adaptive receptive field for each pixel (Wu et al., 2018 Jia et al.) 2016 Han et al. 2018). SAC (Zhang al. (017) was proposed to modify the fixed-size receptive field
3

---

Published as a conference paper at ICLR 2022
by learning -aatiionn scale coefficients. Deformable ConvNets (Dai al. 2017 Zhu al. 2019) learned offsets for each pixel in a regular sampling grid of standard convolution to enlarge the sampling field with an arbitrary form that can discover geometric innetriicinvariantt features. Recently, Chen et al. (020b) proposed a dynamic convolution composed of multiple kemels with the same size increase model complexity without increasing the network depth width. All these existing works learn dynamic filters directly from the input images. They are suitable for segmentation or recognition tasks. For MVS task, they ignore the epipolar constraint between reference and source images. In contrast, our dynamic filtering is guided by normal curvature computed in the direction of epipolar line, which can measure the matching capacity between reference and source image patches. To the best of our knowledge, this is the first work that introduces dynamic filters to MVS.
3 PROPOSED FEATURE EXTRACTION NETWORK
3.1 NORMAL CURVATURE
Normal curvature is used to estimate how much surface is curved specific point along particular direction (Do Carmo 2016). Let p(X, o)) be a patch centered at pixel X and having a scale which is proportional the window size. L(X, Ia79) is denoted as the epipolar line on the reference image fre which passes through X and is related to a source image _ To reduce the matching ambiguity between p in rref and the correct patch p' in Torc, proper scale a should be chosen when the corresponding patch p(X, o)) is less linearly correlated to its surrounding patches along the epipolar line L. From the scale space theory (Lindeberg 1994), the patch p(X, o) can be considered as point the image surface represented at the scale Therefore, the scale is optimal for stereo matching if the image surface represented in that scale is curved significantly at the point X along the epipolar line direction (Xu et al. (020). This requires compute the normal curvature for the patch p( X, 0r) along the direction w M, v] of the epipolar line L. The formula can be expressed in terms of the values of the first and second fundamental forms of the image surface:
T F. Tay
FMI
1
W
Lyy 1yy.
W
curv (X,w)
FM,
V 12
12
+
1
TgT
T
I
1 1 T2
where Lu, Ly: Luns Lay: Jay are the first-order and second-ordes derivatives of the image I along the x-axis and y-axis, respectively.
3.2 SENVATURE-GUDDEE DYNAMIC SCALE CONVOLUTION
This section describes the novel convolutional module CDSConv to extract dynamic scale features. Given a set of K convolutional kernels with different size {C1, C2...., Cx) } corresponding to K candidate scales {œ1, 20 ... .x) }. CDSConv aims to select proper scale for each pixel X. Hence, it can produce robust output feature Fout from the input F.. based on the selected scale. Fig. lb shows our overall pipeline of CDSConv including two steps. First, CDSConv estimates approxi- mately normal curvatures at K candidate scales. Second, it performs a scale selection step to output the optimal scale from K estimated curvatures. This selection is implemented by classification network composed of two convolutional blocks and a Softmax activation for the output.
Learnable normal curvature. The formula of normal curvature of patch centered at X along the direction of epipolar line w = [u, w/ in Eq. can be re-written as follows
u I (X, + Iww σ) + To (X,0)
curv. (X,w)
(2)
V (++2(X 12(X, (1+ (ulz (x,(
where o(X.a)= I(X) G(X, 0) is the image intensity of pixel X in the image scale σ; it is determined by convolving I with a Gaussian kernel G(X, 0) with the window size/scale o (Linde- berg 1994). The derivatives In. Iy. Lans Lay. and Lyy can be computed by convolution between the original image / and the derivatives of Gaussian kernel G(X.o)
gi+s
giti
Oy 1(X, T(X)+ Or G(X,o)
(3)
4

---

Published as a conference paper at ICLR 2022
where is the convolution operator. There are two main drawbacks when embedding the normal curvature in Eq. into a deep neural network. First, the computation is heavy because of five convolution operations for computing the derivatives Las Ly. Laxs Lzys and Iny- Second, using Eq. 2 to compute curvature is infeasible when the pixel X is a latent feature [ee instead of the image intensity (X). For these reasons, we derive an approximate form of the normal curvature, which reduces the computational cost and can handle the nigh-dimensional feature input.
To address the heavy computation issue, we notice that the curvature curti and derivatives Letyo proportional to Gaussian kernel G. following Eq. and Eq. [] Moreover,
a classification network to select the patch scale automatically from the curvature inputs. Therefore, we can perform normalization for the curvatures by rescaling the kernel G. We restrict the gaussian kernel in a small range, i.c., G(X, o) < 1. As result, the derivatives L.. Ly. Iza, Lzy. and Lyy are also much smaller than 1. We can approximate the denominator in Eq. 2 by Eq. 2 can be re-defined as
curto w) u² L' + 2wv (X, σ) + o (X,
(4)
where cure, (X,w), Tua: Jy and I yu is much smaller than 1, which implies that their values should be restricted in a small range. The computation in Eq. 4 is reduced by haft compared to Eq. 2
To make Eq. 4 work with the high-dimensional image feature Fin (X,(X, o), we propose to use learn- able kernels instead of using fixed derivatives of Gaussian kernel. In particular, for each scale a, introduce three learnable convolution kernels KT², Kz" Kyy replace Gzæs Gzy. and Guy respectively. These kernels adapt to the input features to approximately compute the second-order derivatives of the image surface. They are trained implicitly by backpropagation when training the end-to-end networks. Weights of these kernels need to be restricted to small value K. < because of the assumption about the Gaussian kernel. We enforee this constraint by adding regu- larization term to the loss in Section 4.2
In summary, we propose learnable normal curvature having the formula as follows (written in the matrix form)
curv, (X, w) Fin # K Fin * K
Fin K Fin Ky9
(5)
where K:''s are the learnable kernels and | + 0, Fin is the input feature.
Scale selection. After obtaining K normal curvatures {curve, curVezy curvax } in K candi- date scales, this step selects a proper scale (() ).) for each pixel X and then outputs the feature Foou (X) from the corresponding convolution kernel C(.)- The proper scale is selected by analyzing the cur- vatures estimated in K candidate scales. is observed that a small scale cannot capture enough context information to learn discriminative feature, especially in low-textured regions. Mcanwhile, a large scale smooths local structure rich texture regions. MARMVS (Xu et al.) 2020) chose the proper scale by searching the normal curvature with a threshold T = 0.01. However, using of fixed threshold T prevents generalization on various scenes and structures. Instead, we propose a classifi- cation strategy that automatically selects the proper scale from K curvature inputs. This is achieved by using lightweight CNN with two convolutional blocks. For each pixel X, the CNN outputs one-hot vector [W1, U22 ..., wK] for scale selection by applying a Softmax with small temperature parameter, which is a differentiable approximation to argmax operator (Jang et al. 2017).
Finally, the feature output pout is produced from the feature input pin with K candidate kernels {Ci, C2,..., Cx) } by using weighted sum
pout
wi (F * C1) + w2(F'' i C2) + (K(F"" * Ck)
(6)
where * is the convolution operator. Also, we can extract the normal curvature corresponding to the selected scale by
NC Wircuro, + WgcurVoz + + uureuroo
3.3 CURVATURE-GUIDEE DYNAMIC SCALE FEATURE NETWORK
This section introduces the Curvature guided dynamic scale feature network, CDSFNet, which is used as feature extraction step for our MVS framework. CDSFNet is composed of multiple CD- SConv layers instead of standard convolution layers. By expanding the searching scale-space,
5

---

Published as a conference paper at ICLR 2022
for Virv
Elos-ap view.
caplured viewpoints
camera pose sec.
Figure 2: Dynamic scale according different captured viewpoints and relative camera pose between reference and source views. Figure shows the scale maps when images is captured at far view and close-up view. Figure shows the results when the difference of camera poses small (left) and large (right).
CDSFNet select the optimal scale for each pixel to learn robust representation that reduces matching ambiguity.
Architecture. CDSFNet is an Unet-liked architecture (Ronneberger et al. 2015). It is organized into three levels of spatial resolution I E {0, 1 to adapt the coarse-to-fine MVS framework. Given the inputs including image I and its estimated epipole e, the network outputs three features for three levels (FO)) F(1), p(2) and three estimated normal curvatures (NCrstt)l), NN Neeee,())NNeet.. NC
Understanding CDSFNet. The main goal of CDSFNet to select pixel-level scale for robust feature extraction. The scale changes dynamically, depending on the detail of image objects and the epipolar constraint, i.e. the relative camera pose between reference and source images. Therefore, method is more appropriate for MVS than the other dynamic scale networks (He et al. 2019 Wu et al. 2018 Jia et al. 2016) proposed in image segmentation task.
To have low computational complexity, the number of candidate scales in CDSConv layer should be small, we only use 2 or 3 candidates. However, the searching scale-space is expanded profoundly when multiple CDSConv layers are stacked in CDSFNet. Fig. show the patch scale window size for each pixel estimated roughly from the first two layers of CDSFNet. For each reference and source views, we draw the reference and source scale maps respectively. There are main advantages of CDSFNet shown in Fig. 2 First, the estimated scales in rich texture, thin structure, and near-edge regions are often small while those in untextured regions are large. Second, the scales are changed with respect to viewpoints (Fig. (2) and relative camera pose between reference and source views (Fig. [b).
Fig- [... the closer viewpoint is, the larger scales are estimated. In the reference and source scale maps are similar when the difference of camera poses is not large. Otherwise, when the difference is large, the scale map of reference view changed to adapt the source view, which was marked in the red circle. This is superiority of CDSFNet because estimates scale based on epipolar constraint between reference and source views.
CDS-MVSNET
In this section, describe the proposed MVS network referred to as CDS-MVSNet, which predicts a depth map Dree from the reference 7 and N source images {T.) We adopted the cascade structure of CasMVSNet (Gu al. 2020) as the baseline structure of CDS-MVSNet. The network is composed of multiple cascade stages to predict the depth maps in a coarse-to-fine manner. Each stage estimates a depth through three steps: feature extraction, cost volume formulation, cost volume regularization & depth regression.
CDS formulates 3D cost volume based on the output features of CDSFNet. The features of CDSFNet effectively reduce the matching ambiguity by considering the proper pixel scales. Due to the robustness of our feature extraction and cost formulation, CDS-MVSNNe can produce more accurate depths in coarser stages compared to other cascade MVS methods (Gu et al. 2020 Cheng jet al.) 2020; Yang et al., 2020 Wang et al. 2021). There is even little difference the accuracy of the depths estimated at half resolution and full resolution. Therefore, differently from CasMV only computes the depths of half-resolution images using three-step MVS computa- tion and then upsamples the depths to the original resolution in the final stage. The upsampled depths at the original resolution can be refined by a 2D CNN to obtain the final depths. This approach can drastically reduce the computation time and GPU memory requirements.

---

Published as a conference paper at ICLR 2022
4.1 DETAILS OF ARCHITECTURE
Robust Feature Extraction. Given the reference image Io and N source images {11, } CDSFNet extracts a robust feature pair (Fo.,; ( for each image pair (In, k=... Note that the feature of the reference image Fo.s varies according to each source image i due to the guidance of epipolar geometry in normal curvature estimation. This is our superiority compared to the statc-of-the-e -art MVS networks which learn an identical feature for the reference image.
Cost Volume Formulation. We first compute two-view matching costs and then aggregate them into a single cost. The two-view matching cost is computed per sampled depth hypothesis to form cost volume (Yao et al. 2018; Gu al. 2020; Wang et al., 202T). Given a sampled depth hypoth- esis dj I... D) along with all camera parameters, we warp the feature maps of source image (F.) into this depth plane. Let F. (dz) ) be the warped feature map of source image i at the depth hypothesis d, The two-view matching cost Vi (d,) between the reference image In and source image I at the depth hypothesis dj is determined as Vi(d,) (Fo,i, FT (d,) (d,)). where (.. is the pixel-wise inner product for measuring the similarity two feature maps.
Next, we perform cost aggregation over the computed two-view cost volumes. Inspired from visibility-awarenes for cost aggregation (Zhang et al., 2020 Xu Tao 2020e), we employ pixel- wise view weight prediction. However, instead of predicting directly from two-view cost volume (Zhang et al. [2020] Xu & Tao 2020c Yi et al. (020). we additionally utilize the normal curvature maps, NC estimated by CDSFNet in Eq. 7 The normal curvature can implicitly provide in- formation about level details surface. For instance, a pixel with small normal curvature indicates that it belongs to a large untextured region. In this case, its feature is unmatchable and needs to be eliminated from cost aggregation to reduce noises for cost regularization step.
In summary, the estimated normal curvature encodes implicitly the matching ability of learned fea- ture. Let N NO be the estimated normal curvature of the reference view related to source view i, and H, be the entropy computed from two-view matching cost volume V,. We use simple 2D CNN denoted as Vis(.) to predict view weight map W, from two inputs NC and H,. Finally, the aggregated cost is computed by weighted mean.
D Hi H-=-D v.d,
Wi Vis(NC H.), V(d, EN W.Vl()))
EM W.
Cost volume regularization & Depth regression. Following most recent gearning-based MVS methods, 3D CNN is applied to obtain depth probability volume P from the aggregated matching cost volume V. The depth is then regressed from P by Dest dj
4.2 LOSS FUNCTION
We propose feature loss for training CDSFNet effectively. Given ground truth depth D of the reference image, we warp the feature of source images into this depth map and then compute the matching cost (DDe). We label this map positive, hence defines its classification label as e 1. To generate the matching costs with negative labels c = 0. we randomly sample Na neighboring depths D around the ground truth D and then compute the matching costs V(D"es) for these depths. Finally, binary cross entropy is used to define the feature loss. However, we need to add reg- ularization terms restrict the convolutional weights of CDSFNet, WCDSF, and estimated normal curvature st to a small range as mentioned in Section 3.2 Therefore, the final feature loss is defined
1 Σ log X
X M
Creat M
sig(V( + AllWDDSF11
| (8)
where sig(.) is the sigmoid function, M is total number of pixels X, A 0.01 and X2 = 0.1 are the regularized hyperparameters.
Similar to most learning-basee MVS methods (ao et al. 2018 Gu et al. 2020). the depth loss Ldepth is defined by L1 loss between the predicted depth and the ground truth depth. Because our method includes 4 stages, there are four depth loss (C(() 11 Finally, the total loss Liotal is
defined as sum of mentioned losses Liotal 5L Σi (() depth

---

Published as a conference paper at ICLR 2022
Table 1: Quantitative results on the DTU evaluation
Table 2: Evaluation of estimated depth with different image resolutions on DTU (higher is better).
dataset (lower better).
Methods
Mean crrur distance (nun)
Acc.
Comp.
Overall
Methods
552
Resolutions
1152%
1250 x
(wildth
1408
beight
1536 x
Gipume" Coollant ecal
2015
0.383
0,573
0,575
9w6
1152
Colmap"
2016
0.403
0.664
0.552
P-MVSNe
2019
0.420
Prec. 2mm (%)
CusMYSNNx
75.35
74.01
72.22
70.92
Fyst-MYSNN
0.235
0.370
MarchMatchNer
69.53
71.57
71.20
70:59
Chince
0.362
0.363
UCSNNe
24.11
73.85
72.95
71.55
70,49
CasMVSNee
AuMYS Luvgd Bi
0.325
0.383
0.385
0.329
0.355
0.356
CVP SYv
66.80
09.75
70.71
71.03
71.16
UCSNE
Cherger
0.358
0.349
0.344
Ours
PVA-MYSNe
2021
0.352
Ptec. Mamn (%)
PecchMatchNet
CaMVSNa
82.11
76.31
80.47
78.32
76.03
79.02
77.67
17.34
76.15
BP-MVSNa Stemstn of p0z2
CVP-MYSN
Yearee 2220
0.353
0.295
0.405
0.320
0.327
0.351
UCSNes
$2.15
$0.10
75.59
76.91
75.61
Ours (DTU
0.352
0.240
0.316
CVP-MVSNN
77.41
78.18
78.76
Ours (
0.351
0.315
90.99
Untitiosll nethod, untae baiing DTU, (raisel os roddite vension
5 EXPERIMENTAL RESULTS
5.1 DATASETS
The datasets used in our evaluation are DTU (Aanæs et al., 2016). BlendedMVS (Yao al. 2020). and Tanks & Temples (Knapitsch et al. 2017). D. Due to the simple camera trajectory of all scenes in DTU, we additionally utilize the BlendedMVS dataset with diverse camera trajectories for training- Tanks& & Temples is provided as a set of video sequences in realistic environments. We train the model on the training sets of DTU and BlendedMVS, then evaluate the testing set of DTU and intermediate set of Tanks& & Temples. Our source code is available at atlhtps::/ petts::/githann h/gpthb....ttg/gg/// com/ Truongkhang/ -nmiaaannnnndddnneeeeett nvsnet
5.2 BENCHMARK PERFORMANCE
Results for DTU dataset. We predict the depth the highest resolution 1536 1152. The perfor- mance of the proposed method was compared with state-of-the-art MVS methods, including con- ventional and learning-based methods, using the DTU dataset. However, we especially notice the learning-base based methods including CasMMSNet, UCSNet, CVP-MVSNet, and Vis-MVSNet, which adopted the cascade structure for MVS depth estimation. Similar to us, these methods used the same 3D CNN cost regularization. Therefore, we can validate our feature extraction and cost formulation by comparing with them.
Table I presents the quantitative results achieved by the methods on the DTU evaluation dataset. We calculated the three standard error metrics (Aanæs et al.) (016)) given on the official site of DTU: accuracy, completeness, and overall error. Following the same setups with most of the baselines, we trained our model on the DTU training set and then used this model for evaluation. Moreover, we also provided the results which produced by training on both DTU and BlendedMVS simultancously. In both cases, our method exhibited the best performance in terms of completeness and overall error. We also achieved good results for both accuracy and completeness, while the other methods often produced a large trade-off between these two metrics. For example, Gipuma had a high accuracy 0.283 but it produced very low completeness, 0.873. In contrast, PatchMatchNet was with high completeness (0.277) and very low accuracy (0.427). Moreover, our method outperformed all similar methods using the cascade structure. This indicates that our feature representation and cost aggregation guided by normal curvature can remarkably improve MVS depth prediction remarkably.
To validate the dynamic scale property of our CDSFNet, we measure depth precision on different image resolutions. Depth precision is the average percentage of depths with errors lower than a defined threshold (Gu et al. 2020). Here we computed precision with the thresholds of 2mm and 4mm. We compared to CasM WStet, PatchMatchNNer, UCSNet, and CVP-MVSNet, which applied the cascade structure to predict high-resolution depths. Note that all methods are trained on the 512. Table presents T the quantitative results for the depth estimated on five image resolutions. The baseline methods degraded the performance when the maage-resolution increased while method could maintain stable performance any image resolution. The reason is that our feature extraction CDSFNet can utilize dynamic scales for learning features. The dynamic scale adapts to various image-rssolutions and object scales in image.
datasets with image-resolution of 640 x
8

---

Published as a conference paper at ICLR 2022
Table 3: Performance comparisons (F-score) of various reconstruction algorithms on the intermediate
quences of the Tanks Temples benchmark. The higher value of F-score implies better reconstruction results. Our method achieves the best performance in terms of mean F-score.
Method
Mraa
Family
Francis
Horse
Lighthonse
M6ll
Panther
Playground
Irain
MOMME
Na & Tan 2019
57.27
0::24
51.45
46.37
6:3.20
55.07
57.64
60.05
54,45
ACMP
54.06
54,11
61.65
54.16
47.39
39.59
34.98
47717
60.03
77.40
60.23
47:07
53.44
62.21
57.28
60.54
52.07
AnMVS
Lug X0AU
60.05
73.00
62.55
44.08
64,88
56.08
59.39
63.42
56.06
CaIMVSN2r
20120
56.84
76.37
58.45
54.06
(Chesg
PVA-MMSNNE
54,46
46.80
46.01
57.23
54.75
CVP-MVSNI
54.03
76.50
47.74
36.94
55.12
57.28
54,28
57.43
47.54
BP-MVSNes
of MLA1
57.60
77.31
60.90
47.89
58.26
56.00
51.54
58.47
50.41
Ours (timee suming on WeVndedMVVS)
53.12
53.45
Ours OTTUBBiiiIJMSS.
61.58
78,85
63.17
$3,04
61.34
62.63
59.06
62.28
$2.30
Inditiional method, only traiting the traiting ser DTU, traiting on DTU and then line runing on BlendsiMVS, taining molited veriion of DTU
1909
Ones
CaIMVVNN SN
FARMESNNT
CVR-MVSN -EES-MINNE
50
180
Figure 3: Relating GPU memory and run-time to the input resolution on DTU's evaluation set. The original image resolution 1600 x 1200 (100%). Here we evaluate the highest resolution 1536 x 1152 (95.2%).
Generalization on Tanks & Temples Dataset. To verify the generalization capability of our CDS- MVSNet, we also provided two evaluations similar to the evaluation of DTU. First, we fine-tuned the pre-trained model on DTU by using BlendedMS dataset. Second, we used directly the model trained on both datasets for evaluation. The reconstructed point clouds were submitted to the benchmark website of Tanks & Temples (Knapitsch et al. 2017) to receive the F-score, which is a combination of precision and recall of the reconstructed 3D model. Table 3 lists the quantitative results of our method and other state-of-the-art methods. As shown in Table 3 our method achieved the best mean Fscores for both evaluations, 60.82 and 61.58, respectively. Our method outperformed both the state-of-the-art traditional methods such as ACMM, ACMP and the learning- based methods including AttMSS and Vis-MVSNet. The qualitative results of all scenes are shown in detail in the supplemental materials. Compared to CasMVSNet and UCSNet, which are most similar to our method except the feature extraction and cost formulation, we obtained better mean F-score with high margins. This demonstrates the effectiveness of our feature extraction CDSFNet and cost aggregation strategy on complex outdoor scenes.
This section evaluates the memory consumption and run-time compared to several state-of-the-an learning- -baeed methods that achieve competing performance and employ the cascade structure similar to us: CasM UCS-Net, CVP-MVSNet, and FastMVSNet. Fig. shows the memory and run- un-time of all methods measured on DTU evaluation set with different image resolutions. We observe that the run-time and memory of all methods are more required when the resouution increases higher. However, we notice that the memory consumption of our method was dropped from the resolution of 80% while our run-time was slightly increased. This indicates that our method can release significant amount of GPU memory by only sacrificing small amount of run-time, Our method achieves significantly faster run-time and lower required memory at the very high image-resolution compared to the other baselines. For example, resolution of 1536 x 1152 (92.2%), the memory consumption reduced by 54.0%. 34.1%, and 54.0% % compared to CasM UCSNet, and CVP-MVSNet respectively. The corresponding comparison results of run-time are 43.2%, 37.8%, and 74.9% %. Finally, we conclude that our method is more efficient in memory consumption and run-time than most ofateoofthe-arr learning-basc methods.
Run-time and Memory.
6 CONCLUSION
We propose a dynamic scale network CDSFNet for MVS task. CDSFNet can extract the discrimina- tive features to improve the matching ability by analyzing the normal curvature of the image surface. We then design an efficient MVS architecture, CDS MVSNet, which achieves state-of-the. per- formance on various outdoor scenes

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work was supported by the National Research Foundation of Korea (NRF) funded by the Min- istry of Education under Grant 2016RIDIA/B01013377
REFERENCES
Henrik Aanæs, Rasmus Ramsbøl Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl. Large-scale data for multiple-view stereopsis. International Journal of Computer Vision, 120(2): 153-168, 2016.
Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A ran- domized correspondence algorithm for structural image editing. ACM Trans. Graph., 28(3):24, 2009.
Liang-Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4): 834 848, 2017.
Rui Chen, Songfang Han, Jing Xu, and Hao Su. Point-based multi-view stereo network. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1538-1547, 2019.
Rui Chen, Songfang Han, Jing Xu, and Hao Su. Visibility-aware point-based multi-view stereo network. IEEE transactions on pattern analysis and machine intelligence, 43950)33655-3708 2020a.
Yinpeng Chen, Xiyang Dai, Mengehen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention over convolution kernels. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11030-1 1 1039, 2020b.
Shuo Cheng, Zexiang Xu, Shilin Zhu, Zhuwen Li, Li Erran Li, Ravi Ramamoorthi, and Hao Su. Deep stereo using adaptive thin volume representation with uncertainty awareness. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2524 2534, 2020.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 764-773, 2017.
Manfredo P Do Carmo. Differential geomelry of curves and surfaces: revised and updated second edition. Courier Dover Publications, 2016.
Yasutaka Furukawa and Carlos Hernández. Multi-view stereo: A tutorial. Foundations and Trends in Computer Graphics and Vision, 9(1-2):1-148, 2015.
Silvano Galliani, Katrin Lasinger, and Konrad Schindler. Massively parallel multiview stereopsis by surface normal diffusion. In Proceedings of the IEEE International Conference on Computer Vision, pp. 873-881, 2015.
Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai. Feitong Tan, and Ping Tan. Cascade cost vol- ume for high-resolution multi-viww stereo and sterco matching. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2495-2504, 2020.
Chunrui Han, Shiguang Shan, Meina Kan, Shuzhe Wu, and Xilin Chen. Face recognition with contrastive convolution. In Proceedings the European Conference on Computer Vision (ECCV), pp. 118-134, 2018.
Junjun He, Zhongying Deng, and Qiao. Dynamic multi-scale filters for semantic segmentation. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pp. 3562-3572, 2019.
Tak- Wai Hui, Chen Change Loy, and Xiaoou Tang. Depth map super-resolution by deep multi-scale guidance. In European conference computer vision, pp. 353-369. Springer, 2016.
10

---

Published as a conference paper at ICLR 2022
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizatioo with gumbel-softuax. In 5th International Conference on Learning Representations. ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. httptttss///pnnnie net/fo d
Mengqi Ji, Juergen Gall, Haitian Zheng, Yebin Liu, and Lu Fang. Surfacenet: An end-to-end 3d neural network for multiview stereopsis. Proceedings of the IEEE International Conference on Computer Vision, pp. 2307-2315, 2017.
Xu Jia, Bert De Brabandere, Tinne Tuytelaars. and Luc V Gool. Dynamic filter networks. Advances in neural information processing systems, 29:667-675, 2016.
Abhishek Kar, Christian Häne, and Jitendra Malik. Learning multi-view stereo machine. arXiv preprint 050.00375, 2017.
Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking scage scene reconstruction. ACM Transactions on Graphics (ToG), 36(4):1-13, 2017.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference com- puter vision and pattern recognition, pp. 2117-2125, 2017.
Tony Lindeberg. Scale-space theory: A basic tool for analyzing structures at different scales. Journal of applied statistics, 21(1-2):225-270, 1994.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pp. 21-37. Springer, 2016.
Keyang Luo, Tao Guan, Lili Ju. Haipeng Huang, and Yawei Luo. P-mysnet: Learning patch-wizd matching confidence aggregation for multi-view stereo. In Proceedings of the IEEE/CVF Inter- national Conference Computer Vision, pp. 10452-10461, 2019.
Keyang Luo, Tao Guan, Lili Ju, Yuesong Wang, Zhuo Chen, and Yawei Luo. Attention- multi-view stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1590-1599, 2020.
Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Johannes L Schonberger and Man-Michael Frahm. Structure- cenettre-from-moote revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4104-4113, 2016.
Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Mare Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision, pp. 501-518. Springer, 2016.
Christian Sormann, Patrick Knöbelreiter, Andreas Kuhn, Mattia Rossi, Thomas Pock, and Friedrich Fraundorfer. Bp-mvsnet: Belief-propagaatio-layae for multi-view-sterro. In 2020 International Conference on 31) Vision (3DV), pp. 394 403. IEEF, 2020.
Fangjinhua Wang, Silvano Galliani, Christoph Vogel, Pablo Speciale, and Mare Pollefeys. Patch- matchnet: Learned milti-view patchmatch stereo. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14194-14203, 2021.
Jialin Wu, Dai Li, Yu Yang, Chandrajit Bajaj, and Xiangyang Ji. Dynamic filtering with large sampling field for convnets. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 185-200, 2018.
Qingshan Xu and Wenbing Tao. Multi-scale geometric consistency guided multi-view stereo. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5483- 2019.
11

---

Published as a conference paper at ICLR 2022
Qingshan Xu and Wenbing Tao. Learning inverse depth regression for multi-view stereo with corre- lation cost volume. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 12508-12515, 2020a.
Qingshan Xu and Wenbing Tao. Planar prior assisted patchmatch multi-view stereo. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 12516-12523, 2020b.
Qingshan Xu and Wenbing Tao. Pysnet: Pixelwise visibility-awann multi-view stereo network. arXiv preprint arXiv:2007.00) 007 14, 2020c.
Zhenyu Xu, Yiguang Liu, Xuelei Shi, Ying Wang, and Yunan Zheng. Marmvs: Matching ambiguity reduced multiple view stereo for efficient large scale scene reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5981-59999 2020.
Youze Xue, Jiansheng Chen, Weitao Wan, Yiqing Huang, Cheng Yu, Tianpeng Li, and Jiayu Bao. Myserf: Learning multi-view stereo with conditional random fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4312-4321, 2019.
Jiayu Yang, Wei Mao, Jose M Alvarez, and Miaomiao Liu. Cost volume pyramid based depth the IEEE/CVF Conference on Computer Vision
inference for multi-view stereo. In Proceedings the and Pattern Recognition, pp. 3886, 2020.
Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstruc- tured multi-view stereo. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 767-783, 2018.
Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1790-1799, 2020.
Hongwei Yi, Zizhuang Wei, Mingyu Ding, Runze Zhang, Yisong Chen, Guoping Wang. and Yu- Wing Tai. Pyramid multi-view stereo net with self-adaptive view aggregation. In European Conference on Computer Vision, pp. 766-782. Springer, 2020.
Zehao Yu and Shenghua Gao. Fast-mvsnet: Sparse-to-dense multi-view stereo with learned propa- gation and gauss-newton refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1949-1958, 2020.
Jingyang Zhang, Yao Yao, Shiwei Li, Zixin Luo, and Tian Fang. Visibility-aware multi-view stereo network. British Machine Vision Conference (BMVC), 2020.
Rui Zhang, Sheng Tang, Yongdong Zhang, Jintao Li, and Shuicheng Yan. Scale-adaptive convo- lutions for scene parsing. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2031-2039, 2017.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2881-2890, 2017.
Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9308-9316, 2019.
A APPENDIX
A.1 DERIVED FORMULAS OF NORMAL CURVATURE
First, re-derive Eq. proposed by Xu et al. 2020). The image surface can be represented by s(r, y) (x. v, [1x, v)),
12

---

Published as a conference paper at ICLR 2022
where X [z,y] y] is pixel coordinate and [(x, y) is the color information of image. The unit normal vector surface at point X is determined as
8x 8y Se syll
Ix, 1y 1/,
12 19
where 8R and 8y are the first partial derviatives with respect to x-axis and y-axis, A the cross- product operator. Then, the first and second fundamental form of surface point X can be written by
82
8y.8y Sy.8y.
1 12 1gly Ig
Fy
8x.8y
1 1
y.
Sgn-7l
Suy.
Ixx
Ley Lyy
Fir
Sxy-1
+ 13 + 13
Lzg
Note that "." is the dot-product operation between two vectors. Finally, the normal curvature of at point X along the direction w = [u, u]? of epipolar line is defined as
Tuz
xy Ty
w
FM1
1
1xy
curvo (X,w)=
FM,
1
+12
+ T2
I
+ I W
I IaTy
1
u2 I + 2uv 2 Iy
+ 12 +12 22 + 12) + 2uv T1 + x²
1
12 + 2wv y v² Lys
1 + 12 +13 u² + + (ul + v1,)²
Because w = [u, u]"" is unit vector, 2 + 122 1. Therefore,
u² I, + 2uvlay by + v² [y
curv, (X,w)
12 + 1 (1t (ulz + vly)?
This formula can be applied to an image patch p with scale a centered at X. Following the scale space theory (Lindeberg 1994,. the patch p be regarded represented at the scale σ. Let I(X,o) be the image intensity at the scale a. I(X,or can be computed from / ((X) by using Gaussian kernel with size a, /(X, = I(X) G(X,o)
pixel when the image I
Therefore, we can finally write the normal curvature computed for X at the scale a as
I (X, .) + IuvIay ox (X,
curv, (X,w)=
{ (ul,(x,))) uI,
which Eq. 2 mentioned in Section 3.2
A.2 DETAIL OF MVS ARCHITECTURE
Fig. depicts the multi-stage architecture of CDS-MVSNet. First, CDS-MVSNet uses the feature extraction CDSFNet to extract outputs at three resolutions { 1 } corresponding to the first three stages {0, 1,2). For each of these stages 1 {0, 1,2). CDS-MVSNet predicts the depth at the corresponding image-resolution उ
H W - by using three-step MVS described in section 4.1 CDS-MVSNet also uses an independent 3D CNN for each stage to regularize 3D cost volume. The estimated depth of current stage is upsampled and used to reduce the depth hypothesis range of 3D cost volume the next stage. For the last stage 3, DDS-MVSNet outputs the full-resolution depths by performing depth refinement from the upsampled depth and the color image.
13

---

Published as a conference paper at ICLR 2022
lo- 0dl
stage
W
Formalation
W
volumse formudation
CDSFNel
CDC
g
slage
II
W
valume formulation
2
10
depth regression
stage (refinement)
Ref. image fo
X
CNN
D
Figure 4: CDS-MVSNet architecture. For the first three stages E {0, 2), the estimated depth of current stage is used to reduce the depth hypothesis range of cost volume in the stage. For the last stage = 3, we perform depth refinement from the estimated depth previous stage and color information
CDSFNet Given reference image Io and N source images {I, I2....... IN) with correspond- ing camera parameters (Qo...., QN we can compute an epipole pair {(Co, e for each reference-source image pair ((To, T,)] based on the epipolar geometry. We use the epipole to de- termine the epipolar line for each pixel. Therefore given the inputs including image 7 and epipole e, CDSFNet extracts three feature maps corresponding to the three levels (0, 1, 2} of CDS-MVSNet. Fig. 5 describes the Unet-liked architecture of CDSFNet. Fig. [nd 7 the results of dynamic scale estimation and normal curvature estimation respectively.
ref. image
spipale e
half Reature p())
full-nes feature (89)
Seanure
NEC mmage
CDSCaav + IistaaceNouzz - LLligReeUU
stoddrd consotaton, is stiee
CDSCuu RestuarNouzz Tush
Up-sxmpling
coocatoration
Figure 5: The Unet-liked architecture of CDSFNet. organized into three levels: quater-resolution, haft- resolution, and full-resolution
Depth Refinement. Instead of using three-step MVS depth estimation as described for the finest resolution level (stage 3), we directly up-sample the depth estimated in previous stage (from resolu- H to W x H) and refine the upsampled depth with the RGB image. Following MSG-Net (Hui al. 2016) and PatchMatchNet (Wang et al.) (021). implement depth residual network to perform refinement. The network uses 2D CNN to output residual from the upsampled depth. TTii residual depth is then added to the up-sampled depth, get the refined depth map Dref
tion 2 x
W
14

---

Published as a conference paper at ICLR 2022
ref.
Figure 6: Illustrations dynamic scale estimation according epipolar constraint between reference and source views. Consider a pixel of reference view marked red color, its epipolar line, marked in yellow, varies with each source view. Therefore, the normal curvatures estimated at that pixel are different, this leads to the different estimated scale the reference view when it is matched to source view.
0.20
ref view
0.16
0.12
0.08
SPC. Views
0.04
0.00
Figure 7: Visualization of the normal curvature maps estimated from our feature extraction CDSFNet by Eq. For each pixel X, the normal curvature computed X is large when X belongs rich texture region. In this case, estimated scale for X small shown Fig. 6ooo preserve the local detail the scenes. Otherwise, the curvature is small when X belongs to an untextured region; therefore, a large scale chosen to increase the completeness. These figures demonstrate that the normal curvature can provide guidance to choose the optimal scale to improve the matching ability.
7
A.3 IMPLEMENTATION DETAILS
Training. We applied the same image-resolution 640 x 512 in both DTU and BlendedMVSS datasets. The preprocessing of these datasets for view selection strategy and ground-truth depth maps was provided by Yao et al.] (2018). For CDS-MVSNet, we set the number of input views to 3, the number of depth hypothesis planes to {48, 32, 8} with the corresponding depth interval scales (4, 2, 1}. To train CDFSNet effectively, we used the feature loss as mentioned in Section 4.2 Besides, implemented the scale selection step CDSConv by first initializing the Softmax temperature by then decreased over training time until it reached 0.01 to produce the
1. This hyperparameter sparse output for scale selection. The entire network CDS-MVSNet was trained with 30 epochs, using the SGD optimizer with initial learning rate of 0.01, and two NVIDIA Titan V GPU with a batch size of 6. We provided two experimental scenarios. First, we trained the model on DTU training set and then evaluated on DTU test set. This pre-trained model is then fine-tuned on BlendedMVS dataset for 15 epochs with a learning rate of 10-4 to evaluate on Tanks & Temples. Second, we trained a single model on both DTU and BlendedMVS datasets and use this model for evaluation. This training strategy can improve the performance as indicated in Table I and
Point cloud generation. To reconstruct the final point cloud, we filtered out the low confidence depths based on the probability maps estimated from the depth probability volumes of all stages (Zhang al. 2220)) Similar to (Guu et al. [2020] [2020]Zhang et al. 2020). we then integrated all the depth maps based on the fusion method proposed in (Galliani al.
2015 Yao et al. 2018).
A.4 ABLATION STUDY
Effectiveness of learnable curvature. We demonstrate the advantage of our proposed learnable curvature in comparison to the original curvature (Xu et al. 2020). We replace the curvature estima- tion 5 with Eq and then re-train the MVS network on the DTU dataset with identical setups. Table 4 presents four model variants denoted A, B, C, and D. Each variant implements a specific version of curvature (original or learnable) and then uses the visibility aggregation with or without the curvature prior. We consider comparison of the two groups, models A and B versus models C
15

---

Published as a conference paper at ICLR 2022
Imagee
Icarnable original
sic. image
layer, 1152 2nd laver, 1536 1152
3rd layer,
1/h layer,
Figure 8: A qualitative comparison on DTU evaluation set (Aanæss et al. 2016) between the learnallc and original curvature. We visualize the curvature maps at the scale a = 3 (kernel size is 3) extracted from the first four layers of CDSFNet. The first and second layers output the curvature maps the full resolution, the third and fourth layers extract the maps the half-resolution. The red circles denote the effect of attention mechanism, which is benefited from the learnable kernels as presented Section
Table 4: Comparison between our learnable curvature and the original curvature Xu et al. are trained on subset of DTU training set, using the same setups and hyperparameters. The evaluation of depth map and pointcloud is collected DTU validation and test set, respectively.
2020 All models
Design Options
Depth Map
Pointcloud
Time & Memory
Methods
Prec.
Prec.
MAE
Acc.
Comp. (mm)
Overall (mm)
Time
Mem
curv. type
curv.
2mm
4mm
(mm)
(mm)
(6)
(Mb)
Model A
original
74.26
84.89
6.84
0.378
0.315
0.347
0.539
7993
Model
original
74.12
84.77
5.99
0.391
0.327
0.359
0.539
7993
Model
learnable
86.08
6.30
0.302
0.338
0.421
4389
Model D
learnable
76.23
85.92
5.89
0.372
0.305
0.339
0.421
4389
and To evaluate the depth maps, we measure the percentage depth precision with the thresholds {2, 4 Jmmm and Mean Absolute Error (MAE). To evaluate the reconstructed models, we use the ac- curacy, completeness, and overall metries. We also provide the runtime and memory consumption to evaluate the computational efficiency.
Table indicates that the models with learnable curvature (C and D) achieve a better performance than the original curvature (A and B) for all evaluation metrics. The learnable curvature has low computation cost and boosts the quality of depth and reconstruction significantly. The original curvature degrades the performance because it cannot handle the high-dimensional features. The original method treats each feature channel equally and averages all feature channels when comput- ing the derivatives by Gaussian filters. Therefore, it may produce a high curvature estimation though the neighboring feature vectors are visually similar to each other, as shown in Fig.
Fig. shows the normal curvatures estimated by the original method (Xu et al. 2020 and our learn- able method. The curvature maps are extracted from the first four CDSConv layers of CDSFNet. As shown in Fig, [8] the original method provides a ground truth estimation only in the first layer when the input is a color image (i.e., the estimated curvatures are small in untextured regions while large in near-edge or rich-textured regions). However, it produces noisy estimation when the inputs are the feature maps in the subsequent layers, especially in the third and fourth layers where the curvatures do not reflect the edge or texture information. On the other hand, although the learnable curvature approximates the ground truth (the original curvature in the first layer), it consistently preserves the properties of normal curvature concerning the untextured and rich-textured regions in every layer. We further notice that our lcarning- based estimation takes advantage of the attention mechanism, as shown in red circles. focuses on the rich texture region to improve the completeness of reconstruc- tion, which is illustrated in Fig. in Appendix A.5 Fig. g also implies that the estimated curvature tends to be higher at the deeper layers because it encodes à larger patch scale.
Curvature-guider visibility aggregation. We proposed pixel-wise visibility prediction method for cost aggregation Section 4.1 Previous works generally predicted the visibility weights only from the two-view cost volume (Zhang al. 2020 Xu & Tao. 2020c). Compared to these works,
16

---

Published as a conference paper at ICLR 2022
Table 5: Impact of the number of candidate scales in CD- SConv layer. The models are trained on DTU training set at the resolution 320 x 256, without 3D cost regulariza- tion. The statistics below are collected on DTU valida- tion set.
Table The accuracy of estimated depth different stages our CDS-MVSNet. The entire MVS network is trained on DTU training set. The trained model used to evaluate on the DTU test set at the resolution 1536 x 1152
#scales/layer (N,)
Prec.
Prec.
Prec.
MAE
Time
Stages
Prec.
Prec.
Prec.
MAE
4mm
(mm)
2mm
4mm
(mm)
(CDSFNet FPN)
40.87
55.95
66.72
21.41
0.114
39,99
76.35
11.43
58.10
67.89
18.89
58.01
72.92
80.21
10,80
3
45.27
59.14
17.46
0.155
2
63.97
75.43
80.92
10.67
46.77
60.63
70.61
16.10
0.195
3
64.26
75.52
80.94
10.62
our method additionally considers the curvature prior estimated by CDSFNet. To evaluate the effec- tiveness of curvature prior to visibility prediction, we compared the performance of our method with the baseline method [Zhang et al. (2020). To implement the baseline method, we removed the curva- ture input from our visibility prediction network and only used the entropy information two view cost volumes. We compared the models using curvature prior (Models B and D) with those without using it (Models A and C). Table 4 Shows the experimental results. As shown in Table Model B had the worst performance in terms of overall. Because Model B produced the noisy curvature, this noisy estimation degraded the performance of Model B compared to Model A. Model D remarkably improves the depth estimation performances remarkably and achieves the similar performance of reconstruction compared to Model C without increasing runtime and memory consumption. These results demonstrate the effectiveness of the proposed learnable curvature and curvature-guided visi- bility prediction in the MVS process.
Analysis of CDSFNet and the cascade structure. This section first analyzes the effectiveness of proposed CDSFNet when increasing the number of candidate scales. To measure explicitly the matchability of features extracted by CDSFNet, we remove the cost-volume regularization step in the MVS architecture and then regress the depth directly from the aggregated 3D cost volume. Table [5] presents the results when increasing the number candidate scales N₂ in each CDSConv layer. When Ne is equal to 1, CDSFNet is considered Feature Pyramid Network (FPN), which is applied to most of the state-of-the- art learning-base methods (Gu et al.) 2020; Zhang et al.) 2020). As shown in Table 5, CDSFNet achieved better performance when using
candidate
scales. However, suffered from heavy computation. Therefore, we chose the design with two or three scales for each CDSConv (CDSFNet architecture shown in Fig. 5). to guarantee both the performance of reconstruction and computational efficiency.
Next, investigate the improvement of depth estimation the cascade stages. Table provides the accuracy of depth estimation over all stages of our CDS-MVSNet. The first three stages mainly contribute the final quality of estimated depth. The last stage it outputs the full-resolution depth while maintaining the same depth accuracy compared to the previous stage. Suppose this stage is implemented by the three-step step MVS depth estimation pipeline mentioned in Section [. the entire MVS network suffers from a considerable computation and cannot into limited GPU memory. Therefore, we only apply the depth refinement with 2D CNN to the last stage for efficiency in time and memory.
is used for depth upsampling:
17

---

Published as a conference paper at ICLR 2022
A.5
QUALITATIVE RESULTS ON DTU AND TANKS& & TEMPLES
LEA
SER 2919
TUBORG
280
BUBORG
ORG
OIRORG
TIBORG
NTK
DTM
ETK
STM
TUB
TUB
TUB
TUB
TUB C
Ground truth
R-MVSNet
CVP-MVSNN
CacMVVNel
CDS MVSNet (Ours)
Figure 9: Qualitative results on DTU evaluation dataset. Our method achieved the high completeness of recon- structed models. The visualization of reconstructed models for DTU evaluation set are shown in the top figure. A comparison with the other state-of-the-art methods is shown in the bottom figure.
18

---

Published as a conference paper at ICLR 2022
27
3r
AIMVS
LEAN TEAVAN YAN CEOO
75.57
37.31
Vis-VVNN
86.70
54.15
CDS-MV SSNee
Francis
Horse
Panther
Figure 10: Qualitative results on the intermediate set of Tanks & Temples. Our method achieved the high The bottom figure shows the error visualization
completeness of reconstruction for complex outdoor for Recall scores compared to AuMVS (Luo et al. 2020) and Vis-MVSNet (Zhang 20200
19


Reviews:
[{"review_id": "wnq-iRfkEZ8", "reviewer": "Reviewer_aAob", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is basically well written and has several strengths.\n+ Using dynamic scale convolution for learning-based multi-view stereo is new.\n+ Using the curvature of the target scene for determining kernel scale would be adequate.\n+ Well experimented (but only with learning-based methods)\n\n\nMy main concern for this paper is about the main contribution.\nUsing curvature information to select the patch scale is used in an MVS method (Xu et al. (2020)). \nAlthough the method does not use deep learning, it limits the technical contribution of this paper.\n\nIn the related work section, this paper claims that the proposed method should have an advantage of performance compared with Xu et al. (2020) because they do not rely on the handcrafting features.\nHowever, the claim was not confirmed through the experiments.\n\n*** After discussion phase ***\nDuring the discussion phase, the authors added a comparison of the proposed method with Xu et al. (2020).\nFrom the comparison, indeed the proposed approach shows advantages for both accuracy and computation time.\n\nTherefore, compared with the initial rating, I changed my rating to a positive one.\nA dynamic-scale method using curvature has been proposed so far, somehow limiting the novelty of this work.\nMeanwhile, developing a learning-based approach would be beneficial.", "summary_of_the_paper": "This paper proposes a method for deep-learning-based multi-view stereo. The contribution of the paper would be to adjust the scale of convolutional kernels according to the curvature of the target scene. Experiments show the proposed method (slightly) outperforms the existing deep-learning-based multi-view stereo methods. ", "summary_of_the_review": "*** For the initial review***\nI am concerned about the effectiveness of the main claim of this paper because it is not confirmed through the experiment, i.e., it is not compared with the most important existing method. \n\n***As for the final rating,***\nI acknowledged the comparison with the most important method and confirmed the advantage of the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "h6Wfb0jT9kB", "reviewer": "Reviewer_6P6T", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is generally well written and the main novelty of the curvature estimation is explained in a high level allowing an inexpert to follow. The authors properly cite previous works when necessary (except Do Carmo which is not the earliest use of curvature AFAIK)\n\nThis paper pushes the envelope and seem to provide better results in using less compute, which (IMHO) brings it over the conference's bar, but not by much.\n\nThe main area where I believe this paper can be improved is by providing deeper analysis of the novel parts. \nSpecifically:\n- Replacing Gxx with learned Kxx seems arbitrary: first, using G is quite tractable (only 3 derivatives). second, what promise do we have that all the K's are directional like the derivatives? e.g. Kxx can have some y meaning - lastly, this can easily be tested and showed in an experiment (i.e. show performance with G and then learned K)\n- Is the physical curvature/scale actually estimated? can you compare it to the one from the GT?\n- Is the curvature/scale consistent across the cascade? If not, is it getting better/worse compared to GT?\n- Are the CDSConv scale features (C1, ... Ck) constrained to be scaled versions of each other? otherwise it's not really a scale estimation, no?\n- The use of 2D CNN in the last layer seems to be for performance reasons (not clear from the text), but a comparison to another stage of 3D CNN should provide better results - why not show the impact here?\n\n\nTo summarize, most of my concerns can be addressed with either ad deeper explanation, or preferably an ablation (of using visibility-aggregation  K vs G, removing the last 2D CNN, whether to add the epipolar constraint)\n\n\nsmaller notes:\n- \"handcrafting\" --> \"handcrafted\"\n- Ixx, Iyy, Ixy in eq. 1 are used before definition\n- the G<<1 assumption should be better justified\n- regarding eq2 - first, why is it intractable? second, there are only 4 convolutions, not 5 (Ixy==Iyx)\n- Is the visibility-based aggregation in CDS-MVSNet novel or was it done in prior works? not clear from the text\n- are weights shared between cascade scales? (are stages 0,1,2 of Fig 4 identical weights)\n", "summary_of_the_paper": "This paper adds upon the recent line of MVSnet works, (specifically CasMVSNet) by explicitly treating the scale needed to reconstruct each part of the scene. This is done by estimating the local curvature of the surface prior to the depth, which is then used to select (in an attention sense) a kernel with an appropriate scale from a set of scaled features in the CDSConv. The curvature estimations is further constrained \nto the epipolar line of the relevant source image. The image is processed in a coarse to fine manner similar to CasMVSNet, except the finest resolution which is solved in a (naive?) 2D manner. ", "summary_of_the_review": "The paper is novel, well written, and provides nice SOTA results. \nMore justification of the design choices can turn it from weak to strong accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Qtthrfs0qEN", "reviewer": "Reviewer_68iw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*** Strengths of the paper:\n\n(+) Conceptual novelty.\n\nWhile the proposed contributions do partially exist in the literature, their combination to address the problem of MVS per se is novel and interesting. This in turn leads to several interesting experimental discussions that are teased early-on in the paper (but lack in their execution, cf. weaknesses below for the missing expected pieces in the evaluation section in particular).\n\n(+) Reproducibility.\n\nThe amount of disclosed information regarding the method is fair, and makes uses of the supplementary material by providing comprehensive details regarding the curvature-aware formulation and its differentiable approximation, as well as the code of the proposed method that has also been submitted.\n\n(+) Related Work.\n\nThe section (2) is dense, well populated and structured and discusses the main key prior work in the field. While there could have been a broader discussion about the literature overlaps w.r.t aspects of the proposed contributions in difference applicative scopes (eg, the use of 2D curvatures in U-Net based networks), the section is already fairly long (1.2 page long) and serves its purpose relatively well.\n\n(+) Structure, Balance of the Contents.\n\nOverall, the paper is relatively well structured, each section is of pertinent relative size and the text real-estate is well utilized and efficient. To some extent, more qualitative results could make their way to the main paper, instead of predominantly relying on the supplementary material in this regard.\n\n\n*** Weaknesses of the paper.\n\n- (1) The fairness of the experimental setup is in question.\n\n(a) On the DTU dataset (page 8), as explicitly stated by the authors the proposed method is trained on the union of DTU and Blended-MVS datasets (training splits):\n\nAppendix, A3 page 15: \n\"Note that we trained the single model using both DTU and Blended datasets\ninstead of training individually on each dataset as in previous methods (Zhang et al., 2020; Luo\net al., 2020; Sormann et al., 2020). This procedure prevented the model from biasing to each dataset\nin the evaluation step; it guaranteed the generalized property of model.\"\n\nThis is a major issue because the reported performance figures of the competition in Table 1 are being trained on the conventional train split of DTU, without additions.\n\n(b) Appendix, A3 page 15:  It is factually not true that (Zhang et al., 2020; Luoet al., 2020; Sormann et al., 2020) perform training on DTU+Blended-MVS to test on DTU:\n\n(i) Zhang et al., 2020: \nThe experiments on DTU and Blended-MVS are performed by training on the respective conventional training splits, NOT on both training sets simultaneously. This is explicitly stated by the authors Page 7, sec 4.1 of their main paper.\n\n(ii) Luo et al., 2020: \nTheir model is trained and tested solely on DTU, using the conventional splits (sec 5.2 in their main paper).\n\n(iii) Sormann et al., 2020: \nExperiments on DTU: the training is performed solely on the training split of the dataset.\nFor Tanks and Temples and ETH3D (which is not considered in the proposed paper), the authors additional fine-tune on Blended-MVS on top of the conventional DTU training. This can be found in their main paper, sec 5.1.2 and sec 4 at length.\n\n(c) Regarding the provided experiments on Tanks and Temples (ToG 2017), Table 3:\n\nIt is a common practice in the field to train on either DTU  with or without fine-tuning on Blended-MVS (eg, resp. BP-MVS, AttMVS), not by training on the union of both dataset as it is done for the proposed method (Appendix A3).\n\nIn particular, given that AttMVS is the closest-performing competition w.r.t the proposed method, why not provide experiments with the same training setup / data?\n\nThis was an easy way to directly address the relative positioning against the most competitive player in the field. Chosing a different setup can be interpreted as a means to hide potential red flags.\n\n- (2) Certain claims and justifications are either ambiguous, inaccurate or even misleading.\n\nThis is the case in particular regarding the evaluation part above, and the pieces of justification that are specified in the appendix. This agin, is a relatively major concern as is.\n\n- (3) No ablation studies are provided.\n\nGiven the sequential pipeline nature of the proposed contributions, ie, feature extraction, approximate curvature priors, 3D cost volume aggregation and regularization, it is highly expected to provide a comprehensive ablation study (ie, activating/removing pieces of the pipeline) to help understand and assess the relative contribution of each individual piece / ingredient, in particular given:\n\n(a) It is a common practice in the field and within the realm for this particular literature (eg, BP-MVS, AttMVS, or even at BMVC which is a venue which much less text real-estate: Zhang et al. 2020).\n\n(b) This is all-the-more problematic given (1) above, that puts an even greater weight on the ablation analysis and the lack thereof.\n\n(c) Additionally, certain design choices and hyperparameters are not experimentally challenged to support their respective values are (at least near-) optimal, eg, \\lambda_1 and \\lambda_2 in sec 4.2, page 13: the number of different scales / number of stages that are typically considered.\n\n- (4) Missing prior work in the comparative evaluation.\n\nThis is relatively important given the proximity, performance-wise, w.r.t the proposed method, eg, (Sormann et al. 2020) BP-MVS (overall perf: 0.327 vs. 0.315).\n\n- (5) Relative positioning, performance-wise, is unclear.\n\nThis is a direct implication stemming from (1)+(3) above. The reported performance figures are hence uninformative and do not allow to properly assess the relative positioning of the proposed approach w.r.t existing work, and hence, the validate its usefulness.\n\n- (6) Readability.\n\nTo a lesser extent, the text as it currently stands, is still in a relatively rough, tough to digest state. There are numerous grammar issues and typos (a few example below). However and overall, the main ideas and key claims do not suffer substantially from the language-related issues and the articulations of the main ideas is often preserved.\n\nA few examples with suggestions for improvements:\n- abstract l3: by introducing a skilled design to cost formulation -> by designing aggregated 3D cost volumes and their regularization.\n- abstract: our final MVS architecture -> our resulting MVS architecture.\n- abstract: Moreover -> As a result,\n- abstract: can process the high resolution with faster run-time -> can process higher resolution inputs within faster runtimes\n- introduction, first paragraph: an object's scale varies extremely in images -> in presence of high/significant scale variations of objects in images\n- throughout: dynamic scale networks -> scale-adaptive networks", "summary_of_the_paper": "The paper proposes a novel Deep Neural Net to address the problem of Multi-View Stereo (MVS) by introducing several claimed novel ingredients, ie, (i) CDFSNet: a scale-adaptive feature extraction network that takes into account (ii) an approximation of image-based 2D curvatures in a U-Net based network;\n(iii) CDS-MVSNet: is the actual reconstruction network that consider multiple resolutions and input feature scales into an aggregated, regularized multi-view cost-volume.\n\nThe end-to-end sequential pipeline forming the proposed contribution is evaluated on public, popular standard benchmarks in the field, showing promising results, despite the fairness of the experimental setup being in question (as detailed below in the review).", "summary_of_the_review": "*** Summary and justification of the initial rating.\n\nOverall, the paper presents very interesting and relatively new ingredients to address the problem of Multi-View Stereo using supervized learning. The presentation of the contents, despite a few substantial issues, also lies in the list of positives. \n\nHowever, as it currently stands, the proposed package suffers from major drawbacks in the evaluation department, as detailed above in the main review. In particular:\n\n(i) The proposed experimental comparisons are performed in a fair setup, for the most part and regarding the most important dataset / set of experiments.\n\n(ii) As an extension to (i), there are key claims, references and justifications that are inaccurate to say the least, that could easily mislead the reader.\n\n(iii) Given the nature of the proposed contributions forming a sequential pipeline, (ie: feature extraction, approximate curvature priors, 3D cost volume aggregation and regularization) it is highly expected to provide a comprehensive ablative study to analyze, motivate and qualtify the need of each individual ingredient. Such experiments have not been proposed.\n\nAs a result, the proposed package is very interesting and the results promising, but the execution of the evaluation part do not allow to validate the main claims and novelties so far. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EnFodgZjyso", "reviewer": "Reviewer_Qeua", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The rationale behind finding the right scale to disambiguate matching is well written and quite reasonable. It would be good to pinpoint though, the exact example that suffers from the matching ambiguity issue, i.e. highlight some pixels in the image pairs about the wrong feature association.\n- The quantitative results show the method is competitive. \n- Run time is also lower when image resolution increases, compared with CasMVSNet and others. \n\nWeakness:\n- The writing can be improved. For example, the learnable normal curvature part is a bit hard to read.\n- It’d be good to add some ablation studies to show the delta brought by CDSConv and the MVS architectural changes compared with the baseline network, i.e. CasMVSNet.\n\n\nFeedback:\n- For the Tanks&Temple dataset, did you also evaluate the method using high-res images on the advanced cases? I found that the method ranks high on the intermediate cases but not on the advanced cases.\n- Is the major deduction of running time a consequence of only computing depths of half-resolution images and then upsample to the original resolution?\n- Add legend to the left image of Figure 3. Why is the GPU memory significantly lower when resolution is increased? It’s even lower compared with lower-res images, for ‘ours’.\n\n\n", "summary_of_the_paper": "This paper proposes a method named CDSFNet for the multi-view stereo problem. The network is composed of curvature-guided dynamic scale convolution (CDSConv) layers to first estimate approximate normal curvature at different candidate scales, and then do a classification to find the optimal scale. The module is then embedded with cascade structure of CasMVSNet, to reduce the matching ambiguity by considering the proper pixel scales. The method is evaluated on DTU, BlendedMVS and Tanks & Temples datasets, which archives state-of-the-art performance compared with other related methods.\n", "summary_of_the_review": "I recommend Accept for this paper since the strengths outweigh the weaknesses. The paper is generally written well with good performance. One thing that can be improved as I mentioned above is more ablation experiments.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers recommended accept after discussion. I am happy to accept this paper.
Decision: Accept (Poster)