GenS: Generalizable Neural Surface Reconstruction from Multi-View Images
Authors: Rui Peng
Xiaodong Gu
Luyang Tang
Shihe Shen
Fanqi Yu
Ronggang Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Generalizable Neural Surface
Volume Rendering
Signed Distance Function

Abstract: Combining the signed distance function (SDF) and differentiable volume rendering has emerged as a powerful paradigm for surface reconstruction from multi-view images without 3D supervision. However, current methods are impeded by requiring long-time per-scene optimizations and cannot generalize to new scenes. In this paper, we present GenS, an end-to-end generalizable neural surface reconstruction model. Unlike coordinate-based methods that train a separate network for each scene, we construct a generalized multi-scale volume to directly encode all scenes. Compared with existing solutions, our representation is more powerful, which can recover high-frequency details while maintaining global smoothness. Meanwhile, we introduce a multi-scale feature-metric consistency to impose the multi-view consistency in a more discriminative multi-scale feature space, which is robust to the failures of the photometric consistency. And the learnable feature can be self-enhanced to continuously improve the matching accuracy and mitigate aggregation ambiguity. Furthermore, we design a view contrast loss to force the model to be robust to those regions covered by few viewpoints through distilling the geometric prior from dense input to sparse input. Extensive experiments on popular benchmarks show that our model can generalize well to new scenes and outperform existing state-of-the-art methods even those employing ground-truth depth supervision. Code will be available at https://github.com/prstrive/GenS.

GenS: Generalizable Neural Surface Reconstruction from Multi-Viewy Images
Rui Pengl. Xiaodong Gu Luyang Tang' Shihe Shen' Fanqi Yu' Ronggang Wang" 2412 School of Electronic and Computer Engineering, Peking University
Peng Cheng Laboratory AAlibaba Group
ruipengestu, ppk..edu.co rgwanglpkusz.edd edw.aaszzzdd...o cn
Abstract
Combining the signed distance function (SDF) and differentiable volume rendering has emerged powerful paradigm for surface reconstruction from multi-view images without 3D supervision. However, current methods are impeded by requir- ing long-time per-scene optimizations and cannot generalize to new scenes. In this paper, we present GenS, an end-to-end generalizable neural surface reconstruction model. Unlike coordinate-baid based methods that train separate network for each scene, we construct generalized multi-scale volume to directly encode all scenes. Compared with existing solutions, our representation more powerful, which can recover high-freew freuufrreuencyy details while maintaining global smoothness. Mcanwhile, we introduce multi-scale feature-metric consistency to impose the multi-view consistency in more discriminative multi-scale feature space, which is robust the failures of the photometric consistency. And the learnable feature can be self-enhancee to continuously improve the matching accuracy and mitigate aggre- gation ambiguity. Furthermore, we design a view contrast loss foree the model to be robust to those regions covered by few viewpoints through distilling the geometric prior from dense input sparse input. Extensive experiments on popular benchmarks show that our model can generalize well to new scenes and outper- form existing state-of-te -ftth-artt methods even those employing ground-trute depth supervision. Code will be available at https: //gi tithub..a. ctw/coom/////e pttpp//rrtiieees..
1 Introduction
Surface reconstruction from multi-view images cornerstone task in computer vision with many applications in virtual reality, autonomous driving, robotics, etc. Typical solutions [15, 16. 7, 43, 58, 56, 10, 36] in past were mostly based on multi-step pipeline, which includes depth estimation, depth fusion and meshing. Although they have demonstrated their excellent performance, the procedure is cumbersome and inevitably introduces cumulative errors. While several early works [31. 61] used differentiable surface rendering to directly reconstruct surfaces, recent works [34, 49, 60], inspired by the huge success neural radiance field (NeRF) [27] synthesizing novel views, follow the volume rendering [24] to represent the 3D geometry with occupancy field [25] or signed distance function (SDF) [35] and can achieve more impressive results.
The key idea of these approaches is to train a compact multi-layer perceptrons (MLPs) to predict the implicit representation (e.g.. SDF) of each sampled point on camera rays. The density volume rendering is then regarded as function of this implicit representation, and alpha-composition of samples performed to produce the corresponding pixel color and geometry information. However, existing methods are hampered by requiring lengthy per-scene optimization procedure and cannot generalize to new scenes, which makes them infeasible for many application scenarios. A recent method [23] attempts to address these issues through conditioning the SDF-induced model with
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Image
MVSNeRF
SpanseNeuS
Ours
COLMAP
SparsNNuSS (It)
Ours (M)
Figure 1: Qualitative comparisons on DTU and BlendedMVS datasets with sparse inputs.
features extracted from sparse nearby views. Nevertheless, its accuracy is limited due to the smooth reconstruction, and the multi-stage process it relies on is prone introducing cumulative errors. In this paper, we seek to establish an end-to-end generalizable model which can efficiently infer finer 3D structure. Compared with existing methods [49, 60], this generalization system faces more challenging problems. First, it's non-trivial to efficiently represent the scene. Previous methods [29, 23, 3, 50] either build global volume employ feature projections, but they have proven be either lacking in detail or unsuitable for view independent surface reconstruction. Second, relying only on the rendering loss is difficult reconstruct compact geometry, since the multi-view consistency is ignored. And we found that the ordinary photometric consistency also cannot effectively solve this problem for our generalizable model because of the existence of ambiguuus areas such as low-texture and reflection. Last but not least, since generalizable models heavily rely on aggregation quality, how infer smooth geometry when the input is sparse thorny issue.
To this end, we introduce GenS to tackle these challenges. The main ideas behind are as follows: 1) We first construct generalized multi-scale volume represent the scene, which preserves global smoothness through the low-resolution volumes and recovers geometric details from high-resolution volumes. Meanwhile, low-dimensional features make model more lightweight compared to a single large-width volume. 2) We introduce the multi-scale feature-metric consistency, which enforces multi-view consistency the multi-scale feature space, replace the common photometric consistency. Compared with the original image space, Icarnable multi-scale features can provide more discriminative representation, and the feature space can be self-enhanced during the generalization training process to continuously improve the matching accuracy. 3) Inspired by the fact that the reconstruction with dense inputs is more accurate, we propose a view contrast loss to force the model to better perceive the geometry of regions visible by few viewpoints through teaching the reconstruction from sparse inputs with dense inputs.
To demonstrate the quantitative and qualitative effectiveness of GenS, we conduct extensive ex- periments DTU [12] and BlendedMVS [59] datasets. Results show that model out- perform existing state-of-the-art generalizable method [23], and even recent method [40] which adopts the ground-truth depth for supervision. Compared with the per-scene overfitting methods [49, 60, 61, 34, 23], we can also achieve comparable or superior results with dense inputs. Some comparisons shown in Fig. 1. In summary, our main contributions are highlighted below: a) We present a powerful representation based on our generalized multi-scale volume, which can efficiently reconstruct smooth and detail surfaces from multi-view images. b) We introduce more discrim- inative multi-scale feature-ment consistency to successfully constrain the geometry, which helps generalization model converge the optimum. e) We propose view contrast loss improve the geometric smoothness and accuracy when the visible viewpoint is limited. d) Our model can be trained end-to-end and achieve state-of-the-art reconstructions both generic setting and per-scene optimization setting.

---

Banding
Bagge FUter
Bay sirlnce
Figure 2: Illustration of GenS. We first extract multi-scale features through FPN network. The generalized multi-scale volume is then reconstructed with the corresponding scale feature. We employ the same blending strategy as [50] to estimate the appearance of each point on a ray, and adopt the volume rendering recover the color of pixel. We design the multi-scale feature-metric consistency to constrain the geometry as shown in the top right. For convenience, we omit some losses that will be detailed later.
2 Related work
Classical multi-view reconstruction. Reconstructing 3D geometry from multi-view images is a longstanding problem in the field of 3D vision. Classical algorithms mainly adopt depth- based voxel-based methodology to solve this problem. Multi-view stereo (MVS) is typical class of depth-based methods, which takes stereo correspondence from multiple images as the main cue to reconstruct depth maps. While previous traditional MVS methods [1, 43, 7, 6. 56, 44] relied on the hand-crafted similarity metrics, many recent learning-based methods [58, 10, 48, 36] apply deep learning achieve more discriminative matching. These methods go through complicated procedures to retrieve surface, including depth filtering, fusion and meshing [15, 2]. and are prone to cumulative errors. On the other hand, voxcl-based methods [45, 18, 11, 32] directly model objects in a volume, but they are restricted to memory, which is the common drawback of the volumetric representation, and cannot achieve high accuracy.
Neural surface. Due to the notable advantages of being able to achieve high spatial resolution, neural implicit functions have recently gained lot of attention and have emerged as an effective representation of 3D geometry [47, 25, 35, 8, 26, 30, 37, 42] and appearance [20, 27, 21, 33, 46, 62, 38, 28]. Furthermore, some works [27, 22, 31] have proposed to train models without 3D supervision via differentiable rendering, e.g., surface rendering and volume rendering. Methods adopt surface rendering [31, 61, 65] only consider a single surface intersection point for each ray and fail to reconstruct complex objects, and they are restricted by the need of accurate object masks and careful weight initialization. On the contrary, recent methods use volume rendering [34, 49, 60, 64) to take multiple points along the ray into consideration and achieve more impressive results. However, either type of method requires an expensive per-scene optimization and cannot generalize to new scenes.
Generalizable neural surface. the field novel view synthesis, some methods [3, 50, 63. 13] have successfully introduced the generalization into rendering methods. These methods suffer from the same problem NeRF: the geometry ambiguous. Few works have focused on the generalization of neural surface reconstruction. A recent study, SparseNeuS [23], is the first attempt to achieve this by reconstructing the surface from nearby viewpoints multi-stage manner. Nevertheless, its reconstruction lacks details, and same to the classical 3D reconstruction, the multi-stage pipeline may accumulates errors at each stage. On the contrary, our designed model can be trained end-to-end and reconstruct smoother and more refined geometries.
3 Method
Given N posed images of an object taken from different viewpoints, our goal to reconstruct the surface as implicit function without expensive per-scene optimization or only by fast fine-tuning. Our overall framework depicted in Fig We first introduce how to infer the geometry and appearance from the generalized multi-scale volume in Sec. 3.1, then elaborate on the necessity and implementation of the multi-scale feature-metric consistency Sec. 3.2, and finally detail the realization of view contrast loss in Sec. 3.3 and the overall pipeline in Sec. 3.4.

---

3.1 Geometry and color reasoning from the generalized multi-scale volume
Compared with existing solution [23], which relies on single volume and multi- stage strategy. we have three main advantages. First of all, our generalized multi-scale volume is a more powerful representation, which implicitly decouples geometry into base structures in low-resolution volumes and high-frequency details in high-resolution volumes. Second, with the low-dimensional features, we can construct multi-scale volumes with higher resolution and less memory consumption than single large-width volume. Besides, our model can be trained end-to-end, avoiding cumulative errors.
Generalized multi-scale volume construction. Suppose there are N images V-1 of an object, N-1,1-1 for all images with shared weights. and different volumes are then constructed from features at corresponding scales, In this paper, we define a bounding box of interest in the reference frustum like [23] and in the world coordinate system like [49, 66] when dense inputs are available. We adopt combination of volumes (Vj) which cover the same region but with different resolutions Ch D.
we first apply the FPN network [19] to extract multi-scale feature maps {Fi 1i.j=0,0
2 2 2/
Here, we discuss at the first scale and omit the scale subscript j for convenience. Given camera intrin- N-1 and extrinsics {[R, 1
sics { K; h==0
N-1 we first project the voxel e = (æ, y, 2) onto viewpoint i's
pixel position:
(v) - (K, 44)),
where π ((2, y. (()) (= ") an operator to convert homogeneous coordinates to cartesian coordinates. Then we can get the corresponding feature of each voxel on i. viewpoint through N- bilinear interpolation (v) = F. (v) >. To fuse features from all viewpoints {u) we adopt the same aggregation strategies to generate cost volume as in [50] that concatenates mean and variance to simultaneously capture statistical and semantic information: B(e) = eMann Var(v)].
get the multi-scale cost volumes {B L-1 Next, we further design efficient multi-scale 3D network & to refine these cost volumes in one forward, starting from the finest volume and injecting the others into different stages the model to save memory. The output of the 3D network / = e ((BB}{=)) the multi-scale volume that need to infer the geometry.
Simply repeating the above process features and volumes of all L scales,
Geometry reasoning, For an arbitrary 3D point = (x, y. 2). we first get the interpolation of volumes at all scales (V)()) through trilinear sampling, and then concatenate them as the final feature F(p) RCh,, where Ch, I. x Ch. Combining the feature and the point position, MLP network is applied to predict the corresponding SDF value: sdfo R3 x Ch + R. And the surface represented by the zero-level set of the SDF value:
S = {p E R (df F(p)) = =0.
(2)
Color prediction. We refer to the first viewpoint Iu as the reference image. To predict the color of each point on a ray, we employ the blending strategy similar to [50]. We first project the 3D point to source views' pixel position according to Eq. 1, and interpolate the corresponding colors {p) N-1 and features {P) N-1 Here, we only use the highest resolution features to predict the color. Next, an MLP network take the concatenation of features and viewing direction differences Ad = d d; as input, to predict the softmax-activald blending weights [wr(p)))<= N-11 of each source view, and the final color is blended as the weighted sum of source colors:
N
c(p) k(p)ws(p)
SDF-based volume rendering. Given the density Ma and color { of M samples along the ray p(t) + td emitting from camera center o to pixel q in view direction d, NeRF [27] approximates the color using numerical quadrature:
Tac T = (1 a1)
(4)
1=1

---

point
Pointi
Surtice paitt
Belinance view
Refsese Siew
jour
Source view
Soarce
(A)
Figure 3: Multi-view aggregation ambiguity, Here, we take two viewpoints as an example. (a) For those low-texture regions, sampling points near the surface may get the same aggregation and lack discriminability. (b) The aggregation of points away from the surface are random and hard infer the accurate geometry, e.g., two sampling points may get the same aggregation even with different SDF value.
where T; the accumulated transmittance, and a; 1 =1,c in original volume rendering. To better approximation the geometry of the scene, NeuS [49] proposed an unbiased and occlusion-aware weighting method incorporate signed distance, and the a; is formulated as:
T (sdf (p(:))) T1 (pd((()(((++t)) ,0).
a; max
(5)
(s(ss((p(tt))
Here, (x(x) = (1 + e s0)-1 the sigmoid function and s an anneal factor. Readers can refer to [49] for more details.
3.2 Multi-scale feature-metric consistency
Rendering loss tends to trap the model into sub-optimization since it only considers single point and ignores the consistency among multiple viewpoints. To mitigate this problem, straightforward practice is to project the image patches multiple views to the estimated surface location based on the local planar assumption and rely the photometric consistency enforce the multi-view consistency. However, we found this solution works well for per-scene overfitting training [4, 5] but brings limited benefits to generalization training.
We analyze that the main reason may the failure of photometric consistency, which be- comes more challenging for generalization train- ing. As proven in recent supervipe multi- view stereo methods [55, 54, 57, 39], the as- sumption of photometric consistency isn't al- ways effective, and the predicted geometry still has significant holes even in combination with the robust patch similarity like SSIM [52]. As the coordinate-basee methods train models sepa- rately for each scene to directly overfit the scene, they have greater potential converge to the op- timum. However, our generalization model en- codes all scenes with one model, and requires image features to infer geometry, which makes the model rely heavily on the discriminability of features, e.g., regions like low-texture and reflection become more critical for degrading results. As shown in Fig (a), those regions vi- olating photometric consistency not only reduce the accuracy of multi-view matching, but also decrease the discriminability of generalization model's input (we call this aggregation ambiguity), while the input of overfitting methods are distinct (3D) coordinate).
NCC-0.99
0.73
0.99
0.13
0.99
Image space
Multi-scale feature space
Figure 4: Multi-scale feature space. The feature space is more discriminative than ordinary image space, and more potential find the correspond- ing point during matching.
5

---

To overcome these challenges, we propose the multi-scale feature- metric consistency to measure the consistency between views multi-scale feature space, as shown Fig. 4. There are three main advantages of doing this way. First of all, the learnable feature is proven to be more discriminative than the original image [14], especially on those ambiguous regions like low-texture and reflection. Second, due to the larger receptive field, multi-scale information is conducive to improving the matching accuracy, and allows the model to be assisted by global information while recovering details. More importantly the feature discriminability can be continuously self-enhanced in the process of generalization training. The multi-scale feature space can train powerful model through more accurate matching. and the more powerful model can in turn lead to more discriminative feature space. And the enhanced feature can further mitigate the aforementioned aggregation ambiguity. These advantages have been proven in Tab. 3.
To generate the geometry, we adopt the same approximate method as [5] to directly locate the zero-level set of the SDF As shown in Fig. 5, We first find the interval where a ray intersects the surface by checking whether the signs of the SDFs of two adjacent sampling points are different. To handle occlusion, we only extract the surface within the first interval. Suppose the two samples of the interval are and P2, and their distances to the camera center are t and t2 respectively, our goal is to compute the position of pg. Here, rely on an assumption that two adjacent samples are close enough that the near surface can be regarded can get two similar triangles:
=df(rr)
vdf !
local plane. In this way, we
Figure 5: Locating the sur- face of ray.
Ap2p,52 Sp1P,S1:
(6)
Therefore, we can approximate the distance from the surface to the camera center Lg as:
-sdf (p2) to t to sdi
t1 to sdf(p1) (p1)ts It2 sdf (p2)t1
We thus can get the coordinate of the surface point p. = o lgd.
Through the automatic differentiation of the SDF network at ps. we can get the corresponding normal
ng Based on the assumption that the local surface centered at ps is plane of normal ng. we can find
the corresponding pixel position qi in is source view that correspond to the pixel go in reference
view:
qi = H.go H, = K. RR - 10) K5²
(8)
ni'p.
For pixel patch qo in the reference view, we can find the corresponding source patch through passing all pixels to Eq. 8 like q. H Regardless of occlusion, if the estimated surface ps is accurate, then these corresponding patches should also be consistent. In this paper, we measure patch consistency in a multi-scale feature space. We only apply features at the top 3 scales, since features lower scales lose lot of structural information. Therefore, for pixel patch a certain view, we can get the multi-scale patches (F, q >> >>}=0 through bilinear interpolation, and we upsample and concatenate them together as input F", whose channel is Ch₃ = 3 x Ch, for patch similarity measure, Here, we employ the normalization cross correlation (NCC) compute the feature-space consistency:
Cha-1 Σ
Cou( FoL Fa)
NCC; =
Ch2
(ar(Fa)aar(FF)
t=0
where Cov denotes covariance and Var refers to variance. Following the common solution in multi-view stereo field [7], compute the final multi-scale feature-space consistency loss as the average of the best K NCCs:
1 K-1
Lmfe = K. k=0
(1 _ NCC,).
(10)
3.3
View contrast loss
For 3D structure captured by multiple viewpoints, there a fact that some regions are covered by enough viewpoints, while some regions are only visible to few viewpoints. Compared with the

---

former, the aggregated features of the latter are more likely to be polluted by irrelevant rays, making them less predictable. To solve this problem, we design a view contrast loss to improve the accuracy of the reconstruction when visible views are limited, which enforces the geometric estimation to be the same under different inputs of the same scene.
We empirically lets results from dense inputs to supervise results of sparse imputs. Specially, taking a of multi-view images input, we first reconstruct a multi-scale volume as a teacher, which is used infer the finer SDF value s for set of points P. Then we build student multi-scale scale volume from sparse input views and estimate the corresponding SDF value s'. Meanwhile, as shown in Fig. 3 (b), we found that only the sampling points falling on the surface have positive epipolar correspondences, and their aggregated features are more meaningful, while other samples are more random, and may obtain the same aggregation even if their SDF values are different. As shown in Fig. 6, we thus only calculate the consistency loss for near-surface points, whose finer SDF values are more accurate:
p
Figure Visualization of view contrast loss.
1 P'
Type
Σ
1s(p) s((p),.
(11)
where Pr is set of points close to the surface inferred from the fine SDF according to Eq. 7.
3.4 Overall pipeline
This section will introduce some implementation details and crucial components of our model including generalization training and fine-tuning.
Loss function.
The overall loss function is defined as:
Leator + almfe LL + Liegg =
(12)
For batch of sampled pixel set Q, the color loss is computed the LI distance between the rendered color and the ground-truth:
Leolor
Σ - C(g)). GEQ
(13)
Q
To make the geometry more compact and accurate, we apply the regularization loss which is composed of four terms:
Leeg LLek + 1/Lomooth + XLt + ÖLspprsee
(14)
Eikonal loss [9] is employed regularize SDF values of all sampled points P:
Lek
Σ (dl
1
(15)
P
PEP
To maintain the smooth of the surface, introduce a regularization the gradient of the normal:
1
Lemooth
[
(16)
Q
gEQ
where Agraa(g) is the alpha composition of normal gradient V (g) in ray through pixel g. Besides, we also adopt the total variation (TV) regularization [41] for our multi-scale volumes:
h-1
Lev = Σ 1=
+
(17)
To clean the geometric estimation, we introduce a sparsity prior:
La Leporae
1
Σ
exp(-7ladf(p)))
(18)
PP
PEP

---

Method
24
37
40
55
63
65
69
83
97
105
106
110
114
118
122
Mean
VolRecon [40]
1.20
1.56
1.08
1.11
1.05
0.74
1.23
1.38
PixelNerf [63]
5.13
8.07
5.85
4.40
7.11
4.64
5.68
6.76
9.05
6.11
3.95
5.92
6.26
6.89
6.93
6.28
IBRNet (50)
2.66
1.83
2.83
1.77
2.73
1,96
2.05
2.32
MVSNerf [3]
1.96
3.27
2.54
1.93
2.57
2.71
1.82
1.72
2.29
1.75
1.72
1.47
1.29
2.09
2.26
2.09
SparseNeuS [23]
1.68
2.25
1.10
2.18
1.28
1.23
1.19
0.75
1.56
1.64
GenS
1.45
2.77
1,69
0.97
1.54
1.90
1,03
1.49
1.36
0.97
1.07
0.97
0.62
1.14
1.16
1.34
NeuS [49]
4.57
4.49
3.97
4.32
4.63
1.95
4.68
3.83
4.15
2.50
1.52
6.47
1.26
5.57
6.11
4.00
VolSDF [60]
4.03
4.21
6.12
0.91
8.24
1.73
2.74
1.82
5.14
3.09
2.08
4.81
0.60
3.51
2.18
3.41
IBRNet
2.26
1.56
2.30
1.50
1.73
1.66
1.84
1.90
COLMAP [44]
0.90
2.89
1.63
1,08
2.18
1.94
1.6]
1,30
2.34
1.28
1.10
1.42
0.76
1.17
1.14
1.52
SparseNeuS (ft)
1.57
0.88
1.06
1.27
1.07
0.99
0.54
1.15
1.27
GenS (ft)
0.91
2.33
1,46
0.75
1.02
1.58
0.74
1.16
1,05
0.77
0.88
0.56
0,49
0.78
0.93
1,03
Table 1: Quantitative results of Chamfer Distance on DTU dataset with sparse inputs. denotes that the method needs the ground-truth depth for supervision.
'81
Generalization training. We select N = 4 for sparse setting and N = 19 for dense setting. We use Adam optimizer [17] with the base learning rate of 1e-3 for feature network and 5e-4 for other MLPs. We train the joint loss for 16 epochs on two A100 GPUs. We increase the value of from 0 to and in the first 2 epochs. In our implementation, we generate the surface points P" of each image of the model trained with dense input first, and then distill the model with sparse input, with 3 set to
We build the generalized multi-scale volume with scales, whose resolution increase from 24 to 2 Each volume is equipped with thin features with only 4 feature channels, which allows to save memory compared to general single volume methods.
Fine-tuning. After generalization training, we first reconstruct the generalized multi-scale volume, which has encoded the geometry information. Then we sparse the multi-scale volume by pruning voxels far from the surface. During fine-tuning, we abandon the feature network, and directly optimize the multi-scale volume and MLPs. With the generalization prior, can achieve state-of oftte-of-the-rtt performance in only about 20 minutes of fine-tuning.
4 Experiments
We demonstrate the state-of-the-art performance of GenS with comprehensive experiments and verify the effectiveness of each module through ablation studies. We first introduce the datasets and then analyze our results.
Datasets. We conduct experiments both DTU [12] and BlendedMVV [59] datasets previous methods [49, 60, 23]. Our generalization model is trained on DTU dataset, which is an indoor MVS dataset with 124 different scenes scaned from 49 or 64 views with fixed camera trajectories. Following [61, 49, 23], we take the same 15 scenes for testing. The training set is defined as in [58, 36). and the test scenes contained therein are removed. We also evaluate our model on BlendedMVS, which is large-scale synthetic dataset. Each scene is scaned from different number of views, and all images has resolution of 768 x 576. We report the Chamfer Distance for DTU, and show some visual effects for BlendedMVV..
4.1 Comparisons
Results on DTU. We first adopt the same testing split and configuration as [23] compare with existing generalizable methods [3, 50, 23, 63). The results shown in Tab. indicate that our model outperforms existing methods by significant margin, and this advantage can be amplified after rapid fine-tuning (about 20 mins). Even compared with recent method [40], which adopts the ground-truie depth for supervision, our model can achieve superior results. The qualitative results in Fig. show that our reconstruction exhibits finer details. We further conducted more experiments on DTU to compare with per-scene overfitting methods with more input views. The quantitative comparisons in Tab. 2 show that our model can surpass some methods [61, 34, 49, 60] just through very fast network inference, i.e., we can achieve more than 34% improvement on scene 24 compared with [49]. After fast fine-tuning, the performance can be significantly improved, and even surpassing recent SOTA works [4, 5, 51]. Some visualization results Fig. depict that our model trained on large amounts of data is more robust to ambiguous regions.
8

---

Method
55
110
114
Mean
IDR [61]
1.63
1.87
0.63
0.48
1.04
0.79
0.77
1.33
1.16
0.76
0.67
0.90
0.42
0.51
0.53
0.90
MVSDF [65]
0.83
1.76
0.88
0.44
1.11
0.90
0.75
1.26
1.02
1.35
0.87
0.84
0.34
0.47
0.46
0.88
COLMAR
0.45
0.91
0.37
0.37
0.90
1.00
0.54
1.22
1.08
0.64
0.48
0.59
0.32
0.45
0.43
0.65
NoRF [27]
1.30
1.60
1.85
0.58
2.28
1.27
1.47
1.67
2.05
1.07
0.88
2.53
1.06
1.13
0.96
1.49
UNISURF [34]
1.32
1.36
1.72
0.44
1.35
0.79
0.80
1.49
1.37
0.89
0.59
1.47
0.46
0.59
0.62
1.02
VolSDF [60]
1.14
1,26
0.81
0.49
1,25
0.70
0.72
1.29
1.18
0.70
0.66
1.08
0.42
0.61
0.55
0.86
NeuS [49]
HF-NeuS [51]
0.76
1.32
0.70
0.39
1.06
0.63
0.63
1.15
1.12
0.80
0.52
1.22
0.33
0.49
0.50
0.77
Voxurf
0.35
NeuralWarp [4]
0.49
0.71
0.38
0.38
0.79
0.81
0.82
1.20
1.06
0.68
0.66
0.74
0.41
0.63
0.51
0.68
Goo-NeuS* [5]
GenS
0.66
1.01
0.71
0.43
1,06
0.99
0.73
1.4
1.18
0.78
0.64
0.93
0.38
0.54
0.54
0.80
(ft)
Table 2:
Quantitative results of Chamfer Distance on DTU dataset with dense inputs. that we retrain the Geo-NeuS without sparse geometric supervision.
denotes
Image
VOISE
Nau
Chuss
Ours (ts
Figure 7: Qualitative comparisons with per-scene overfitting methods DTU dataset with dense inputs.
Generalizing to BlendedMVS. Like [23], we conduct additional verification on BlendedMVVS to showcase the generalization ability of our model. We also employ the same evaluation strategy as [23] for a fair comparison. As shown in Fig. 1, just through fast network inference, our model can recover more geometric details than SparseNeuS [23]. Through the fine-tuning of 5k iterations, the effect will be significantly improved and better than SparseNeuS with 10k iterations.
4.2 Analysis
Ablation studies. We conduct ablation stud- ies DTU dataset understand how the ponents of our model contribute to the overall performance. We start with our baseline model SparseNeuS [23], and gradually insert our con- tributions. The results in Tab. 4 show that our full model combining all components has the best mean score, and the baseline model, without any of our contributions, performs the worst. Multi-scale Feature-metric Consis- tency (MFC): Our self-enhanced MFC can con- tinuously improve the multi-view consistency of the model, and we also elaborated the ablation results its three main characteristics in Tab. 3. And the baseline is based on the pixel-wise feature consistency proposed [65]. It can
Patch-similarity
Multi-scale
Self-enhanced
Mean
1.86
1.76
1.73
1.62
Table Some ablation studies on MFC.
Single 962
Single 1922
Our GMY
Figure 8: Reconstruction from volumes with different resolution.
seen that strategy is more robust and efficiency. Generalized Multi-scale Volume (GMV):

---

MFC
GMV
VCL
24
37
40
55
63
65
69
97
105
106
110
114
118
122
Mcan
X
2.26
3.39
2.04
1.27
2.47
2.65
1.62
1.84
1.61
1.32
1.82
1.94
0.91
1.78
1.62
1.90
1.51
3.07
1.88
0.97
1.36
2.11
1.12
145
1.31
0.95
1.20
1.02
0.64
1.32
1.24
1.42
1.45
2.77
1.69
0.97
1.54
1,90
1.03
1.49
1.36
0.97
1,07
0.97
0.62
1.14
1.16
1.34
Table 4: Ablation results on DTU.
Imagge
Baseline
MFC
MEC-GMV
MEC-GMY-VEL
Chanter Distasee
2.1
151
1.2
1.0
Clamfir Distance
1.91
1.54
1.29
1.14
Figure 9: Visualization of some ablation results DTU.
We show some results of the models have different resolutions Fig. We can see that the reconstruction of single high-resolution volume is unbearably noisy (higher-resolution volume will lead more empty voxels, which is more tricky for generalizable models.) and overly smooth at low resolution, whereas our GMV reconstructs clean and detailed geometry. And our representation is lighter than a single volume with a resolution 32 x 192 due to our thin feature. View Contrast Loss (VCL): As the results shown in Tab. and Tab. 2, the reconstruction with dense inputs is more accurate than sparse reconstruction, we therefore treat the former as a teacher to teach the latter. The results shown in Tab. 4 validate that this strategy of the model. The visualization the ablation results are shown Fig. 9, which further depicts that every contributions propose continuously improve performance.
indeed improve the reconstruction quality
Limitation. Although model exhibits excellent generalization performance in multi-view reconstruction, we found that it cannot satisfactorily handle scenes with large camera motion, such as surrounding cases. Because in these scenarios, the aggregation features will be polluted by the ray features shooting from behind. Our current solution is to first predict the local structure covered by some adjacent viewpoints like [23, 58, 36), and finally fuse them together.
5 Conclusion
In this paper, we introduced GenS, an end-to-end generalizable neural surface reconstruction model. We first encode all scenes into our generalized multi-scale volume, more powerful representation that can reconstruct clean and detailed 3D structures. Then we introduce the multi-scale feature- metric consistency to combat the challenge of the photometric consistency failure. The learnable multi-scale feature can provide more discriminative representation and can be self-enhanced during the generalization training. And we finally designed a view contrast loss improve the accuracy of the reconstruction through distilling the finer reconstruction from dense inputs to the reconstruction from sparse inputs. Experimental results on both DTU and BlendedMVS datasets show that our model possess stronger generalization ability and can achieve start-of-the -hhe-art reconstruction through fast network inference efficient fine-tuning. In the future, we will focus on improving the performance of the model in difficult scenarios.
10

---

Acknowledgments and Disclosure of Funding
This work is financially supported by National Natural Science Foundation of China U21B2012 and 62072013, Shenzhen Science and Technology Program-Shenzlen Cultivation of Excellent Scientific and Technological Innovation Talents project(Grant No. RCJC202007 141 14435057). Shenzhen Science and Technology Program-Shenzherr Hong Kong joint funding project (Grant No. SGDX2021 123 1444440000)), this work is also financially supported for Outstanding Talents Training Fund in Shenzhen. In addition, thank our collaborators in Alibaba Group and the anonymous reviewers for their valuable comments.
References
[1] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan Goldman. Patchmatch: A randomized correspondence algorithm for structural image editing. ACM TOG, 28(3):24, 2009.
[2] Fausto Bernardini, Joshua Mittleman, Holly Rushmeier, Cláudio Silva, and Gabriel Taubin. The ball-pivoting algorithm for surface reconstruction. IEEE TVCG, 5(4):349-359 1999.
[3] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In ICCV, pages 14133-14133, 2021.
[4] François Darmon, Bénédicte Bascle, Jean-Clément Devaux, Pascal Monasse, and Mathieu Aubry. Improving neural implicit surfaces geometry with patch warping. In CVPR, pages 6260-6269, 2022.
[5] Qiancheng Fu, Qingshan Xu, Yew-Soon Ong, and Wenbing Tao. Geo-neus: geometry-consisten neural implicit surfaces learning for multi-view reconstruction. NeurIPS, 2022.
[6] Yasutaka Furukawa and Jean Ponce. Accurate, dense, and robust multiview stereopsis. IEEE TPAMI, (000):1362-3333 2009.
[7] Silvano Galliani, Katrin Lasinger, and Konrad Schindler. Massively parallel multiview stereopsis by surface normal diffusion. In ICCV, pages 873-881, 2015.
[8] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In ICCV, pages 7154-7164, 2019.
[9] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. arXiv preprint arXiv:2002. 2020.
[10] Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan. Cascade cost volume for high-resolution multi-view stereo and stereo matching. In CVPR, pages 2495-2504, 2020.
[11] Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, et al. Kinectfusion: real-time 3d reconstruction and interaction using moving depth camera. In UIST, pages 559 569,-56,220 2011.
[12] Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large scale multi-view stereopsis evaluation. In CVPR, pages 406-413, 2014.
[13] Mohammad Mahdi Johari, Yann Lepoittevin, and François Fleuret. Geonerf: Generalizing nerf with geometry priors. CVPR, pages 18365-18375. 2022.
[14] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, pages 694-711. Springer, 2016.
[15] Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. Poisson surface reconstruction. In SGP, volume 7, page 0, 2006.

---

[16] Michael Kazhdan and Hugues Hoppe. Screened poisson surface reconstruction. ACM ToG, 32(3):1-13, 2013.
[17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[18] Kiriakos N Kutulakos and Steven M Seitz. A theory of shape by space carving. IJC,, 39:199- 218, 2000.
[19] Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, pages 21 2117-2125, 2017.
[20] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. NeurlPS, 23:15651-15666 2020.
[21] Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Mare Pollefeys, and Zhaopeng Cui. Dist: Rendering deep implicit signed distance function with differentiable sphere tracing. In CVPR, pages 2019-2028, 2020.
[22] Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li. Learning to infer implicit surfaces without 3d supervision. NeurIPS, 32, 2019.
[23] Xiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, and Wenping Wang. Sparseneus: Fast generalizable neural surface reconstruction from sparse views. In ECCV, pages 210-227. Springer, 2022.
[24] Nelson Max. Optical models for direct volume rendering. IEEE TVCG, 1(2):99-008, 1995.
[25] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger, Occupancy networks: Learning 3d reconstruction in function space. In CVPR, pages 4460 4470, 2019.
[26] Mateusz Michalkiewicz, Jhony K Pontes, Dominic Jack, Mahsa Baktashmotlagh. and Anders Eriksson. Implicit surface representations as layers in neural networks. In ICCV, pages 4743-4752, 2019.
[27] Ben Mildenhall, Pratul P Srinivasan, Matthew Taneik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, pages 405-42J 2020.
[28] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM ToG, 41(4):1-15, 2022.
[29] Zak Murez, Tarrence Van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Rabinovich. Atlas: End-to-end 3d scene reconstruction from posed images. In ECCV, pages 414-431. Springer, 2020.
[30] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Occupancy flow:
4d reconstruction by learning particle dynamics. In ICCV, pages 5379-5389 2019.
[31] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In CVPR. pages 3504-3515, 2020.
[32] Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger. Real-time 3d reconstruction at scale using voxel hashing. ACM ToG, 32(6):1-11, 2013.
[33] Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. Texture fields: Learning texture representations in function space. In ICCV, pages 4531-4540, 2019.
[34] Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In ICCV, pages 5589-5599, 2021.
12

---

[35] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. CVPR. pages 165-174, 2019.
[36] Rui Peng, Rongjie Wang, Zhenyu Wang, Yawen Lai, and Ronggang Wang. Rethinking depth estimation for multi-view stereo: A unified representation. In CVPR, pages 8645-8654 2022.
[37] Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In ECCV, pages 523-540. Springer, 2020.
[38] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno- Noguer. D-nerf: Neural radiance fields for dynamic scenes. In CVPR, pages 10318-10327, 2021.
[39] Ke Qiu, Yawen Lai, Shiyi Liu, and Ronggang Wang. Self-super multi-view stereo via inter and intra network pseudo depth. In ACMMM, pages 2305-2313, 2022.
[40] Yufan Ren, Fangjinhua Wang, Tong Zhang, Mare Pollefeys, and Sabine Süsstrunkk. Volrecon: Volume rendering of signed ray distance functions for generalizable multi-view reconstruction. In CVPR, 2023.
[41] Leonid Rudin and Stanley Osher. Total variation based image restoration with free local constraints. In IC/P, volume I, pages 31-35. IEEE, 1994.
[42] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In ICCV, pages 2304-2314, 2019.
[43] Johannes L Schonberger and Jan-Michael Frahm. Structure-from-mottin revisited. In CVPR, pages 4 4104-4113, 2016.
[44] Johannes I. Schönberger, Enliang Zheng, Jachhal Frahm, and Marc Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In ECCV, pages 501-518. Springer, 2016.
[45] Steven M Seitz and Charles Dyer. Photorealistic scene reconstruction by voxel coloring. IJCV, 35:151-173, 1999.
[46] Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In CVPR, pages 5469, 2022.
[47] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In CVPR, pages
2021.
[48] Fangjinhua Wang, Silvano Galliani, Christoph Vogel, Pablo Speciale, and Mare Pollefeys. Patchmatchnet: Learned multi-view patchmatch stereo. In CVPR, pages 14194-14203, 2021.
[49] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurlPS, 2021.
[50] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In CVPR, pages 4690-4699, 2021.
[51] Yiqun Wang, Ivan Skorokhodov, and Peter Wonka. Hf-neus: Improved surface reconstruction using high-frequency details. NeurlPS, 1e21116661978,,, 2022.
[52] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 13(4):600-61) 612, 2004.
[53] Tong WU, Jiaqi Wang, Xingang Pan, Xudong Xu, Christian Theobalt, Ziwei Liu, and Dahua Lin. Voxurf: Voxel-based efficient and accurate neural surface reconstruction. In arXiv preprint 2208. 2022.
13

---

[54] Hongbin Xu, Zhipeng Zhou, Yu Qiao, Wenxiong Kang, and Qiuxia Wu. Self-supervised multi-view stereo via effective co-segmentation and data augmmmtammnnnn
[55] Hongbin Xu, Zhipeng Zhou, Yali Wang, Wenxiong Kang, Baigui Sun, Hao Li, and Yu Qiao. Digging into uncertainty in self-supervised multi-view stereo. In ICCV, pages 6078-6087 2021.
[56] Qingshan Xu and Wenbing Tao. Multi-scale geometric consistency guided multi-view stereo. In CVPR, pages 5483-5492, 2019.
[57] Jiayu Yang, Jose Alvarez, and Miaomiao Liu. Self-supervised learning of depth inference for multi-view stereo. In CVPR, pages 7526-7534, 2021.
[58] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstructured multi-view stereo. In ECCV, pages 767-783, 2018.
[59] Yao Yao. Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. In CVPR, pages 1790- 1799, 2020.
[60] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. NeurIPS, 34:4805-4811 2021.
[61] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. NeurlPS, 33:2492-2502, 2020.
[62] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In ICCV, pages 5752-5761, 2021.
[63] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one few images. In CVPR, pages 4578 4587, 2021.
[64] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. In NeurIPS, volume 35, pages 25018-25032, 2022.
[65] Jingyang Zhang, Yao Yao, and Long Quan. Learning signed distance field for multi-view surface reconstruction. In ICCV, pages 6525 6534, 2021.
[66] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint axXiv:2010.04492, 2020.
14


Reviews:
[{"review_id": "dOV2QWvtKW", "reviewer": "Reviewer_Tu79", "summary": "This paper proposes a generalisable neural surface reconstruction approach that is able to learn an SDF function from volume features and does not need per-scene training.", "questions": "1. In Eq. 2, the MLP also takes position as input. Doesn't it make the function not generalisable to other scenes, since position is scene-specific?\n2. In Eq. 3, does it make sense to blend the colors of all view projections? What if the point is not visible in certain views?\n3. Eq. 10 is proposed to encourage multi-view feature consistency. But how will it handle occlusion cases? If there are occlusions, the features are naturally inconsistent.", "limitations": "Limitations are already revealed in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper uses a multi-scale volume to model the feature field.\n2. The view contrast loss is novel to me which helps to enhance the reconstruction quality over regions that lack multi-view observations.", "weaknesses": "1. This paper does not mention some of the latest advanced approaches that also work on the generalisable neural surface reconstruction:\n     VolRecon: Volume Rendering of Signed Ray Distance Functions for Generalizable Multi-View Reconstruction\n     Rethinking Rendering in Generalizable Neural Surface Reconstruction: A Learning-based Solution\n2. The experiments are not complete. It does not compare with the two SOTA methods mentioned above. Also it does not compare with the learned MVS methods, such as MVSNet.\n3. The results of the reconstruction without fine-tuning is actually not looking good enough to me. For example, in figure 8, the mesh results are quite noisy. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer Tu79 for recognizing our work and the valuable comments. Here, we will explain your concerns point by point.\n\n**Q1: Lacking comparison with latest advanced approaches: VolRecon [1] and ReTR [2].**\n\nThanks for your suggestion, and the reasons why we don’t compare with these two methods are:  \n1) these two methods are contemporaneous with ours, and ReTR was made public after the NeurIPS deadline.  \n2) they are not unsupervised methods like ours, which require the ground-truth depth for supervision.  \n3) they are target at the sparse adjacent views, and only reconstruct part of the geometry. \n\nEven though, we think these comparisons are valuable and we conduct these comparisons here. For fair comparisons, we adopt the same training and testing configuration with SparseNeuS . Due to the rush of the rebuttal, we only train our model for 10W iterations, which is half of SparseNeuS, and we will supplement the results with the same training iterations as SparseNeuS in our final version. The results are shown in the table below. Without any fine-tuning, our model can achieve superior performance than existing unsupervised method SparseNeuS, and perform a little worse than supervised method (VolRecon and ReTR). Through fast fine-tuning, our model can achieve the state-of-the-art performance and even outperform the supervised methods. Note that this experiment is performed under the test set split of SparseNeuS, where each scene contains only three images. And we will cite these works in our final version.\n\n|  | 24 | 37 |  40 | 55 | 63 | 65 | 69 | 83 | 97 | 105 | 106 | 110 | 114 | 118 | 122 | Mean |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| VolRecon [1] | 1.20 | 2.59 | 1.56 | 1.08 | 1.43 | 1.92 | 1.11 | 1.48 | 1.42 | 1.05 | 1.19 | 1.38 | 0.74 | 1.23 | 1.27 | 1.38|\n| ReTR [2] | 1.05 | 2.31 | 1.44 | 0.98 | 1.18 | 1.52 | 0.88 | 1.35 | 1.30 | 0.87 | 1.07 | 0.77 | 0.59 | 1.05 | 1.12 | 1.17 |\n| SparseNeuS (20W)\t | 1.68 | 3.06 | 2.25 | 1.10 | 2.37 | 2.18 | 1.28 | 1.47 | 1.80 | 1.23 | 1.19 | 1.17 | 0.75 | 1.56 | 1.55 | 1.64 |\nOurs (10W) | 1.28|3.04|2.05|0.97|1.38|2.16|1.07|1.41|1.36|0.93|1.22|1.02|0.64|1.24|1.25|1.40 |\nSparseNeuS (ft) | 1.29 | 2.27 | 1.57 | 0.88 | 1.61 | 1.86 | 1.06 | 1.27 | 1.42 | 1.07 | 0.99 | 0.87 | 0.54 | 1.15 | 1.18 | 1.27 |\nOurs (ft) | 0.75 | 1.99 | 1.34 | 0.75 | 1.12 | 1.68 | 0.77 | 1.21 | 0.98 | 0.74 | 0.89 | 0.54 | 0.43 | 0.82 | 0.91 | 0.99 |\n\n**Q2: Lacking the comparison with learning-based MVS method.**\n\nFollowing your suggestion, we add the comparison with VisMVSNet [3], which is an efficient learning based method. The comparison is shown in the table below. It can be seen that our model can achieve superior results even without fine-tuning. \n\n|  | 24 | 37 |  40 | 55 | 63 | 65 | 69 | 83 | 97 | 105 | 106 | 110 | 114 | 118 | 122 | Mean |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\nVisMVSNet [3] | 0.98 | 2.10 | 0.93 | 0.46 | 1.89 | 0.67 | 0.67 | 1.08 | 0.67 | 0.96 | 0.66 | 0.85 | 0.30 | 0.45 | 0.51 | 0.88 |\nOurs | 0.66 | 1.01 | 0.71 | 0.43 | 1.06 | 0.99 | 0.73 | 1.43 | 1.18 | 0.78 | 0.64 | 0.93 | 0.38 | 0.54 | 0.54 | 0.80 |\nOurs (ft) | 0.55 | 0.71 | 0.39 | 0.38 | 0.79 | 0.65 | 0.67 | 1.29 | 0.96 | 0.64 | 0.49 | 0.59 | 0.33 | 0.44 | 0.45 | 0.62 |\n\n**Q3: The reconstruction without finetuning in Fig.8 is quite noisy**\n\n1) Nosie is unavoidable in the reconstruction of generalizable model, and often requires some post-processing to filter, for example, photometric consistency and geometric consistency are required to filter in multi-view stereo.\n2) Inferring accurate geometry from only three images is challenging, and the model is prone to ambiguity due to missing multi-view correspondence information. Even though, with our designed modules, we can recover more details than SparseNeuS, such as the eyes, mouth, and nose.\n\n**Q4: Does using position as input to MLP affect the generalization of the model?**\n\nThis is like the positional encoding in the transformer. Our experiments show that it will not affect the generalization of the model, and it will speed up the convergence of the model, which has also been proved by SparseNeuS.\n\n**Q5: Does it make sense to blend the colors of all view projection? What if the point is not visible in certain view?**\n\nThis is a crucial problem we faced during our implementation. From Eq. (3), we can see that the final color is blended as the weighted sum of the colors projected from the source views. The network is trained to estimate higher weights for those visible views and lower weights for others. Theoretically, it will make sense to blend the colors of all view projection as long as there is at least one visible view. However, as your concern, the existing of the occlusion will inevitably degrade the performance of the model. To mitigate this drawback, We adopt a robust training strategy that randomly selects N source views from the pool with both adjacent and non-adjacent views. In this way, the model is more robust to blend accurate color from situations with occluded views.\n\n**Q6: How to handle occlusion cases in the multi-scale feature consistency?**\n\nAs you say, the features are naturally inconsistent if there are occlusions, and such bad cases will pollute the model training if no operation is done. To solve this problem, we only select the top K views with the largest feature consistency (the smallest NCC loss) to calculate the consistency loss, which is an effective empirical strategy assuming that the consistency with occluded views is small.\n\n**Reference**:\n\n[1] VolRecon: Volume Rendering of Signed Ray Distance Functions for Generalizable Multi-View Reconstruction, CVPR 2023.\n\n[2] Rethinking Rendering in Generalizable Neural Surface Reconstruction: A Learning-based Solution, arxiv 2023. \n\n[3] Visibility-aware Multi-view Stereo Network, BMVC 2020.\n\n"}, {"review_id": "KMeTvryBku", "reviewer": "Reviewer_EdqW", "summary": "This paper aims to address the problem of multi-view reconstruction using neural implicit representation. The key to improving both efficiency and efficacy is a deep network that predicts a generalizable multi-scale feature volume encapsulating the geometry and appearance of the scene given a set of posed images. The generalizable feature volume circumvent the need of optimizing the neural imiplicit representation from the scratch for each testing scene. The authors also devise two losses -- multi-view feature consistency and view contrast loss -- to further improve the reconstruction performance. Experiments demonstrate that the proposed method outperform several state-of-the-art methods in the DTU dataset.", "questions": "See questions in the weaknesses section.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- View contrast loss is novel and interesting in the context of NeRF training.\n\n- Ablation studies on multi-scale feature volume, , multi-view feature consistency, and view contrast loss validates the contribution of these critical components in the proposed system.", "weaknesses": "- My major concern of this paper is that the key ideas presented have been explored separately in several works. \n    1. Multi-scale feature volume for neural implicit field has been widely used in MonoSDF, NGP, and several other works. Generalizable multi-scale feature volume has been a go-to representation for deep learning based Multi-view Stereo.\n    2. Feature consistency has been investigated in MVSDF.\n    3. The surface approximation method is borrowed from Geo-NeuS.\nAs a result, the novelties are somewhat limited.\n\n- What is the annealing factor in (eq. 5) in training? I understand the authors provided a detailed description of the annealing factor for fine-tuning on a particular scene, but I wonder how to control the annealing factor in a batch-wise training setup.\n\n- Some questions regarding experiments:\n    1. The baseline results in Table 2 are surprisingly low. Without equipping all the novelties presented in the paper, the baseline seems to be a NeuS with a large feature volume. Can the authors explain why this baseline is much worse than NeuS (e.g., 3.7 vs 1 in scene 24 in DTU)?\n    2. Missing comparisons to recent efficient neural implicit methods. Although works like Voxurf do not have a generalizable deep network to obtain learnt feature volume given a new scene, they also claim to achieve state-of-the-art results on DTU after optimizing on a scene for several minutes.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer EdqW for recognizing our work and the valuable comments. Here, we will explain your concerns point by point.\n\n**Q1: The multi-scale mechanism has been explored in existing method.**\n\nThe multi-scale strategy is effective and has inspired and advanced many fields. We are the first to apply the multi-scale strategy to generalizable neural implicit surface reconstruction to extend SparseNeuS, and the similar strategy has also been proposed in the recent generalizable work ReTR [1]. Although the recent MonoSDF and NGP also use multi-scale ideas, they differ from our method in the following ways: 1) Our feature volume is constructed from image features through a model inference, which has strong generalization, while Both MonoSDF and NGP require scene-by-scene learning. 2) Our multi-scale feature volume can be sparsed with the learned generalization prior to further reduce storage and improve fine-tuning speed. \n\n[1] Rethinking Rendering in Generalizable Neural Surface Reconstruction: A Learning-based Solution, arxiv 2023.\n\n**Q2: The difference between our self-enhanced multi-scale feature consistency and the feature consistency in MVSDF**\n\nThe feature consistency of MVSDF and ours are quite different. Our self-enhanced multi-scale feature-metric consistency (MFC) has three main advantages. \n1) Our feature-metric consistency can be self-enhanced during the training. The robust feature consistency can train a better reconstruction model, and the better reconstruction model can extract better feature spaces, leading to more robust feature-metric consistency. To demonstrate this, we add a experiment using feature consistency without self-enhancement (pre-trained features). The results in the table below indicates that our consistency has significant advantage.\n2) Our consistency is based on the local planar assumption and is measured by the structural similarity losses within patches like SSIM or NCC, which is more robust and effective than the pixel-wise L1 loss used in MVSDF. As the comparison shown in the table below, the model with pixel-wise L1 feature loss like MVSDF performs worse than ours.\n3) Our consistency is measured in a multi-scale manner. The high-resolution feature space can provide finer and context-complete information, while the low-resolution feature space has larger receptive field to achieve more discriminative matching as shown in our Fig. 4. The comparison of single scale models with ours again demonstrates the superiority of our method.\n\nNote that the structure of the baseline is just like SparseNeuS, which has a single 96-resolution volume, and all experiments are conducted with three images as input.\n\n||24|37|40|55|63|65|69|83|97|105|106|110|114|118|122|Mean|\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Baseline |2.26|3.39|2.04|1.27|2.47|2.65|1.62|1.84|1.61|1.32|1.82|1.94|0.91|1.78|1.62|1.90|\n| Pixel-wise L1 |2.11|3.29|2.20|1.26|2.39|2.54|1.62|1.78|1.59|1.29|1.76|2.01|0.88|1.66|1.58|1.86|\n| Single-scale |1.64|3.39|2.16|1.27|2.29|2.25|1.48|1.76|1.55|1.17| 1.70 | 1.75 | 0.86 | 1.65 | 1.54 | 1.76 |\n| No self-enhanced | 1.77 | 3.12 | 2.03 | 1.21 | 2.09 | 2.34 | 1.49 | 1.74 | 1.54 | 1.26 | 1.60 | 1.76 | 0.91 | 1.61 | 1.48 | 1.73 |\n| Our MFC | 1.51 | 3.22 | 1.99 | 1.16 | 2.00 | 2.21 | 1.30 | 1.58 | 1.45 | 1.18 | 1.48 | 1.53 | 0.80 | 1.54 | 1.43 | 1.62 |\n\n**Q3: The surface approximation method is borrowed from Geo-NeuS.**\n\nIn our multi-scale feature-metric consistency, we do not focus on the surface approximation method, but on applying the multi-scale feature space to achieve more robust matching among multi-view images. Actually, our solution is general and can be applied to any surface approximation method.\n\n**Q4: How to control the annealing factor in a batch-wise training setup?**\n\nWe adopt the same strategy as NeuS and SparseNeuS , setting the annealing factor (s) as a learnable parameter, and as the network converges, (1/s) will approach 0.\n\n**Q5: Why the baseline (NeuS with a large feature volume) is much worse than NeuS?**\n\n1. Table 2 shows the generalization results of the model trained on the training set, and it is reasonable to be worse than NeuS, an overfitting model that has been optimized for a long time in each scene. \n2. We did find that a large volume does not work well in generalization training. The main reason we think is that the receptive field of each voxel in a large feature volume is small, and the number of the empty voxel in a scene is large, which is even worse in the large volume. This makes it difficult to train the feature extraction network. The results of the small volume are also not satisfactory due to the lack of high frequency detail. Our experiments in the able below just verfy these, and it can be seen that our multi-scale volume is the best. Note that these experiments are based on dense inputs.\n\n||24|37|40|55|63|65|69|83|97|105|106|110|114|118|122|Mean|\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| SparseNeuS (96 res)|2.41|3.11|2.14|1.22|2.38|2.80|1.70|2.64|2.15|1.68|1.52|2.80|1.04|1.91|1.72|2.08|\n| SparseNeuS (192 res)|3.70|2.87|1.91|1.03|1.72|1.67|1.59|2.35|2.03|1.40|1.33|3.57|0.72|1.37|1.52|1.92|\n| Our GMV |0.92|1.91|1.12|0.77|1.37|1.21|1.09|1.55|1.35|1.09|0.96|1.14|0.41|0.70|0.67|1.08|\n\n**Q6: Missing comparisons to recent methods like Voxsurf.**\n\nSorry for missing this comparison due to the fact that Voxsurf is not a generalizable model. Voxsurf is a very valuable work, which can greatly speed up the training compared to previous methods. Even though, our model has two main advantages than Voxsurf: 1) Our model is generalizable which can infer the geometry  just through a fast network inference and achieve satisfactory results. 2) In a specific scene, we can achieve much better performance after fine-tuning for similar periods of time. We will cite Voxsurf in our final version.\n"}, {"review_id": "NGa7mGIeyh", "reviewer": "Reviewer_cro4", "summary": "This paper proposed a generalizable Nerf method. The novelty of this method lies in a proposed multi-scale feature-metric consistency loss and a proposed view contrast loss. The feature-metric consistency loss encourages the features extracted from different images at the same surface point to be consistent with each other. Doing this in feature space is claimed to be more discriminative. The view contrastive loss enforces that the recovered surfaces from subsampled image sets be aligned with that of a full image set, to encourage smoothness n surface regions with poor visibility. This method can be applied directly to predict surfaces or have its output fine-tuned for a short time to arrive at a stronger final solution. With this method, the authors achieve state-of-the-art results on the DTU and BlendedMVS datasets. ", "questions": "Some requests I have for the paper: \n-> Provide a numerical ablation over removing the proposed losses \n-> Provide standard deviations in the means for Table 1 \n-> Bold the best-performing model in each column for Table 1 \n\nWith respect to comparing to prior works, why do you not provide the performance of models such as SparseNeus and PixelNerf in Table 1? You make claims that these will perform worse, but this is not demonstrated quantitatively. Please provide the results of training at least one other generalizable method with the same fine-tuning process applied to your method in this table. ", "limitations": "-", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "The strengths of this paper are that the proposed losses are novel and well-founded and that their inclusion in a generalizable Nerf pipeline leads to state-of-the-art results both qualitatively and quantitatively. ", "weaknesses": "The main weakness of the paper is the grammar and sentence structure. I would heavily advise that some polishing be done to the paper before the final submission. Other than that the proposed method seems sound and the results are impressive. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer cro4 for recognizing our work and the valuable comments. Here, we will explain your concerns point by point.\n\n**Q1: The grammar and sentence structure should be further polished.**\n\nThanks for pointing them out, and we appreciate these suggestions. We will seriously enrich and correct our paper in the final version.\n\n**Q2: Provide a numerical ablation over removing the proposed losses.**\n\nDue to the rush of the deadline and the limitation of the page, we only conduct the ablation studies of our main innovations, e.g., view contrast loss and multi-scale feature-metric consistency loss. We include more ablation analysis on the Supp., but some ablation experiments on regularization loss, i.e., sparse loss $L_{sp}$, smooth loss $L_{sm}$ and total variation loss $L_{tv}$ are still ignored as you point out. Here, we further append these ablation experiments. To quickly complete these experiments within a limited rebuttal time, we perform the experiments on the model with single small feature volume and 3 views as input, which is just like the first stage of SparseNeuS. The results are shown in the table below, we can see that each regularization term can steadily improve the model performance. And the ablation for other losses are shown in Tab.2 of our main paper.\n\n|  | 24 | 37 |  40 | 55 | 63 | 65 | 69 | 83 | 97 | 105 | 106 | 110 | 114 | 118 | 122 | Mean |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Base | 2.17 | 3.50 | 2.45 | 1.36 | 2.64 | 2.51 | 1.75 | 1.92 | 1.79 | 1.43 | 1.89 | 2.03 | 1.05 | 2.01 | 1.71 | 2.01 |\n| +$L_{sp}$ | 2.12 | 3.50 | 2.15 | 1.34 | 2.52 | 2.67 | 1.72 | 1.95 | 1.70 | 1.39 | 2.05 | 2.01 | 1.03 | 1.84 | 1.66 | 1.98 |\n| +$L_{sp}$+$L_{sm}$ | 2.11 | 3.43 | 2.14 | 1.32 | 2.50 | 2.59 | 1.69 | 1.95 | 1.72 | 1.38 | 1.88 | 2.02 | 0.97 | 1.91 | 1.60 | 1.95 |\n| +$L_{sp}$+$L_{sm}$+$L_{tv}$ | 2.26 | 3.39 | 2.04 | 1.27 | 2.47 | 2.65 | 1.62 | 1.84 | 1.61 | 1.32 | 1.82 | 1.94 | 0.91 | 1.78 | 1.62 | 1.90 |\n\n**Q3: Provide standard deviations in the means for Tab. 1.**\n\nThanks for this constructive suggestion, it’s an important strategy to support our model. Since it takes a long time to retrain multiple sets of generalizable models, in this limited rebuttal time, we do multiple sets of fine-tuning experiments  to obtain the variance of the fine-tuning results, and the variance of the generalizable model will be supplemented in the final version later. We fine-tune each scene four more times, and the results are shown in the Table below. It can be seen that our model can get an performance of about 0.62 after fine-tuning, and the ones with relatively large standard deviations are mostly concentrated in difficult scenes.\n\n|  | 24 | 37 |  40 | 55 | 63 | 65 | 69 | 83 | 97 | 105 | 106 | 110 | 114 | 118 | 122 | Mean |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Exam1 | 0.55 | 0.71 | 0.39 | 0.38 | 0.79 | 0.65 | 0.57 | 1.29 | 0.96 | 0.64 | 0.49 | 0.59 | 0.33 | 0.44 | 0.45 | 0.615 |\n| Exam2 | 0.54 | 0.74 | 0.40 | 0.38 | 0.82 | 0.62 | 0.61 | 1.35 | 0.94 | 0.64 | 0.50 | 0.64 | 0.32 | 0.44 | 0.43 | 0.625 |\n| Exam3 | 0.57 | 0.71 | 0.40 | 0.39 | 0.77 | 0.63 | 0.58 | 1.32 | 0.94 | 0.65 | 0.49 | 0.60 | 0.34 | 0.45 | 0.45 | 0.619 |\n| Exam4 | 0.56 | 0.73 | 0.41 | 0.37 | 0.79 | 0.69 | 0.56 | 1.30 | 0.97 | 0.67 | 0.52 | 0.62 | 0.33 | 0.43 | 0.44 | 0.626 |\n| Exam5 | 0.55 | 0.72 | 0.38 | 0.37 | 0.80 | 0.64 | 0.59 | 1.29 | 0.94 | 0.64 | 0.51 | 0.61 | 0.34 | 0.44 | 0.44 | 0.617 |\n| Mean | 0.554 | 0.722 | 0.396 | 0.378 | 0.794 | 0.646 | 0.582 | 1.310 | 0.950 | 0.648 | 0.502 | 0.612 | 0.332 | 0.440 | 0.442 | 0.621 |\n|Std | 0.011 | 0.013 | 0.011 | 0.008 | 0.018 | 0.027 | 0.019 | 0.025 | 0.014 | 0.013 | 0.013 | 0.019 | 0.008 | 0.007 | 0.008 | 0.004 |\n\n**Q4: Bold the best-performance model in each column for Table 1.**\n\nThanks for this valuable suggestion, and we will revise our paper according to your suggestions.\n\n**Q5: There are no quantitative comparison with existing generalizable methods like SparseNeuS, and please provide these results.**\n\nSorry for missing these important quantitative comparisons, and we only do qualitative comparisons on BlendedMVS dataset. Here, we conduct two groups of experiments in the case of sparse and dense input to fully prove our advantages. \n\n1. Sparse input: We adopt the same sparse configuration as SparseNeuS and only train our model for10W iterations, which is half of SparseNeuS, to save rebuttal time. The results are shown in Table 2 in our uploaded PDF. We can see that our model has a significant advantage over existing methods, e.g., our fine-tuned model can outperform SparseNeuS by more than 20%.. \n\n2. Dense input: We train our model with dense inputs to reconstruct complete geometry. We implemented SparseNeuS in two configurations, one with feature volume resolution of 96 and one with resolution of 192. And we train all models with the same procedure. The results are shown in the table below. We can see that our model can outperform SparseNeuS by more than 60%.\n|  | 24 | 37 |  40 | 55 | 63 | 65 | 69 | 83 | 97 | 105 | 106 | 110 | 114 | 118 | 122 | Mean |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| SparseNeuS (96 res) |2.41|3.11| 2.14|1.22|2.38|2.80|1.70|2.64| 2.15 | 1.68 | 1.52 | 2.80 | 1.04 | 1.91 | 1.72 | 2.08 |\n| SparseNeuS (192 res)|3.70|2.87|1.91|1.03|1.72|1.67|1.59|2.35|2.03|1.40|1.33|3.57|0.72|1.37|1.52|1.92|\n| Our GeNIS | 0.66|1.01|0.71|0.43|1.06|0.99|0.73|1.43|1.18|0.78|0.64|0.93|0.38|0.54|0.54|0.80|"}, {"review_id": "H6clRFPXlF", "reviewer": "Reviewer_XxbL", "summary": "This paper proposed a method for neural implicit surface reconstruction from multi-view images. The proposed method is generalizable without per-scene overfitting. The key component of the proposed method is a multi-view feature volume constructed from a multi-scale FPN network. During training process, the multi-scale feature is used to constrain the view-consistency as well as geometry smoothness. Experiment results show that the proposed method achieves better results than some previous pre-scene methods and achieve state-of-the-art results on DTU dataset with finetuning.", "questions": "- How long does it take for finetuning? A long finetuning process would make the proposed method less interesting for \"generalizable\" representation.\n- I still failed to understand why the proposed view-contrast loss can improve smoothness. Is the teacher volume and student volume constructed with the same network? And why add consistency between student and teacher can improve smoothness?\n- I notice that ablation studies are only conducted on three data points instead of full set in DTU dataset. Is there any reason?\n- Is multi-view volume a must for generalizable representations? Current experiments only demonstrate multi-view feature consistency is useful. What about using single volume but with all the feature consistency and contrast constraints? i.e., a naive extension of SparseNeuS with the loss proposed in the paper.", "limitations": "The limitations and societal impact have been discussed in the paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A new approach that targets generalizable neural surface representation is interesting. The proposed method is reasonable and technical sound.\n- Writing is overall clear.\n- Result quality is not bad.", "weaknesses": "- The current proposed method (from my POV) is a direct extension of SparseNeuS with more input view and multi-scale volumes (and two additional loss term that using multi-scale features). Neural geometry representation with multi-scale approaches has been also proposed in [1] (BTW this was also missing in related works of this paper). The combination of these methods has not appeared before, but the overall novelty would still be limited, as the paper does not demonstrate the key challenges (either theoretical or practical) when combining these techniques. It more sounds like \"we put this and that together, and it works better than previous methods.\"\n- Experiment is a little bit weak. Quantitative results are only reported on DTU dataset. From results on Tab.1, retraining Geo-NeuS with the same dense-view input works really well (almost same as GeNIS with finetune). Fig.8 demonstrates qualitative results on BlendedMVS dataset; however, it is hard to say which one is better for Ours(ft) and SparseNeuS(ft). These results suggest the improvement of the proposed method (especially, over its most related baseline, SparseNeuS) is marginal.\n\n[1] Takikawa, Towaki, et al. \"Neural geometric level of detail: Real-time rendering with implicit 3D shapes.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer XxbL for your time and valuable comments. Here, we will explain your concerns point by point. Due to words limitation, we put most experiments in the uploaded PDF.\n\n**Q1: Missing the citation of [1], which also adopts a multi-scale strategy**\n\nSorry for missing this important citation since we only focused on the volume rendering-based methods, and we will cite this paper in our final version. Meanwhile, our method and [1] are quite different: our multi-scale volume are constructed from the image feature, and that of [1] are learnable parameters.  We will cite [1] in our final version.\n\n[1] Neural geometric level of detail: Real-time rendering with implicit 3D shapes, CVPR 2021.\n\n**Q2: There is a lack of the demonstration of the key challenges when designing modules.**\n\nWe sincerely thank the reviewer for this suggestion, which we believe is very helpful in enhancing the persuasiveness of our article. In fact, due to the limitation of the page and the rush of deadline, we only show the main implementation and main experiments in the paper, and put many necessity analyzes of each modules in the Supp. We will adjust the structure of the article and enrich the analysis of key challenges in the final version according to your suggestions. We mainly focus and deal with three key challenges: 1) how to make the model recover geometric details while maintaining global shape consistency. 2) how to reconstruct the geometry that is consistent across different views. 3) how to make the model be robust to those regions only visible to few views. We have positively proposed generalized multi-scale volume, multi-scale feature-metric consistency and view contrast loss to solve these problems respectively, and verified their effectiveness through extensive experiments.\n\n**Q3: Experiment is a little bit weak. Quantitative results are only reported on DTU dataset.**\n\nMost of the previous methods including our baseline NeuS and SparseNeuS only conduct the quantitative results on DTU dataset, and show the qualitative results on BlendedMVS dataset. We just follow the same evaluation strategy as them.\n\n**Q4: From results on Tab.1, retraining Geo-NeuS with the same dense-view input works really well (almost same as GeNIS with finetune).**\n\nYou overlooked one important factor of the training time, which is one advantage of our model. Our model only needs to be fine-tuned in 5K iterations per-scene (about 20 mins), while Geo-NeuS needs to be trained in 300K iterations (about 10 hours).\n\n**Q5: Fig.8 demonstrates qualitative results on BlendedMVS dataset; however, it is hard to say which one is better for Ours(ft) and SparseNeuS(ft). These results suggest the improvement of the proposed method (especially, over its most related baseline, SparseNeuS) is marginal.**\n\n1. Actually, in the results depicted in Fig. 8, Our qualitative results are significantly superior to SparseNeuS with or without fine-tuning. In order to see them more clearly, we purposely zoomed in on the local details as shown in Figure 1 in the PDF we uploaded. From which we can clearly see that our reconstruction results are superior, especially in details such as eyes, ears and nose. \n2. To further demonstrate that the improvement of our model is not marginal, we conduct the quantitative comparison with existing methods SparseNeuS using both dense and sparse inputs. As the results shown in the table in our global response channel and Table 2 in our uploaded PDF, our model can achieve more than 60% improvement than SparseNeuS with dense inputs, and more than 20% with sparse inputs.\n\n**Q6: How long does it take for fine-tuning?**\n\nAs described in our paper, we fine-tune each scene for about 20 minutes. And even with only about 10 minutes of fine-tuning, we can already achieve very desirable performance and outperform most SOTA methods.\n\n**Q7: Why the proposed view-contrast loss can improve smoothness?  Is the teacher volume and student volume constructed with the same network?**\n\n1. As we state in our paper, there are regions visible only to a few viewpoints, which are easily polluted by irrelevant rays when aggregating features from multiple viewpoints, leading to geometric prediction errors. We show an naïve example in Figure 2 in our uploaded PDF. To make the model be robust to these tricky regions, we use results from dense views to teach results from the randomly selected sparse views. In this way, our model can infer more accurate geometry on these regions and make the reconstruction smoother. Especially when the resolution of the mesh is high, this smoothing effect is more obvious, as shown in Figure 2 in our uploaded PDF. \n2. Yes, we use the shared network for student volume and teacher volume.\n\n**Q8: Why only conduct ablation studies on three scenes?**\n\nThe reason is because of page limitations and layout considerations. We randomly select the results of three scenes, which can already demonstrate the effectiveness of each module. We show you the results of all scenes in Table 1 in our uploaded PDF.\n\n**Q9: Is multi-scale volume a must for generalizable representations? Current experiments only demonstrate multi-view feature consistency is useful. What about using single volume but with all the feature consistency and contrast constraints? i.e., a naive extension of SparseNeuS with the loss proposed in the paper.**\n\n1. The generalized multi-scale volume (GMV) indeed plays an important role in generalization training, and we have proved this through the ablation analyzes shown in our main paper and Supp, e.g., our GMV can be improved by more than 40% from 1.92 to 1.08 compared with single high-scale volume model as shown in  Table 1 in our uploaded PDF.\n2. As your suggestion, we further conduct experiments combining our proposed loss with the single-volume model SparseNeuS. The results are shown in Table 3 in our uploaded PDF.  It can bee seen that our innovation can indeed lead to significant improvements.\n\n\n"}]

Author rebuttal:
We sincerely thank all reviewers for their detail and constructive comments. We are glad that the reviewers appreciate the novelty (Tu79, EdqW, cro4), state-of-the-art and impressive results (cro4), reasonable ablation studies (EdqW), good technical soundness (cro4, XxbL), clear writing (XxbL). We highlight and summarize the advantages of our model here.

1. We are the first to successfully build a generalizable model that can reconstruct geometries comparable to or even superior than recent per-scene overfitting methods like NeuS and VolSDF. Meanwhile, we can achieve state-of-the-art performance with fast per-scene finetuning. Compared with existing generalizable model SparseNeuS, our model can achieve more than 60% improvement in the case of dense input (from 2.08 to 0.80), and more than 20% improvement in the case of sparse input (from 1.27 to 0.99). Moreover, our model can be trained end-to-end, while SparseNeuS needs two training stages.
|  | 24 | 37 |  40 | 55 | 63 | 65 | 69 | 83 | 97 | 105 | 106 | 110 | 114 | 118 | 122 | Mean |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SparseNeuS ft (Sparse) | 1.29 | 2.27|1.57|0.88|1.61|1.86|1.06|1.27|1.42|1.07|0.99|0.87|0.54|1.15|1.18|1.27|
| Ours ft (Sparse) | 0.75|1.99|1.34|0.75|1.12|1.68|0.77|1.21|0.98|0.74|0.89|0.54|0.43|0.82|0.91|0.99|
| SparseNeuS (dense) | 2.41|3.11|2.14|1.22|2.38|2.80|1.70|2.64|2.15|1.68|1.52|2.80|1.04|1.91|1.72|2.08|
| Ours (dense) | 0.66|1.01|0.71|0.43|1.06|0.99|0.73|1.43|1.18|0.78|0.64|0.93|0.38|0.54|0.54|0.80|

2. We are the first to introduce the multi-scale volume to the generalizable neural implicit surface reconstruction model to extend the single-volume model SparseNeuS. We found that increasing the number of input images leads to more severe occlusions and more noise, making geometric reconstruction more challenging. (This phenomenon has been proved in learning-based MVS methods (PVSNet)). We therefore introduce the low-resolution volume to maintain the globally coherent shape through the large receptive field, and adopt the high-resolution volume to recover finer detail. Meanwhile, each volume only equipped with a thin feature, which is more efficient than a single heavy high-resolution volume.
3. We introduced a self-enhanced multi-scale feature-metric consistency to improve the geometric consistency across multiple views. On the one hand, it can provide more accurate matching through the more discriminative multi-scale feature space and more robust patch similarity measure. On the other hand, the feature space can be self-enhanced during the training period, which will be continuously improved as the model converges.
4. We observe that different regions of the model have different numbers of visible viewpoints. The aggregated features of regions visible to a few viewpoints are likely to be polluted by irrelevant rays, making them less predictable. We thus design a view contrast loss to improve the model's reconstruction accuracy for these regions, thereby improving reconstruction smoothness.

The main concerns of all reviewers are concentrated on including more comparisons with existing methods. Here we have conducted more quantitative and qualitative experiments to further verify our effectiveness, and we will add these experiments to our final version. We put some experimental results in each response and put some in the uploaded PDF.

Meta-review: This is a technically solid paper that received unanimous accept recommendations. The AC concurs and feels that the presented advances in generalizable sparse-view 3D inference approach would be beneficial to the community.
Decision: Accept (poster)