Geo-Neus: Geometry-Consistent Neural Implicit Surfaces Learning for Multi-view Reconstruction
Authors: Qiancheng Fu
Qingshan Xu
Yew-Soon Ong
Wenbing Tao
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3d reconstruction
neural implicit surfaces learning
multi-view geometry consistency

Abstract: Recently, neural implicit surfaces learning by volume rendering has become popular for multi-view reconstruction. However, one key challenge remains: existing approaches lack explicit multi-view geometry constraints, hence usually fail to generate geometry-consistent surface reconstruction. To address this challenge, we propose geometry-consistent neural implicit surfaces learning for multi-view reconstruction. We theoretically analyze that there exists a gap between the volume rendering integral and point-based signed distance function (SDF) modeling. To bridge this gap, we directly locate the zero-level set of SDF networks and explicitly perform multi-view geometry optimization by leveraging the sparse geometry from structure from motion (SFM) and photometric consistency in multi-view stereo. This makes our SDF optimization unbiased and allows the multi-view geometry constraints to focus on the true surface optimization. Extensive experiments show that our proposed method achieves high-quality surface reconstruction in both complex thin structures and large smooth regions, thus outperforming the state-of-the-arts by a large margin.

Geo-Neus: Geometry-Consisten Neural Implicit Surfaces Learning for Multi-view Reconstruction
Qiancheng Ful
Qingshan Xu
Yew-Soon Ong
Wenbing Tao
Huazhong University of Science and Technology
Nanyynng Technological University
STAR,, Singapore
1 {(fqc98, wenbieitwwwnggt edu. en 2 "[qingshan. xu, asy.aogg//tuuu.ee
Abstract
Recently, neural implicit surfaces learning by volume rendering has become popular for multi-view reconstruction. However, one key challenge remains: existing approaches lack explicit multi-view geometry constraints, hence usually fail generate geometry-consister surface reconstruction. To address this challenge, we propose coometry--cnsisten neural implicit surfaces learning for multi multi-view reconstruction. We theoretically analyze that there exists gap between the volume rendering integral and point-bate based signed distance function (SDF) modeling. To bridge this gap. directly locate the zero-Ievel set of SDF networks and explicitly perform multi-view geometry optimization by leveraging the sparse geometry from structure from motion (SFM) and photometric consistency in multi-view stereo. This makes our SDF optimization unbiased and allows the multi-view geometry constraints to focus the true surface optimization. Extensive experiments show that our proposed method achieves high-quality surface reconstruction in both complex thin structures and large smooth regions, thus outperforming the state-of- the-arts by large margin. Code: https:/ //gt::///ht //tp://gtttttt thub.....c com/ cwm//oiit/
Introduction
Reconstructing surfaces from calibrated multi-view images Long-standing problem in computer vision and graphics. In the past years, traditional methods [30, 37, 15, 17] have adopted a multi-step pipeline to achieve impressive reconstruction results. Such pipeline requires depth maps point clouds to generate surface meshes. These intermediate representations inevitably introduce accu- mulated errors for the final reconstructed geometry. Recently, directly reconstructing surfaces from images [25, 43, 35, 42, 26] has attracted great interest for its potential to alleviate the accumulated errors and produce high-quality reconstructions.
achieve this, existing approaches represent surfaces as neural implicit representations and leverage volume rendering [21] to optimize them.
Inspired by neural volume rendering [23, 45] that simultaneously learns volume density and radiance field from input images, recent works [35, 42] use signed distance functions (SDF) [27] for surface representation and introduce the SDF-induced density function enable the volume rendering to learn an implicit SDF representation. In essence, these works still focus on direct color field modeling by volume rendering integral rather than explicit multi-view geometry optimization. Therefore, existing approaches usually fail to generate geometry consstttt surface reconstruction. Intuitively. volume rendering samples multiple points along each ray and expresses the output pixel colors as the integral of the radiance field, or the weighted sum of sampled colors along the ray (cf. Fig. 1(a)). It means that the volume rendering integral directly optimizes the integral of geometry instead of
"Equal contribution. 'Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Neest
Reference image
Rendened colors
Colors of estimated, surface
Geometry
(a) Volume rendering
(b) Visual example
Figure 1: (a) Illustration of volume rendering. (b) A visual example. The volume rendering in NeuS uses integral colors to implicitly supervise surface modeling. Although its rendered colors achieve good results, the colors of estimated surface fail to preserve object geometry information. This shows the bias between rendered colors and geometry. In contrast, our approach achieves structure- preserving colors of estimated surface and produces seometry-consistee surface reconstruction.
single surface intersection along the ray. This obviously introduces bias for geometry modeling, thus hindering true surface optimization. In Fig. ((b), we show the reconstruction case of NeuS [35], in which the bias between rendered colors and object geometry can be observed intuitively. Rendered colors are obtained by the color network via volume rendering. Surface colors are formed by the predicted colors of the surface where the SDF values are zeroes. can be easily seen that there exists gap between the rendered colors and the surface colors. Thus the reconstructed surface is imprecise despite the high-quality rendered image, indicating the bias between the color rendering and implicit geometry. (Detailed theoretical analysis will be elaborated later).
To address the above problem, we propose Geo-Neus to devise explicit and accurate neural geometry optimization model for geometry-consisterr neural implicit surfaces learning by volume rendering, leading to better multi-view 3D reconstruction. Specifically, directly locate the zero- level set of SDF networks and explicitly perform multi-view geometry optimization by leveraging the sparse geometry from structure from motion (SFM) and photometric consistency in multi-view stereo. This model has several benefits. First, directly locating the zero-level set of SDF networks guarantees that our geometry modeling is unbiased. This enables our method to focus on true surface optimization. Second, we show that explicitly enforcing multi-view geometry constraints on the located zero-level set of SDF networks allows our method to generate geometry-consister surface reconstruction. Previous neural implicit surfaces learning mainly uses the rendering loss to implicitly optimize SDF networks. This results geometry ambiguity during the training optimization. Our introduced two types of explicit multi-view constraints encourage our SDF networks to reason about the correct geometry, including both complex thin structures and large smooth regions.
In summary, our contributions are: 1) We theoretically analyze that there exists a gap between volume rendering integral and point-based SDF modeling. This demonstrates that is necessary directly supervise the SDF networks to boost the neural implicit surfaces learning. 2) Based on our theoretical analysis, we propose directly locate the zero-level set of SDF networks and leverage multi-view geometry constraints to explicitly supervise the training of SDF networks. In this way, the SDF networks are encouraged to focus on true surface optimization. Extensive experiments further validate the effectiveness of our theoretical analysis and the proposed direct optimization of SDF networks. We show that our proposed Geo-Neus is capable to reconstruct both complex thin structures and large smooth regions. Therefore, it greatly outperforms the state-of-the-an surface reconstruction methods, including traditional methods and neural implicit surface learning methods.
Related work
Traditional multi-view 3D reconstruction. Traditional multi-view 3D reconstruction is the clas- sical pipeline of surface reconstruction from multi-view images. Given multi- viet input images, traditional multi-view 3D reconstruction uses structure from motion (SFM) [33, 29] to extract and match features of neighbor views, and estimate camera parameters and sparse 3D points. After that, multi-view stereo (MVS) [30, 9, 37, 38, 36] is applied to estimate dense depth maps for each view and then all the depth maps are fused into dense point clouds. Finally, the surface reconstruction method [15, 17, 6), e.g., screened Poisson Surface Reconstruction [15) used to reconstruct surfaces from point clouds. Traditional methods have achieved great success various occasions, but there

---

exists incompleteness of surface in some cases because their multiple intermediate steps are not made into an ensemble. With the development of deep learning, many attempts have been made on learning-based multi-view reconstruction [14, 40, 39, 27, 22]. but the problem still exists.
Implicit representation of surface. Surface reconstruction methods can be generally divided into explicit methods and implicit methods, depending the representation of surface. Explicit representation includes voxels [5, 31] and triangular mesh [3, 4, 16), which are limited by the resolution. Implicit representation uses an implicit function to represent the surface and thus is continuous. The surface can be extracted using the implicit function at any resolution. Traditional reconstruction methods, c.g., screened Poisson Surface Reconstruction [15], use basic functions to form the implicit function. As for learning- baaedd methods, the most commonly used forms are the occupancy function [22, 28] and the signed distance function (SDF) [27] represented by the network. Based on these functions, many implicit surface reconstruction methods from point clouds have been proposed, e.g., ONet [22]. DeepSDF [27]. Point2Surf [8] and etc. For these methods, point clouds are obtained by scanner devices multi-view stereo methods. That is, these point clouds are uniform and complete. Therefore, these methods are rarely applied on the sparse point clouds produced by SFM to reconstruct surfaces (In fact, we show that these methods degrade on sparse point clouds from SFM in our supplementary material). In this work, we use the sparse points from SFM as an explicit geometry supervision and show that they can facilitate the neural implicit surface reconstruction.
Neural implicit surface reconstruction. Neural implicit field is a new way to represent the geometry of objects. With NeRF [23] first using the neural radiance field represented by Multi-Layer Perceptron (MLP) in novel view synthesis, plenty of works [32, 18, 20] have sprung up using neural networks to represent scenes. IDR [43] reconstructs surfaces with neural networks by representing
the geometry as the zero level set of an MLP that is considered to be an SDF MVSDF [44] imports information from the MVS network to arrive at more geometry priors. VolSDF [42] and NeuS [35] use the weight function that involved SDF during the rendering process to make colors and geometry closer. UNISURF [26] explores the balance between surface rendering and volume rendering. The surface reconstructed by the neural network shows better completeness compared with the traditional multi-view reconstruction methods, especially when dealing with Lam-erabnt cases. However, complex structures not handled well. Meanwhile, flat planes and sharp corners could not be guaranteed. NeuralWary [7], a concurrent work, also explores the use of patch-match on neural surface reconstruction. It combines volumetric rendering with a patch warping integration technique, which aggregates colors from points sampled along the camera ray from source views with patch warping. This way of patch aggregation is similar with volume rendering and shares the same sampled points and the same weights with those used by color integration. Note that NeuralWarp uses patch match with the color aggregation to optimize weights of sampled points, and thus to optimize the geometry indirectly. As we will analyze later, this kind of color integration operation will cause bias in the colors and the geometry. Therefore, Warp could not be trained from scratch and relies on the pre-trained model of VoISDF. Differently from Neural our method locates the predicted surface of the SDF network using bDF-dased interpolation and uses patch match to measure the photometric consistency among neighboring views. In this way, Geo-Neus can be trained from scratch and gets much better performance.
3 Method
Given posed multi-view images of an object, we aim at reconstructing the surface by neural volume rendering without mask supervision. The spatial field of the object is represented by signed distance function (SDF), and the corresponding surface is extracted using the zero level set of the SDF In the process of volume rendering, first analyze the inherent bias in color rendering which causes the inconsistency between rendered colors and implicit geometry. Then introduce explicit SDF optimization to achieve geometry consistency. An overview of our approach is shown in Fig. 2.
goal is to optimize the signed distance function. In this section,
3.1 Bias in color rendering
In the process of volume rending, there is a gap between the rendered colors and the geometry of the object. The rendered colors are not consistent with the real colors of the surface.
3

---

Sparse 3D points
SDF constraints
SDF Loss (Sec. 3.2)
MLP
SDF
Multi-view stereo constraints
Photometric Consistency Loss (Sec. 3.3) +
Volumetric Integration
MLP
Radiance
Color Loss (Sec. 3.4)
Σ
Figure 2: Overview of Geo-Neus. Previous neural implicit surfaces learning methods mainly depend on the color loss to implicitly supervise the SDF network. Our proposed Geo-Neus explicitly supervises the SDF network by introducing the SDF loss from sparse 3D points and photometric consistency loss from multi-view stereo.
For an opaque solid object f E R33 the opacity can be represented by an indicator function O(p):
1. p 0, 0
O(p)
When some colors capture some colors with cameras, the colors the light that transfers along the light ray into our eyes or cameras. Based on the inherent optical properties of the opaque solid object, approximately assume that the colors C of image {T, the colors of object intersecting with the light ray from the corresponding camera position o:
(o v) coo+
+ ttv):
(2)
where =" argmin (tlo+ t 8Ω, E (0, oo)]. on represents geometry surfaces. The assumption is appropriate because light that transmits through the opaque object can be omitted. The intensity of light decays to about zero drastically when passing through the surface of the opaque object. Let us represent the surface of the object mathematically with the signed distance function. The signed distance function sdf (p) is the signed distance between a spatial point and the surface 0Ω. In this way, the surface 0Ω can be represented as:
+
ds = {d] (p) =
(3)
With neural volume rendering, we estimate the signed distance function sdf and color field è by Multi-Layer Perceptron (MIP) networks Fe and G
saf (p) = Fe (p):
(4)
(0, v,t) = G. (o, v, Thus the estimated colors of the image with camera position be represented as:
(5)
è c=ʃ (t)
where is the depth along the ray that comes from o with the direction and w(t) is a weight for the point at t. For simplicity, the notes and omitted. To obtain discrete counterparts of and è we also sample & discretely along the ray and use the Riemann sum:
Σ (t)) 1=1
Following NeuS [35], w<(4) is computed as w(1)) ((4)) where = = 11; (1())) 3, (sdf(p,) (adf(ra
represents accumulated transmittance, a(t)) max represents the sampled spatial point 4, (.(xr) = (1 + --=====----- a Sigmoid function, where is learnalle parameter which controls the smoothness of the transition the surface.
0) is opacity, and p;
4. (sdl (.))

---

Notably, the goal of novel view synthesis is to make an accurate prediction of the colors Ć, and bend efforts to minimize the difference between the colors of ground truth images C. and the prediction C:
C C=Č= w (t))
(8)
In surface reconstruction tasks, what we concentrate more is the surface of the object rather than the color. In this way, the above formula can be rewritten as:
j-1 Σ w ((4) + (1)) + (2(4) +
C
Σ w (44)
i=f+1
= w (()))((())) + samplle + Σ w (4))
(9)
w (tz) + E Esample + Eweight:
where sf+ 0, 1 denotes the nearest sample point from 6t, Esampte denotes the bias caused by sampling operation and E weight denotes the bias caused by weighted sum operation of volume rendering. With Formula (2), it can be rewritten
(1z) 2 (1+) + Esample + Ewnight c (L*),
(10)
(0+) c c(I)) w (tz)
Esample Emeight
(11)
There the total bias between the colors of object surface and estimated surface is:
Ac = ē ((+) c((") (1 - w (t*)) E sample Eweight
(12)
(12)
The relative bias is:
Ac
6 sampie + tweightt
8c c (t*)
(13)
w (tz)
w c()))(())))
When w (ty) approaches to 1, Eweight approaches to 0 and de approaches to Exampls/c(t"). In this case, the total bias only caused by discrete sam- pling, which is small (but still exists). Simulated weights of some existing neural reconstruction methods shown in Fig. 3. As can be seen, it is nearly impossible to get right there in practice, especially without any geometric constraints. Fur- thermore, the problem becomes more intractable when dealing with cases of occlusion. Therefore, the weighted manner of volume rendering integral introduces bias to implicit geometry modeling. Because the supervision of the whole network al- most depends exclusively tween rendered colors and ground truth colors, the bias would make it difficult supervise the colors of surface and the SDF network, leading gap between the colors and the geometry.
witii
MAS
(a) Single plane
(b) Multiplane
Figure 3: Simulated weight in color rendering
the difference be-
process of neural reconstruction methods.
A trivial solution is directly supervise the geometry of the object. In this way, design explicit supervision on the SDF network and geometry-consiste supervision with multi-view constraints.
3.2 Explicit supervision on SDF network
The SDF network, which estimates the signed distance from any spatial point to the surface of the object, is the key network that we need optimize. So we propose an explicit supervision method on the SDF network to ensure its accuracy directly with points in 3D space.
5

---

For less extra cost, we use points generated by structure from motion (SFM) [29, 33] to supervise the SDF network. In fact, SFM is canonical solution to compute the camera parameters of input images, where 2D feature matches X and sparse 3D points P are also generated as byproducts. Thus, these sparse 31D points can be used as "free" explicit geometry information. Approximately, we suppose that these sparse points are on the surface of the object. That is, the SDF values of the sparse points are zeroes: sdf (p:) = O, where p; E P. In practice, after obtaining sparse 3D points, radius filter is applied to exclude some outliers [46].
Occlusion handling. Because we focus on opaque objects, some parts of objects are invisible from view of a certain camera position. Therefore, there are only some of the sparse points visible for each view. For an image 1, with camera position o,, the visible points P. are consistent with feature points X; of I;:
X; = K, [R
(14)
where K; is the internal calibration matrix, R, is the rotation matrix and I, is the translation vector for image 4. The coordinates of X, and P; are all homogeneous coordinates. The scale index before X; is omitted for simplicity. According to feature points of each image, we get visible points for each view and use them to supervise the SDF network while rendering image from the corresponding view.
Vwew-aware SDF loss. While rendering image I, from view V,, we use the SDF network to estimate SDF values for the visible points P; of V; (see supplementary for the SDF loss by random sampling from sparse 3D points). Based on the approximation that the SDF values of sparse points are zeroes, we propose the view-aware SDF loss:
LSDF =E P.
1 såf (D,) sdf (p;) Σ No
1 Pj N,
1saf sâf (p,)
(15)
where N; is the number of points in P; and denotes the Tn distance. It is worth noting that the loss we use to supervise the SDF network varies according to the view being rendered. In this way, the introduced SDF loss is consistent with the process of color rendering.
With the explicit supervision on the SDF network, our network could converge faster owing to the use of geometry prior. Besides, because the complex geometric structures with strong textures are the concentrated distribution areas of the sparse points, our method could capture more meticulous geometries.
3.3 Geometry-consistent supervision with multi-view constraints
With SDF loss, our network could eapture complex geometric details with strong textures. Since the sparse 3D points mainly provide the explicit constraints the areas with rich textures, large smooth regions still lack explicit geometry constraints. To go a step further, we design geometry-consistent supervision on the implicit surface with multi-view stereo constraints.
Occlusion-aware implicit surface capture. We use the implicit representation of the surface, and extract surface with the zero-level set of the implicit function. So the question is: Where is our implicit surface? According Formula (3), the estimated surface is:
on plsdr - =03-
(16)
We aim to optimize 052 with coometry-consisenn constraints among different views. Because the number of points on the surface is infinite, we need to sample points from Q52 in practice. To maintain consistency with the process of color rendering using view rays, we sample the surface points on these rays. As mentioned in 3.1, we sample t discretely along the view ray and use the Riemann sum to obtain the rendered colors. Based on the sampled points, we use linear interpolation to get the surface points, which is similar to the root-finding used in [25, 26] to estimate the surface.
Specifically, with sampled point on the ray, the corresponding 3D point is p = 0 + tv, and the predicted SDF value is sdf (p). For simplicity, we further represent (p) as sdf(t). which is the function of t. We find the sample point t, the sign of whose SDF value is different from the next sample point 4.41. The sample points set T formed by t, is:
T = {4lbad (1.) zdf (4+=1) < 0J.
(17)

---

In this situation, the line t,t,-1 intersects with the surface 0l.. The intersection points set T- is:
T t/t sdf(4))++1- - (df(4i+))
sdf - sdf((t tt eT]
(18)
The ray that interacts with the object may have more than one intersection with the surface. Specifi- cally speaking, there may be least two intersections. Similar to the SDF supervision mechanism, we just use the first intersection point along the ray considering the occlusion problem:
1' argmin {1le E }
(19)
The selection of 1* guarantees the sample points of the implicit surface are all visible for the corresponding view and makes the supervision consistent with the process of color rendering.
Multi-view photometric consistency constraints. We capture our estimated implicit surface, of which the geometric structures are supposed to be consistent among different views. Based this intuition, we use the photometric consistency constraints in multi-view stereo (MVS) [9, 37, 10] to supervise extracted implicit surface.
For a small area 8 on the surface, the projection of 8 on the image is a small pixel patch q. The patches corresponding to are supposed to be geometry-consistent among different views, except for occlusion occasions. Similar to patch warping traditional MVS methods, we use the central point and its normal to represent For convenience, represent the plane equation of in the camera coordinate of the reference image 4,:: n² p + -
(20)
where p is the intersection point computed through Formula (19) and n is the normal computed with automatic differentiation of SDF network at p. Then the image point x in the pixel patch qi of reference image L, is related to the corresponding point in the pixel patch qss the source image Ty via the plane-induced homography H [12]:
x= == Hx', H RR.R"" RR
I. K
K.
(21)
where donates the intrinsic matrix, R donates the rotation matrix and f donates the translation vector. The index indicates which image the donation belongs to. To concentrate on the geometric information, we convert color images {1,} into gray images {7/) and supervise our implicit surface with the photometric consistency among patches in {T,) (see supplementary for RGB image settings).
To measure the photometric consistency, we use the normalization cross correlation (NCC) of patches in the reference gray image {I,} and the source gray image {T]}:
Photometric consistency loss.
Cov (I; (g:), (g:s))
NCC(T,())
(22)
(Var(P,(q:)) )V Waraar(a (1(())))
where Coov denotes covariance and Var donates variance. While rendering colors for an image, we the patches which take the pixels being rendered as center and the patch size is 11 x 11. We take the rendered image as the reference image and compute NCC scores between its sampled patches and their corresponding patches on all source images. To handle occlusions, of the computed NCC scores for each sampled patch following [10], and use them to compute the photometric consistency loss for the corresponding view:
find the best four
Cphoto EN 1 NCC(T, T((gss))
N
(23)
4N
where N is the number of sampled pixels on the rendered image. With the photometric consistency loss, the geometric consistency of the implicit surface among multiple views is guaranteed.
3.4 Loss function
During rendering colors from a specific view, our total loss is:
C Lootor + aLreg + 3LSDF + Cphoto.
(24)
7

---

Ccolor is the difference between the ground truth colors and the rendered colors:
1 N
Caolor
(25)
And Creg is an eikonal term [11] to regularize the gradients of SDF network:
N
Lreg
Σ
((vdf(
(p,) 1
(26)
N
1=1
In our experiments, we choose a, 3 and as 0.1, 1.0 and 0.5 respectively.
Experiments
4.1 Experimental setting
Following previous practices [43, 35, 42], reconstruct surfaces from 15 scans of DTU dataset [1] to evaluate our method. DTU dataset has objects of various categories, which are quite different in terms of appearance and geometries. There are 49 64 images a resolution of 1200 x 1600 in each scan with camera parameters. We also test on challenging scenes from the low-res set of the BlendedMVS dataset [41] (CC-4 License). Scenes in BlendedMVS have various numbers of views and camera parameters. The scenes are captured by images at a resolution of 768 x 576, and the numbers of views vary from 31 to 143. We evaluate reconstructed surfaces DTU dataset with the Chamfer Distance provided by DTU evaluation metrics [1]. For the BlendedMVS dataset, we show the visual effects of the reconstructed surfaces.
Datasets.
Baselines. To better evaluate our method, we compare it with the-state-of-art learning based methods and the traditional reconstruction method, colmap [30]. For learning-baat methods, we compare with IDR [43]. VoISDF [42], NeuS [35] and NeuralWarp [7]. For colmap, we use the reconstructed surface with trim parameter 7 (the best performance) [26].
Implementation details. Similar to [43, 35, 42], the SDF network is modeled by an 8-layer MI.P with 256 hidden units and a skip connection the middle. is initialized by the geometric initialization presented in [2]. The radiance network is parameterized by 4-layer MLP with 256 hidden units. Positional encoding [20] is applied to 3D location with frequencies and to viewing direction with 4 frequencies. We sample 512 rays per batch and follow the hierarchical sampling strategy in NeuS [35] to sample points for each ray. We train our model for 300k iterations for around 16 hours single NVIDIA RTX2080Ti GPU. After network training, mesh can be extracted from the SDF in predefined bounding box by the Marching Cube [19] with the volume size of 5123
4.2 Comparisons
We compare the reconstruction quality of our method and baselines on DTU dataset. Table 1 shows the quantitative results. No- tably, our method outperforms baselines by large margin. Specifically, it out- performs state-of-the-art neural implicit surfaces learning methods by over 25% and outperforms the traditional method colmap by 22%. As shown qualitatively Fig. 4, our method achieves high-qualitg surface reconstruction thin structures and large smooth regions. For example, our method can recover abrupt depth changes in Scan 37 and re- construct planar structures in Sean and 40. To test the capability of handling
mask Neu8
Scan
IDR
VoISDF
NeuS
NeuralWarp
colmap
Ours
24
1.53
1.15
1.14
1.37
0.49
0.45
0.375
37
1.87
0.95
1.26
1.21
0.71
0.91
0.537
40
0.63
0.80
0.81
0.73
0.38
0.37
0.336
55
0.48
0.39
0.40
0.38
0.357
63
1.04
1.26
1.25
1.20
0.79
0.90
0.800
65
0.79
0.72
0.70
0,70
0.81
1.00
0.454
83
1.33
0.94
1.29
1.01
1.20
1.22
1.032
97
1.16
1.14
1.18
1.16
1.06
1.08
0,843
105
0.76
0.77
0.70
0.82
0.68
0.64
0.548
106
0.67
0.66
0.66
0.66
0.66
0.48
0.460
110
1.35
1.69
0.74
0.473
both complex
114
0.42
0.39
0.42
0.39
0.41
0.32
0.294
118
0.51
0.51
0.61
0.49
0.63
01.45
0.355
mcan
0:90
0.82
0.86
0.87
0.68
0.65
0.508
Table 1: Results on DTU scenes. The surfaces produced by colmap are trimmed with trimming value 7.
8

---

(O00 Sent 24
(DU) O0 Sacc
(S Meceeei Saatee Saa
Reference Image
IDR
NeuS
colmap
Ours
Figure Surfaces reconstructed DTU and BlendedMVS. We NeuS trained with mask supervision and colmap with trimming value 7 (see supplementary for comparison with NeuralWarpy
Reterenoe Image
Model-A Figure 5: Surface quality of ablation models.
Baseline
Model-B
Geo-Neus
various test on challenging
scenes of the BlendedMVS dataset. Qualitative results Fig. 4 show that our method yields more smooth and consistent surface quality than other methods.
4.3 Analysis
Ablation study. To evaluate the effect our proposed contributions, we conduct an ablation study on DTU dataset. NeuS is adopted as our baseline. Different modules progressively added to the baseline to investigate their efficacy. Results are reported in Table 2. We see that, with very sparse 3D supervision SDF networks, Model-A has begun to outperform colmap (0.62 0.65). This demonstrates that explicit SDF optimization very beneficial to improve geometries. With the proposed photometric consistency loss, Model-B B can optimize SDF networks more completely, leading to much more performance improvement, Fig. 5 shows how the proposed loss functions improve the surface quality. Model-A reconstructs the apple stem finely but the surface is not smooth enough. Model-B reconstructs the smooth surface but the apple stem is lost. That is, the SDF loss is better to improve the reconstruction of complex thin structures, while the photometric loss is better for the reconstruction of large smooth regions. Moreover, our full model, Geo-Neus absorbs their individual advantages and achieves the best performance.
Geometry bias of volumetric integration. To further investigate the geometric bias of volumetric integration, we render the depth images from a particular pose similar fashion to rendering RGB pixels [20] (see supplementary for details), and then use the depth images to construct sparse 3D points and photometric consistency constraints. NeuS also used the baseline. Comparison results shown in Table 3. As can be seen, compared with bascline (0.87), multi-view geometry

---

Method
Looter
LSDF
ppooto
mean
Constraint
Depth integral
Setting
mean
0,85
Bascline
0.87
Sparse 3D points
SDF location
0.62
Model-A
0.62
Depth integral
1.08
Model-B
0.54
Photometric consistency
SDF location
0.57
Geo-Neus
0.51
Table 3: Comparison results between depth inte-
'abble 2: Ablation study DTU scenes.
gral and SDF location.
constraints with depth integral bring little performance improvement or even degradation. It is remarkable fact that photometric consistency supervision with depth integral surface location could not converge because of the iniiial immense bias while the SDF location model converges smoothly. As an alternative, we train these two models based on the baseline model pretrained with 200k iterations. The result of the depth integral model still degrades compared with the baseline. This verifies the existence of geometric bias in volumetric integration. With our proposed SDF-oriented optimization, surface reconstruction quality can be significantly boosted.
Convergence speed. We further study the convergence speed of our proposed method, Geo-Neus, and baseline, NeuS. As shown in Fig. 6, our method converges rapidly from scratch and becomes stable after 200k iterations. In contrast, NeuS cannot extract the reasonable surface from SDF networks in the beginning and starts to become stable after 250k iterations. This demonstrates that our proposed explicit SDF optimization also improves the efficiency of neural surfaces learning by volume rendering, reducing the training time from around 16 hours to around 10 hours.
Cen
Figure 6: Convergence speed.
Limitation. We show failure cases of our method on scenes with strong specular highlights and transparent objects. Fig. 7(a) shows our reconstructed surfaces on the scene with strong specular high- lights, DTU scan77. this case, the strong specular highlights lead to large view-dependent effects, making the multi-view photometric consistency loss unable to reliably measure multi-view geometry constraints. Thus, our method cannot produce satisfactory surfaces in this case. In addition, Fig. 7(b) shows our reconstructed surfaces on the scene with transparent objects [13]. As proposed in Sec. 3.1, Figure Failure cases. we assume the target objects are all opaque and solid. For transparent objects, the proposed bias be- tween rendered colors and implicit geometry is invalid. Furthermore, the geometric loss we proposed may not work well, just like the photometric consistency used in traditional reconstruction methods.
Referasc image
Becocstrsiied Suraacs
5 Conclusion
We have proposed Geo-Neus, new method to perform neural implicit surfaces learning by enforcing explicit SDF optimization. our paper. we first provide the theoretical analysis that there exists gap between volume rendering integration and neural SDF learning. With this theoretical support, we propose to explicitly optimize neural SDF learning by introducing two multi-view geometry constraints: sparse 3D points in structure from motion and photometric consistency in multi-view stereo. In this way, Geo-Neus produces high-quality surface reconstruction in both complex thin structures and large smooth regions. Therefore, it outperforms the state-of-the-arts by large margin, including both traditional and neural implicit surfaces learning methods. We note that although our method greatly improves reconstruction quality, its efficiency is still limited. In the future, it will be interesting to explore accelerating neural implicit surfaces learning by volume rendering through super-fast per-scene radiance field optimization methods [34, 24]. We don't 't see an immediate negative socictal impact our work, but accurate 3D models may be used from malevolence.
Acknowledgments This work was in part supported by the National Natural Science Foundation of China under Grants 62176099 and 61991412, the Data Science and Artificial Intelligence Research Center (DSAIR), School of Computer Science and Engineering, Nanyang Technological University and the A Star Center for Frontier AI Research (CFAR).
10

---

References
[1] Henrik Aanæs, Rasmus Ramsbøl Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl. Large-scale data for multiple-view stereopsis. International Journal of Computer Vision, 120(2):153-168) 2016.
[2] Matan Atzmon and Yaron Lipman. Sal: Sign agnostic learning of shapes from raw data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2565 2574, 2020.
[3] Bruce G Baumgart. A polyhedron representation for computer vision. In Proceedings of the National Computer Conference and Exposition, pages 589-596, 1975.
[4] Jean-Daniel Boissonnat and Bernhard Geiger. Three-dimensional reconstruction of complex shapes based on the delaunay triangulation. In Biomedical Image Processing and Biomedical Visualization, volume 1905, pages 964-975, 1993.
[5] Adrian Broadhurst, Tom W Drummond, and Roberto Cipolla. A probabilistic framework for space carving. In Proceedings of the IEEE International Conference on Computer Vision, volume 1, pages 388-393, 2001.
[6] Brian Curless and Marc Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 303-312, 1996.
[7] François Darmon, Bénédicte Bascle, Jean-Clément Devaux, Pascal Monasse, and Mathieu Aubry. Improving neural implicit surfaces geometry with patch warping. arXiv preprint arXiv:2112.00644,, 2021.
[8] Philipp Erler, Paul Guerrero, Stefan Ohrhallinger, Niloy J Mitra, and Michael Wimmer. Points2surf learning implicit surfaces from point clouds. In Proceedings of the European Conference on Computer Vision, pages 108-124. Springer, 2020.
[9] Yasutaka Furukawa and Jean Ponce. Accurate dense, and robust multiview stereopsis. IEEE Trunsactions on Puttern Analysis and Machine Intelligence, 1369):136 2010.
[10] S. Galliani, K. Lasinger, and K. Schindler. Massively parallel multiview stereopsis by surface normal diffusion. In Proceedings of the IEEE International Conference on Computer Vision, pages 873-881, 2015.
[11] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. arXiv preprint arXiv:2002.10000 2020.
[12] Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cam- bridge University Press, 2 edition, 2004.
[13] Jeffrey Ichnowski, Yahav Avigal. Justin KerT, and Ken Goldberg. Dex-nerf: Using neural radiance field to grasp transparent objects. arXiv preprint 1r. 14.11 2021.
[14] Abhishek Kar, Christian Häne, and Jitendra Malik. Learning a multi-view stereo machine. Advances neural information processing systems, 30, 2017.
[15] Michael Kazhdan and Hugues Hoppe. Screened poisson surface reconstruction. ACM Transac- tions on Graphics, 32(3):1-13, 2013.
[16] Patrick Labatut, J-P Pons, and Renaud Keriven. Robust and efficient surface reconstruction from range data. In Computer Graphics Forum, volume 28, pages 2275-2290, 2009.
[17] Patrick Labatut, Jean- -ean-Philippe Pons, and Renaud Keriven. Efficient multi-view reconstruction of large-scale scenes using interest points, delaunay triangulation and graph cuts. In Proceedings of the IEEE International Conference on Computer Vision, pages 1-8, 2007.
[18] Lingje Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 13:15551-15666 2020.

---

[19] William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics. 21(4):163-16) 1987.
[20] Ricardo Martin- Bruaala, Noha Radwan, Mehdi SM Sajjadi, Jonathan Barron, Alexey Doso- vitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE Conference on Computer Vision and Pultern Recognition, pages 7210-7219, 2021.
[21] Nelson Max. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics, (12)999-18,, 1995.
[22] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4460 1470-4470, 2019.
[23] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In Proceedingss of the European Conference on Computer Vision, pages 405-421, 2020.
[24] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. arXiv preprini urXiv:2201.05989, 2022.
[25] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volu- metric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3504-33 -015,, 2020.
[26] Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf:
Unifying neural implicit sur- faces and radiance fields for multi-view reconstruction. In Proceedings the Conference Computer Vision, pages 5589, 2021.
the IEEE International
[27] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 165-174, 2019.
[28] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In Proceedings of the European Conference on Computer Vision, pages 523- 540, 2020.
[29] Johannes L Schonberger and Jan-Michael Frahm. Structure-fromt from-motion revisited. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 4104 4113, 2016.
[30] Johannes L. Schönberger, Enliang Zheng, Jan-Michae Frahm, and Mare Pollefeys. Pixelwise view selection for unstructured multi-view stereo. Proceedings of the European Conference on Computer Vision, pages 501-518, 2016.
[31] Steven M Seitz and Charles Dyer. Photorealistic scene reconstruction by voxel coloring. International Journal of Computer Vision, 35(2):151-173, 1999.
[32] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020.
[33] Noah Snavely, Steven M Seitz, and Richard Szeliski. Photo tourism: exploring photo collections in 3d. In ACM SIGGRAPH, pages 335-846, 2006.
[34] Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. arXiv preprint arXiv:2111.12//5 2021.
[35] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.1066 2021.
12

---

[36] Qingshan Xu, Weihang Kong, Wenbing Tao, and Mare Pollefeys. Multi-scale geometric consistency guided and planar prior assisted multi-view stereo. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
[37] Qingshan Xu and Wenbing Tao. Multi-scale geometric consistency guided multi-view stereo. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5483-5492, 2019.
[38] Qingshan Xu and Wenbing Tao. Planar prior assisted patchmatch multi-view stereo. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 12516-12523, 2020.
[39] Qingshan Xu and Wenbing Tao. Pysnet: Pixelwise visibility- multi-view stereo network. arXiv preprint wrXiv:2007.07111 2020.
[40] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mysnet: Depth inference for unstructured multi-view stereo. In Proceedings the European Conference Computer Vision, pages 767-783, 2018.
[41] Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: large-scale dataset for generalized multi-view stereo networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 1790-1799, 2020.
[42] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. In Advances in Neural Information Processing Systems, volume 34, 2021.
[43] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. In Advances in Neural Information Processing Systems, volume 33, pages 2492-2502, 2020.
[44] Jingyang Zhang, Yao Yao, and Long Quan. Learning signed distance field for multi-view surface reconstruction. Proceedings of the IEEE International Conference on Computer Vision, pages 6525-6534, 2021.
[45] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
[46] Qian- Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3d: A modern library for 3d data processing. arXiv preprint arXiv: 1801.09877 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 4.3.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 3.1.
(b) Did you include complete proofs of all theoretical results? [Yes] See Section 3.1.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either the supplemental material or as a URL.)? [No] We plan to release the code completely but have not yet with submission.
13

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.1.
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section 4.1.
(b) Did you mention the license of the assets? [Yes] See Section 4.1.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you' 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "i0pMhJoZHkZ", "reviewer": "Reviewer_1oxk", "summary": "This paper introduces a new method for geometry-consistent neural implicit surfaces learning. The proposed method includes a theoretical analysis of the gap between volume rendering and point-based SDF modeling and a solution by leveraging sparse points from SfM and utilizing patchmatch to provide geometry consistent supervision. This method is evaluated on DTU and BlendedMVS and achieves improvements compared with existing methods. ", "questions": "1. What’s the key difference between NeuraWarp and Geo-NeuS? Both methods adopted patch-match based optimization for better geometry reconstruction. The authors should provide a detailed discussion between these two methods and show their strengths, especially in the introduction and related works. \n\n2. Sparse points from SfM are utilized to supervise the SDF network, where the sparse points with a radius filter are supposed on the surface (line 162). However, in practice, the sparse points from SfM are usually noisy, and wrongly incorporating these points with large errors may decrease the reconstruction quality.  \n\n3. The authors claim that discrete sampling can cause bias (line 138) and linear interpolation is adopted to get surface points (line 195). What’s the error of the extracted surface points by these two methods, i.e., the proposed method of Geo-NeuS and the hierarchical sampling method in NeuS? This experiment may support the author’s assumption and theoretical analysis. \n\n4. The authors are encouraged to conduct a qualitative comparison between NeuralWarp and Geo-NeuS.  \n\n5. The training time is 16h with 300k iterations on NVIDIA 2080TI (line 259) while the training time of NeuS reported in its original paper on NVIDIA 2080TI is also about 16h. However, Geo-NeuS requires extra computation for depth and patch match supervision. I’m not sure whether this number is correct. Besides, what about the accuracy with fewer iterations, as discussed in line 313, such as 150k or 200k iterations?   \n\n ", "limitations": "Limitations or failure cases are not discussed in this paper. ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In this paper, the main contribution is incorporating depth supervision (from sparse points) and patch match supervision into NeuS’s optimization framework, which leads to better reconstruction results. It seems that the improvement mainly comes from integrating patch match, shown in Tab. 2, which has been demonstrated by NeuralWarp. So the discussion about these two methods is expected. \n\nThe paper is well organized, and the evaluations are sufficient to support the proposed method. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Uak6pSh69P", "reviewer": "Reviewer_Xndx", "summary": "This paper presents neural implicit surfaces learning by enforcing explicit SDF constraints. It analyzed the gap between volume rendering integration and implicit SDF learning. Based on that, it proposes an explicit surface point SDF supervision where the surface points are estimated with SFM. in addition, it locates the zero-level set of SDF networks and a multiview photometric consistency loss is proposed to explicitly supervise the training of SDF networks. The results demonstrate high quality 3d scene geometry reconstruction, even on thin structures and large flat regions. ", "questions": "- For the multiview photometric consistency loss, it is not clear to me how the reference view is picked w.r.t. the source view. Is it pure random or within a perturbation range? Eq. 19 determines the ray intersection with the implicit surface. However, it seems to me it does not guarantee the intersection point will be visible for all the reference views. If the point occluded in the reference view, will the multiview constraint still be valid?  \n\n- It is also not clear to me why this approach does not require foreground mask while still being able to reconstruct the 3d scene without background. \n\n- Please clarify how the ray sample weights w are converted from SDF values. \n\n- Both the SDF loss and the multiview consistency loss seem reasonable to help regularize the geometry learning. However, it is not clear to me how they are directly linked to the motivation given in sec 3.1. Will it be possible to evaluate the bias w/wo the proposed losses? \n", "limitations": "- The multiview photometric consistency loss does not consider the view dependent effects. it remains questionable for reflective or specular objects. \n\n- It will be interesting to show the effect of number of reference cameras on the rendering quality. On one hand, with less reference views, the SFM might produce noisy result and affect the SDF guidance loss. It is not clear so far how the quality of SFM points affect the 3d scene reconstruction. On the other hand, multiview consistency loss might help regularize the neural field distribution and compared to baseline models, it might still converge with sparser training cameras. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The paper is well organized and written. \n- Comprehensive experiments demonstrate the geometry improvements brought by the proposed 2 losses.\n- It provides theoretical analysis for the geometry bias, although I would like to see an evaluation of the bias after introducing the 2 losses (see details below)\n- Such warping patch losses have been explored in recent papers, such as NeuralWarp and MVCGAN, CVPR 2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "TxBxAKaOubw", "reviewer": "Reviewer_b5Ve", "summary": "Implicit representation has become a popular technique for 3D scene reconstruction. Existing methods do not utilize the multi-view geometry constraints in the learning. In this paper, by leveraging sparse geometry from structure-from-motion (SFM) and photometric constraints in multi-view stereo. \n \nThe paper compares the proposed method with both classical (colmap [28]) and recent deep learning baselines (IDR [40], VolSDF [39], NeuS [33], and NeuralWarp [7]).\n", "questions": "The proposed method uses an 8-layer MLP for generating features from 3d points. We can also use a permutation-invariant method such as pointNet to generate features from 3D points. All the baselines are somewhat constrained to IDF-style algorithm. It would be good to also consider other baselines like Point2Surf or DVR in addition to the baselines shown:\n", "limitations": "The paper clearly mentions the limitations, especially the challenge in improving computationally efficiency. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1) The paper builds on top of other methods such as IDF [40] in handling multi-view geometry constraints while learning the model. \n\nWeaknesses\n\n1) Some of the top-performing SDF methods from poinclouds are not cited. Using the pointclouds from SfM methods, we can generate implicit 3D representations using methods such as points2surf:\n\nErler et al. Points2Surf: Learning Implicit Surfaces from Point Cloud Patches, ECCV 2020.\n\n2) The chamfer distance shown in Table 1 has minor issues. For example, the proposed method is not the best performing method on scan 83. NeuS achieves better results. \n\n3) The paper heavily relies on existing structure-from-motion pipelines  [51, 47, 22, 19] to compute the camera parameters.  In some sense, the MLP networks are essentially utilized to further refine the multi-view geometry constraints. Furthermore, if the classical methods fail on a scene it would be hard to use this method. The improvement in the results need to be explained more carefully since the geometry constraints are already implicitly incorporated while using the  structure-from-motion pipelines. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GidTo6MYlec", "reviewer": "Reviewer_ee2G", "summary": "The authors of this manuscript propose Geo-NeuS, a novel neural implicit 3D reconstruction method.\nThey combine a SDF-based neural implicit surface representation, that is usually only optimized with the reconstruction loss, with explicit multi-view constraints.\nMore specifically, they incorporate supervision in the form of i) the sparse point cloud obtained from SfM, ii) photometric consistency from classic multi-view stereo.\nThey first theoretically analyze the difference of volume rendering and surface rendering-based approaches wrt. 3D reconstruction, and then show experimentally that adding the proposed constraints leads to better 3D reconstruction and more accurate surfaces. ", "questions": "1.) Why is BlendedMVS not evaluated quantitatively?\n\n2.) Why is the explicit SDF supervision (Sec. 3.2) implemented with occlusion handing and view-aware, instead of a simple loss via sampling in 3D space?\n\n3.) Why are grey-scale images used for the photoconsistency loss? Could an ablation study help to support this design choice?\n\n4.) Why did the authors use a patch size of 11x11?\n\n5.) Where / For which type of surfaces do the proposed losses help in particular (e.g. specular surfaces, homogeneous areas, etc)?", "limitations": "The authors discussion on limitations and negative societal impact (L. 331 - 334) is quite limited. The authors could tackle more complex datasets / more complex scenes if the proposed system does not fail on DTU and the selected BlendedMVS scenes. What happens if photoconsistency is not given, e.g., because of strong specular highlights? How does the model perform in the presence of transparent surfaces? How could the model handle sparse set of inputs instead of the considered dense coverage of scenes? These could be starting points for an interesting limitation discussion.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n1.) The proposed method clearly improves quantitatively (Tab 1) as well as qualitatively (Fig 4) over the state-of-the-art.\n\n2.) The proposed additional losses are well-grounded, not too complicated in general, and clearly have benefits for the task of 3D reconstruction (Tab. 2)\n\n3.) The ablation study (Tab. 3) of applying the proposed losses either via expected depth maps or via surface points found via root finding is very interesting. \n\n4.) While the findings of the theoretical analysis of volume vs surface rendering approaches for 3D reconstruction (Sec. 3.1) are expected, it is still valuable to have it.\n\n5.) The manuscript is written clearly, has a good structure, and experimental evaluation is enough to show the benefits of the proposed system.\n\n# Weaknesses\n\n1.) BlendedMVS is not evaluated quantitatively, and I couldn't find an argument for this.\n\n2.) It is not clear to me why the explicit SDF supervision (Sec. 3.2) is done with occlusion handling (L. 165) and view-aware (L. 173). It is only stated that \"the introduced SDF loss is consistent with the process of color rendering\" (L. 177 - 178). Instead, in every iteration, a subset of the sparse point cloud could be sampled and the loss can be applied on the 3D location without any occlusion reasoning etc. which seems simpler and more straight-forward. I believe a good reason / ablation study for this complicated setup is missing\n\n3.) The described process of finding the surface intersection (L. 197ff) is very similar to proposed root-finding methods from ray marching-based approaches for neural implicit surfaces like [23] and a short note on this+citation on this would be helpful for the reader.\n\n4.) The fact that the photometric consistency loss is only applied on grey-scale images (L. 211ff) is interesting, and an ablation study on this would be helpful.\n\n5.) NCC is used as the phometric consistency metric. Have the authors investigated other measures as well? This could be an interesting ablation study (but not a must in this manuscript).\n\n6.) It is not clear how the patch size of 11x11 (L. 227) was determined.\n\n7.) The fact that Colmap's best trim parameter is 7 (L. 253) should be cited, e.g. [23]. \n\n8.) The visual ablation study (Fig 5) could be bigger with zoom-in windows to better see the differences, similar to Fig 1 of Supp Mat.\n\n9.) Table 3 / \"Geometry bias of volumetric integration\": very interesting, but details are missing. Are here the expected depth maps used obtained via volume rendering? I think at least the supp mat should contain relevant formulas how the quantities are obtained.\n\n10.) Appendix C: Why is NeRF's rendering quality worse than NeuS / Geo NeuS?\n\n11.) Would be interesting to further discuss or which situations the losses help in particular, e.g. mostly for specular areas?\n\n12.) Fig 4 caption typo: NueS -> NeuS\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept