NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction
Authors: Peng Wang
Lingjie Liu
Yuan Liu
Christian Theobalt
Taku Komura
Wenping Wang
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: neural rendering
3d reconstruction

Abstract: We present a novel neural surface reconstruction method, called NeuS, for reconstructing objects and scenes with high fidelity from 2D image inputs. Existing neural surface reconstruction approaches, such as DVR [Niemeyer et al., 2020] and IDR [Yariv et al., 2020], require foreground mask as supervision, easily get trapped in local minima, and therefore struggle with the reconstruction of objects with severe self-occlusion or thin structures. Meanwhile, recent neural methods for novel view synthesis, such as NeRF [Mildenhall et al., 2020] and its variants, use volume rendering to produce a neural scene representation with robustness of optimization, even for highly complex objects. However, extracting high-quality surfaces from this learned implicit representation is difficult because there are not sufficient surface constraints in the representation. In NeuS, we propose to represent a surface as the zero-level set of a signed distance function (SDF) and develop a new volume rendering method to train a neural SDF representation. We observe that the conventional volume rendering method causes inherent geometric errors (i.e. bias) for surface reconstruction, and therefore propose a new formulation that is free of bias in the first order of approximation, thus leading to more accurate surface reconstruction even without the mask supervision. Experiments on the DTU dataset and the BlendedMVS dataset show that NeuS outperforms the state-of-the-arts in high-quality surface reconstruction, especially for objects and scenes with complex structures and self-occlusion. 

NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction
Peng Wang', Lingjie Liut*, Yuan Liu", Christian Theobalt', Taku Komura', Wenping Wang**
The University of Hong Kong !ax Planck Institute for Informatics Texas A&M University
(pwang3, yliu, taku]@cs. hku. *(lliu, theobalt (aabball@@mmpp tmpiiinn mmg..p.......... de "eepinneeammu. eduu
Abstract
We present a novel neural surface reconstruction method, called NeuS, for recon- structing objects and scenes with high fidelity from image inputs. Existing neural surface reconstruction approaches, such as DVR [Niemeyer et al., 2020] and IDR (Yariv et al., 2020), require foreground mask as supervision, easily get trapped in local minima, and therefore struggle with the reconstruction of objects with severe self-occlusion or thin structures. Meanwhile, recent neural methods for novel view synthesis, such NeRF [Mildenhall al., 2020] and its variants, use volume rendering to produce a neural scene representation with robustness optimization, even for highly complex objects. However, extracting high-quality surfaces from this learned implicit representation is difficult because there are not sufficient surface constraints in the representation. In NeuS, we propose represent a surface as the zero-level set of a signed distance function (SDF) and develop
volume rendering method train neural SDF representation. We observe that the conventional volume rendering method causes inherent geometric errors (i.e. bias) for surface reconstruction, and therefore propose new formula- tion that is free of bias in the first order of approximation, thus leading to more accurate surface reconstruction even without the mask supervision. Experiments on the DTU dataset and the BlendedMVV dataset show that NeuS outperforms the state-of- the-arts in high-quality surface reconstruction, especially for objects and scenes with complex structures and self-occlusion.
Introduction
Reconstructing surfaces from multi-view images is fundamental problem in computer vision and computer graphics. 3D reconstruction with neural implicit representations has recently become highly promising alternative classical reconstruction approaches [35] 2 due to its high reconstruction quality and its potential to reconstruct complex objects that are difficult for classical approaches, such non-Lambertian surfaces and thin structures. Recent works represent surfaces as signed distance functions (SDF) [46] 49 17 (22 or occupancy (29 30 To train their neural models,
these methods use differentiable surface rendering method render 3D object into images and compare them against input images for supervision. For example, IDR [46) produces impressive reconstruction results, but it fails reconstruct objects with complex structures that causes abrupt depth changes. The cause of this limitation is that the surface rendering method used in IDR only considers single surface intersection point for each ray. Consequently, the gradient only exists at this single point, which is too local for effective back propagation and would get optimization stuck in poor local minimum when there are abrupt changes of depth on images. Furthermore, object
"Coresponding authors.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

masks are needed supervision for converging to a valid surface. As illustrated in Fig. (a) top, with the radical depth change caused by the hole, the neural network would incorrectly predict the points near the front surface to be blue, failing to find the far-back blue surface. The actual test example in Fig. (b) shows that IDR fails correctly reconstruct the surfaces near the edges with abrupt depth changes.
Recently, NeRF (28] and its variants have explored use a volume rendering method to learn volumetric radiance field for novel view synthesis. This volume rendering approach samples multiple points along each ray and perform a-composition of the colors of the sampled points produce the output pixel colors for training purposes. The advantage of the volume rendering approach is that it can handle abrupt depth changes, because it considers multiple points along the ray and so all the sample points, either near the surface or on the far surface, produce gradient signals for back propagation. For example, referring Fig. (a) bottom, when the near surface (yellow) found to have inconsistent colors with the input image, the volume rendering approach ís capable of training the network to find the far-back surface to produce the correct scene representation. However, since intended for novel view synthesis rather than surface reconstruction, NeRF only learns volume density field, from which it is difficult to extract a high-quality surface. Fig. (b) shows surface extracted level-set surface of the density field learned by NeRF, Although the surface correctly accounts for abrupt depth changes, it contains conspicuous noise in some planar regions.
warta aurfantneedeen neneetu
surface
volkee woknnnseeeeee
sempled points
-- o -0-00-0000 0
Reference Image
IDR
NeRF
Ours
(a) Illustration
(b) Example
Figure/ I: (a) Illustration of the surface rendering and volume rendering. (b) A toy example of bamboo planter, where there are occlusions on the top of the planter. Compared to the state-of-the-ar methods, approach can handle the occlusions and achieve better reconstruction quality.
In this work, we present a new neural rendering scheme, called NeuS, for multi-view surface recon- struction. NeuS uses the signed distance function (SDF) for surface representation and uses novel volume rendering scheme to learn a neural SDF representation. Specifically, by introducing a density distribution induced by SDF, make it possible apply the volume rendering approach to learning an implicit SDF representation and thus have the best of both worlds, i.e. an accurate surface represen- tation using a neural SDF model and robust network training the presence abrupt depth changes as enabled by volume rendering. Note that simply applying a standard volume rendering method the density associated with SDF would lead to discernible bias (i.e. inherent geometric errors) in the reconstructed surfaces. This is new and important observation that we will elaborate later. Therefore we propose novel volume rendering scheme to ensure unbiased surface reconstruction in the first-order approximation of SDF. Experiments on both DTU dataset and BlendedMVS dataset demonstrated that NeuS is capable of reconstructing complex 3D objects and scenes with severe occlusions and delicate structures, even without foreground masks as supervision. It outperforms the state-of-the-art neural scene representation methods, namely IDR (46) and NeRF 28 terms of reconstruction quality.
2 Related Works
Classical Multi-view Surface and Volumetric Reconstruction. Traditional multi-view 3D recon- struction methods can be roughly classified into two categories: point- and surface-bseed reconstruc-
tion 2
35 and volumetric reconstruction 6 36 Point- and surface-based reconstruction
methods estimate the depth map of each pixel by exploiting inter-image photometric consistency [ and then fuse the depth maps into a global dense point cloud [25]
48 The surface reconstruction
usually done post processing with methods like screened Poisson surface reconstruction [16]. The

---

reconstruction quality heavily relies on the quality of correspondence matching, and the difficulties in matching correspondence for objects without rich textures often lead to severe artifacts and missing parts in the reconstruction results. Alternatively, volumetric reconstruction methods circumvent the difficulty of explicit correspondence matching by estimating occupancy and color voxel grid from multi-view images and evaluating the color consistency of each voxel. Due to limited achievable voxel resolution, these methods cannot achieve high accuracy.
Neural Implicit Representation. Some methods enforce 3D understanding in a deep learning framework by introducing inductive biases. These inductive biases can be explicit representations, such as voxel grids [13 44 point cloud 77 [4 18), meshes [4] 43 [4]. and implicit repre- sentations. The implicit representations encoded by a neural network has gained lot of attention recently, since it is continuous and can achieve high spatial resolution. This representation has
been applied successfully to shape representation 226 27 31
novel view synthe-
sis 38 23
15
28
21
33
34
40
37 and multi-view 3D reconstruction
46
22
Our work mainly focuses learning implicit neural representation encoding both geometry and appearance in 3D space from 2D images via classical rendering techniques. Limited in this scope, the related works can be roughly categorized based the rendering techniques used, i.e. surface rendering based methods and volume rendering based methods. Surface rendering based methods [29] assume that the color of ray only relies on the color of an intersection of the ray with the scene geometry, which makes the gradient only backpropagated to local region near the intersection. Therefore, such methods struggle with reconstructing complex objects with severe self-occlusions and sudden depth changes. Furthermore, they usually require object masks as supervision. On the contrary, our method performs well for such challenging cases without the need of masks.
17 46 [22
Volume rendering based methods, such as NcRF[28), render an image by a-compositing colors of the sampled points along each ray. As explained in the introduction, it can handle sudden depth changes and synthesize high-quality images. However, extracting high-fidelity surface from the learned implicit field is difficult because the density-hased scene representation lacks sufficient constraints on its level sets. In contrast, our method combines the advantages of surface rendering based and volume rendering based methods by constraining the scene space signed distance function but applying volume rendering to train this representation with robustness. UNISURF 300 a concurrent work, also learns an implicit surface via volume rendering. It improves the reconstruction quality by shrinking the sample region of volume rendering during the optimization. Our method differs from UNISURF in that UNISURF represents the surface by occupancy values, while our method represents the scene by an SDF and thus can naturally extract the surface as the zero-level set of it, yielding better reconstruction accuracy than UNISURF, as will be seen later the experiment section.
3 Method
Given a set of posed images {Ix} of a 3D object, our goal is to reconstruct the surface S if. The surface is represented by the zero-level set of neural implicit SDF. In order to learn the weights of the neural network, we developed a novel volume rendering method to render images from the implicit SDF and minimize the difference between the rendered images and the input images. This volume rendering approach ensures robust optimization in NeuS for reconstructing objects of complex structures.
3.1 Rendering Procedure
Scene representation. With NeuS, the scene of an object to be reconstructed is represented by two functions: f R + R that maps a spatial position x E R3 to its signed distance to the object, and e: 3 x g² - R that encodes the color associated with a point x E TR3 and a viewing direction E $ Both functions are encoded by Multi-layer Perceptrons (MIP). The surface S of the object represented by the zero-level set of its SDF, that is.
S = fx ff(x)
In order to apply a volume rendering method to training the SDF network, we first introduce a probability density function (x)),, called S-density, where f(x), R3, is the signed distance
3

---

function and (x(x) se
se s1 +1 e= commonly known as the logistic density distribution, is (2). In principle (x) can be any unimodal (i.e. bell-shaped) density distribution centered at 0; here we choose the
the derivative of the Sigmoid function (,(x) te - ie.. 0,(x)
logistic density distribution for its computational convenience. Note that the standard deviation of (xrr) is given by 1/s, which is also trainable parameter, that is, 1/8 approaches to zero as the network training converges.
Intuitively, the main idea of NeuS is that, with the aid of the S-density field (x)),)), volume rendering used to train the SDF network with only 2D input images as supervision. Upon successful minimization of loss function based on this supervision. the zero-level set of the retwork-encodeer SDF is expected to represent an accurately reconstructed surface S, with its induced S-density (f))) assuming prominently high values near the surface.
Rendering. To learn the parameters of the neural SDF and color field, advise volume rendering scheme to render images from the proposed SDF representation. Given a pixel, we denote the ray emitted from this pixel (p(t) + tv|t 2 0}, where is the center of the camera and is the unit direction vector of the ray. We accumulate the colors along the ray by
[(o.v)= w(t)c(p(t), v)de,
(2)
where Cooo, v) is the output color for this pixel, wz(t) a weight for the point p(t), and c(p(t), v) the color at the point p along the viewing direction
Requirements on weight function. The key learn an accurate SDF representation from 2D images is to build an appropriate connection between output colors and SDE, i.e., to derive an appropriate weight function w(l) on the ray based on the SDF f of the scene. In the following, we list the requirements on the weight function w(l).
1. Unbiased. Given camera ray p(t), w(t) attains a locally maximal value at a surface intersection point p(t"), i.c. with fp((t") = 0, that is, the point p(t*) is on the zero-level set of the SDF (x).
2. Occlusion-awan 0, w(4)) > 0, and lo < 4, there is w((o)= > w(()). That is, when two points have the same SDF value (thus the same SDF-induced S-density value), the point nearer to the view point should have a larger contribution to the final output color than does the other point.
Given any two depth values in and I satisfying f(lo f(1),, w(4o) >
An unbiased weight function w(t) guarantees that the intersection the camera ray with the zero-level set
700
f(t)
of SDF contributes most to the pixel color. The occlusion-awar ensures that when a ray sequentially passes multiple surfaces, the render- ing procedure will correctly use the color of the surface nearest to the cam- era to compute the output color.
w(t)
w(t)
m(t)
P(()
property
Ray
Ray
Next, we will first introduce a naive way of defining the weight function w((t) that directly using the standard pipeline of volume rendering, and ex- plain why it is not appropriate for re- construction before introducing our novel construction of w().
SDF
SDF
Surface
Surface
(a) Neive Solution
(b) Our Solution
Figure 2: Illustration of (a) weight bias of naive solution, and (b) the weight function defined in our solution, which is unbiased in the first-order approximation of SDF.
Naive solution. To make the weight function occlusion-aware natural solution is based on the standard volume rendering formula- tion 28 which defines the weight function by
w(() (I)
where (t) is the so-called volume density in classical volume rendering and T(t) exp(- So (u))du) here denotes the accumulated transmittance along the ray. To adopt the stan-

---

dard volume density formulation (28), here (t) is set to be equal to the S-density value, i.c. (t) (f()tt())) and the weight function w(t) is computed by Eqn. B Although the result- ing weight function is occlusion-aware, it is biased as it introduces inherent errors in the reconstructed surfaces. As illustrated in Fig- 2 (a), the weight function w(t) attains a local maximum at a point before the ray reaches the surface point p(t"), satisfying (p((t)) 0. This fact will be proved in the supplementary material.
Our solution. To introduce our solution, we first introduce a straightforward way to construct an unbiased weight function, which directly uses the normalized S-density weights
Φ (f(((t))) wtt)) = So f (>(((p(u))du
(4)
This construction of weight function is unbiased, but not occlusion-aware. For example, if the ray penetrates surfaces, the SDF function will have two zero points on the ray, which leads two peaks on the weight function w(t) and the resulting weight function will equally blend the colors of two surfaces without considering occlusions.
To this end, now we shall design the weight function w(t) that is both occlusion-awarg and unbiased in the first-order approximation of SDF, based on the aforementioned straightforward construction. To ensure an occlusion aware property of the weight function w(t), we will still follow the basic framework of volume rendering Eqn. 3 However, different from the conventional treatment as in naive solution above, we define our function w(() from the S-density in a new manner. We first define opaque density function p(!), which is the counterpart of the volume density in standard volume rendering. Then we compute the new weight function vc(1) by
w(t) = (t)) where T(t)= = cxp (-% p(u))u)
(5)
How we derive opaque density p. We will first consider a simple case where there is only one surface intersection, and the surface is simply a plane. Since Eqn. 4 indeed satisfies the above requirements under this assumption, we derive the underlying opaque density corresponding to the weight definition of Eqn. 4 usiig the framework of volume rendering. Then we will generalize this opaque density to the general case of multiple surface intersections.
Specifically, in the simple case of single plane intersection, it is easy to see that the signed distance function of (p(()) is co) (t - t"), where (P((t)) = 0, and / is the angle between the view direction and the outward surface normal vector Because the surface is assumed locally, | cos( a constant. It follows from Eqn. that
(f(((()))
wo(l de (f(())))
Φ. (f(((tt))
&, (-]cos(0) (u - )d
(6)
(.(((((t))
(00s())-1 1 (u - t*
((u-t))du
Recall that the weight function within the framework of volume rendering is given by w(t) (t)),(t), where T(t) = exp(- fo p(u)du) denotes the accumulated transmittance. Therefore, to derive p(t), we have
((t) | coos())l
Since T(t) exp(- So du)) is easy verify that T(t)p(t) - (t). Further, note that | cos(()) @x (f(((tt)) de, (f()())). It follows that di' dt (t) de (f(p(t)). Integrating both
dē,
sides of this equation yields
T(t) 4. (f(((t)))
(8)
5

---

Taking the logarithm and then differentiating both sides, we have
[ p(u)du In (f
40, dr (f(((t))
p(t) D. (x(f(p(t))
This is the formula of the opaque density p(t) in case of single plane intersection. The weight function w(t) in- duced by p(t) is shown in Figure Now generalize the opaque density to the general case where there are multiple surface intersections along the ray p(t). In this case, dd (ft)) becomes negative the segment of dt
Vidble
Ira-ssblee
Surface
Surfann
Outside
Imsile
Object
Ray
the ray with increasing SDF values. Thus we clip against zero ensure that the value of is always non-negative. This gives the following opaque density function p(l) in general cases.
SDF
Visible Rang
Visible
Rang
p(t) max (f(P(t) 2.(s(p())) o)
Weight W(E)
(10)
Based on this equation, the weight function wt() can be computed with standard volume rendering as Equ. [ The illustration in the case of multiple surface intersection
Figure 3: Illustration of weight
distribution in case of multiple surface intersection.
shown in Figure 3
The following theorem states that in general cases (i.e., including both single surface intersection and multiple surface intersections) the weight function defined by Eqn. [0 and Eqn. [5 unbiased in the first-order approximation of SDF, The proof is given the supplementary material.
Theorem Suppose that a smooth surface is defined by the zero-level set of the signed distance function fxx O. and ray p(t) + tv enters the surface from outside inside, with the intersection point at p(t)), that is, f(p(t*)) = 0 and there exists an interval [t,, ty] such that t" E [fr, and (p(()) is monotonically decreasing in [1, Suppose that in this local interval (tr, the surface can be tangentially approximated by sufficiently small planar patch, i.e. V is regarded as fixed. Then, the weight function w(t) computed by Eqn. and Eqn. in [t, tr] attains
its maximum at t"
Discretization. To obtain discrete counterparts of the opacity and weight function, we adopt the same approximation scheme as used in NeRF (28). This scheme samples n points {p: = o t 1. n, t < t++ along the ray to compute the approximate pixel color of the ray as
Tiac,
(1I)
where T; is the discrete accumulated transmittance defined by T, I1 aa)) and a, is discrete opacity values defined by
(-/("t
phii p(t)ct)
04 1=
(12)
which can further be shown to
L (f .0) 4 -
0: max
(13)
(.(f(p(t)))
The detailed derivation of this formula for a, is given in the supplementary material.

---

3.2 Training
To train NeuS, we minimize the difference between the rendered colors and the ground truth colors, without any 3D supervision. Besides colors, we can also utilize the masks for supervision if provided.
Specifically, optimize our neural networks and inverse standard deviation by randomly sampling a batch of pixels and their corresponding rays in world space P = {Ck, Mg, oą, V& where Ck is its pixel color and Mr E (0, 1} is its optional mask value, from image in every iteration. We assume the point sampling size is n and the batch size m. The loss function is defined as
L= Ceolor + >rreg + mmmaek
(14)
The color loss Leotor defined as
Eootor
1
Σ R(Ck, Ck).
(15)
k
Same as IDR(46, we empirically choose R as L1 loss, which in our observation is robust to outliers and stable in training.
We add an Eikonal term 10 on the sampled points to regularize the SDF of So by
Lreg
((IVS(DH,))l 1)
(16)
The optional mask loss mmak is defined as
Lmask BCE(ME; Ok);
(17)
where Ô₂ = Em= i== T is the sum of weights along the camera ray, and BCE is the binary cross entropy loss.
Hierarchical sampling. In this work, we follow a similar hierarchical sampling strategy as in NeRF (28) We first uniformly sample the points on the ray and then iteratively conduct importance sampling on top of the coarse probability estimation. The difference that, unlike NeRF which simultaneously optimizes coarse network and fine network, only maintain the probability in coarse sampling is computed based on the S-density (f()))) with fixed standard deviations while the probability of fine sampling is computed based on o (x()) with the learned Details of hierarchical sampling strategy are provided supplementary materials.
network, where
Experiments
4.1 Experimental settings
Datasets. To evaluate our approach and baseline methods, we use 15 scenes from the DTU dataset [Л]. same as those used in IDR [46), with wide variety of materials, appearance and geometry, including challenging cases for reconstruction algorithms, such as non non-Lambertian surfaces and thin structures. Each scene contains 49 or 64 images with the image resolution of 1600 x 1200. Each scene was tested with and without foreground masks provided by IDR (46] We further tested on 7 challenging scenes from the low-res set of the BlendedMVS dataset [5) License). Each scene has 31 143 images at 768 x 576 pixels and masks are provided by the BlendedMVS dataset. We further captured two thin objects with 32 input images to test our approach on thin structure reconstruction.
Baselines. (1) The state-of-the-art surface rendering approach - IDR 46 IDR can reconstruct
surface with high quality but requires foreground masks supervision; Since IDR has demonstrated superior quality compared to another surface rendering based - 22), we did not conduct comparison with DVR. (2) The state-of-the-an volume rendering approach NeRF (88:: We use a threshold of 25 to extract mesh from the learned density field. We validate this choice in the supplementary material. (3) A widely-used classical MVS
355) We reconstruct mesh from the output point cloud of COLMAP with Screened Poisson Surface Reconstruction [16]. (4) The concurrent work which unifies surface rendering and volume rendering with an occupancy field as scene representation UNISURF 30). More details of the baseline methods are included in the supplementary material.
7

---

Implementation details. We assume the re- gion of interest is inside a unit sphere. We sam- ple 512 rays per batch and train our model for
w/ mask
w/o mask
ScandD
IDK
NeRF
Ours
COLMAP
NeRF
UNISURF
Dus
scun24
1.3
1.83
0.83
0.81
1.50
1.32
1.00
scan37
1.87
0.98
scan40
300k iterations for hours (for the 'w/ mask'
scan55
0.88
0.37
0,43
setting) and 16 hours (for the 'w/o mask' setting) on a single NVIDIA RTX2080Ti GPU. For the "w/o mask' setting, model the background by NeRF++ [50]. Our network architecture and initialization scheme are similar to those of IDR [46]. More details of the network archi- tecture and training parameters be found in
scan63
1.04
1.79
1:13
1.79
2.28
1.35
1.10
scan65
0.79
1.44
0.59
1.58
1.27
0.79
0.65
scanñ99
0.77
1.50
0.60
1.02
1.47
0.80
0.57
1.33
1.20
1:45
3.05
1,45
1.16
0.67
scan100
0.90
2.61
1.43
1.32
2.53
1.47
1.20
scam14
0.42
1,04
9.36
0.49
1.06
0.46
0.35
scan118
0.51
1.13
0.45
0.78
1.15
0.59
0.49
scun122
0.53
0.99
0.45
0.54
the supplementary material.
Table Quantitative evaluation on DTU dataset. COLMAP results are achieved by trim=0.
4.2 Comparisons
We conducted the comparisons in settings, with mask supervision (w/ mask) and without mask supervision (w/o mask). We measure the reconstruction quality with the Chamfer distances the same way as UNISURF [3] and IDR [46] and report the scores in [ The results show that our approach outperforms the baseline methods on the DTU dataset in both settings w/ and w/o mask in terms of the Chamfer distance. Note that the reported scores of IDR in the setting of w/ mask and NeRF and UNISURF in the w/o mask setting are from IDR 46 and UNISURF 30
We conduct the qualitative comparisons on the DTU dataset and the BlendedMVS dataset in both settings, w/ mask and w/o mask, in Figure and Figure 5 respectively. As shown in Figure for the setting of w/ mask, IDR shows limited performance for reconstructing thin metals parts in Scan 37 (DTU), and fails to handle sudden depth changes in Stone (BlendedMVS) due the local optimization process in surface rendering. The extracted meshes of NeRF are noisy since the volume density field has not sufficient constraint on its 3D geometry. Regarding the w/o mask setting, we visually compare our method with NeRF and COLMAP in the setting of w/o mask in Figure 5 which shows our reconstructed surfaces are with more fidelity than baselines. We further show comparison
Sean 24
(D10
(DTUU)
Scan
Dog (BlendedMMVS)
(BlendedM
Stone
Reference Image
Ours
IDR
NeRF
Figure 4: Comparions on surface reconstruction with mask supervision.

---

Scan 69
(BlendedM (DTU)
Clock
(BlendedMVS3
Sculpture
Bear (BlendedM
Reference Image
Ours
NeRF
COLMAP
Figure 5: Comparions on surface reconstruction without mask supervision.
with UNISURF 30 on two examples in the w/o mask setting. Note that we use the qualitative results of UNISURF reported their paper for comparison. Our method works better for the objects with abrupt depth changes. More qualitative images are included in the supplementary material.
4.3 Analysis
Ablation study.
To evaluate the effect of the weight calculation, we test three different kinds of weight constructions described in Sec. [3.1; (a) Naive Solution. (b) Straightforward Construction shown in Egn. 4 (e) Full Model. As shown
Figure the quantitative result of naive solution worse than our weight choice (e) in terms of the Chamfer distance. This is because introduces bias the surface reconstruction. If direct construction used, there are severe artifacts.
(a) Naive Solution (b) Direct Solution (c) wo Eikonal
(d) w/ Gno-Int
(e) Full Model
Reference Image
1.49
4.45
0.64
0.62
0.59
Chamfer Distance
1.75
44.34
88.94
6.19
0.93
MAE
Figure 6: Ablation studies. We show the qualitative results and report the quantitative metrics in Chamfer distance and MAE (mean absolute error) between the ground-trute and predicted SDF values.

---

We also studied the effect of Eikonal regulariza- tion [T0] and geometric initialization [T. With- out Eikonal regularization or geometric initial- ization, the result on Chamfer distance on par with that of the full model. However, neither of them can correctly output signed distance function. This is indicated by the MAE(mean absolute error) between the SDF predictions and corresponding ground-truth SDF, shown in the bottom line of Figure 6 The MAE is com- puted uniformly-sampln points in the ob- ject's bounding sphere. Qualitative results of SDF predictions provided in the supplementary material.
Refermcc Image
UNISURF
Figure 7: Visual comparisons with UNISURE.
Thin structures. We additionally show results on two challenging thin objects with 32 input images. The plane with rich texture under the object is used for camera calibration. As shown in Fig. our method is able to accurately reconstruct these thin structures. especially on the edges with abrupt depth changes. Furthermore, different from the methods (39 19] 42 20 which only target at high-quality thin structure reconstruction, our method can handle the scenes which have mixture of thin structures and general objects.
Reference Image
(a) Ours
(b) COLMAPotte
(c) COLMA
Figure 8: Comparison on scenes with thin structure objects. Left half is the depth map while right half the reconstructed surface.
Conclusion
We have proposed NeuS, a new approach to multiview surface reconstruction that represents 3D surfaces neural SDF and developed representation. NeuS produces high-quality reconstruction and successfully reconstructs objects with severe occlusions and complex structures. outperforms the state-of-the-an both qualitatively and quantitatively. One limitation of our method is that although our method does not heavily rely on correspondence matching of texture features, the performance would still degrade for textureless objects (we show the failure cases in the supplementary material). Moreover, NeuS has only single scale parameter that used model the standard deviation of the probability distribution for all the spatial location. Hence, an interesting future research topic is to model the probability with different variances for different spatial locations together with the optimization scene representation, depending on different local geometric characteristics. Negative societal impact: like many other learning- works, our method requires large amount computational resources for network training, which can be concern for global climate change.
volume rendering method for training the implicit SDF
10

---

Acknowlegements
We thank Michael Oechsle for providing the results of UNISURF Christian Theobalt was supported by ERC Consolidator Grant 770784. Lingjie Liu was supported by Lise Meitner Postdoctoral Fellowship. Computational resources are mainly provided by HKU GPU Farm.
References
[1] Matan Atzmon and Yaron Lipman. Sal: Sign agnostic learning of shapes from raw data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2020.
[2] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A ran- domized correspondence algorithm for structural image editing. ACM Trans. Graph., 28(3):24, 2009.
[3] Adrian Broadhurst, Tom W Drummond, and Roberto Cipolla. A probabilistic framework for space carving. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 1, pages 388-393. IEEE, 2001.
[4] Z. Chen and H. Zhang. Learning implicit fields for generative shape modeling. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5932-5941. 2019.
[5] Christopher B Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: unified approach for single and multi-view 3d object reconstruction. In European conference on computer vision, pages 628 644. Springer, 2016.
[6] Jeremy De Bonet and Paul Viola. Poxels: Probabilistic voxelized volume reconstruction. In Proceedings of International Conference on Computer Vision (ICCV), pages 418-425, 1999.
[7] Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 605-613 2017.
[8] Yasutaka Furukawa and Jean Ponce. Accurate, dense, and robust multiview stereopsis. IEEE transactions on pattern analysis and machine intelligence, 32(8): 1362-1376, 2009.
[9] Silvano Galliani, Katrin Lasinger, and Konrad Schindler. Gipuma: Massively parallel multi- view stereo reconstruction. Publikationen der Deutschen Gesellschaft für Photogrammetrie, Fernerunnung und Geoinformation V, 2,25361-369))22 2016.
[10] Amos Gropp. Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. arXiv preprint arXiv:2002. 10099, 2020.
[11] Rasmus Jensen, Anders Dahl, George Vogiatzis, Engil Tola, and Henrik Aanæs. Large scale multi-view stereopsis evaluation. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 906-413, 2014.
[12] Yue Jiang, Dantong Ji, Zhizhong Han, and Matthias Zwicker. Sdfdifff: Differentiable rendering of signed distance fields for 3d shape optimization. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[13] Abhishck Kar, Christian Häne, and Jitendra Malik. Learning a multi-view stereo machine. arXiv preprint vtXiv:1708.000 05375, 2017.
[14] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3907-3916, 2018.
[15] Srinivas Kaza et al. Differentiable volume rendering using signed distance functions. PhD thesis, Massachusetts Institute Technology. 2019.
[16] Michael Kazhdan and Hugues Hoppe. Screened poisson surface reconstruction. ACM Trans.
Graph., 32(3), July 2013.
[17] Petr Kellnhofer, Lars Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, and Gordon Wetzstein. Neural lumigraph rendering. arXiv preprint 2131=22003.11571.000
11

---

[18] Chen-Hsuan Lin, Chen Kong, and Simon Lucey, Learning efficient point cloud generation for dense 3d object reconstruction. In proceedings of the AAAI Conference on Artificial Intelligence. volume 32, 2018.
[19] Lingjie Liu, Duygu Ceylan, Cheng Lin, Wenping Wang, and Niloy J. Mitra. Image-based reconstruction of wire art. 36(4):63:1 2017.
[20] Lingjie Liu, Nenglun Chen, Duygu Ceylan, Christian Theobalt, Wenping Wang, and Niloy J. Mitra. Curvefusion: Reconstructing thin structures from rghd sequences. 37(6), 2018.
[21] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33, 2020.
[22] Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, and Zhaopeng Cui. Dist: Rendering deep implicit signed distance function with differentiable sphere tracing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2019-2028, 2020.
[23] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. ACM Trunsactions on Graphics (TOG), 38(4):65, 2019.
[24] Priyanka Mandikal, K L Navancet, Mayank Agarwal, and R Venkatesh Babu. 3D-LMNet: Latent embedding matching for accurate and diverse 3d point cloud reconstruction from single image. In Proceedings of the British Machine Vision Conference (BMVC), 2018.
[25] Paul Merrell, Amir Akbarzadeh, Liang Wang, Philippos Mordohai, Jan-Michael Frahm, Ruigang Yang, David Nistér, and Marc Pollefeys. Real-time visibility-based fusion of depth maps. pages 1-8, 01 2007.
[26] Lars Mescheder, Michael Oechsle. Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4460-4470, 2019.
[27] Mateusz Michalkiewicz, Jhony K. Pontes, Dominic Jack, Mahsa Baktashmotlagh,, and An- ders Eriksson. Implicit surface representations as layers in neural networks. In The IEEE International Conference Computer Vision (ICCV), October 2019.
(28) Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference Computer Vision, pages 405-421. Springer, 2020.
[29] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volu- metric rendering: Learning implicit 3d representations without 3d supervision. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3505,
2020.
[30] Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: surfaces and radiance fields for multi-view reconstruction. arXiv preprint 104. 2021.
Unifying neural implicit
[31] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165-174, 2019.
[32] Songyou Peng, Michael Niemeyer, Lars M. Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. ArXiv, Abs/22003.04618 2020.
[33] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao
Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. ICCV, 2019.
[34] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. Pifuhd: Multi-level pixel- aligned implicit function for high-resolution 3d human digitization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 84-93, 2020.
[35] Johannes I. Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. Pixelwise view selection for unstructured milti-view stereo. European Conference on Computer Vision pages 501-518. Springer, 2016.
12

---

[36] Steven M Seitz and Charles R Dyer. Photorealistic scene reconstruction by voxel coloring. International Journal of Computer Vision, 35(2):151-173, 1999.
[37] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2437-2446, 2019.
[38] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure- aware neural scene representations. In Advances in Neural Information Processing Systems, pages 1119-1130, 2019.
[39] A. Tabb. Shape from silhouette probability maps: Reconstruction of thin objects in the presence of silhouette extraction and calibration error. pages 161-168, June 2013.
[40] Alex Trevithick and Bo Yang. Grf: Learning general radiance field for 3d scene representation and rendering. arXiv preprint arXiv:2010.04595, 2020.
[41] Nanyang Wang. Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu- Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European Conference on Computer Vision (ECCV), pages 52-67, 2018.
[42] Peng Wang. Lingjie Liu, Nenglun Chen, Hung-Kuo Chu, Christian Theobalt, and Wenping Wang. Vid2curve: Simultaneous camera motion estimation and thin structure reconstruction from an rgb video. ACM Trans. Graph., 39(4). July 2020.
[43] Chao Wen, Yinda Zhang, Zhuwen Li, and Yanwei Fu. Pixel2mesh++: Multi-view 3d mesh generation via deformation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1042-1051, 2019.
[44] Haozhe Xie, Hongxun Yao, Xiaoshuai Sun, Shangchen Zhou, and Shengping Zhang. Pix2vox: Context-aware 3d reconstruction from single and multi-view images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2690-2698, 2019.
[45] Yao Yao. Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: large-scale dataset for generalized multi-view stereo networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1790-1799, 2020.
[46] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems, 33, 2020.
[47] Wang Yifan, Shihao Wu, Cengiz Oztireli, and Olga Sorkine-Hornune Iso-points: Optimizing neural implicit surfaces with hybrid representations. arXiv preprint arXiv:2012.06434, 2020.
[48] Christopher Zach, Thomas Pock, and Horst Bischof. A globally optimal algorithm for robust tv-11 range image integration. In 2007 IEEE 11th International Conference Computer Vision, pages 1-8, 2007.
[49] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. arXiv preprint AXXv:2104.0067.. 2021.
[50] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
13


Reviews:
[{"review_id": "cCLL9t3nVSP", "reviewer": "Reviewer_MRq8", "summary": "This paper proposes a neural surface reconstruction method which they call called NeuS. The novelty of this method is derived from their use of signed distance functions to represent geometry, and a proposed volume rendering method for accumulating information along rays. With these changes they are better able to estimate depth in more complicated surface geometry, demonstrate clearly superior shape predictions, and achieve state of the art quantitative results on the DTU dataset. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is quite strong. It identified an issue with NERF style volumetric rendering method, and solved  it by representing geometry with an SDF. The proposed volume rendering method which allows for this change is well motivated and explained. Both the qualitative and quantitative results are convincing and compelling. \n\nOne point on concern is the comparison to other methods. I am not an expert in this field but it seems there has been quite an explosion of nerf-like papers since its release, and from a quick lit review of them it seems like NERF would no longer be considered state of the art in novel view synthesis, as referenced in the baselines section. NERF was released only last year and so there is leeway here for concurrent, or close to concurrent research, but I would perhaps revise this claim. If you wanted to raise my score higher it might be nice to compare directly to papers like \"Neural Sparse Voxel Fields\" which I see you reference, or other more recent papers which seem to tackle the same shortcomings in the original paper. \n\nother comments: \n- lines 67 to 70 are true of some methods but not all. This overly general statement is misleading. \n- There needs to be some editing for grammar, especially in the intro section. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.\n", "review_text": "", "extra_scores": {"ethical_concerns": "No. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "Dt8IvwLpXGz", "reviewer": "Reviewer_Hvd1", "summary": "This paper presents a method for implicit 3D surface reconstruction from posed 2D images, where the 3D surface is represented with an SDF. The authors advocate a new way of using neural volume rendering methods for surface reconstruction, where the density field is induced by the optimized SDF instead of direct MLP outputs (as in e.g. NeRF). In addition, the authors described two key properties that should be satisfied for volume rendering with SDF representations and proposed a novel solution. Experimental results on the DTU multiview dataset shows its advantage over state-of-the-art baseline methods.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The paper is very well-written, with a clear problem statement and a thorough overview of related literature.\n+ To take advantage of the nice optimization properties in neural volume rendering methods e.g. NeRF, the authors pointed out two important properties which should be satisfied for extracting surfaces with SDF as the representation: unbiasedness and occlusion-awareness. The authors suggested two simple solutions satisfying only each of the property, and then propose a nice solution to blend the two together. This makes a complicated concept easy to understand. The authors also provided a nice theoretical proof on how the proposed solution satisfy both properties.\n+ The proposed method NeuS is compared with extensive recent baseline methods and significantly outperforms both qualitatively and quantitatively. Ablation studies and failure cases are also provided, which helps understand the contribution of each component.\n\nWeaknesses:\n- It was mentioned that a proof on how Eq. 6 is not unbiased would be provided, but only an empirical example was provided in the supplementary materials. It is still unclear to me how the construction of Eq. 6 is not unbiased.\n- Although the results are superior, it would be helpful to provide a comparison on the training/inference speed. Since NeuS is still a volume rendering method with hierarchical sampling as adopted in NeRF, I would expect it to be almost as slow as NeRF to render. How slow would NeuS be compared to IDR?\n- It would be good to also evaluate how well the recovered 3D representation actually satisfy an SDF, as SDF recovery is also one main goal of the paper. This would be important since efficient rendering (e.g. with sphere tracing) and relighting could be applied on true SDF shapes. The current evaluation seem to focus on 3D shape reconstruction in terms of surface accuracy (measured by Chamfer distances). In Fig. 4 of the supplementary material, the exterior region does not seem to correctly satisfy an SDF. \n\nI think it would also be interesting to see whether Neus would be able to reconstruct surfaces on forward-facing scenes, such as in the LLFF dataset originally considered in NeRF. Does the reparametrization of coordinates using NDC affect the solution?\n\nOther minor problems:\n- It is unclear what $s$ in L123 -- is it a hyperparameter that should be tuned?\n- $\\delta_i$ is undefined in Eq. 6. (I'm guessing it corresponds to $t_{i+1}-t_i$?)\n- L183: the equation seems off, as $\\bar{\\phi} = \\phi_s \\circ f$ (Eq. 2).\n- L192: the differential is missing.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequately addressed", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4hr"}, "extra_text": {}}, {"review_id": "DIRD4kRt2EP", "reviewer": "Reviewer_f9A2", "summary": "This paper proposes a method that combines volume rendering with surface rendering techniques for multi-view reconstruction. The method represents the geometry of objects using a neural implicit representation that outputs the SDF at query locations. In order to enable volume rendering, the output SDF is mapped to a density function centered around the surface. Furthermore, they also encode the appearance of the object into the neural implicit representation that is needed for multi-view reconstruction. The main contribution is a novel weight function that is centered (unbiased) around the surface while being occlusion-aware. Therefore, the paper resolves major limitations of volume rendering (to few constraints on surface) and surface rendering (very sparse gradients). The paper extensively discusses different formulations of the weight function (properties, weaknesses, strengths). The method is evaluated on the DTU as well as the BlendedMVS dataset and compared to existing state-of-the-art methods like NeRF (volume rendering) and IDR (surface rendering). Moreover, they evaluate some aspects of the method in an ablation study.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Here, I order the strengths and weaknesses according to their importance and contribution to the final score as well as additional comments that should be resolved in the final version of the paper.\n\n#Strengths:\n\n1. The paper proposes a simple yet elegant formulation for multi-view surface reconstruction using neural implicit representations. The results are impressive and show that it is very beneficial to combine volume rendering and surface rendering to solve the well known problems of noisy surfaces (NeRF) and requiring a mask for supervision (IDR). This is particularly shown in Figure 6.\n\n2. The paper extensively discusses the strengths and limitations of existing weight functions used in volumetric rendering. Especially, the illustrations in Figure 2 and Figure 3 are very useful to understand the problem and the proposed solution. \n\n3. The introduction as well as the related work very well motivate the paper with respect to existing surface and volume rendering methods. This helps the reader to understand the importance of the problem and the proposed solution.\n\n4. By providing an intuition behind different weight functions, the different solutions and their strengths/weaknesses are very well motivated. \n\n#Weaknesses:\n\n1. Clarity:\nThe paragraph, where the solution is explained (L185 - L193) is very unclear. It is very hard to understand how the solution is derived from Equation 5 and Equation 6. This should be definitely made clearer for the final version of the paper as it is crucial for the reader to understand the derivation of the main contribution. I would like to see how you derive and combine equation 7 from equation 5 and 6. This could then be also added to the supplementary material for the final paper with an intuition in the main paper.\n\n2. Results/Ablations \n\n- Firstly, as you also encode the appearance and especially NeRF is primarily designed for novel view synthesis, it would be interesting to see how this method performs on novel view synthesis. For doing this, I suggest to visualize novel views from the scenes and also run a quantitative evaluation using PSNR and SSIM.\n\n- Secondly, as you already compare to UNISURF, it would also be interesting to see the experiments on the Indoor Scene Dataset as shown in UNISURF (Figure 8, p. 8).\n\n- Thirdly, one missing experiment is the ablation of all loss terms. I would be particularly interested in an ablation of the Eikonal term to see whether this is also the reason for the better performance compared to UNISURF. \n\n\n3. Related Work:\nThere are some relevant works missing on multi-view reconstruction. Particularly, the two works below that are one of the first works using machine learning for multi-view reconstruction.\n\n- Kar et al., Learning a multi-view stereo machine, NeurIPS 2017\n- Choy et al., 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction, ECCV 2016\n\nIn the section about Neural Implicit Representations, the following work is missing:\n\n- Sitzmann et al., DeepVoxels: Learning Persistent 3D Feature Embeddings, CVPR 2019\n\nMoreover, I do only partially agree with the discussion of the differences between this work and the concurrent work UNISURF. Although it is correct that UNSIRF represents the geometry as occupancy values, you can also naturally extract the surface at the decision boundary (0.5), which is just a different level set. I doubt that this difference is the reason for the performance improvement over UNISURF. A more in depth discussion would be helpful for the community to understand what improves performance and what does not. \n\n4. Lacking justification of certain design choices:\n\n- To me, the choice of the logistic density distribution is not motivated enough. I assume the main benefit is the easy integration to the sigmoid function that is used in the final solution (Equation 7). I would like to see a brief comment on that. \n\n\n5. Clarity/Flow of Reading\nThe clarity of the writing and flow of reading is limited by very long sentences (e.g. Abstract L11 - L14), obscure formulations (e.g. L87 'which makes the gradient only be backpropagated to a local region near the intersection', L118 'neural networks of Multi-layer Perceptron (MLP), L), and wrong/inconsistent prepositions (e.g. requirements of weight function (L147-L148) vs. requirements on weight function (L149)). This significantly breaks the flow while reading the paper and makes it harder to understand everything.\n\n\n#Additional Comments:\n\n- The writing of the paper should be reviewed and revised in general (uncountable vs. countable nouns, third person s, prepositions, missing or too many a/the (e.g. L56, L86), etc.). This would make the reading much easier and redirects the mental capacity of the reader to the content. \n\n- A few formulations could be improved for better clarity.\n\t- L31 - L32: would get optimization stuck --> other formulation \n\t- L100: sample region --> sampling region \n\t- L108: zero-set --> zero level set\n\t- L111: train the network of SDF --> other formulation\n\t- L120: method to training the SDF network --> method to train the SDF network\n\t- L134: To learn the parameters of MLPS of the SDF and the color field --> To learn the parameters of the SDF and color networks (or something similar)\n\t- L157 - 158: near the view point --> closer to the view point\n\t- Theorem 1: depth --> depth values\n\t- Theorem 1: going from the outside of the surface --> going from the outside of the object (there is no inside outside of a surface)\n\n- It would be very good to visualize the meshes (especially in Figure 7) with some sort of error color coding (similar to Tatarchenko et al. 2019, Figure 12). This would guide the reader to better see where the different configuration fail/work.\n\n- Reference 5 and 6 are the same. They should be merged.\n\n- The failure case in the appendix is not really informative (Fig. 6). Some more failure cases caused by lack of texture should be shown together with the input views (to see the lack of texture). Guiding the reader more (directly in the figure) would help as well. \n\n\n\n\n#Questions\n\n- What is the reason for using different sampling points for the color and alpha values? (Appendix L57-59)\n- Why do the results for some scenes get worse when using the mask supervision? What is the intuition behind that? (Table 1)\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "They are discussed sufficiently.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "BS5uFVAUyYp", "reviewer": "Reviewer_3fHi", "summary": "This paper presents a novel multi-view reconstruction model that couples an implicit SDF representation (as in IDR [31]) with an unbiased volumetric rendering function (as in NeRF [20]). The volumetric rendering function is deliberately \"de-biased\" by redefining the opacity values $\\alpha_i$ such that the weight is maximized exactly on the surface defined by the zero level set of the SDF.\n\nThe resulting model enjoys the benefits of both surface-based representation for high-fidelity surface reconstruction, and dense volumetric rendering which provides effective gradients that are not restricted locally to the surface and facilitate surface details. The reconstructed surfaces are much better compared to IDR and NeRF. This result can potentially also be very useful for further downstream applications that require smooth effective gradient signals.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n### S1 - method\nThe idea of combining volumetric rendering with SDF is actually very neat, which is better than the analytical gradient used in DVR and IDR that is only defined on surface and hence very local. This is very well implemented by carefully designing the alpha values to observe the surface properties of the SDF, driven by mathematical proofs. The resulting model is clean and easily interpretable.\n\n### S2 - results\nThe reconstructed surfaces are clearly better than existing methods, eg IDR, NeRF, UNISURF, with smooth surfaces (unlike noisy shapes extracted from NeRF) and fine details, handling thin structures well (much better than IDR). The model can also perform reconstruction without object masks without much loss in quality, whereas DVR and IDR require mask supervision.\n\nThorough ablation and analysis results, including visualizations of the training progression, are presented and provide a lot of insights to the model.\n\n### S3 - writing\nThe paper is very well written and highly polished. I really enjoyed reading it. The motivations are clearly explained and illustrated with examples and great figures. Technical descriptions are followed by motivation/intuition and careful proofs, and very well structured such that the readers could easily follow through.\n\n\n## Weaknesses\n### W1 - image synthesis\nThere are no comparison on the image synthesis quality. It seems the rendered image appear less visually appealing compared to NeRF, which however is a trade-off for obtaining explicit surface.\n\n### W2 - (minor comment) more applications\nAlthough the paper is already very complete in itself, it mostly follows the experimental setup of previous method (which is good). There are also many other interesting applications now that the model can obtain smooth, non-local gradients with volumetric rendering while maintaining a surface representation. For example, does that helps with the speed of convergence? Have the authors tried optimizing camera poses from coarse estimates as IDR does, and would it tolerate more noise? Does it help with the cases with fewer views?\n\n## Typos\n- Line 176 & 183: $\\bar{\\phi}$ should be $\\phi$\n- Line 152: remove \"{\"\n- Line 162: \"a unbiased\" -> \"an unbiased\"\n\n\n---\n## Post-Rebuttal\nI appreciate the authors' efforts in the rebuttal. This is a solid submission with a novel method and great results. I will keep my original rating.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A brief example of slightly inaccurate reconstructions is provided in the supplementary material. Similar imperfections are also visible in other examples, such as in purely specular or saturated regions. But I wonder whether there are more extreme major cases.\n", "review_text": "", "extra_scores": {"ethical_concerns": "I do not see immediate concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presented a novel idea of combining volumetric rendering with 3D surface representation (SDF) and achieved SOTA results. All the reviewers liked the paper and gave very constructive comments. Please include the results on novel view synthesis, evaluation of SDF quality, details of training  & inference time in the final version. Also, please carefully proof read the paper.
Decision: Accept (Spotlight)