Volume Rendering of Neural Implicit Surfaces
Authors: Lior Yariv
Jiatao Gu
Yoni Kasten
Yaron Lipman
Venue: NeurIPS 2021 Oral (2021)
Keywords: neural volume rendering
implicit neural representations

Abstract: Neural volume rendering became increasingly popular recently due to its success in synthesizing novel views of a scene from a sparse set of input images. 
So far, the geometry learned by neural volume rendering techniques was modeled using a generic density function. Furthermore, the geometry itself was extracted using an arbitrary level set of the density function leading to a noisy, often low fidelity reconstruction.
The goal of this paper is to improve geometry representation and reconstruction in neural volume rendering. We achieve that by modeling the volume density as a function of the geometry. This is in contrast to previous work modeling the geometry as a function of the volume density. 
In more detail, we define the volume density function as Laplace's cumulative distribution function (CDF) applied to a signed distance function (SDF) representation. This simple density representation has three benefits: (i) it provides a useful inductive bias to the geometry learned in the neural volume rendering process; (ii) it facilitates a bound on the opacity approximation error, leading to an accurate sampling of the viewing ray. Accurate sampling is important to provide a precise coupling of geometry and radiance; and (iii) it allows efficient unsupervised disentanglement of shape and appearance in volume rendering.
Applying this new density representation to challenging scene multiview datasets produced high quality geometry reconstructions, outperforming relevant baselines. Furthermore, switching shape and appearance between scenes is possible due to the disentanglement of the two. 

Volume Rendering of Neural Implicit Surfaces
Lior Yariv!
Jiatao Gu
Yoni Kasten'
Yaron Lipman" 1.2
Weizmann Institute of Science
Facebook Al Research
Abstract
Neural volume rendering became increasingly popular recently due to its success in synthesizing novel views scene from a sparse set of input images. So far, the geometry learned by neural volume rendering techniques a generic density function. Furthermore, the geometry itself was extracted using arbitrary level set of the density function leading reconstruction. The goal of this paper is to improve geometry representation and reconstruction in neural volume rendering. We achieve that by modeling the vol- ume density as function of the geometry. This is in contrast to previous work modeling the geometry as function of the volume density. In detail, we define the volume density function as Laplace's ' cumulative distribution function (CDF) applied signed distance function (SDF) representation. This simple density representation has three benefits: (i) it provides useful inductive bias the geometry learned in the neural volume rendering process; (ii) facilitates a bound on the opacity approximation error, leading to an accurate sampling of the viewing ray. Accurate sampling is important to provide precise coupling geometry and radiance; and (iii) it allows efficient unsupervised disentanglement shape and appearance volume rendering. Applying this new density repre- sentation to challenging scene multiview datasets produced high quality geometry reconstructions, outperforming relevant baselines. Furthermore, switching shape and appearance between scenes is possible due to the disentanglement of the two.
modeled using
noisy, often low fidelity
Introduction
Volume rendering [8] is set of techniques that renders volume density in radiance fields by the so called volume rendering integral. It has recently been shown that representing both the density and radiance fields as neural networks can lead to excellent prediction of novel views by learning only from sparse set of input images. This neural volume rendering approach, presented in [21 and approximates the integral alpha-composition in a differentiable way, allowing to learn simultancously both from input images. Although this coupling indeed leads to good generalization of novel viewing directions, the density part is not as successful in faithfully predicting the scene's actual geometry, often producing noisy, low fidelity geometry approximation.
developed by its follow-ups 34 2
We propose VoISDF to devise different model for the density in neural volume rendering, leading to better approximation of the scene's 's geometry while maintaining the quality of view synthesis. The key idea is to represent the density as a function of the signed distance to the scene's 's surface, see Figure Such density function enjoys several benefits. First, guarantees the existence of well-defined surface that generates the density. This provides useful inductive bias for disentangling density and radiance fields, which in turn provides more accurate geometry approximation. Second, show this density formulation allows bounding the approximation error of the opacity along rays. This bound is used to sample the viewing ray so to provide a faithful coupling of density and radiance field in the volume rendering integral. E.g., without such bound the computed radiance along a ray (pixel color) can potentially miss or extend surface parts leading to incorrect radiance approximation.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Figure I: VoISDF: given a set of input images (left) we learn a volumetric density (center-left, sliced) defined by signed distance function (center-right, sliced) to produce neural rendering (right). This definition of density facilitates high quality geometry reconstruction (gray surfaces, middle).
A closely related line of research, often referred to neural implicit surfaces [2 38 14,, have been focusing on representing the scene's s geometry implicitly using a neural network, making the surface rendering process differentiable. The main drawback of these methods is their requirement of masks that separate objects from the background. Also, learning to render surfaces directly tends to grow extraneous parts due to optimization problems, which are avoided by volume rendering. In sense, our work combines the best of both worlds: volume rendering with neural implicit surfaces.
We demonstrate the efficacy of VoISDF by reconstructing surfaces from the DTU [12 and Blended- MVS 37 datasets. VoISDF produces more accurate surface reconstructions compared to NeRF (21) and NeRF+++ [39], and comparable reconstruction compared to IDR 38] while avoiding the use of object masks. Furthermore, we show disentanglement results with our method, i.c., switching the density and radiance fields of different scenes, which is shown to fail in NeRF-basee models.
2
Related work
Neural Scene Representation & Rendering Implicit functions are traditionally adopted in modeling Recent studies have been focusing on model implicit functions with multi-layer perceptron (MLP) due its expressive representation power and low memory foot-print, including
3D scenes 24
scene (geometry & appearance) representation 9 20 23
25
29
36
28
35)
and free-view
rendering 33 16
39 34 21. In particular, NeRF 21 has opened up a line of
research (see 6 for an overview) combining neural implicit functions together with volume rendering to achieve photo-realistic rendering results. However, is non-trivial find a proper threshold to extract surfaces from the predicted density, and the recovered geometry is far from satisfactory. Furthermore, sampling of points along a ray for rendering a pixel is done using an opacity function that is approximated from another network without any guarantee for correct approximation.
Multi-view 3D Reconstruction Image-based 3D surface reconstruction (multi-view stereo) has been a longstanding problem in the past decades. Classical multi-view stereo approaches are generally either depth-based [1 31 8 7 or voxcl-based [5 32 For instance, in COLMAP [3]] typical depth-based method) image features are extracted and matched across different views to estimate depth. Then the predicted depth maps are fused to obtain dense point clouds. To obtain the surface, an additional meshing step e.g. Poisson surface reconstruction [33 is applied. However, these methods with complex pipelines may accumulate errors at each stage and usually result in incomplete 3D models, especially for non-Lambertian surfaces they can not handle view dependent colors. On the contrary, although it produces complete models by directly modeling objects in a volume, voxel-based approaches are limited to low resolution due to high memory consumption. Recently, neural-based approaches such as DVR (22), IDR (38), NLR [4] have also been proposed to reconstruct scene geometry from multi-view images. However, these methods require accurate object masks and appropriate weight initialization due to the difficulty of propagating gradients.
Independently from and concurrently with our work here, 27 also use implicit surface representation incorporated into volume rendering. In particular, they replace the local transparency function with an occupancy network [19] This allows adding surface smoothing term to the loss, improving the quality of the resulting surfaces. Differently from their approach, we use signed distance representation, regularized with an Eikonal loss [38 10 without any explicit smoothing term. Furthermore, we show that the choice of using signed distance allows bounding the opacity approximation error, facilitating the approximation of the volume rendering integral for the suggested family of densities.

---

3 Method
In this section we introduce a novel parameterization for volume density, defined as transformed signed distance function. Then show how this definition facilitates the volume rendering process. In particular, we derive bound of the error in the opacity approximation and consequently devise a sampling procedure for approximating the volume rendering integral.
3.1 Density as transformed SDF
Let the set S C R3 represent the space occupied by some object in R and M 0Ω its boundary surface. We denote by la the Ω indicator function, and by de the Signed Distance Function (SDF) to its boundary M,
{
if E
(x(x)
0 if & Ω2 and do(x) (-1))a(m) min - yl1 yEM
(1)
where ||-[| is the standard Euclidean 2-norm. In neural volume rendering the volume density IR3 + IR., a scalar volumetric function, where (xx) is the rate that light is occluded at point æ; a is called density since it is proportional to the particle count per unit volume at [18]. In previous neural volumetric rendering approaches 21 15 39], the density function, a, was modeled with a
general-purpose Multi-Layer Perceptron (MLP). In this work we suggest to model the density using certain transformation of a learnable Signed Distance Function (SDF) dg, namely
σ (x) = az (-dq(x)),
where oz, are learnable parameters, and 1 is the Cumulative Distribution Function (CDF) of the Laplace distribution with zero mean and (3 scale (i.e., mean absolute deviation, which is intuitively the L version of the standard deviation).
1 exp
if 0
2
1 exp
0
Figure [ (center left and right) depicts an example of such a density and SDF. As can be readily checked from this definition, as | approach zero, the density a converges scaled indicator function of Ω that is a + alg for all points æ E Ω \ .M.
Intuitively, the density models homogeneous object with constant density that smoothly decreases near the object's boundary, where the smoothing amount is controlled by B. The benefit in defining the density in equation 2 is two-fold: First, it provides useful inductive bias for the surface geometry M, and provides principled way to reconstruct the surface, i.e., as the zero level-set of dg. This in contrast to previous work where the reconstruction was chosen an arbitrary level set of the learned density. Second, the particular form of the density as defined in equation
2 facilitates bound the error of the opacity (or, equivalently the transparency) of the rendered volume, a crucial component in the volumetric rendering pipeline. In contrast, such a bound will be hard to devise for a generic MLP densities.
3.2 Volume rendering of a
In this section we review the volume rendering integral and the numerical integration commonly used to approximate it, requiring a set S of sample points per ray. In the following section (Section 3.3). we explore the properties of the density and derive bound on the opacity approximation error along viewing rays. Finally, in Section 3.4 we derive an algorithm for producing sample S to be used the volume rendering numerical integration.
In volume rendering we consider a ray æ emanating from a camera position e E R in direction R3, |||||| 1, defined by xxt) c tv, t Z 0. In essence, volume rendering all about approximating the integrated (i.e., summed) light radiance along this ray reaching the camera. There are two important quantities that participate in this computation: the volume's opacity O, or equivalently, its transperancy T, and the radiance field L.
The transparency function of the volume along ray æ, denoted T, indicates, for each f. 2 0, the probability a light particle succeeds traversing the segment (c, at)) without bouncing off,
T(t)= = exp (-f- (
(4)
3

---

and the opacity O is the complement probability,
O(1) = - (()).
(5)
Note that O is a monotonic increasing function where (() = O, and assuming that every ray is eventually oc- cluded O( 1. In that sense we can think of O as CDF, and
d0
20 - (0)= di
7tt)
(6)
NeRF
is its Probability Density Function (PDF). The volume rendering equation the expected light along the ray,
I(c, 0) [/o L(x(t), nlt), (t)dd,
where L(x, n., v) is the radiance field, namely the amount of light emanating from point x in direction o; in our formulation we also allow I, to depend on the level-set's normal, i.e., ntt) = (=d)()=()).. Adding this depen- dency is motivated by the fact that BRDFs of common materials are often encoded with respect to the surface normal, facilitating disentanglement as done in surface rendering (38). We will get back to disentanglement in the experiments section. The integral in equation approximated using a numerical quadrature, namely the rectangle rule, at some discrete M, where M is some large constant:
VoISDF
Qualitative comparison to NeRF VoISDF shows less artifacts.
Figure 2:
samples S [s = $1 < $2 <.... < 5m
m
[(c,v) = is(c,
Σ AL
(8)
where we use the subscript S in is to highlight the dependence of the approximation on the sample set S, ft = As is the approximated PDF multiplied by the interval length, and L; [ax(8;)) n(s)), v) is the sampled radiance field. We provide full derivation and detail of is in the supplementary.
Sampling. Since the PDF is typically extremely concentrated near the object's boundary (see e.g.. Figure 3 right) the choice of the sample points S has a crucial effect on the approximation quality of equation 8 One solution is to use an adaptive sample, e.g., S computed with the inverse CDF, i.e. O-1 However, O depends on the density model a and is not given explicitly. In 21 second, coarse network was trained specifically for the approximation of the opacity O, and was used for inverse sampling. However, the second network's 's density does not necessarily faithfully represents the first network's density, for which we wish to compute the volume integral. Furthermore, as we show later, one level of sampling could be insufficient to produce an accurate sample S. Using naive or crude approximation of O would lead to a sub-optimal sample set S that misses, or over extends non-negligible 7 values. Consequently, incorrect radiance approximation can occur (i.e., pixel color), potentially harming the learned density- works with single density and the sampling S is computed by sampling algorithm based on an error bound for the opacity approximation. Figure 2 compares the NeRF and VolSDF renderings for the same scene. Note the salt and pepper artifacts in the NeRF rendering caused by the random samples; using fixed (uniformly spaced) sampling in NeRF leads to a different type of artifacts shown in the supplementary.
field decomposition. Our solution
3.3 Bound on the opacity approximation error
In this section we develop a bound on the opacity approximation error using the rectangle rule. For a set of samples T = (ta (t=)=zz0 0 = t < t2 < < f. = M, we let d; = f++1 - 44 and o; axx(t:)) )) Given some E (0, M.. assume & [tx, tx+1], and apply the rectangle rule (i.e., left Riemann sum) to get the approximation:
L a(o(s))dss -T(() + E(1), where A(1)= = Σ
k-1
8,0; + (1 (6-1x)) 12)o
1=1

---

is the rectangle rule approximation, and E(t) denotes the error in this approximation. The conr- sponding approximation of the opacity function (equation [5) is
Õ(1) = - - (-R(1).
(10)
Our goal in this section to derive a uniform bound over [0, M] to the approximation Ô Rs O. The key is the following bound on the derivative@of the density inside an interval along the ray a:l):
Theorem 1. The derivative of the density a within a segment [44, [4+1] satisfies
(20())
a
exp
(-8)
di 3
where df = min
[<(()) yl1,
(11)
ds
28
seltitin U B.U,1
and B; {2| || ||- x(t))l < |d.1.. d, = dn(x((t)).
B,
But
The proof of this theorem, which is provided in the supplementary, makes a principled use of the signed distance function's 's unique prop- erties; the explicit formula for de is a bit cumbersome and therefore deferred the supplementary as- well. The inset depicts the bound- ary of the open balls union B, U B the interval (a=(4:), a0(++1)) and the bound is defined in terms of the minimal distance between these two sets, i.e., dr-
xxt)))
x x((141)
di
Id
Ide4al
The benefit in Theorem is that allows bound the density's 's derivative in each interval [t,, 4- based only on the unsigned distance at the interval's end points, |de], |ds+1), I. and the density parameters a, B. This bound can be used derive an error bound for the rectangle rule's approximation of the opacity,
|E()) < E(1)
a
+
(12)
4,3
Details are in the supplementary. Equation 12 leads the following opacity error bound, also proved in the supplementary:
Theorem 2. For E [0, M). the error of the approximated opacity Ô can be bounded as follows:
[(()) Ō(1) I exp (-f(()) (exp (f(1)) -1)
(13)
Finally, we can bound the opacity error for E [lx, [x+1] by noting that É(t), and consequently also (x)) are monotonically increasing in while exp( exp(-f((() is monotonically decreasing in 4 and therefore
(o()) -(t) s exp (-f((te) (xxp(E((x+1)) -1)
max AE(Be A
(14)
Taking the maximum over all intervals furnishes bound BT function of T and 3,
10(t) - ((1) s BT,9 max kE(n-1]
(exp (-R(tt)) (cxp(E(te+1)) -
max AE (
(15)
where by convention R(tn) 0, and [{] ......... f). See Figure where this bound is visualized
in faint-red.
To conclude this section derive useful properties, proved in the supplementary. The first, is that sufficiently dense sampling is guaranteed to reduce the error bound BT,
Lemma 1. Fix 3 O. For any a sufficient dense sampling TT will provide BT,
Second, with a fixed number of samples we can set 3 such that the error bound is below e:
Lemma 2. Fix > 0. For any > 0 a sufficiently large 3 that satisfies
2M2
B 2 4(n l)
(16)
will provide B S. As da is not differentiable everywhere the bound is the Lipschitz constant of a, see supplementary.
5

---

Opacity
Enoor
iteration
iteration
2
iteration 5
iteration 1
iteration 2
iteration 5
after 1, 2 and 5 iterations. Left-bottom: per-pixel B, heatmap; Left-top: rendering of areas marked with black squares. Right-top: for single ray indicated by white pixel we show the approximated (orange), true opacity (blue), the SDF (black), and Ô-1 sample example (yellow dots). Right-bottom: for the same ray we now show the true opacity error (red), and error bound (faint red). After 5 iterations most of the rays converged, as can be inspected by the blue colors in the heatmap, providing guaranteed approximation to the opacity, resulting in a crisp and more accurate rendering (center-left, top).
Figure 3: Qualitative evaluation of Algorithm D
3.4 Sampling algorithm
In this section we develop an algorithm for computing the sampling S to be used in equation g This
is done by first utilizing the bound in equation 15 to find samples T so that Ô (via equation 10
provides an approximation to the true opacity O, where e a hyper-parameter, that BT,8
Second, we perform inverse CDF sampling with Ó, as described in Section 3.2
Note that from Lemma Tit follows that However, this would lead to prohibitively large number of samples. Instead, we suggest a simple algorithm to reduce the number of required samples in practice and allows working with limited budget of sample points. In a nutshell, we start with a uniform sampling T = To. and use Lemma to initially set 3+ > 3 that satisfies BT,,, I Then, repeatedly upsample T to reduce 14 while maintaining B S. e. Even though this simple strategy is not guaranteed to converge, find that 3_ usually converges
simply choose large enough n to ensure BT, <
(typically 85%, see also Figure (3), and even in cases it does not, the algorithm provides 84 for which the opacity approximation still maintains an e error. The algorithm is presented below (Algorithm T].
We initialize T (Line in Algorithm I with uniform sampling To = {(,)== where 44 = (k - n-1
k E [n]] (we use n = 128 in our implementation) Given this sampling we next pick 3, > 3 according to Lemma 2 so that the error bound satisfies the required bound (Line in Algorithm I
In order to reduce B4 while keep BT.0, S. G, n sam- ples are added to T (Line 4 in Algorithm[]] []] where the number of points sampled from each interval is proportional to its current error bound, equation [4] Assuming T was sufficiently upsampled and satisfy By,
Algorithm 1: Sampling algorithm.
Input: error threshold > 0; B
Initialize T T=
Initialize 34 such that BT, , Se
decrease 34 towards B. Since the algorithm did not stop we have that By,8 > e. There- fore the Mean Value Theorem implies the existence of B. E (3,8+) such that BT.s. 6. We use the bisection method (with maximum of 10 iterations) efficiently search for 3, and update 34 accordingly (Lines and in Algorithm| D.. The algorithm runs iteratively until BT < or a maximal number of 5 iterations is reached. Either way, we use the final and B+ (guaranteed to provide By ,,, I <) to estimate
3 while By,s > e and not max_iter do
4
upsample T
5
if BT,a, < thehn
Find B, E (3, ) so that 6
Br,
7
Update 34 + 3,
8
end end
the current opacity Ô, Line 10 Algorithm I]. Fi- nally we return a fresh set of m = 64 samples O using inverse transform sampling (Line 11 in Algorithm T). BigureBshow qualitative illustration of Algorithm]] for 3 = 0.001 and 0.1 (typical values).
Estimate Ô using T and B+
1 S + get fresh m samples using Ô-1 return

---

Colmap
NeRF
VoISDE
IDR
Figure 4: Qualitative results for reconstructed geometries of objects from the DTU dataset.
3.5
Training
Our system consists of two Multi-Layer Perceptrons (MLP): (i) f.p approximating the SDF of the learned geometry, as well as global geometry feature z of dimension 256, i.e., f (xx (d(æ), =xx)) IR2255 where yp denotes its learnable parameters; (ii) (x n, U, z) R rep- resenting the scene's radiance field with learnable parameters v. In addition we have two scalar learnable parameters a, 3 E IR. In fact, in our implementation we make the choice a 8-1. We denote by 0 E R" the collection of all learnable parameters of the model, 0 = (4p, v, 3). To facilitate the learning of high frequency details of the geometry and radiance field, we exploit positional encoding [1 for the position æ and view direction v in the geometry and radiance field. The influence different positional encoding choices are presented in the supplementary.
Our data consists of collection of images with camera parameters, From this data we extract pixel level data: for each pixel p we have a triplet (4p, Cp, Up). where I, E R& its intensity (RGB color), Cp IR3 is its camera location, and Up RR3 is the viewing direction (camera pixel). Our training loss consists of two terms: (()) LRCB (0) + {LSDD(P)), where
(17)
CRGA (0) Ep I, is(c,, un) and (SDr((p) E. ([]V)()) - 1)2,
(18)
where LRGB is the color loss; ||-11; denotes the 1-norm, S is computed with Algorithm 1 and is is the numerical approximation to the volume rendering integral in equation 8 here we also incorporate the global feature in the radiance field, i.e., L Tog (2::)), r((s). Up. =<x(s)))) LSDF is the Eikonal loss encouraging to approximate a signed distance function [0]: the samples z are taken to combine single random uniform space point and single point from S for each pixel p. We train with batches of size 1024 pixels p. a Syper-parameter set to 0.1 throughout the the experiments. Further implementation details are provided in the supplementary.
Experiments
We evaluate our method on the challenging task multiview 3D surface reconstruction. We use two datasets: DTU [12] and BlendedMVS [37], both containing real objects with different materials that are captured from multiple views. Section 4.1 we show qualitative and quantitative 3D surface reconstruction results of VolSDF, comparing favorably to relevant baselines. In Section 4.2 we demonstrate that, in contrast to NeRF (21), our model is able to successfully disentangle the geometry and appearance of the captured objects.
7

---

Scan
24
37
40
55
63
65
69
83
97
105
106
110
114
118
122
Mean
IDR
1.63
1.87
0.63
0.48
1.04
0.79
0.77
1.33
1.16
0.76
0.67
0.90
0.42
0.51
0.53
0.90
colmap:
0.45
0.91
0.37
0.37
0.90
1.00
0.54
1.22
1.08
0.64
0.48
0.59
0.32
0.45
0.43
0.65
colmapo
0.81
2.05
0.73
1.22
1.79
1.58
1.02
3.05
1.40
2.05
1.00
1.32
0.49
0.78
1.17
1.36
PSNR Chamier
NeRF
1.92
1.73
1.92
0.80
3.41
1.39
1.51
5.44
2.04
1.10
1.01
2.88
0.91
1.00
0.79
1.89
VoISDF
1.14
1.26
0.81
0.49
1.25
0.70
0.72
1.29
1.18
0.70
0.66
1.08
0.42
0.61
0.55
0.86
NeRF
26.24
25.74
26.79
27.
27.57
31.96
31.50
29.58
32.78
28.35
32.08
33.49
31.54
31.0
35.59
35.51
30.65
VoISDF
26.28
25.61
26.55
26.76
76
31.57
31.5
29.38
23
28.03
32.13
33.16
31.49
30.33
34.9
34.75
30.38
Table 1: Quantitative results for the DTU dataset.
GT
NeRP++
VoISDE
Figure 5: Qualitative results sampled from the BlendedMVS dataset. For each scan we present visualization of a rendered image and the reconstructed 3D geometry.
4.1 Multi-view 3D reconstruction
DTU The DTU dataset contains multi-view image (49 or 64) of different objects with fixed camera and lighting parameters. We evaluate our method on the 15 scans that were selected by [38].
We compare our surface accuracy using the Chamfer l loss (measured in mm) to COLMAP, (which watertight reconstruction; COLMAP; is not watertight and provided only for reference) [3],
NeRF (21) and IDR (38), where for fair comparison with IDR we only evaluate the reconstruction
inside the visual hull of the objects (defined by the segmentation masks of (38) We further evaluate the PSNR of our rendering compared to [21]. Quantitative results are presented in Table It can be
observed that our method on par with IDR (that uses object masks for all images) and outperforms NeRF and COLMAP in terms of reconstruction accuracy. Our rendering quality is comparable to NcRF's.
BlendedMIVS The BlendedMVS dataset 37 contains large collection of 113 scenes captured from multiple views. It supplies high quality ground truth 3D models for evaluation, various camera configurations, and variety of indoor/outdoor real environments. We selected 9 different scenes and used our method to reconstruct the surface of each object. In contrast the DTU dataset, BlendedM scenes have complex backgrounds. Therefore we use NeRF++ [39 as baseline for this dataset. Table we present our results compared to NeRF++. Qualitative comparisons are presented in Fig- 5 since the units are unknown in this case we present relative improvement of Chamfer distance (in %) compared to NcRF, Also in this case, we improve NeRF reconstructions considerably, while being on-par in terms of the rendering quality (PSNR).
Comparison to 388 IDR 38 is the state of the art 3D surface reconstruction method using implicit representation. However, suffers from two drawback backsas: first, it requires object masks for training, which strong supervision signal. Second, since it sets the pixel color based only on the single point of intersection of the corresponding viewing ray, more pruned to local minima that sometimes appear in the form of extrancous surface parts. Figure 6
IDR
VoISDF
Figure 6: IDR extraneous parts.
compares the same scene trained

---

Egg
Head
Angel
Bull
Robot
Dog
Camera
Mean
Chamfer 4
Our Improvement (%)
$4.0
91.2
24.3
75.1
60.7
27.2
47.7
34.6
51.8
51.8
N+RF++
26.95
27.34
27.23
30.06
26.65
26.73
27.90
31.68
23.44
27.56
PSNR
VoISDF
25.49
27.18
26.36
29.79
26.01
26.03
28.65
31.24
22.97
27.08
Table 2: Quantitative results for the BlendedMVS dataset.
with IDR with the addition of ground truth masks, and VoISDF trained without masks. Note that IDR introduces extraneous surface parts (e.g., in marked red), while VoISDF provides more faithful result in this case.
4.2 Disentanglement of geometry and appearance
We have tested the disentanglement of scenes to geometry (density) and appearance (radiance field) by switching the radiance fields of two trained scenes. For VoISDF we switched Txp. For NeRF 21 note that the radiance field is computed (z, v), where L fully connected network with one hidden layer (of width 128 and ReLU activation) and is a feature vector. We tested two versions of NeRF disentanglement: First, by switching the original radiance fields Ly of trained NeRF networks. Second, by switching the radiance fields of trained NeRF models with an identical radiance field model ours, namely Lyj n, v, z). As shown in Figure 7 both versions of NeRF fail to produce a correct disentanglement in these scenes, while VolSDF successfully switches the materials of the two objects. We attribute this the specific inductive bias injected with the use of the density in equation?
VoISDF Figure 7: Geometry and radiance disentanglement is physically plausible with VoISDE.
NeRF
NeRF with normal
Conclusions
We introduce VolSDF, a volume rendering framework for implicit neural surfaces. We represent the volume density transformed version of the signed distance function the Iearned surface geometry. This seemingly simple definition provides a useful inductive bias, allowing disentangiement of geometry (i.e., density) and radiance field, and improves the geometry approximation over previous neural volume rendering techniques. Furthermore, it allows to bound the opacity approximation error leading to high fidelity sampling of the volume rendering integral.
Some limitations of our method present interesting future research opportunities. First, although working well in practice, helieve providing such a proof, or finding a version of this algorithm that has proof would be a useful contribution. In general, we believe working with bounds in volume rendering could improve learning and disentanglement and push the field forward. Second, representing non- watertight manifolds and/or manifolds with boundaries, such as zero thickness surfaces, is not possible with an SDF, Generalizations such multiple implicits and unsigned fields could be proven valuable. Third, our current formulation assumes homogeneous density; extending it to more general density models would allow representing a broader class of geometries. Fourth, now that high quality geometries can be learned in an unsupervised manner will interesting to learn dynamic geometries and shape spaces directly from collections of images. Lastly, although we don't see immediate negative societal impact our work, we do note that accurate geometry reconstruction from images can used for malice purposes.
do not have proof of correctness for the sampling algorithm. We

---

Acknowledgments
LY is supported by the European Research Council (ERC Consolidator Grant, "LiftMatch" 771136), the Israel Science Foundation (Grant No. 1830/17), and Carolito Stiftung (WAIC), YK is supported by the U.S.- Israel Binational Science Foundation, grant number 2018680, Carolito Stiftun (WAIC), and by the Kahn foundation.
References
[1] C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Gruphics (Proc. SIGGRAPH)) 28(3), Aug. 2009.
[2] M. Boss, R. Braun, V. Jampani, J. T. Barron, C. Liu, and H. P. Lensch. Nerd: Neural reflectance decomposition from image collections, 2020.
[3] A. Broadhurst, T. Drummond, and R. Cipolla. A probabilistic framework for space carving. In Proceedings Eighth IEEE International Conference on Computer Vision ICCV 2001, volume 1, pages 388-393 vol.1, 2001.
[4] A. Dai, M. Nießner, M. Zollhöfer, S. Izadi, and C. Theobalt. Bundlefusion: Real-time globally consistent 3d reconstruction using on- n--the-fly surface reintegration. ACM Transactions on Graphics (ToG), 36(4):1, 2017.
[5] J. S. De Bonet and P. Viola. Poxels: Probabilistic voxelized volume reconstruction. In Proceedings of the IEFE International Conference on Computer Vision. ICCV 1999, 1999.
[6] F. Dellaert and L. Yen-Chen. Neural volume rendering: Nerf and beyond. arXiv preprint 20X5042000000000 2020.
[7] Y. Furukawa and J. Ponce. Accurate, dense, and robust multiview steroopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(8):1362-1377 2010.
[8] S. Galliani, K. Lasinger, and K. Schindler. Massively paraliel multiview stereopsis by surface normal diffusion. In Proceedings of the IEEE International Conference on Computer Vision, pages 873-881, 2015.
[9] K. Genova, F. Cole, D. Vlasic, A. Sarna, W. T. Freeman, and T. Funkhouser. Learning shape templates with structured implicit functions. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 7164, 2019.
[10] A. Gropp. L. Yariv, N. Haim, M. Atzmon, and Y. Lipman. Implicit geometric regularization for lcarning shapes. arXiv preprint arXiv:2002. 10099, 2020.
[11] S. Izadi, D. Kim, O. Hilliges, D. Molyncaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison, al. Kinectfusion: real-time 3d reconstruction and interaction using moving depth camera. In Proceedingg of the 24th annual ACM symposium on User interface software and technology. pages 559-568, 2011.
[12] R. Jensen, A. Dahl, G. Vogiatzis, Tola, and Aanæs. Large scale multi-viee view stereopsis evaluation. In 2014 IEFE Conference on Computer Vision and Pottern Recognition, pages 406-413. TEEE, 2014.
[13] M. Kazhdan, M. Bolitho, and H. Hoppe. Poisson Surface Reconstruction. In A. Sheffer and K. Polthier, editors, Symposium on Geometry Processing. The Eurographics Association, 2006.
[14] P. Kellnhofer, L. Jebe, A. Jones, R. Spicer, K. Pulli, and G. Wetzstein. Neural lumigraph rendering. In CVPR, 2021.
[15] L. Liu, Gu, K. Zaw Lin, T.S.. Chua, and C. Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33, 2020.
[16] S. Liu, Y. Zhang, S. Peng, B. Shi, M. Pollefeys, and Z. Cui. Dist: Rendering deep implicit signed distance function with differentiable sphere tracing, arXiv preprint 1rXiv:1911.13225. 2019.
[17] S. Lombardi, T. Simon. J. Saragih, G. Schwartz. A. Lehrmann, and Y. Sheikh. Neural volumes: Learning 2019.
dynamic renderable volumes from images. arXiv preprint
[18] N. Max. Optical models for direct volume rendering, IEEE Transactions on Visualization and Computer Graphics, 1(2):99-100) 1995.
10

---

[19] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pages 4460 4470, 2019.
[20] M. Michalkiewicz, J. K. Pontes, D. Jack, M. Baktashmotlagh, and A. Eriksson. Implicit surface representa- tions as layers in neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4743-4752, 2019.
[21] B. Mildenhall, P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[22] M. Niemeyer, I.. Mescheder, M. Oechsle, and A. Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. arXiv preprint arXiv:19/2.07372. 2019.
[23] M. Niemeyer, I.. Mescheder, M. Oechsle, and A. Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5379-5389, 2019.
[24] M. Nießner, M. Zollhöfer, S. Izadi, and M. Stamminger. Real-time 3d reconstruction at scale using voxel hashing. ACM Trans. Graph., 32(6), Nov. 2013.
[25] M. Oechsle, L. Mescheder, M. Niemeyer, T. Strauss, and A. Geiger. Texture fields: Learning texture representations in function space. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 4531-4540, 2019.
[26] M. Oechsle, M. Niemeyer, L. Mescheder, T. Strauss, and A. Geiger. Learning implicit surface light fields. arXiv preprint ar0iv:2003, /2406, 2020.
[27] M. Oechsle, S. Peng. and A. Geiger. Unisurf: Unitying neural implicit surfaces and radiance fields for multi-view reconstruction. arXiv preprint arXiv:2104. 10078, 2021.
[28] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
2019.
[29] S. Peng, M. Niemeyer. L. Mescheder, M. Pollefeys, and A. Geiger. Convolutional occupancy networks. In A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, editors, Computer Vision ECCV 2020, pages Cham, 2020. Springer International Publishing.
[30] S. Saito, Z. Huang, R. Natsume, S. Morishima, A. Kanazawa, and H. Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2304-2314, 2314, 2019.
[31] J. L. Schönberger, E. Zheng, M. Pollefeys, and J.-M. Frahm, Pixelwise view selection for unstructured multi-view stereo. In European Conference Computer Vision (ECCV), 2016.
[32] S. M. Seitz and C. R. Dyer. Photorealistic scene reconstruction by voxel coloring. International Journal of Computer Vision, 1512)1 1999.
[33] V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene representation networks: Continuous 3d-structure- aware neural scene representations. In Advances in Neural Information Processing Systems, pages 1119-1130, 2019.
[34] P. P. Srinivasan, B. Deng, X. Zhang, M. Tancik, B. Mildenhall, and J. T. Barron. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In CVPR, 2021.
[35] T. Takikawa, Litalien, K. Yin, K. Kreis, C. Loop, D. Nowrouzezahrai, Jacobson, M. McGuire, and S. Fidler. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. arXiv preprint (tt:21001.109., 2021.
[36] Q. Xu, W. Wang. D. Ceylan, R. Mech, and U. Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. arXiv preprint arXiv: /905. 10711, 2019.
[37] Y. Yao, Z. Luo, S. Li, J. Zhang, Y. Ren, Zhou, T. Fang, and I.. Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. Computer Vision and Pattern Recognition (CVPR), 2020.
[38] L. Yariv. Y. Kasten, D. Moran, M. Galun, M. Atzmon, B. Ronen, and Y. Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems, 33, 2020.
[39] K. Zhang, G. Riegler, N. Snavely, and V. Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv:2010.07492.200 2020.


Reviews:
[{"review_id": "lOTe7L1Xx6T", "reviewer": "Reviewer_rpsv", "summary": "The paper proposed a method to recover both geometry and appearance of objects. They \nachieve this by combing both SDF representations and volume rendering. Given a set of \nmulti-view images, they use a network to predict the SDF and radiance filed of an arbitrary point, which \nis mapped to the volume density using a bell-shaped analytic function. Then they perform \nvolume rendering to predict the color for the pixel. They minimize the color difference as \nwell as an Eikonal loss to train the networks. They also propose a sampling scheme to  adaptively sample points along the ray based on the derived theoretical bound. The results \nshow that the proposed method can generate both accurate 3D mesh and faithful view-dependent \nappearance.\n", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n### Strengths \n1. Compared to previous volume-based methods, the proposed method can generate better mesh for the \ncaptures scenes and produces comparable appearance.  \n\n2. Compared to previous SDF-based methods, the proposed method can accurate reconstruct the geometry \nwith the need for foreground masks, which makes the acquisition setup much simpler.    \n\n3. The idea of mapping SDF to volume density is interesting and novel.\n\n4. The paper develops a novel method for adaptively sampling points along the ray, and it gives \na theoretical proof for the sampling scheme. Such a sampling scheme get rids of using coarse-fine \nnetworks as in NeRF, and also leads to better accuracy. \n\n### Weakness\n1. In the sampling scheme, it's not clear what the upsampling means exactly. It seems to me \nthat the method is gradually adding new samples to the current sampling points. My concern is \nthat whether such a scheme will cause memory and efficiency problem when there are many sampling \npoints considering that the network needs to be evaluated at each point. How much time does \nthe sampling take? \n\n2. As far as I understanding, after the sampling iterations finish, the sampled points  \n$\\mathcal{T}$ and $\\beta_{+}$ satisfying the desired bound. However, in Line 11, the algorithm \ngenerates a new set of samples. Do this new set of samples also meet the bound? If not, \nhow does this affect the performance? Also when there are more than 64 intervals, how these \n$64$ samples are generated? Does it mean some of the intervals will have zero samples?\n\n3. How does the method work for scenes with a lot of self-occlusions and thin structures? \nMost of the objects in the scene have relatively simple convex geometry.  It is desirable to show \nresults on the NeRF dataset such as the Lego, the Ship and the Plant scene. \n\n4. The paper only shows object-level reconstructions. Since the method does not need image masks,  \nI would expect the method to also work on complex scenes.  How does the method work on large-scale scenes as those shows in the NeRF paper? \n\n5. In the results in Figure A4 (supp), it seems that the recovered appearance tends to lose \ndetails and is a bit blurry, as shown in the Bull and the Dog scene. What is the reason\nfor it?\n\n6. The proposed method can generate accurate meshes from the SDF. One question is that whether it's possible to direct use the colors of points on the isosurface to represent the appearance of the objects? Or volume rendering is still needed to predict the colors?  \n\n7. Another limitation of the highlighted is that the texture/normal ambiguity. The method tends \nto treat the texture changes as normal changes in the reconstructions, such as \nthe floor in the second scene of Figure 5.  \n\nOverall, I appreciate the idea of the paper, and the results shown in the paper are convincing.  \nI expect the authors to answer the questions above and add further clarifications to the paper. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see the main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "dABuXaMXlFb", "reviewer": "Reviewer_L3KT", "summary": "NeRF and follow-up papers have led to impressive improvements in view synthesis when trained from multiple views of a single scene. This is achieved through volumetric rendering of an opacity field and a decoupled reflectance network. A limitation of NeRF is that the opacity field might not be faithful to the true geometry, as long as the rendered RGB images reproduce the correct appearance.\n\nThe proposed work builds on ideas developed in the NeRF literature and adapts them to 3D surface reconstruction. Contributions include deriving opacity from a truncated signed distance function, which explicity models the inside-outside topology of the surface. The validity of the signed distance function is enforced though an Eikonal loss term. The authors present a novel sampling technique for volumetric rendering, based on an approximation of the current opacity with bounded error. \n", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I consider this paper a strong submission that clearly identifies the limited applicability of NeRF-like goemetry representations to surface reconstruction. The authors present an intuitive geometry representation based on a truncated signed distance function, and an opacity field that is derived from the distance using a Laplace distribution. The width of the distribution is a learnable global parameter, which appears to allow for a gradual optimization from a relatively transparent initial geometry, to a well-defined, fully-opaque surface upon convergence.\n\nThe paper includes a thorough theoretical discussion of an adaptive sampling technique along the viewing ray, as well as a proof of error bounds on the opacity estimate that is used for sampling. The sampling technique is verified empirically in the supplemental material.\n\nThe presented experiments are comprehensive, comparing the accuracy of the reconstructed geometry to a state-of-the-art traditional technique (COLMAP), a recent learned technique (IDR), as well as a surface extracted from NeRF's opacity field. Both qualitatively and quantitatively, the presented 3D reconstructions compare very favourably to COLMAP and NeRF. It is exciting that the technique does not require silhouette masks as input, which promises to significantly increase the technique's practical viability.\n\nThe paper also includes an experiment that demonstrates improved disentanglement of geometry and reflectance compared to NeRF. It is surprising to this reviewer that the radiance network yields reliable predictions when sampled for positions near the surface of the new geometry, even though at training time the samples are placed near the original surface (guided by adaptive sampling). Could the authors comment why this apparent lack of supervision is not an issue?\n\n### Limitations\n\nI would expect a surface-based representation to show limitations for reconstruction problems that involve scenes without clearly defined surfaces, for example smoke or flames. The presented results do not include results on highly detailed geometry, such as hair or plants, which would require a level of detail that might exceed the resolution of the SDF, but could be represented with \"soft visibility\" for parts of the scene. Could the authors elaborate on the design space between representations with unconstrained opacity, and surface-based representation?\n\nIn direct comparison to baseline NeRF (see video), novel views appear to have somewhat smoother geometry and duller specular highlights. The paper would benefit from an analysis of implicit smoothness priors in the current implementation (Eikonal loss? L1 regularization in opacity transform?)\n\n### Further comments\n\n - The relative impact of the Eikonal loss term should be verified with an ablation study.\n\n - The sampling ablation from the supplemental would be useful addition to the main paper.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The potential for negative societal impact of this work is adequately discussed. See Main Review for a discussion of other limitations.", "review_text": "", "extra_scores": {"ethical_concerns": "The authors have adequately adressed the ethical concerns surrounding this kind of work.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "ZDp1MhHANJI", "reviewer": "Reviewer_EPLB", "summary": "This paper presents a neural approach based on NeRF, for the purpose of learning a 3D representation of a shape from captured images. Instead of allowing volume density to be freely allocated along rays, as in NeRF, this paper proposes concentrating volume density around a single surface. To do this, the method represents the shape as a neural SDF, and defines a volumetric density around the zero level set of the SDF using the CDF of the Laplace distribution. This choice enables the paper to derive a bound on the estimated opacity for any segment along a ray, and this is used to draw samples such that the error is limited to some epsilon. The paper experimentally shows that the resulting recovered surfaces are closer to the ground-truth geometry than surfaces extracted from level sets of NeRF's volumetric geometry, as well as meshes produced by COLMAP. Additionally, the paper experimentally shows that the proposed method is able to render novel views that are close in quality to those rendered by NeRF.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like the paper and the general research direction, but I have some concerns regarding the experimental evaluation.\n\nStrengths:\n\nOverall, think that the idea of hybrid surface-volume representation and rendering is well-motivated, and this paper develops a well thought-out approach to recover NeRF-like models whose volume densities are concentrated around a surface. I intellectually like the idea of choosing a volumetric function of the SDF that lets us bound the opacity estimation.\n\nThe proposed method clearly outperforms IDR and level sets extracted from NeRF, in terms of surface reconstruction quality.\n\nI think that the disentanglement results are very cool and thought-provoking.\n\nWeaknesses:\n\nI think that the main weaknesses of the paper are in the thoroughness of the experimental evaluation. I am listing a few of my questions/concerns below:\n\nIt is unclear how many actual MLP evaluations along a ray are used for each of the baselines and ablations in A1. I am concerned that the experiments are not controlling for the number of MLP evaluations along each ray when comparing different methods. For example, is it correct that VolSDF with 5 iterations is using more than 832 MLP evaluations per ray (5*128 in order to compute the error bounds for each sample for each iteration + 128 for estimating \\hat{O} + final 64 \"fresh\" samples, and this is without accounting for the \"upsampling\" done in each iteration)? If this is true, then I think that the comparisons are potentially quite unfair, because NeRF results would probably be much better with this number of samples per ray (instead of 64+128 as in the default NeRF implementation). Could the authors please clarify this?\n\nI think it would be useful to make the ablation figure in A1 into a more complete comparison across multiple scenes, and additionally clarify how many actual MLP queries along a ray are used by each of these methods, so readers can form a reasonable basis for comparison.\n\nI've noticed these same noisy salt-and-pepper artifacts (like those seen in Figure 2 and mentioned on line 158) when experimenting with NeRF if I use stratified random sampling of points along the ray for the \"coarse\" MLP at test time (as opposed to training time, where NeRF draws random stratified samples along the ray for the \"coarse\" MLP, at test time, NeRF uses regularly-spaced samples for the \"coarse\" MLP). Is it possible that these artifacts are actually due to the NeRF implementation forgetting to turn off random sampling at test time?\n\nThe scenes shown in the supplementary video are composed of materials that have pretty simple view-dependent appearance (not too shiny and not illuminated by obviously high-frequency lighting). Furthermore, they are only rendered from a limited range of views, so the view-dependent appearance showcased in these results is not very complex. I would be very interested to see how this method performs on scenes like the Blender material ball samples (in NeRF's Blender dataset) that are much shinier and contain reflections of high-frequency lighting. Also, I'd suggest including rendered camera path results from the BlendedMVS dataset and camera paths with much larger motion (full 360s around an object maybe?) in the supplementary video, because it would help readers get a better idea how VolSDF performs.\n\nI'm confused by the statement on line 290, which says that NeRF computes L(z,v) with a linear function. My understanding is that NeRF uses a single layer with a ReLU linearity (width = 128) that takes in z and v, followed by a linear layer that outputs the view-dependent emitted radiance RGB value. Could the authors please comment on the NeRF baseline implementation?\n\nAnother concern I have regarding comparisons to the NeRF baseline: line 59 of the supplementary PDF says that the VolSDF model has 4 layers of width 256 that represent the view-dependent emitted radiance. I think it would make sense to use the same higher-capacity architecture for the NeRF baseline for the experiments in Table 1 and the disentanglement comparisons.\n\nA few other questions:\n\nDoes the representation still need to be initialized as a unit sphere SDF (supplementary line 57) in this hybrid surface/volume rendering setup? I would guess that this initialization is less important than it would be in a purely surface-based representation like IDR.\n\nMy understanding is that the alpha and beta parameters of the Laplace CDF are global for the whole scene. Would it make sense to instead have them be local, so that different regions of the scene could be more volumetric vs. more surface-like? \n\nThe choice of setting alpha to the reciprocal of beta seems unintuitive, can the authors please elaborate why both can't be free parameters during optimization? What happens to the results if both are free?\n\nEdits and suggestions:\n\nSince the main goal of this paper is to improve the surface representation recovered from NeRF-like models (as opposed to improving view synthesis quality), I think it would be good to include more motivation why we should care about better surface reconstruction from these methods. Maybe mentioning robotics applications or other scenarios where we would like to use differentiable rendering to recover geometry for some application that actually needs high-quality geometry (one could argue that in view synthesis, geometry doesn't matter if the views are rendered well)?\n\nI think that the usage of the term \"light field\" in this paper is slightly different from how it is typically used in graphics, and this may be confusing to readers. This paper uses \"light field\" to mean the view-dependent emitted radiance function in NeRF (how much radiance is emitted in any direction at each point), which is a quantity that is accumulated to render the radiance travelling along a ray. However, the typical usage of \"light field\" within graphics (see [Levoy and Hanrahan, SIGGRAPH 1996]) refers to the radiance along rays, which is actually the quantity that is produced from volume rendering. These usages of the term \"light field\" are slightly different, because the actual light field function would represent the accumulated opacity along all rays instead of just what is being emitted by particles at each location. I suggest changing the usage of \"light field\" in this paper to avoid confusion.\n\nLine 22-23: remove \"so called\"\n\nLine 69: add dash between non and Lambertian\n\nLines 105, 139, 147, etc.: capitalize \"Equation\" to match capitalization of \"Figure\" and \"Section\"\n\nLine 114: it --> the volume rendering integral\n\nFirst line of Fig. 3 caption: missing space after \"1\"\n\nLine 313: malice --> malicious\n\nI'd recommend proofreading the references. For example, CVPR is cited differently in [12] and [14], [17] was published at SIGGRAPH, NeRF [21] should have the R and F capitalized, NeRD [2] should have the R and D capitalized, NeRV [34] should have the R and V capitalized, the Adam optimizer was published at ICLR and not just arXiv, etc.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper includes an interesting discussion of potential future directions and a figure with example failure cases. I think that it would be nice to include a more thorough discussion of limitations in the updated draft.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "NPb4dMiSWb_", "reviewer": "Reviewer_ybo7", "summary": "The paper introduces a novel method for multi-view scene reconstruction and image synthesis. The authors combine volumetric and surface based neural rendering techniques to obtain highly accurate surface models without need for object mask annotation. The paper presents theoretical framework motivating the SDF-based density field parametrization and the adaptive sampling approach. The validation shows that the reconstructed shapes are surpassing the state of the art.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Originality] \nThe idea of parametrizing volumetric density through signed distance function is novel, very elegant and effective. I expect it to have significant impact on the community.\n\n[Quality]\nThe algorithms are well motivated and based on theoretical analysis.\n\n[Clarity]\nThe paper is clearly written and easy to follow. I particularly like the elegant definition of the SDF and surface manifold in Eq. 1. The proofs of theoretical relations from the paper are provided in the supplement though they could be expanded to cover more details. E.g. I am not entirely sure how the second inequality in the proof of Lemma 2 emerges from the derivation of Lemma 1.\nThe paper covers implementation details needed for reproduction and, furthermore, code was also provided.\n\n[Significance]\nThe capability of surface-based inverse neural rendering without mask requirement lifts one of the major practical limitations from this category of methods and it should allow for easier and higher quality surface reconstruction methods in the future.\n\n[Questions, Comments and Concerns]\n\n1) Why is the density around the surface modeled using Laplacian distribution? Is there a physical reason for this particular distribution or is it just mathematical convenience?\n\n2) The normal as input parameter of L makes sense but I would still expect explicit ablation testing the same model with and without this new feature. Does it affect reconstruction quality or is it only useful for the appearance swap demo in Sec 4.2?\n\n3) The quantitative evaluation does not fully capture the image reconstruction quality. The shape reconstruction without masks alone is a great achievement but it would still be useful to learn limits of the image reconstruction quality. The PSNR scores seem competitive with Nerf(++), however, closer inspection of Fig. 5 shows that image features are rather smeared. The images are also of relatively low resolution. It would be useful to provide a metric more sensitive to perceptual qualities of the images such as PSNR. Further, how would the proposed technique compare to some of the recent surface-based techniques focusing on image reconstruction quality such as \"Stable View Synthesis\" [Riegler and Koltun 2021]? (I believe it should also be cited)\n\n\n4) Minor comments:\n- L122: transperancy -> transparency\n- L209: while keep -> while keeping\n\n\n[Conclusion]\n\nOverall, this is a very high quality and high impact potential paper. I believe that the proposed hybrid formulation is a very promising way forward for surface reconstruction methods. My only notable concern with the paper is the extent of image reconstruction quality evaluation. However, it does not have a direct impact on the main contribution and it can easily be improved in the final version. I strongly recommend accepting.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "[Societal impact]\nThe authors (and me) do not expect negative societal impacts. It could potentially lead to more effective 3D data capture, storage and transfer which is useful in many fields including cultural heritage preservation.\n\n\n\n[Limitations]\nThe paper lacks listing of limitations (Or I could not find it). Perhaps with the exception of future work discussion in the conclusion. In the video I noticed some instability of the ray-traced surface in the skull example (may or may not be related to the representation) and potentially lack of spatial fidelity of the reconstructed image details. The method also shares the encoding/decoding computational complexity with its predecessors.\n", "review_text": "", "extra_scores": {"ethical_concerns": "Do not see any.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission introduces a timely and important contribution that integrates SDF representations with neural volume rendering.  All reviewers are positive, and three of the four reviewers strongly recommend acceptance.  The AC agrees.  The authors should try to address the reviewers' concerns in the camera-ready version. This includes adding the ablation studies from the rebuttal period, incorporating the discussion as L3KT suggested, among others. 
Decision: Accept (Oral)