Volume Rendering of Neural Implicit Surfaces

1 Introduction . Volume rendering [ 18 ] is a set of techniques that renders volume density in radiance fields by the so called volume rendering integral . It has recently been shown that representing both the density and radiance fields as neural networks can lead to excellent prediction of novel views by learning only from a sparse set of input images . This neural volume rendering approach , presented in [ 21 ] and developed by its follow-ups [ 34 , 2 ] approximates the integral as alpha-composition in a differentiable way , allowing to learn simultaneously both from input images . Although this coupling indeed leads to good generalization of novel viewing directions , the density part is not as successful in faithfully predicting the scene ’ s actual geometry , often producing noisy , low fidelity geometry approximation . We propose VolSDF to devise a different model for the density in neural volume rendering , leading to better approximation of the scene ’ s geometry while maintaining the quality of view synthesis . The key idea is to represent the density as a function of the signed distance to the scene ’ s surface , see Figure 1 . Such density function enjoys several benefits . First , it guarantees the existence of a well-defined surface that generates the density . This provides a useful inductive bias for disentangling density and radiance fields , which in turn provides a more accurate geometry approximation . Second , we show this density formulation allows bounding the approximation error of the opacity along rays . This bound is used to sample the viewing ray so to provide a faithful coupling of density and radiance field in the volume rendering integral . E.g. , without such a bound the computed radiance along a ray ( pixel color ) can potentially miss or extend surface parts leading to incorrect radiance approximation . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . A closely related line of research , often referred to as neural implicit surfaces [ 22 , 38 , 14 ] , have been focusing on representing the scene ’ s geometry implicitly using a neural network , making the surface rendering process differentiable . The main drawback of these methods is their requirement of masks that separate objects from the background . Also , learning to render surfaces directly tends to grow extraneous parts due to optimization problems , which are avoided by volume rendering . In a sense , our work combines the best of both worlds : volume rendering with neural implicit surfaces . We demonstrate the efficacy of VolSDF by reconstructing surfaces from the DTU [ 12 ] and BlendedMVS [ 37 ] datasets . VolSDF produces more accurate surface reconstructions compared to NeRF [ 21 ] and NeRF++ [ 39 ] , and comparable reconstruction compared to IDR [ 38 ] , while avoiding the use of object masks . Furthermore , we show disentanglement results with our method , i.e. , switching the density and radiance fields of different scenes , which is shown to fail in NeRF-based models . 2 Related work . Neural Scene Representation & Rendering Implicit functions are traditionally adopted in modeling 3D scenes [ 24 , 11 , 4 ] . Recent studies have been focusing on model implicit functions with multi-layer perceptron ( MLP ) due to its expressive representation power and low memory foot-print , including scene ( geometry & appearance ) representation [ 9 , 20 , 19 , 23 , 25 , 29 , 36 , 28 , 35 ] and free-view rendering [ 33 , 16 , 30 , 26 , 17 , 21 , 15 , 39 , 34 , 2 ] . In particular , NeRF [ 21 ] has opened up a line of research ( see [ 6 ] for an overview ) combining neural implicit functions together with volume rendering to achieve photo-realistic rendering results . However , it is non-trivial to find a proper threshold to extract surfaces from the predicted density , and the recovered geometry is far from satisfactory . Furthermore , sampling of points along a ray for rendering a pixel is done using an opacity function that is approximated from another network without any guarantee for correct approximation . Multi-view 3D Reconstruction Image-based 3D surface reconstruction ( multi-view stereo ) has been a longstanding problem in the past decades . Classical multi-view stereo approaches are generally either depth-based [ 1 , 31 , 8 , 7 ] or voxel-based [ 5 , 3 , 32 ] . For instance , in COLMAP [ 31 ] ( a typical depth-based method ) image features are extracted and matched across different views to estimate depth . Then the predicted depth maps are fused to obtain dense point clouds . To obtain the surface , an additional meshing step e.g . Poisson surface reconstruction [ 13 ] is applied . However , these methods with complex pipelines may accumulate errors at each stage and usually result in incomplete 3D models , especially for non-Lambertian surfaces as they can not handle view dependent colors . On the contrary , although it produces complete models by directly modeling objects in a volume , voxel-based approaches are limited to low resolution due to high memory consumption . Recently , neural-based approaches such as DVR [ 22 ] , IDR [ 38 ] , NLR [ 14 ] have also been proposed to reconstruct scene geometry from multi-view images . However , these methods require accurate object masks and appropriate weight initialization due to the difficulty of propagating gradients . Independently from and concurrently with our work here , [ 27 ] also use implicit surface representation incorporated into volume rendering . In particular , they replace the local transparency function with an occupancy network [ 19 ] . This allows adding surface smoothing term to the loss , improving the quality of the resulting surfaces . Differently from their approach , we use signed distance representation , regularized with an Eikonal loss [ 38 , 10 ] without any explicit smoothing term . Furthermore , we show that the choice of using signed distance allows bounding the opacity approximation error , facilitating the approximation of the volume rendering integral for the suggested family of densities . 3 Method . In this section we introduce a novel parameterization for volume density , defined as transformed signed distance function . Then we show how this definition facilitates the volume rendering process . In particular , we derive a bound of the error in the opacity approximation and consequently devise a sampling procedure for approximating the volume rendering integral . 3.1 Density as transformed SDF . Let the set Ω ⊂ R3 represent the space occupied by some object in R3 , andM = ∂Ω its boundary surface . We denote by 1Ω the Ω indicator function , and by dΩ the Signed Distance Function ( SDF ) to its boundaryM , 1Ω ( x ) = { 1 if x ∈ Ω 0 if x /∈ Ω , and dΩ ( x ) = ( −1 ) 1Ω ( x ) min y∈M ‖x− y‖ , ( 1 ) where ‖·‖ is the standard Euclidean 2-norm . In neural volume rendering the volume density σ : R3 → R+ is a scalar volumetric function , where σ ( x ) is the rate that light is occluded at point x ; σ is called density since it is proportional to the particle count per unit volume at x [ 18 ] . In previous neural volumetric rendering approaches [ 21 , 15 , 39 ] , the density function , σ , was modeled with a general-purpose Multi-Layer Perceptron ( MLP ) . In this work we suggest to model the density using a certain transformation of a learnable Signed Distance Function ( SDF ) dΩ , namely σ ( x ) = αΨβ ( −dΩ ( x ) ) , ( 2 ) where α , β > 0 are learnable parameters , and Ψβ is the Cumulative Distribution Function ( CDF ) of the Laplace distribution with zero mean and β scale ( i.e. , mean absolute deviation , which is intuitively the L1 version of the standard deviation ) , Ψβ ( s ) = 1 2 exp ( s β ) if s ≤ 0 1− 12 exp ( − sβ ) if s > 0 ( 3 ) Figure 1 ( center left and right ) depicts an example of such a density and SDF . As can be readily checked from this definition , as β approach zero , the density σ converges to a scaled indicator function of Ω , that is σ → α1Ω for all points x ∈ Ω \M . Intuitively , the density σ models a homogeneous object with a constant density α that smoothly decreases near the object ’ s boundary , where the smoothing amount is controlled by β . The benefit in defining the density as in equation 2 is two-fold : First , it provides a useful inductive bias for the surface geometryM , and provides a principled way to reconstruct the surface , i.e. , as the zero level-set of dΩ . This is in contrast to previous work where the reconstruction was chosen as an arbitrary level set of the learned density . Second , the particular form of the density as defined in equation 2 facilitates a bound on the error of the opacity ( or , equivalently the transparency ) of the rendered volume , a crucial component in the volumetric rendering pipeline . In contrast , such a bound will be hard to devise for a generic MLP densities . 3.2 Volume rendering of σ . In this section we review the volume rendering integral and the numerical integration commonly used to approximate it , requiring a set S of sample points per ray . In the following section ( Section 3.3 ) , we explore the properties of the density σ and derive a bound on the opacity approximation error along viewing rays . Finally , in Section 3.4 we derive an algorithm for producing a sample S to be used in the volume rendering numerical integration . In volume rendering we consider a ray x emanating from a camera position c ∈ R3 in direction v ∈ R3 , ‖v‖ = 1 , defined by x ( t ) = c + tv , t ≥ 0 . In essence , volume rendering is all about approximating the integrated ( i.e. , summed ) light radiance along this ray reaching the camera . There are two important quantities that participate in this computation : the volume ’ s opacity O , or equivalently , its transperancy T , and the radiance field L. The transparency function of the volume along a ray x , denoted T , indicates , for each t ≥ 0 , the probability a light particle succeeds traversing the segment [ c , x ( t ) ] without bouncing off , T ( t ) = exp ( − ∫ t 0 σ ( x ( s ) ) ds ) , ( 4 ) and the opacity O is the complement probability , O ( t ) = 1− T ( t ) . ( 5 ) Note that O is a monotonic increasing function where O ( 0 ) = 0 , and assuming that every ray is eventually occluded O ( ∞ ) = 1 . In that sense we can think of O as a CDF , and τ ( t ) = dO dt ( t ) = σ ( x ( t ) ) T ( t ) ( 6 ) is its Probability Density Function ( PDF ) . The volume rendering equation is the expected light along the ray , I ( c , v ) = ∫ ∞ 0 L ( x ( t ) , n ( t ) , v ) τ ( t ) dt , ( 7 ) where L ( x , n , v ) is the radiance field , namely the amount of light emanating from point x in direction v ; in our formulation we also allow L to depend on the level-set ’ s normal , i.e. , n ( t ) = ∇xdΩ ( x ( t ) ) . Adding this dependency is motivated by the fact that BRDFs of common materials are often encoded with respect to the surface normal , facilitating disentanglement as done in surface rendering [ 38 ] . We will get back to disentanglement in the experiments section . The integral in equation 7 is approximated using a numerical quadrature , namely the rectangle rule , at some discrete samples S = { si } mi=1 , 0 = s1 < s2 < . . . < sm = M , where M is some large constant : I ( c , v ) ≈ ÎS ( c , v ) = m−1∑ i=1 τ̂iLi , ( 8 ) where we use the subscript S in ÎS to highlight the dependence of the approximation on the sample set S , τ̂i ≈ τ ( si ) ∆s is the approximated PDF multiplied by the interval length , and Li = L ( x ( si ) , n ( si ) , v ) is the sampled radiance field . We provide full derivation and detail of τ̂i in the supplementary . Sampling . Since the PDF τ is typically extremely concentrated near the object ’ s boundary ( see e.g. , Figure 3 , right ) the choice of the sample points S has a crucial effect on the approximation quality of equation 8 . One solution is to use an adaptive sample , e.g. , S computed with the inverse CDF , i.e. , O−1 . However , O depends on the density model σ and is not given explicitly . In [ 21 ] a second , coarse network was trained specifically for the approximation of the opacity O , and was used for inverse sampling . However , the second network ’ s density does not necessarily faithfully represents the first network ’ s density , for which we wish to compute the volume integral . Furthermore , as we show later , one level of sampling could be insufficient to produce an accurate sample S. Using a naive or crude approximation of O would lead to a sub-optimal sample set S that misses , or over extends non-negligible τ values . Consequently , incorrect radiance approximations can occur ( i.e. , pixel color ) , potentially harming the learned density-radiance field decomposition . Our solution works with a single density σ , and the sampling S is computed by a sampling algorithm based on an error bound for the opacity approximation . Figure 2 compares the NeRF and VolSDF renderings for the same scene . Note the salt and pepper artifacts in the NeRF rendering caused by the random samples ; using fixed ( uniformly spaced ) sampling in NeRF leads to a different type of artifacts shown in the supplementary .

Summary: The paper introduces a novel method for multi-view scene reconstruction and image synthesis. The authors combine volumetric and surface based neural rendering techniques to obtain highly accurate surface models without need for object mask annotation. The paper presents theoretical framework motivating the SDF-based density field parametrization and the adaptive sampling approach. The validation shows that the reconstructed shapes are surpassing the state of the art.