NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction

1 Introduction . Reconstructing surfaces from multi-view images is a fundamental problem in computer vision and computer graphics . 3D reconstruction with neural implicit representations has recently become a highly promising alternative to classical reconstruction approaches [ 35 , 8 , 2 ] due to its high reconstruction quality and its potential to reconstruct complex objects that are difficult for classical approaches , such as non-Lambertian surfaces and thin structures . Recent works represent surfaces as signed distance functions ( SDF ) [ 46 , 49 , 17 , 22 ] or occupancy [ 29 , 30 ] . To train their neural models , these methods use a differentiable surface rendering method to render a 3D object into images and compare them against input images for supervision . For example , IDR [ 46 ] produces impressive reconstruction results , but it fails to reconstruct objects with complex structures that causes abrupt depth changes . The cause of this limitation is that the surface rendering method used in IDR only considers a single surface intersection point for each ray . Consequently , the gradient only exists at this single point , which is too local for effective back propagation and would get optimization stuck in a poor local minimum when there are abrupt changes of depth on images . Furthermore , object ∗Corresponding authors . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . masks are needed as supervision for converging to a valid surface . As illustrated in Fig . 1 ( a ) top , with the radical depth change caused by the hole , the neural network would incorrectly predict the points near the front surface to be blue , failing to find the far-back blue surface . The actual test example in Fig . 1 ( b ) shows that IDR fails to correctly reconstruct the surfaces near the edges with abrupt depth changes . Recently , NeRF [ 28 ] and its variants have explored to use a volume rendering method to learn a volumetric radiance field for novel view synthesis . This volume rendering approach samples multiple points along each ray and perform α-composition of the colors of the sampled points to produce the output pixel colors for training purposes . The advantage of the volume rendering approach is that it can handle abrupt depth changes , because it considers multiple points along the ray and so all the sample points , either near the surface or on the far surface , produce gradient signals for back propagation . For example , referring Fig . 1 ( a ) bottom , when the near surface ( yellow ) is found to have inconsistent colors with the input image , the volume rendering approach is capable of training the network to find the far-back surface to produce the correct scene representation . However , since it is intended for novel view synthesis rather than surface reconstruction , NeRF only learns a volume density field , from which it is difficult to extract a high-quality surface . Fig . 1 ( b ) shows a surface extracted as a level-set surface of the density field learned by NeRF . Although the surface correctly accounts for abrupt depth changes , it contains conspicuous noise in some planar regions . In this work , we present a new neural rendering scheme , called NeuS , for multi-view surface reconstruction . NeuS uses the signed distance function ( SDF ) for surface representation and uses a novel volume rendering scheme to learn a neural SDF representation . Specifically , by introducing a density distribution induced by SDF , we make it possible to apply the volume rendering approach to learning an implicit SDF representation and thus have the best of both worlds , i.e . an accurate surface representation using a neural SDF model and robust network training in the presence of abrupt depth changes as enabled by volume rendering . Note that simply applying a standard volume rendering method to the density associated with SDF would lead to discernible bias ( i.e . inherent geometric errors ) in the reconstructed surfaces . This is a new and important observation that we will elaborate later . Therefore we propose a novel volume rendering scheme to ensure unbiased surface reconstruction in the first-order approximation of SDF . Experiments on both DTU dataset and BlendedMVS dataset demonstrated that NeuS is capable of reconstructing complex 3D objects and scenes with severe occlusions and delicate structures , even without foreground masks as supervision . It outperforms the state-of-the-art neural scene representation methods , namely IDR [ 46 ] and NeRF [ 28 ] , in terms of reconstruction quality . 2 Related Works . Classical Multi-view Surface and Volumetric Reconstruction . Traditional multi-view 3D reconstruction methods can be roughly classified into two categories : point- and surface-based reconstruction [ 2 , 8 , 9 , 35 ] and volumetric reconstruction [ 6 , 3 , 36 ] . Point- and surface-based reconstruction methods estimate the depth map of each pixel by exploiting inter-image photometric consistency [ 8 ] and then fuse the depth maps into a global dense point cloud [ 25 , 48 ] . The surface reconstruction is usually done as a post processing with methods like screened Poisson surface reconstruction [ 16 ] . The reconstruction quality heavily relies on the quality of correspondence matching , and the difficulties in matching correspondence for objects without rich textures often lead to severe artifacts and missing parts in the reconstruction results . Alternatively , volumetric reconstruction methods circumvent the difficulty of explicit correspondence matching by estimating occupancy and color in a voxel grid from multi-view images and evaluating the color consistency of each voxel . Due to limited achievable voxel resolution , these methods can not achieve high accuracy . Neural Implicit Representation . Some methods enforce 3D understanding in a deep learning framework by introducing inductive biases . These inductive biases can be explicit representations , such as voxel grids [ 13 , 5 , 44 ] , point cloud [ 7 , 24 , 18 ] , meshes [ 41 , 43 , 14 ] , and implicit representations . The implicit representations encoded by a neural network has gained a lot of attention recently , since it is continuous and can achieve high spatial resolution . This representation has been applied successfully to shape representation [ 26 , 27 , 31 , 4 , 1 , 10 , 47 , 32 ] , novel view synthesis [ 38 , 23 , 15 , 28 , 21 , 33 , 34 , 40 , 37 ] and multi-view 3D reconstruction [ 46 , 29 , 17 , 12 , 22 ] . Our work mainly focuses on learning implicit neural representation encoding both geometry and appearance in 3D space from 2D images via classical rendering techniques . Limited in this scope , the related works can be roughly categorized based on the rendering techniques used , i.e . surface rendering based methods and volume rendering based methods . Surface rendering based methods [ 29 , 17 , 46 , 22 ] assume that the color of ray only relies on the color of an intersection of the ray with the scene geometry , which makes the gradient only backpropagated to a local region near the intersection . Therefore , such methods struggle with reconstructing complex objects with severe self-occlusions and sudden depth changes . Furthermore , they usually require object masks as supervision . On the contrary , our method performs well for such challenging cases without the need of masks . Volume rendering based methods , such as NeRF [ 28 ] , render an image by α-compositing colors of the sampled points along each ray . As explained in the introduction , it can handle sudden depth changes and synthesize high-quality images . However , extracting high-fidelity surface from the learned implicit field is difficult because the density-based scene representation lacks sufficient constraints on its level sets . In contrast , our method combines the advantages of surface rendering based and volume rendering based methods by constraining the scene space as a signed distance function but applying volume rendering to train this representation with robustness . UNISURF [ 30 ] , a concurrent work , also learns an implicit surface via volume rendering . It improves the reconstruction quality by shrinking the sample region of volume rendering during the optimization . Our method differs from UNISURF in that UNISURF represents the surface by occupancy values , while our method represents the scene by an SDF and thus can naturally extract the surface as the zero-level set of it , yielding better reconstruction accuracy than UNISURF , as will be seen later in the experiment section . 3 Method . Given a set of posed images { Ik } of a 3D object , our goal is to reconstruct the surface S of it . The surface is represented by the zero-level set of a neural implicit SDF . In order to learn the weights of the neural network , we developed a novel volume rendering method to render images from the implicit SDF and minimize the difference between the rendered images and the input images . This volume rendering approach ensures robust optimization in NeuS for reconstructing objects of complex structures . 3.1 Rendering Procedure . Scene representation . With NeuS , the scene of an object to be reconstructed is represented by two functions : f : R3 → R that maps a spatial position x ∈ R3 to its signed distance to the object , and c : R3 × S2 → R3 that encodes the color associated with a point x ∈ R3 and a viewing direction v ∈ S2 . Both functions are encoded by Multi-layer Perceptrons ( MLP ) . The surface S of the object is represented by the zero-level set of its SDF , that is , S = { x ∈ R3|f ( x ) = 0 } . ( 1 ) In order to apply a volume rendering method to training the SDF network , we first introduce a probability density function φs ( f ( x ) ) , called S-density , where f ( x ) , x ∈ R3 , is the signed distance function and φs ( x ) = se−sx/ ( 1 + e−sx ) 2 , commonly known as the logistic density distribution , is the derivative of the Sigmoid function Φs ( x ) = ( 1 + e−sx ) −1 , i.e. , φs ( x ) = Φ′s ( x ) . In principle φs ( x ) can be any unimodal ( i.e . bell-shaped ) density distribution centered at 0 ; here we choose the logistic density distribution for its computational convenience . Note that the standard deviation of φs ( x ) is given by 1/s , which is also a trainable parameter , that is , 1/s approaches to zero as the network training converges . Intuitively , the main idea of NeuS is that , with the aid of the S-density field φs ( f ( x ) ) , volume rendering is used to train the SDF network with only 2D input images as supervision . Upon successful minimization of a loss function based on this supervision , the zero-level set of the network-encoded SDF is expected to represent an accurately reconstructed surface S , with its induced S-density φs ( f ( x ) ) assuming prominently high values near the surface . Rendering . To learn the parameters of the neural SDF and color field , we advise a volume rendering scheme to render images from the proposed SDF representation . Given a pixel , we denote the ray emitted from this pixel as { p ( t ) = o + tv|t ≥ 0 } , where o is the center of the camera and v is the unit direction vector of the ray . We accumulate the colors along the ray by C ( o , v ) = ∫ +∞ 0 w ( t ) c ( p ( t ) , v ) dt , ( 2 ) where C ( o , v ) is the output color for this pixel , w ( t ) a weight for the point p ( t ) , and c ( p ( t ) , v ) the color at the point p along the viewing direction v. Requirements on weight function . The key to learn an accurate SDF representation from 2D images is to build an appropriate connection between output colors and SDF , i.e. , to derive an appropriate weight function w ( t ) on the ray based on the SDF f of the scene . In the following , we list the requirements on the weight function w ( t ) . 1 . Unbiased . Given a camera ray p ( t ) , w ( t ) attains a locally maximal value at a surface intersection point p ( t∗ ) , i.e . with f ( p ( t∗ ) ) = 0 , that is , the point p ( t∗ ) is on the zero-level set of the SDF ( x ) . 2 . Occlusion-aware . Given any two depth values t0 and t1 satisfying f ( t0 ) = f ( t1 ) , w ( t0 ) > 0 , w ( t1 ) > 0 , and t0 < t1 , there is w ( t0 ) > w ( t1 ) . That is , when two points have the same SDF value ( thus the same SDF-induced S-density value ) , the point nearer to the view point should have a larger contribution to the final output color than does the other point . An unbiased weight function w ( t ) guarantees that the intersection of the camera ray with the zero-level set of SDF contributes most to the pixel color . The occlusion-aware property ensures that when a ray sequentially passes multiple surfaces , the rendering procedure will correctly use the color of the surface nearest to the camera to compute the output color . Next , we will first introduce a naive way of defining the weight function w ( t ) that directly using the standard pipeline of volume rendering , and explain why it is not appropriate for reconstruction before introducing our novel construction of w ( t ) . Naive solution . To make the weight function occlusion-aware , a natural solution is based on the standard volume rendering formulation [ 28 ] which defines the weight function by w ( t ) = T ( t ) σ ( t ) , ( 3 ) where σ ( t ) is the so-called volume density in classical volume rendering and T ( t ) = exp ( − ∫ t 0 σ ( u ) du ) here denotes the accumulated transmittance along the ray . To adopt the stan- dard volume density formulation [ 28 ] , here σ ( t ) is set to be equal to the S-density value , i.e . σ ( t ) = φs ( f ( p ( t ) ) ) and the weight function w ( t ) is computed by Eqn . 3 . Although the resulting weight function is occlusion-aware , it is biased as it introduces inherent errors in the reconstructed surfaces . As illustrated in Fig . 2 ( a ) , the weight function w ( t ) attains a local maximum at a point before the ray reaches the surface point p ( t∗ ) , satisfying f ( p ( t∗ ) ) = 0 . This fact will be proved in the supplementary material . Our solution . To introduce our solution , we first introduce a straightforward way to construct an unbiased weight function , which directly uses the normalized S-density as weights w ( t ) = φs ( f ( p ( t ) ) ) ∫ +∞ 0 φs ( f ( p ( u ) ) ) du . ( 4 ) This construction of weight function is unbiased , but not occlusion-aware . For example , if the ray penetrates two surfaces , the SDF function f will have two zero points on the ray , which leads to two peaks on the weight function w ( t ) and the resulting weight function will equally blend the colors of two surfaces without considering occlusions . To this end , now we shall design the weight function w ( t ) that is both occlusion-aware and unbiased in the first-order approximation of SDF , based on the aforementioned straightforward construction . To ensure an occlusion-aware property of the weight function w ( t ) , we will still follow the basic framework of volume rendering as Eqn . 3 . However , different from the conventional treatment as in naive solution above , we define our function w ( t ) from the S-density in a new manner . We first define an opaque density function ρ ( t ) , which is the counterpart of the volume density σ in standard volume rendering . Then we compute the new weight function w ( t ) by w ( t ) = T ( t ) ρ ( t ) , where T ( t ) = exp ( − ∫ t 0 ρ ( u ) du ) . ( 5 ) How we derive opaque density ρ . We will first consider a simple case where there is only one surface intersection , and the surface is simply a plane . Since Eqn . 4 indeed satisfies the above requirements under this assumption , we derive the underlying opaque density ρ corresponding to the weight definition of Eqn . 4 using the framework of volume rendering . Then we will generalize this opaque density to the general case of multiple surface intersections . Specifically , in the simple case of a single plane intersection , it is easy to see that the signed distance function f ( p ( t ) ) is −| cos ( θ ) | · ( t− t∗ ) , where f ( p ( t∗ ) ) = 0 , and θ is the angle between the view direction v and the outward surface normal vector n. Because the surface is assumed locally , | cos ( θ ) | is a constant . It follows from Eqn . 4 that w ( t ) = φs ( f ( p ( t ) ) ) ∫ +∞ −∞ φs ( f ( p ( u ) ) ) du = φs ( f ( p ( t ) ) ) ∫ +∞ −∞ φs ( −| cos ( θ ) | · ( u− t∗ ) ) du = φs ( f ( p ( t ) ) ) | cos ( θ ) |−1 · ∫ +∞ −∞ φs ( u− t∗ ) du =| cos ( θ ) |φs ( f ( p ( t ) ) ) . ( 6 ) Recall that the weight function within the framework of volume rendering is given by w ( t ) = T ( t ) ρ ( t ) , where T ( t ) = exp ( − ∫ t 0 ρ ( u ) du ) denotes the accumulated transmittance . Therefore , to derive ρ ( t ) , we have T ( t ) ρ ( t ) = | cos ( θ ) |φs ( f ( p ( t ) ) ) . ( 7 ) Since T ( t ) = exp ( − ∫ t 0 ρ ( u ) du ) , it is easy to verify that T ( t ) ρ ( t ) = −dTdt ( t ) . Further , note that | cos ( θ ) |φs ( f ( p ( t ) ) ) = −dΦsdt ( f ( p ( t ) ) ) . It follows that dT dt ( t ) = dΦs dt ( f ( p ( t ) ) ) . Integrating both sides of this equation yields T ( t ) = Φs ( f ( p ( t ) ) ) . ( 8 ) Taking the logarithm and then differentiating both sides , we have∫ t −∞ ρ ( u ) du =− ln ( Φs ( f ( p ( t ) ) ) ) ⇒ ρ ( t ) = −dΦsdt ( f ( p ( t ) ) ) Φs ( f ( p ( t ) ) ) . ( 9 ) This is the formula of the opaque density ρ ( t ) in case of single plane intersection . The weight function w ( t ) induced by ρ ( t ) is shown in Figure 2 ( b ) . Now we generalize the opaque density to the general case where there are multiple surface intersections along the ray p ( t ) . In this case , −dΦsdt ( f ( p ( t ) ) ) becomes negative on the segment of the ray with increasing SDF values . Thus we clip it against zero to ensure that the value of ρ is always non-negative . This gives the following opaque density function ρ ( t ) in general cases . ρ ( t ) = max ( −dΦsdt ( f ( p ( t ) ) ) Φs ( f ( p ( t ) ) ) , 0 ) . ( 10 ) Based on this equation , the weight function w ( t ) can be computed with standard volume rendering as in Eqn . 5 . The illustration in the case of multiple surface intersection is shown in Figure 3 . The following theorem states that in general cases ( i.e. , including both single surface intersection and multiple surface intersections ) the weight function defined by Eqn . 10 and Eqn . 5 is unbiased in the first-order approximation of SDF . The proof is given in the supplementary material . Theorem 1 Suppose that a smooth surface S is defined by the zero-level set of the signed distance function f ( x ) = 0 , and a ray p ( t ) = o + tv enters the surface S from outside to inside , with the intersection point at p ( t∗ ) , that is , f ( p ( t∗ ) ) = 0 and there exists an interval [ tl , tr ] such that t∗ ∈ [ tl , tr ] and f ( p ( t ) ) is monotonically decreasing in [ tl , tr ] . Suppose that in this local interval [ tl , tr ] , the surface can be tangentially approximated by a sufficiently small planar patch , i.e. , ∇f is regarded as fixed . Then , the weight function w ( t ) computed by Eqn . 10 and Eqn . 5 in [ tl , tr ] attains its maximum at t∗ . Discretization . To obtain discrete counterparts of the opacity and weight function , we adopt the same approximation scheme as used in NeRF [ 28 ] , This scheme samples n points { pi = o+ tiv|i = 1 , ... , n , ti < ti+1 } along the ray to compute the approximate pixel color of the ray as Ĉ = n∑ i=1 Tiαici , ( 11 ) where Ti is the discrete accumulated transmittance defined by Ti = ∏i−1 j=1 ( 1−αj ) , and αi is discrete opacity values defined by αi = 1− exp ( − ∫ ti+1 ti ρ ( t ) dt ) , ( 12 ) which can further be shown to be αi = max ( Φs ( f ( p ( ti ) ) ) − Φs ( f ( p ( ti+1 ) ) ) Φs ( f ( p ( ti ) ) ) , 0 ) . ( 13 ) The detailed derivation of this formula for αi is given in the supplementary material .

Summary: This paper proposes a method that combines volume rendering with surface rendering techniques for multi-view reconstruction. The method represents the geometry of objects using a neural implicit representation that outputs the SDF at query locations. In order to enable volume rendering, the output SDF is mapped to a density function centered around the surface. Furthermore, they also encode the appearance of the object into the neural implicit representation that is needed for multi-view reconstruction. The main contribution is a novel weight function that is centered (unbiased) around the surface while being occlusion-aware. Therefore, the paper resolves major limitations of volume rendering (to few constraints on surface) and surface rendering (very sparse gradients). The paper extensively discusses different formulations of the weight function (properties, weaknesses, strengths). The method is evaluated on the DTU as well as the BlendedMVS dataset and compared to existing state-of-the-art methods like NeRF (volume rendering) and IDR (surface rendering). Moreover, they evaluate some aspects of the method in an ablation study.