Differentiable Gradient Sampling for Learning Implicit 3D Scene Reconstructions from a Single Image

Implicit neural shape functions , e.g . occupancy fields or signed distance functions ( SDF ) , are promising 3D representations for modeling clean mesh surfaces . Existing approaches that use these representations for single-view object reconstruction require supervision signals from every location in the scene , posing difficulties when extending to real-world scenarios where ideal watertight geometric training data is difficult to obtain . In such cases , constraints on the spatial gradient of the implicit field , rather than the value itself , can provide a training signal , but this has not been employed as a source of supervision for single-view reconstruction in part due to the difficulties of differentiably sampling a spatial gradient from a feature map . In this paper , we derive a novel closed-form Differentiable Gradient Sampling ( DGS ) solution that enables backpropagation of the loss on spatial gradients to the feature maps , thus allowing training on large-scale scenes without dense 3D supervision . As a result , we demonstrate single view implicit surface reconstructions on real-world scenes via learning directly from a scanned dataset . Our model performs well when generalizing to unseen images from Pix3D or downloaded directly from the internet ( Fig . 1 ) . We comfortably advance the state-of-the-art with results on several established datasets including ShapeNet and ScannetV2 ; extensive quantitative analysis confirms that our proposed DGS module plays an essential role in achieving this performance improvement . Full codes are available in MaskedURL . 1 INTRODUCTION . Recent studies have shown promising learning capabilities of implicit models for 3D geometry representations ( Park et al. , 2019 ; Mescheder et al. , 2019 ; Mildenhall et al. , 2020 ; Sitzmann et al. , 2019 ; Gropp et al. , 2020 ; Saito et al. , 2019 ; Yu et al. , 2020 ) . Rather than explicitly representing 3D geometry with a textured mesh ( Kato et al. , 2018 ) or point cloud ( Fan et al. , 2017 ) , implicit representations express a function of points in the space . Among many recent promising implicit models , the occupancy field ( Mescheder et al. , 2019 ; Popov et al. , 2020 ; Chen & Zhang , 2019b ) and the Signed Distance Field ( SDF ) ( Park et al. , 2019 ; Xu et al. , 2019 ; Li & Zhang , 2021 ) are particularly suitable for high resolution mesh surface reconstruction . Given a query point ( x , y , z ) ∈ R3 in the space , the occupancy function returns the occupancy status of the point o ( x , y , z ) ∈ { + , − } , while the SDF value encompasses the closest distance from a point to the surface of the given 3D geometry , as well as its sign for its occupancy . The mesh surface is defined by the classification boundary of the occupancy field or the zero level-set of the SDF ( Lorensen & Cline , 1987 ) . Despite the strong representational power of these models and their success on predicting single view objects , learning to predict complex 3D geometry , such as scenes , from a real world image remains a challenge for this class of methods . Obtaining the occupancy label , or the sign , from noisy non-watertight meshes is non-trivial ( Atzmon & Lipman , 2020 ) ( Fig . 2 ) . Furthermore , producing the label on the fly from a large number of triangle meshes , as encountered in typical real-world scenes with complex geometry , can be computationally prohibitive ( Straub et al. , 2019 ; Chang et al. , 2017 ; Dai et al. , 2017 ; Song et al. , 2017 ) . Storing sufficiently dense query points with pre-computed occupancies or SDF for complex geometries , or meta data of meshes like octrees or hashing , is a possibility , but incurs severe engineering and computational burdens on storage and runtime loading ( Yu et al. , 2021 ) . The challenge is exacerbated for predicting scenes in particular , since unlike objects scenes in the view frustrum typically can not be enclosed by a pre-defined range of space and do not have a categorical canonical coordinate system ( Popov et al. , 2020 ) . Recent studies on fitting a single scene geometry ( Gropp et al. , 2020 ; Sitzmann et al. , 2020 ) have pointed out the value of the 3D spatial gradient of the implicit field . For instance , the gradients of the SDF on surface points are actually the surface normals and the 2-norm of the spatial gradients for any point in the space is 1 , known as the Eikonal equation ( Crandall & Lions , 1983 ) . Similarly , for the occupancy case , we propose to regularize the gradient of the occupancy field to be zero away from the surface ( Fig . 3 ( c ) ) . Thus , one can train the full occupancy field or SDF with only the surface point clouds , thanks to these constraints on the gradients of the implicit shape models , even when the value of the occupancy or the SDF itself is not available everywhere . However , existing approaches ( Gropp et al. , 2020 ; Sitzmann et al. , 2020 ) have only demonstrated this benefit in the cases of geometry only inputs , where a network is fit to a particular scene per model without any conditioning on input images . When the locally sampled image features are used for the feedforward prediction , it is necessary to derive a differentiable gradient sampling solution over the feature pixel map . In this paper , we tackle this challenge by deriving a closed-form differentiable gradient sampling ( DGS ) solution for learning a single-view 3D implicit reconstruction model ( Fig . 4 ) . Our novel propagation of the loss gradient of the spatial gradient back to the feature maps ( Eq . 6 ) serves in addition to the existing spatial gradient sampling ( Eq . 5 ) and loss gradient back-propagation ( Eq . 4 ) used in existing deep learning frameworks ( Paszke et al. , 2019 ; Abadi et al. , 2016 ) . The resulting end-to-end learning scheme with supervision over the spatial gradients opens the potential to train a model that can generalize to unseen test cases of complex scenes with only surface point cloud supervision — a typical setting for real-world scene data — without requiring ground truth perquery-point labels . Our contributions include a framework to propagate spatial gradients through a spatial feature sampling procedure , and a novel closed-form DGS solution . We further show that DGS enables training a single-image 3D implicit reconstruction model that can generalize to unseen scenes , with only imperfect surface annotations as the supervision ( e.g . non-watertight meshes ) . Finally , experiments on synthetic ( ShapeNet Chang et al . ( 2015 ) ) and real-scanned data ( ScannetV2 ( Dai et al. , 2017 ) ) indicate that we clearly advanced state-of-the-art accuracy . To the best of our knowledge , DGS-enabled shape inference provides the first single-view implicit shape reconstruction on real scene datasets which can generalize accurately to unseen scenes from different datasets or domains ( see Figure 1 ) . 2 RELATED WORKS 3D Implicit Representations Among all the 3D representations , implicit models are advantageous for arbitrarily high resolution modeling ( unlike voxels with fixed resolution and no detailed surface modeling ) and easy learning ( unlike mesh with varying topological structures ) . Implicit models typically learn to establish a mapping between a query point and the prediction of the point . Groueix et al . proposed to learn a mapping from the uv texture map to the 3D surface point . More recent works ( Park et al. , 2019 ; Mescheder et al. , 2019 ; Xu et al. , 2019 ; Popov et al. , 2020 ; Chen & Zhang , 2019a ) focus on mapping the query point coordinates to the signed distance field or the occupancy field . In addition to these pure geometry modeling , recent works like ( Saito et al. , 2019 ) also model the surface texture via learning a mapping from the surface point to the RGB value , or use the RGB loss as the supervison signal ( Niemeyer et al. , 2020 ; Yariv et al. , 2020 ) . In contrast to defining textures explicitly on the surface , Mildenhall et al . ( 2020 ) proposed the volumetric rendering representation which maps the point coordinate and the viewing angle to volumetric textures . Oechsle et al . ( 2021 ) ; Wang et al . ( 2021 ) ; Yariv et al . ( 2021 ) use volume renderings for fitting high quality geometry of single scenes , but these are not generalizable to unseen scenes . In multi-view stereo setting , Murez et al . ( 2020 ) ; Sun et al . ( 2021 ) ; Božič et al . ( 2021 ) proposed to accumulate the multi-view predictions in a TSDF voxel volume . We believe our proposed DGS module is necessary when extending most of the existing implicit surface representations into feed-forward models that are conditioned on input image ( s ) , and training these models with a loss on the gradient of the predicted field . In this work , we focus on pure geometrical reconstructions from single view , as an example of the application of the proposed DGS module . Differentiable Operations End-to-end deep learning requires all modules in a computation path to be differentiable . Many differentiable modules are proposed to serve a particular functionality . The spatial transformer layer ( Jaderberg et al. , 2015 ) is one of the early attempts for differentiably sam- pling the feature map . Kato et al . ( 2018 ) proposed to render the mesh into image in a differentiable manner so that we can optimize the mesh or the camera parameters to fit to a particular image . A similar idea has also been applied for rendering point clouds ( Wiles et al. , 2020 ; Yifan et al. , 2019 ) . Tulsiani et al . ( 2017 ) and Mildenhall et al . ( 2020 ) proposed to compute the ray rendering in an accumulating manner . One common feature of these differentiable operations is a strategy to soften the existing explicit modeling ( Liu et al. , 2019 ) , and replace the gradient of the loss with approximate signals , making sure those signals provide guidance in the right direction . Our differentiable gradient sampler also seeks to soften the spatial gradient between the adjacent pixels . 3 LEARNING FRAMEWORK . Problem definition and notations . We aim to train a feed-forward deep model for predicting the 3D implicit surface reconstruction conditioned on a single image . We denote the input RGB image as I ∈ Rm×n×3 . Like most feed-forward models , our model contains an image encoder . We denote the extracted 2D features as ϕ . Our model f̂Θ ( · ) , parameterized by Θ , takes in the image feature ϕ and predicts for each 3D location ( x , y , z ) the implicit value f̂Θ ( ϕ ; x , y , z ) . We denote the ground truth value as f̃ ( ϕ ; x , y , z ) . For the occupancy field , the implicit field value f represents the occupancy probability . We denote the predicted occupancy probability as ôΘ ( ϕ ; x , y , z ) ∈ [ 0 , 1 ] and the ground truth binary occupancy label as õ ( ϕ ; x , y , z ) ∈ { “ − ” , “ + ” } , where “ − ” represents unoccupied space and “ + ” vice versa . For SDF , the implicit field value f represents the signed distance between the query point and its projection on the surface . We denote the predicted and the ground truth signed distance as ŝΘ ( ϕ ; x , y , z ) ∈ R and s̃ ( ϕ ; x , y , z ) ∈ R respectively . Background . Typical fully supervised training ( Xu et al. , 2019 ; Popov et al. , 2020 ) imposes the loss to each individual sampled query points by comparing the predicted field value f̂Θ ( ϕ ; x , y , z ) with the ground truth value f̃ ( ϕ ; x , y , z ) . For occupancy predictions , we compute the loss as∑ { I , PI } ∈D ∑ ( x , y , z ) ∈PI BCE ( ôΘ ( ϕ ; x , y , z ) , õΘ ( ϕ ; x , y , z ) ) ( Fig . 3 ( a ) ) , where “ BCE ” represents the binary cross entropy loss , D represents the whole single-view image training set , and PI represents the set of all the possible query point sampling locations ( x , y , z ) within the view frustum . For SDF , we compute the loss as ∑ { I , PI } ∈D ∑ ( x , y , z ) ∈PI |ŝΘ ( ϕ ; x , y , z ) − s̃ ( ϕ ; x , y , z ) | .

Summary: This paper presents a new method to learn implicit 3D scene reconstructions from single image input. The main improvement is a closed-form Differentiable Gradient Sampling.  By taking spatial gradient into  consideration, the proposed method can apply back-propagation of the  loss on spatial gradients to feature maps and allow the training for the case of without dense 3D supervision.