Differentiable Gradient Sampling for Learning Implicit 3D Scene Reconstructions from a Single Image

Implicit neural shape functions , e.g . occupancy fields or signed distance functions ( SDF ) , are promising 3D representations for modeling clean mesh surfaces . Existing approaches that use these representations for single-view object reconstruction require supervision signals from every location in the scene , posing difficulties when extending to real-world scenarios where ideal watertight geometric training data is difficult to obtain . In such cases , constraints on the spatial gradient of the implicit field , rather than the value itself , can provide a training signal , but this has not been employed as a source of supervision for single-view reconstruction in part due to the difficulties of differentiably sampling a spatial gradient from a feature map . In this paper , we derive a novel closed-form Differentiable Gradient Sampling ( DGS ) solution that enables backpropagation of the loss on spatial gradients to the feature maps , thus allowing training on large-scale scenes without dense 3D supervision . As a result , we demonstrate single view implicit surface reconstructions on real-world scenes via learning directly from a scanned dataset . Our model performs well when generalizing to unseen images from Pix3D or downloaded directly from the internet ( Fig . 1 ) . We comfortably advance the state-of-the-art with results on several established datasets including ShapeNet and ScannetV2 ; extensive quantitative analysis confirms that our proposed DGS module plays an essential role in achieving this performance improvement . Full codes are available in MaskedURL . 1 INTRODUCTION . Recent studies have shown promising learning capabilities of implicit models for 3D geometry representations ( Park et al. , 2019 ; Mescheder et al. , 2019 ; Mildenhall et al. , 2020 ; Sitzmann et al. , 2019 ; Gropp et al. , 2020 ; Saito et al. , 2019 ; Yu et al. , 2020 ) . Rather than explicitly representing 3D geometry with a textured mesh ( Kato et al. , 2018 ) or point cloud ( Fan et al. , 2017 ) , implicit representations express a function of points in the space . Among many recent promising implicit models , the occupancy field ( Mescheder et al. , 2019 ; Popov et al. , 2020 ; Chen & Zhang , 2019b ) and the Signed Distance Field ( SDF ) ( Park et al. , 2019 ; Xu et al. , 2019 ; Li & Zhang , 2021 ) are particularly suitable for high resolution mesh surface reconstruction . Given a query point ( x , y , z ) ∈ R3 in the space , the occupancy function returns the occupancy status of the point o ( x , y , z ) ∈ { + , − } , while the SDF value encompasses the closest distance from a point to the surface of the given 3D geometry , as well as its sign for its occupancy . The mesh surface is defined by the classification boundary of the occupancy field or the zero level-set of the SDF ( Lorensen & Cline , 1987 ) . Despite the strong representational power of these models and their success on predicting single view objects , learning to predict complex 3D geometry , such as scenes , from a real world image remains a challenge for this class of methods . Obtaining the occupancy label , or the sign , from noisy non-watertight meshes is non-trivial ( Atzmon & Lipman , 2020 ) ( Fig . 2 ) . Furthermore , producing the label on the fly from a large number of triangle meshes , as encountered in typical real-world scenes with complex geometry , can be computationally prohibitive ( Straub et al. , 2019 ; Chang et al. , 2017 ; Dai et al. , 2017 ; Song et al. , 2017 ) . Storing sufficiently dense query points with pre-computed occupancies or SDF for complex geometries , or meta data of meshes like octrees or hashing , is a possibility , but incurs severe engineering and computational burdens on storage and runtime loading ( Yu et al. , 2021 ) . The challenge is exacerbated for predicting scenes in particular , since unlike objects scenes in the view frustrum typically can not be enclosed by a pre-defined range of space and do not have a categorical canonical coordinate system ( Popov et al. , 2020 ) . Recent studies on fitting a single scene geometry ( Gropp et al. , 2020 ; Sitzmann et al. , 2020 ) have pointed out the value of the 3D spatial gradient of the implicit field . For instance , the gradients of the SDF on surface points are actually the surface normals and the 2-norm of the spatial gradients for any point in the space is 1 , known as the Eikonal equation ( Crandall & Lions , 1983 ) . Similarly , for the occupancy case , we propose to regularize the gradient of the occupancy field to be zero away from the surface ( Fig . 3 ( c ) ) . Thus , one can train the full occupancy field or SDF with only the surface point clouds , thanks to these constraints on the gradients of the implicit shape models , even when the value of the occupancy or the SDF itself is not available everywhere . However , existing approaches ( Gropp et al. , 2020 ; Sitzmann et al. , 2020 ) have only demonstrated this benefit in the cases of geometry only inputs , where a network is fit to a particular scene per model without any conditioning on input images . When the locally sampled image features are used for the feedforward prediction , it is necessary to derive a differentiable gradient sampling solution over the feature pixel map . In this paper , we tackle this challenge by deriving a closed-form differentiable gradient sampling ( DGS ) solution for learning a single-view 3D implicit reconstruction model ( Fig . 4 ) . Our novel propagation of the loss gradient of the spatial gradient back to the feature maps ( Eq . 6 ) serves in addition to the existing spatial gradient sampling ( Eq . 5 ) and loss gradient back-propagation ( Eq . 4 ) used in existing deep learning frameworks ( Paszke et al. , 2019 ; Abadi et al. , 2016 ) . The resulting end-to-end learning scheme with supervision over the spatial gradients opens the potential to train a model that can generalize to unseen test cases of complex scenes with only surface point cloud supervision — a typical setting for real-world scene data — without requiring ground truth perquery-point labels . Our contributions include a framework to propagate spatial gradients through a spatial feature sampling procedure , and a novel closed-form DGS solution . We further show that DGS enables training a single-image 3D implicit reconstruction model that can generalize to unseen scenes , with only imperfect surface annotations as the supervision ( e.g . non-watertight meshes ) . Finally , experiments on synthetic ( ShapeNet Chang et al . ( 2015 ) ) and real-scanned data ( ScannetV2 ( Dai et al. , 2017 ) ) indicate that we clearly advanced state-of-the-art accuracy . To the best of our knowledge , DGS-enabled shape inference provides the first single-view implicit shape reconstruction on real scene datasets which can generalize accurately to unseen scenes from different datasets or domains ( see Figure 1 ) . 2 RELATED WORKS 3D Implicit Representations Among all the 3D representations , implicit models are advantageous for arbitrarily high resolution modeling ( unlike voxels with fixed resolution and no detailed surface modeling ) and easy learning ( unlike mesh with varying topological structures ) . Implicit models typically learn to establish a mapping between a query point and the prediction of the point . Groueix et al . proposed to learn a mapping from the uv texture map to the 3D surface point . More recent works ( Park et al. , 2019 ; Mescheder et al. , 2019 ; Xu et al. , 2019 ; Popov et al. , 2020 ; Chen & Zhang , 2019a ) focus on mapping the query point coordinates to the signed distance field or the occupancy field . In addition to these pure geometry modeling , recent works like ( Saito et al. , 2019 ) also model the surface texture via learning a mapping from the surface point to the RGB value , or use the RGB loss as the supervison signal ( Niemeyer et al. , 2020 ; Yariv et al. , 2020 ) . In contrast to defining textures explicitly on the surface , Mildenhall et al . ( 2020 ) proposed the volumetric rendering representation which maps the point coordinate and the viewing angle to volumetric textures . Oechsle et al . ( 2021 ) ; Wang et al . ( 2021 ) ; Yariv et al . ( 2021 ) use volume renderings for fitting high quality geometry of single scenes , but these are not generalizable to unseen scenes . In multi-view stereo setting , Murez et al . ( 2020 ) ; Sun et al . ( 2021 ) ; Božič et al . ( 2021 ) proposed to accumulate the multi-view predictions in a TSDF voxel volume . We believe our proposed DGS module is necessary when extending most of the existing implicit surface representations into feed-forward models that are conditioned on input image ( s ) , and training these models with a loss on the gradient of the predicted field . In this work , we focus on pure geometrical reconstructions from single view , as an example of the application of the proposed DGS module . Differentiable Operations End-to-end deep learning requires all modules in a computation path to be differentiable . Many differentiable modules are proposed to serve a particular functionality . The spatial transformer layer ( Jaderberg et al. , 2015 ) is one of the early attempts for differentiably sam- pling the feature map . Kato et al . ( 2018 ) proposed to render the mesh into image in a differentiable manner so that we can optimize the mesh or the camera parameters to fit to a particular image . A similar idea has also been applied for rendering point clouds ( Wiles et al. , 2020 ; Yifan et al. , 2019 ) . Tulsiani et al . ( 2017 ) and Mildenhall et al . ( 2020 ) proposed to compute the ray rendering in an accumulating manner . One common feature of these differentiable operations is a strategy to soften the existing explicit modeling ( Liu et al. , 2019 ) , and replace the gradient of the loss with approximate signals , making sure those signals provide guidance in the right direction . Our differentiable gradient sampler also seeks to soften the spatial gradient between the adjacent pixels . 3 LEARNING FRAMEWORK . Problem definition and notations . We aim to train a feed-forward deep model for predicting the 3D implicit surface reconstruction conditioned on a single image . We denote the input RGB image as I ∈ Rm×n×3 . Like most feed-forward models , our model contains an image encoder . We denote the extracted 2D features as ϕ . Our model f̂Θ ( · ) , parameterized by Θ , takes in the image feature ϕ and predicts for each 3D location ( x , y , z ) the implicit value f̂Θ ( ϕ ; x , y , z ) . We denote the ground truth value as f̃ ( ϕ ; x , y , z ) . For the occupancy field , the implicit field value f represents the occupancy probability . We denote the predicted occupancy probability as ôΘ ( ϕ ; x , y , z ) ∈ [ 0 , 1 ] and the ground truth binary occupancy label as õ ( ϕ ; x , y , z ) ∈ { “ − ” , “ + ” } , where “ − ” represents unoccupied space and “ + ” vice versa . For SDF , the implicit field value f represents the signed distance between the query point and its projection on the surface . We denote the predicted and the ground truth signed distance as ŝΘ ( ϕ ; x , y , z ) ∈ R and s̃ ( ϕ ; x , y , z ) ∈ R respectively . Background . Typical fully supervised training ( Xu et al. , 2019 ; Popov et al. , 2020 ) imposes the loss to each individual sampled query points by comparing the predicted field value f̂Θ ( ϕ ; x , y , z ) with the ground truth value f̃ ( ϕ ; x , y , z ) . For occupancy predictions , we compute the loss as∑ { I , PI } ∈D ∑ ( x , y , z ) ∈PI BCE ( ôΘ ( ϕ ; x , y , z ) , õΘ ( ϕ ; x , y , z ) ) ( Fig . 3 ( a ) ) , where “ BCE ” represents the binary cross entropy loss , D represents the whole single-view image training set , and PI represents the set of all the possible query point sampling locations ( x , y , z ) within the view frustum . For SDF , we compute the loss as ∑ { I , PI } ∈D ∑ ( x , y , z ) ∈PI |ŝΘ ( ϕ ; x , y , z ) − s̃ ( ϕ ; x , y , z ) | .

Summary: This paper describes novel loss functions for learning to predict an implicit 3D scene representation from a single image.  They argue that when working with real scan data of scenes (rather than single objects) it is difficult to generate accurate occupancy or signed distance function (SDF) ground truth as would be required for supervised learning.  Instead, they propose to only use occupancy or SDF supervision near the surfaces of objects; elsewhere, they rely on constraints on the gradient of the occupancy or SDF adapted from Gropp et al. 2020.  They perform a thorough evaluation on several benchmark datasets and compare against state-of-the-art competing methods.  They show that they outperform competing methods, even though in some cases their method has access to less supervisory data.    They also perform an ablation study to show the importance of various parts of the loss function.