DIFFERENTIABLE GRADIENT SAMPLING FOR LEARNING IMPLICIT 3D SCENE RECONSTRUCTIONS
Year: 2021

Abstract:
Implicit neural shape functions, e.g. occupancy fields or signed distance functions (SDF), are promising 3D representations for modeling clean mesh surfaces. Existing approaches that use these representations for single-view object reconstruction require supervision signals from every location in the scene, posing difficulties when extending to real-world scenarios where ideal watertight geometric training data is difficult to obtain. In such cases, constraints on the spatial gradient of the implicit field, rather than the value itself, can provide a training signal, but this has not been employed as a source of supervision for single-view reconstruction in part due to the difficulties of differentiably sampling a spatial gradient from a feature map. In this paper, we derive a novel closed-form Differentiable Gradient Sampling (DGS) solution that enables backpropagation of the loss on spatial gradients to the feature maps, thus allowing training on large-scale scenes without dense 3D supervision. As a result, we demonstrate single view implicit surface reconstructions on real-world scenes via learning directly from a scanned dataset. Our model performs well when generalizing to unseen images from Pix3D or downloaded directly from the internet (Fig. 1). We comfortably advance the state-of-the-art with results on several established datasets including ShapeNet and ScannetV2; extensive quantitative analysis confirms that our proposed DGS module plays an essential role in achieving this performance improvement. Full codes are available in MaskedURL. View 0 View 1 View 2 View 3 View 4 View 5 Figure 1: Given a single RGB image as the input (the first column in each row), our model can predict its 3D implicit surface reconstruction (shown in six novel views in the last six columns). The test images for the first 3 rows are downloaded from the Internet and the last 2 rows are from the pix3d dataset. 1 Under review as a conference paper at ICLR 2022

text: Implicit neural shape functions, e.g. occupancy fields or signed distance functions (SDF), are promising 3D representations for modeling clean mesh surfaces. Existing approaches that use these representations for single-view object reconstruction require supervision signals from every location in the scene, posing difficulties when extending to real-world scenarios where ideal watertight geometric training data is difficult to obtain. In such cases, constraints on the spatial gradient of the implicit field, rather than the value itself, can provide a training signal, but this has not been employed as a source of supervision for single-view reconstruction in part due to the difficulties of differentiably sampling a spatial gradient from a feature map. In this paper, we derive a novel closed-form Differentiable Gradient Sampling (DGS) solution that enables backpropagation of the loss on spatial gradients to the feature maps, thus allowing training on large-scale scenes without dense 3D supervision. As a result, we demonstrate single view implicit surface reconstructions on real-world scenes via learning directly from a scanned dataset. Our model performs well when generalizing to unseen images from Pix3D or downloaded directly from the internet (Fig. 1). We comfortably advance the state-of-the-art with results on several established datasets including ShapeNet and ScannetV2; extensive quantitative analysis confirms that our proposed DGS module plays an essential role in achieving this performance improvement. Full codes are available in MaskedURL.

heading: 1 INTRODUCTION
text: Recent studies have shown promising learning capabilities of implicit models for 3D geometry representations (Park et al., 2019; Mescheder et al., 2019; Mildenhall et al., 2020; Sitzmann et al., 2019; Gropp et al., 2020; Saito et al., 2019; Yu et al., 2020). Rather than explicitly representing 3D geometry with a textured mesh (Kato et al., 2018) or point cloud (Fan et al., 2017), implicit representations express a function of points in the space. Among many recent promising implicit models, the occupancy field (Mescheder et al., 2019; Popov et al., 2020; Chen & Zhang, 2019b) and the Signed Distance Field (SDF) (Park et al., 2019; Xu et al., 2019; Li & Zhang, 2021) are particularly suitable for high resolution mesh surface reconstruction. Given a query point (x, y, z) ∈ R3 in the space, the occupancy function returns the occupancy status of the point o(x, y, z) ∈ {+,−}, while the SDF value encompasses the closest distance from a point to the surface of the given 3D geometry, as well as its sign for its occupancy. The mesh surface is defined by the classification boundary of the occupancy field or the zero level-set of the SDF (Lorensen & Cline, 1987).
Despite the strong representational power of these models and their success on predicting single view objects, learning to predict complex 3D geometry, such as scenes, from a real world image remains a challenge for this class of methods. Obtaining the occupancy label, or the sign, from noisy non-watertight meshes is non-trivial (Atzmon & Lipman, 2020) (Fig. 2). Furthermore, producing the label on the fly from a large number of triangle meshes, as encountered in typical real-world scenes with complex geometry, can be computationally prohibitive (Straub et al., 2019; Chang et al., 2017; Dai et al., 2017; Song et al., 2017). Storing sufficiently dense query points with pre-computed occupancies or SDF for complex geometries, or meta data of meshes like octrees or hashing, is a possibility, but incurs severe engineering and computational burdens on storage and runtime loading (Yu et al., 2021). The challenge is exacerbated for predicting scenes in particular, since unlike objects scenes in the view frustrum typically cannot be enclosed by a pre-defined range of space and do not have a categorical canonical coordinate system (Popov et al., 2020).
Recent studies on fitting a single scene geometry (Gropp et al., 2020; Sitzmann et al., 2020) have pointed out the value of the 3D spatial gradient of the implicit field. For instance, the gradients of the SDF on surface points are actually the surface normals and the 2-norm of the spatial gradients for any point in the space is 1, known as the Eikonal equation (Crandall & Lions, 1983). Similarly, for the occupancy case, we propose to regularize the gradient of the occupancy field to be zero away from the surface (Fig. 3(c)). Thus, one can train the full occupancy field or SDF with only the surface point clouds, thanks to these constraints on the gradients of the implicit shape models, even when the value of the occupancy or the SDF itself is not available everywhere. However, existing approaches (Gropp et al., 2020; Sitzmann et al., 2020) have only demonstrated this benefit in the cases of geometry only inputs, where a network is fit to a particular scene per model without any conditioning on input images. When the locally sampled image features are used for the feedforward prediction, it is necessary to derive a differentiable gradient sampling solution over the feature pixel map.
In this paper, we tackle this challenge by deriving a closed-form differentiable gradient sampling (DGS) solution for learning a single-view 3D implicit reconstruction model (Fig. 4). Our novel propagation of the loss gradient of the spatial gradient back to the feature maps (Eq. 6) serves in addition to the existing spatial gradient sampling (Eq. 5) and loss gradient back-propagation (Eq. 4) used in existing deep learning frameworks (Paszke et al., 2019; Abadi et al., 2016). The resulting end-to-end learning scheme with supervision over the spatial gradients opens the potential to train a model that can generalize to unseen test cases of complex scenes with only surface point cloud supervision — a typical setting for real-world scene data — without requiring ground truth perquery-point labels.
Our contributions include a framework to propagate spatial gradients through a spatial feature sampling procedure, and a novel closed-form DGS solution. We further show that DGS enables training a single-image 3D implicit reconstruction model that can generalize to unseen scenes, with only imperfect surface annotations as the supervision (e.g. non-watertight meshes). Finally, experiments on synthetic (ShapeNet Chang et al. (2015)) and real-scanned data (ScannetV2 (Dai et al., 2017)) indicate that we clearly advanced state-of-the-art accuracy. To the best of our knowledge, DGS-enabled shape inference provides the first single-view implicit shape reconstruction on real scene datasets which can generalize accurately to unseen scenes from different datasets or domains (see Figure 1).
2 RELATED WORKS
3D Implicit Representations Among all the 3D representations, implicit models are advantageous for arbitrarily high resolution modeling (unlike voxels with fixed resolution and no detailed surface modeling) and easy learning (unlike mesh with varying topological structures). Implicit models typically learn to establish a mapping between a query point and the prediction of the point. Groueix et al. proposed to learn a mapping from the uv texture map to the 3D surface point. More recent works (Park et al., 2019; Mescheder et al., 2019; Xu et al., 2019; Popov et al., 2020; Chen & Zhang, 2019a) focus on mapping the query point coordinates to the signed distance field or the occupancy field. In addition to these pure geometry modeling, recent works like (Saito et al., 2019) also model the surface texture via learning a mapping from the surface point to
the RGB value, or use the RGB loss as the supervison signal (Niemeyer et al., 2020; Yariv et al., 2020). In contrast to defining textures explicitly on the surface, Mildenhall et al. (2020) proposed the volumetric rendering representation which maps the point coordinate and the viewing angle to volumetric textures. Oechsle et al. (2021); Wang et al. (2021); Yariv et al. (2021) use volume renderings for fitting high quality geometry of single scenes, but these are not generalizable to unseen scenes. In multi-view stereo setting, Murez et al. (2020); Sun et al. (2021); Božič et al. (2021) proposed to accumulate the multi-view predictions in a TSDF voxel volume. We believe our proposed DGS module is necessary when extending most of the existing implicit surface representations into feed-forward models that are conditioned on input image(s), and training these models with a loss on the gradient of the predicted field. In this work, we focus on pure geometrical reconstructions from single view, as an example of the application of the proposed DGS module.
Differentiable Operations End-to-end deep learning requires all modules in a computation path to be differentiable. Many differentiable modules are proposed to serve a particular functionality. The spatial transformer layer (Jaderberg et al., 2015) is one of the early attempts for differentiably sam-
pling the feature map. Kato et al. (2018) proposed to render the mesh into image in a differentiable manner so that we can optimize the mesh or the camera parameters to fit to a particular image. A similar idea has also been applied for rendering point clouds (Wiles et al., 2020; Yifan et al., 2019). Tulsiani et al. (2017) and Mildenhall et al. (2020) proposed to compute the ray rendering in an accumulating manner. One common feature of these differentiable operations is a strategy to soften the existing explicit modeling (Liu et al., 2019), and replace the gradient of the loss with approximate signals, making sure those signals provide guidance in the right direction. Our differentiable gradient sampler also seeks to soften the spatial gradient between the adjacent pixels.

heading: 3 LEARNING FRAMEWORK
text: Problem definition and notations. We aim to train a feed-forward deep model for predicting the 3D implicit surface reconstruction conditioned on a single image. We denote the input RGB image as I ∈ Rm×n×3. Like most feed-forward models, our model contains an image encoder. We denote the extracted 2D features as ϕ. Our model f̂Θ(·), parameterized by Θ, takes in the image feature ϕ and predicts for each 3D location (x, y, z) the implicit value f̂Θ(ϕ;x, y, z). We denote the ground truth value as f̃(ϕ;x, y, z). For the occupancy field, the implicit field value f represents the occupancy probability. We denote the predicted occupancy probability as ôΘ(ϕ;x, y, z) ∈ [0, 1] and the ground truth binary occupancy label as õ(ϕ;x, y, z) ∈ {“ − ”, “ + ”}, where “−” represents unoccupied space and “+” vice versa. For SDF, the implicit field value f represents the signed distance between the query point and its projection on the surface. We denote the predicted and the ground truth signed distance as ŝΘ(ϕ;x, y, z) ∈ R and s̃(ϕ;x, y, z) ∈ R respectively. Background. Typical fully supervised training (Xu et al., 2019; Popov et al., 2020) imposes the loss to each individual sampled query points by comparing the predicted field value f̂Θ(ϕ;x, y, z) with the ground truth value f̃(ϕ;x, y, z). For occupancy predictions, we compute the loss as∑
{I,PI}∈D ∑
(x,y,z)∈PI BCE(ôΘ(ϕ;x, y, z), õΘ(ϕ;x, y, z)) (Fig. 3(a)), where “BCE” represents the binary cross entropy loss, D represents the whole single-view image training set, and PI represents the set of all the possible query point sampling locations (x, y, z) within the view frustum. For SDF, we compute the loss as ∑ {I,PI}∈D ∑ (x,y,z)∈PI |ŝΘ(ϕ;x, y, z)− s̃(ϕ;x, y, z)|.

heading: 3.1 PROPOSED FRAMEWORK
text: As obtaining the full labels f̃(ϕ;x, y, z) for query points (x, y, z) from all the locations PI is nontrivial for complex geometry from real-world (Fig. 2) and due to additional engineering constraints on runtime loading and memory bottleneck for the sufficintly dense pre-computed query point occypancy labels, we propose to incorporate the loss w.r.t. the spatial gradients. For occupancy,
L = ∑
{I,PI}∈D
( ∑
(x,y,z)∈PI−P0I
λor∥∇x,y,z ôΘ(ϕ;x, y, z)∥
+ ∑
(x,y,z)∈P0+I
BCE(ôΘ(ϕ;x, y, z), “ + ”) + ∑
(x,y,z)∈P0−I
BCE(ôΘ(ϕ;x, y, z), “− ”)) (1)
where P0+I and P 0− I denotes inward and outward near-surface query points, and λor represents the loss weight of the occupancy geometrical regularization. The facing direction of the mesh surface (determining inward or outward) can be determined by normals (if available), or via rendering the surface in all views as in the RGBD captures (Dai et al., 2017) (with the surface facing toward camera as the “outward” side). The three terms in Eq. 1 correspond to the three types of losses in Fig. 3(c) (“∇”, “+”, “−”) respectively. For SDF, we set the loss similar to Gropp et al. (2020):
L = ∑
{I,PI}∈D
( ∑
(x,y,z)∈PI
λsr|∥∇x,y,z ŝΘ(ϕ;x, y, z)∥2 − 1|+ ∑
(x,y,z)∈P0I
|ŝΘ(ϕ;x, y, z)|
+ ∑
(x,y,z)∈P0I
λsn∥∇x,y,z ŝΘ(ϕ;x, y, z)−∇x,y,z s̃(ϕ;x, y, z)∥) (2)
where P0I denotes the query points on the ground truth surface only, and λsr and λsn represents the loss weight for the signed distance geometric regularization term and normal term respectively. The three terms represents the Eikonal regularization (Crandall & Lions, 1983), surface zero-SDF loss, and the surface normal loss respectively, with the last term optional (Gropp et al., 2020). Note now
the loss functions in Eq. 1, 2 no longer require the implicit ground truth label for query points far away from the ground truth surface, enabling training with the real-world imperfect scanned data. In both cases, the availability of surface normals is optional.
Practically, we found only our proposed loss for the occupancy version (Fig. 3(c), Eq. 1) demonstrates capability to be applicable for the real-world single-view feed-forward scenario, and use Eq. 1 rather than Eq. 2 in all of our experiments. First, learning with the Eikonal term (Eq. 2) suffers severely from its sensitivity of model initialization. We found either the SDF or the Truncated SDF (TSDF) representation model cannot easily converge during training. Specifically, we found it poses numerical difficulties when the model is learned to predict a large distance value or a constant truncation value for the majority of the query points in the air that is far from any surfaces. This becomes a major problem when we extend from the single objects scenario (Mescheder et al., 2019; Popov et al., 2020) or single scene fitting (Gropp et al., 2020). Second, our loss function (Eq. 1) can significantly save memory footprint during training and enable large batch size training, which is crucial in our learning framework. Unlike in Eq. 2 where all the query points require spatial gradient computation, which leads to 3× higher of memory footprint, in our loss function (Eq. 1), only the non-surface query points do. We set the query point batch size of the non-surface points (512 in practice) to be much smaller than the critical near-surface points (4096 in practice), enabling learning with the image batch size of 32 in our real-scene training.

heading: 3.2 SAMPLING WITH DIFFERENTIABLE GRADIENTS
text: A distinct difference compared to existing works is that our spatial gradient ∇x,y,z f̂Θ(ϕ;x, y, z) is also conditioned on the pixels in ϕ. This is in stark contrast to existing works. Gropp et al. (2020) devised the model fΘ fits to a single scene, and the spatial gradient ∇x,y,z f̂Θ(x, y, z) can be conveniently computed because it does not involve the sampling procedure. Jiang et al. (2020); Lin et al. (2020) used a non-spatial global feature for inference and hence bypassed sampling. In our learning framework, the spatial gradient computation must undergo the sampling procedure.
We name our gradient computation involving the sampling operation as the Pixel Conditioned Gradients and derive a closed-form solution, Differentiable Gradient Sampling (DGS), for handling forward and backward propagation. Figure 4(a) provides an illustration of our training pipeline. Each layer in our network tracks both the response of the layer as well as its spatial gradient w.r.t. (x, y, z). While it is well established to track the layer-wise spatial gradient for fully-connected (FC) layers or convolutions in existing works (Gropp et al., 2020), tracking the spatial gradients and back-propagating the loss to the feature map pixels ϕ through the sampling module has not been studied. To this end, we derive the closed-form sampling scheme for tracking and propagating the spatial gradients ∂ϕ∂x,y,z through the sampling layer.
Background - 2D Differentiable Sampling. Differentiably sampling pixel values from a grid of 2D feature map with the given pixel locations (i, j) is a common operation. Throughout our paper, we define the pixel coordinates in the Normalized Device Coordinate (NDC) system that ranges from -1 to 1. As illustrated in Fig. 4(d), given the feature map ϕ and the sampling locations (i, j), the resulting sampled value ϕ(i, j) is
ϕ(i, j) = (1− α)((1− β)ϕA + βϕB) + α((1− β)ϕC + βϕD) (3)
Please refer to Fig. 4(c) for the definitions of α, β and ϕA, ϕB , ϕC , ϕD. Without loss of generality we use bilinear interpolation. During training, the gradient from the loss can be back-propagated via
∂L ∂ϕA = (1− α)(1− β) ∂L ∂ϕ(i, j) . (4)
2D Differentiable Gradient Sampling. Our learning framework (Sec. 3.1, Fig. 4(a)) requires the extension of the sampling capability from just the feature value response ϕi,j to its spatial gradient ∇i,jϕ(i, j). During the forward and the backward propagation, both the sampled feature response ϕi,j and its spatial gradient ∇i,jϕ(i, j) are recorded for further propagation (Fig. 4(e)):
∂ϕ(i, j)
∂i = (1− β)(ϕC − ϕA) + β(ϕD − ϕB) h . (5)
Hence, during the forward pass, we compute the spatial gradient via Eq. 5 in addition to the existing value sampling (Eq. 3). During the backward pass, we compute the loss gradient over the spatial gradient via
∂L ∂ϕA = (1− α)(1− β) ∂L ∂ϕ(i, j) + (−1− β h ) ∂L ∂ ∂ϕ(i,j)∂i + (−1− α w ) ∂L ∂ ∂ϕ(i,j)∂j , (6)
where w and h are the width and height of a pixel in NDC, s.t. w = 2/W and h = 2/H for the feature map with the size W ×H . 3D Differentiable Gradient Sampling. We now extend the sampling to 3D. We model the camera as the pin-hole camera. For any point (x, y, z) in the camera space, we seek for its projected 2D locations (i, j) based on the camera focal length f0 via i = yf0/z, j = xf0/z. The feed forward pass is
∂ϕ(x, y, z) ∂x = ∂ϕ(i, j) ∂j · f0 z , ∂ϕ(x, y, z) ∂y = ∂ϕ(i, j) ∂i · f0 z ,
∂ϕ(x, y, z) ∂z = ∂ϕ(i, j) ∂i · (−yf0 z2 ) + ∂ϕ(i, j) ∂j · (−xf0 z2 ).
(7)
During the backward propagation procedure, the DGS accumulates the gradient via
∂L
∂ϕA =
∂L ∂ϕ(x, y, z) · (1− α)(1− β) + ∂L ∂ ∂ϕ(x,y,z)∂x · (−1− α w ) f0 z + ∂L ∂ ∂ϕ(x,y,z)∂y · (−1− β h ) f0 z
+ ∂L ∂ ∂ϕ(x,y,z)∂z · ((1− β h )( yf0 z2 ) + ( 1− α w )( xf0 z2 )).
(8)

heading: 4 EXPERIMENTS

heading: 4.1 LEARNING FROM SYNTHETIC DATA (SHAPENET)
text: Dataset. Following Popov et al. (2020), we conduct shapeNet experiments with the low-realism and the high-realsim renderings. For low-realism, we use the renderings of various models from Choy et al. (2016). Following Mescheder et al. (2019), we split the dataset into the training set of 30661 models, the validation set of 4371 models, and the test set of 8751 models. During testing, we follow Mescheder et al. (2019) to only test the first rendering of each model. Similar to all the prior works, we use the same 13 classes to report any performance via grouping all the test cases into the same class. For high-realism, we follow the split and evaluation protocol of the single-object scenes Popov et al. (2020) - we only use the 13 categories used by Choy et al. (2016), filter out repeated samples, and finally construct the data with 666,565 models from the training set, 96,084 from the validation set, and 189,748 from the test set. Following Popov et al. (2020), we only evaluate the first 1% test cases of the test set (1898 samples).
Comparison with state-of-the-art approaches. We choose to compare to recent representative state-of-the-art approaches, including OccNet (Mescheder et al., 2019), DISN (Xu et al., 2019),
CoReNet (Popov et al., 2020) and D2IM-Net (Li & Zhang, 2021). Quantitative results are reported in Tab. 3 for the low-realism evaluation setting, and in Tab. 1 for the high-realism setting, where our results outperform all the state-of-the-art results in all the settings. Please refer to Sec. A for qualitative comparison, implementation details and the evaluation metrics.
For our low-realism evaluation setting, all the baseline approaches reported their results in the papers. For OccNet and DISN, the reported results are based on knowing the category canonical view prior, which demonstrates considerable privilege with respect to accuracy Popov et al. (2020). Hence, we mark the results from the literature as OccNet-Privilege and DISN-Privilege (OccNetPriv. and DISN-Priv. for short). Compared to their privileged setting, our results demonstrate superior results even without the category canonical view prior privilege. To further provide the the baseline results where these two approaches are without the category canonical view prior, we retrain their models with the released codes, and report the results in the “OccNet” and “DISN” row respectively. The results further provide evidence that the category canonical view prior demonstrates privilege on accuracy, as observed in Popov et al. (2020).
For our high-realism evaluating setting, since the results for Occnet and DISN are not directly available from the paper, we retrain their model from the released codes (without the category canonical view prior) on the high-realism data provided by Popov et al. (2020). We also retrain CoReNet using their released codes and obtain results better than reported in their original paper (61.5 as mIoU instead of 59.5). Our model still consistently outperforms all the baselines on almost all the categories. We emphasize that our comparison to CoReNet is a direct ablation comparison where we directly follow their experimental setup and model size. Furthermore, our approach only utilizes the closeto-surface occupancy labels, which is a subset of the whole occupancy labels for all the query point. Our model still outperforms the baseline and provides convincing evidence that our supervision loss function demonstrate better results than the original CoReNet approach. We attribute the superiority of our approach to our particular focus toward the difficult close-to-surface region similar to hard negative mining. Ablation Study. We further conduct ablation studies to further validate the importance of derived DGS module, via attempting work-around training methods without DGS. i) NoGrad - To test the performance of the baseline when only the surface data are provided, we train this ablation model in exactly the same way compared to its original model, with the only exceptions that only the near-surface points are equipped with training labels, and we mask out all the training labels from the non-surface query points (where it is not necessary to backpropagate the loss gradient of the spatial gradient using DGS). To further evaluate how the rate of known voxels affect the learning performance, we enlarge the near-surface region and evaluate when the rate is 10%, 30% and 50%. An increase in the performances among these baselines would indicate the importance of knowing more voxel labels if our proposed gradient loss (Eq. 1) is not imposed. ii) FixedE - We train with both the near-surface as well as the spatial gradient supervision, without DGS - meaning the loss gradient of the spatial gradient would not back-propagate to any module before the sampling module - in our case, the feature encoder network. Note all the other losses without gradient sampling can still back-propagated to the feature maps. We report the ablation results in Tab. 1. Both experiments are conducted with the high-realism ShapeNet data. Once again, we comfortably outperform all the baselines, validating the essential roles of DGS in our learning framework.

heading: 4.2 LEARNING FROM REAL SCANNED DATASETS (SCANNETV2)
text: Dataset. We use ScannetV2 (Dai et al., 2017) for training and evaluating the performance of the models on the real images. We follow the standard training / testing split as used in Sun et al. (2021) and Murez et al. (2020), where 1513 scenes are for trainval (with 1201 for training and 312 for validation), and 100 for testing. Each scene is provided with multiple image capture as well as the associated camera pose. We train the models with all the views given in the training / validation set (2423872 frames in total, after filtering out frames with invalid extrinsic poses), while for testing, we select 10 frames with different extrinsic poses for each test scene. Practically, since all the frames of the scenes are in the form of video clips, with adjacent frames associated with similar extrinsic cameras poses, we select the 10 frames for each frame via extracting every 100 frames from each scene video (e.g. Frame 1, 101, 201, ..., 901), resulting in 1000 frames in total in our test set (10 frames per scene, with 100 test scenes).
Metrics. We use the same evaluation metrics following Sun et al. (2021) and Murez et al. (2020). Since we are the first, to our knowledge, to evaluate single view 3D implicit reconstruction on the ScannetV2 benchmark, and here we only evaluate the geometries within the camera view frustum rather than the whole scene geometries as in Sun et al. (2021); Murez et al. (2020). In addition, we also only evaluate the geometries in front of the amodal depth for each pixel ray. We define the amodal depth for a pixel ray to be delineated by the minimum between the closest structure-category surface (e.g. walls and doors, etc) and the farthest surface. In practice, in order to accommodate the evaluation of the surfaces right on the amodal depth, we slack the evaluation scope with a factor λ (1.05 in our case) multiplied with the amodal depth. This evaluation protocol would be equivalent to the “single-layered” protocol used in Sun et al. (2021), within our single-view scenario. Due to the inherent ambiguity of the scaling and shifting of the predicted 3D single view geometry (Ranftl et al., 2019; Yin et al., 2021), we follow them by regressing the best scale and shift comparing the predicted depth with the ground truth depth. For approaches that predict 3D surfaces, the rendered depth map from the predicted mesh would be used for calculating the scale and shift. Baselines. Since we demonstrate the first attempt to conduct the implicit reconstruction of singleview scenes, very few exiting works provide a direct baseline performance to our task. For OccNet (Mescheder et al., 2019), since its image feature extraction is not local, and its gradient propa-
gation does not require sampling, we use its direct application with Gropp et al. (2020) “OccNet + GeoReg” as one baseline. We train DISN (Xu et al., 2019) with the TSDF voxelization labels (Murez et al., 2020; Sun et al., 2021; Božič et al., 2021). For CoReNet (Popov et al., 2020), we stick to its own voxelization and internal filling toolbox for obtaining the training labels. Since CoReNet can only predict geometries within the fixed range of space, we tried our best to pick the best cube location based on the dataset statistics. We also incorporate depth approaches (Ranftl et al., 2019; Yin et al., 2021) for comparison by finetuning their weights on ScannetV2. Lastly, we compare with the two ablation models NoGrad and FixedE as introduced in Sec. 4.1. Results. We provide quantitative comparison in Tab. 2 and qualitative comparison in Fig. 5 respectively. Our model outperforms all state-of-the-art approaches as well as the ablation models. Compared to the synthetic data scenario in Tab. 3 and 1, our model demonstrates even larger advantages when compared to existing state-of-the-art approaches, as our motivation stems from addressing learning from imperfect 3D labels directly from the real scan data. Compared to single-view depth prediction approaches trained on massive data (Lasinger et al., 2019; Yin et al., 2021), our approach does not prevail on “Acc” and “Prec”. This is due to the fact that these two metrics only project the predicted surface to the ground truth surface, giving advantages to approaches that only predict the visible surface. Our approach still prevails for other metrics (Chamfer Distance and F1), which are considered as the most important metrics (Božič et al., 2021; Sun et al., 2021; Murez et al., 2020). Generalization to Unseen Scenes (Pix3d and Open-Domain Images). We further test our model without further finetuning directly on unseen scenes for evaluating the generalizability of the learned model. We tried our model on Pix3d Sun et al. (2018) as well as test images downloaded directly from the internet. We provide a detailed qualitative comparison in Fig. 6 and more results in Fig. 1. The results further indicate our learning framework exhibits promise for unseen scene generalization. Failure Cases. We provide two representative failure cases of our approach in Fig. 7. The first case (the first row) demonstrates difficulties in predicting the floor occupancy as a result of the noisy and non-watertight mesh during training. The second case (the second row) shows that our model cannot identify objects (e.g. chairs) and well predicts the invisible part of them.

heading: 5 CONCLUSIONS
text: We have presented our learning framework for real-world 3D implicit surface reconstruction from a single view image. Owing to our unique learning framework that directly train the model from the raw scan data and our novel occupancy loss function over the gradients, we are able to go beyond the existing works for single-objects reconstruction or single view fitting. Thanks to our differentiable gradient sampling module we enable efficient and memory-economic end-to-end training and achieved scene-level single view reconstruction for the first time, to our knowledge.

heading: A ADDITIONAL DETAILS AND RESULTS ON SHAPENET
text: Metrics. We use the Intersection-of-Union (IoU) for evaluating the model performances. For lowrealism renderings, we follow the protocol of IoU benchmarking from Mescheder et al. (2019) that evaluating occupancy of the specified 100K query points in the space. For high-realism renderings, we follow the protocol from Popov et al. (2020) to evaluate occupancy prediction of the 128×128× 128 grid. Following Popov et al. (2020), the grid cube is a unit cube that spans from −0.5 to 0.5 for x and y, and from f/2 to 1 + f/2 for z in the camera coordinate system.
Implementation details of our model (DGS). We build our model on top of one representative state-of-the-art model - CoReNet (Popov et al., 2020) for its 3D volumetric implicit modeling capability. This is because the space of the geometry predictions are always within the unit cube volume. We maintain exactly the same encoder and decoder architecture (ResNet50 (He et al., 2016) and the ray traced skip connection respectively), optimization parameters (e.g. the learning rate and the epsilon of Adam Kingma & Ba (2014)) and the prediction format (per-query occupancy probability). We set λor to be 0.01 in our loss function (Eq. 1).
Qualitative Results. We provide qualitative results in Fig. 9-13 as the additional illustration of the performance of all the approaches on ShapeNet.

heading: B ADDITIONAL RESULTS ON SCANNETV2
text: Implementation details for DGS. Since the DISN variant architectures (Xu et al., 2019; Niemeyer et al., 2020; Yu et al., 2020) demonstrate higher degress of flexibility of representing any point in the space, in contrast to 3D convolutions that are fixed for a particular range of space (Popov et al., 2020), we build our model based on the former architectures. Similar to Niemeyer et al. (2020); Yu et al. (2020), we use 5 residual blocks in the fully connected part with the first 3 blocks receiving the 2D features and its spatial gradients. Our image encoder uses the same RexNext101 architecture (Xie et al., 2017) as in Yin et al. (2021) and starts the training with the pre-trained weights. We use a batch size of 32 images with 2048 near-surface points and 512 non-surface points per image. Similar to the ShapeNet experiments, we set λor to be 0.01. We found in the real-world setting, this parameter setting is crucial for the convergence of the learning procedure.
Quantitative Results for 2.5D evaluation metrics. We provide the quantitative 2.5D evaluation results in Tab. 4. Our approach constantly outperforms all other implicit based baselines. While our approach falls short slightly when compared to the existing depth-based approaches, we claim that our approach is not directly trained with the massive depth training data. Our results indicate that our depth performance is still on par with the state-of-the-art depth prediciton approaches.
Additional qualitative results. We provide additional qualitative results on ScannetV2 Fig. 14-18.

heading: C ANALYSIS WITH NUMERICAL GRADIENT APPROXIMATION
text: As part of our proposed learning framework with the loss imposed on the spatial gradient, the differentiable gradient sampling module (Sec. 3.2) has a numerical alternative where we can perturb the query points to get its simulated numerical gradients. We further compare the closed-form performance with the numerical counterpart in Tab. 5. Please find qualitative comparison in Fig. 9-13.
We provide a closer look at our comparison to the simulated gradient baseline. We found in our experiments that the simulated gradient baseline demonstrates relatively severe convergence difficulties. As shown in Fig. 8, the simulated gradient model (red) does not observe loss drop in the first 10k training iterations. Despite its subsequent loss drop, which indicates the simulated gradient model can still learn to predict the geometry in moderate accuracy, its converged training loss is still higher than our DGS (blue). This aligns with our benchmarking evaluations in Tab. 5 that the closed-form variant (DGS) achieve better performance.
Table 5: Intersection over Union % (IoU ↑) benchmarking comparison between the closed-form solution and the numerical gradients on the high-realism ShapeNet.
Category Craft Rifle Disp. Lamp Spk. Box Chair Bench Car Plane Sofa Table Phone Mean
Numerical 62.4 68.3 63.7 46.2 60.6 54.9 61.0 49.0 70.7 61.1 69.2 55.6 77.2 61.5 Closed-form 63.3 70.4 65.7 49.0 61.2 57.0 62.1 50.9 70.1 62.8 70.3 56.9 78.3 62.9
0 0.5 1 1.5 2 2.5 3 Iterations 105
0.3
0.4
0.5
0.6
0.7
0.8
Th e
To ta
l L os
s Numerical Closed-Form
Figure 8: Convergence Analysis for the comparison between the closed-form (blue) and the numerical counterpart (red). Notably, the numerical counterpart does not observe loss drop in the first 10k iterations.

heading: D ADDITIONAL GENERALIZABILITY QUALITATIVE RESULTS
text: We provide more qualitative results on our model generalizing to unseen images downloaded from the Internet in Fig. 19. Each result is visualized in 6 views. Once again, these additional visual results further indicate that our model demonstrates good generalizability capability for handling unseen indoor scene images.

heading: E EVALUATION OF THE GENERALIZABILITY TO MATTERPORT3D DATASET
text: To quantatively evaluate our generaizability to new datasets, we provide our results on the Matterport3D dataset (Chang et al., 2017) using our model trained from ScannetV2 as used in Sec. 4.2.
The Matterport3d dataset (Chang et al., 2017) collects indoor scenes of 90 full houses. Textured meshes are provided along with the scanned real images. Since the Matterport3D dataset does not provide an official train/test split, and our model is not trained on Matterport3d, we evaluate our model over all the 90 scenes in the dataset. In particular, we use the 129600 images (without the elevated pitch views where the camera is looking up at the ceiling) from the dataset, use the 10 first images from each scene (resulting in 900 test images in total), and test their reconstruction performance in the same way as we tested on ScannetV2. We provide the quantative evaluation result in Tab. 6. We can see from the table that the results align with our evaluation in Tab. 2 that our model demonstrates clear advantages. We also noticed that the AdelaiDepth Yin et al. (2021) baseline (finetuned on ScannetV2) performance is closer to our results (F1) score compared to our ScannetV2 evaluation. This is probably due to the fact that AdelaiDepth was directly pre-trained from a massive number of training images and potentially gives it some privilege when generalize to a new dataset.

heading: F DETAILS OF THE FORMULATION AND DERIVATION OF DGS
text: In this section, we provide the full formulation as well as the derivation procedure of our DGS in this section as the supplementary to Sec. 3.2.
In Eq. 4, we provided the backward gradient for differentiable sampling for Pixel A. The backward gradient for Pixel B, C and D are
∂L ∂ϕB = (1− α)β · ∂L ∂ϕ(i, j)
∂L ∂ϕC = α(1− β) · ∂L ∂ϕ(i, j)
∂L ∂ϕD = αβ · ∂L ∂ϕ(i, j)
(9)
In Eq. 5, we provided the forward gradient in the vertical pixel direction. The forward gradient of the horizontal pixel direction is
∂ϕ(i, j)
∂j = (1− α)(ϕB − ϕA) + α(ϕD − ϕC) w
(10)
In Eq. 6, we provided the backward gradient for only Pixel A. The backward gradient for Pixel B, C and D are
∂L ∂ϕB = (1− α)β · ∂L ∂ϕ(i, j) + (−β h ) · ∂L
∂ ϕ(i,j)∂i +
1− α w · ∂L ∂ ϕ(i,j)∂j
∂L ∂ϕC = α(1− β) · ∂L ∂ϕ(i, j) + 1− β h · ∂L ∂ ϕ(i,j)∂i + (−α w ) · ∂L
∂ ϕ(i,j)∂j ∂L
∂ϕD = αβ · ∂L ∂ϕ(i, j) + β h · ∂L ∂ ϕ(i,j)∂i + α w · ∂L ∂ ϕ(i,j)∂j
(11)
In Eq. 8, we provided the backward gradient for only Pixel A in the 3D setting. The backward gradient for Pixel B, C and D are
∂L
∂ϕB =
∂L ∂ϕ(x, y, z) · (1− α)β + ∂L ∂ ∂ϕ(x,y,z)∂x · 1− α w · f0 z + ∂L ∂ ∂ϕ(x,y,z)∂y · (−β h ) · f0 z
+ ∂L ∂ ∂ϕ(x,y,z)∂z · (β h · yf0 z2 − 1− α w · xf0 z2 )
∂L
∂ϕC =
∂L ∂ϕ(x, y, z) · α(1− β) + ∂L ∂ ∂ϕ(x,y,z)∂x · (−α w ) · f0 z +
∂L ∂ ∂ϕ(x,y,z)∂y · 1− β h · f0 z
+ ∂L ∂ ∂ϕ(x,y,z)∂z · (−1− β h · yf0 z2 + α w · xf0 z2 )
∂L
∂ϕD =
∂L ∂ϕ(x, y, z) · αβ + ∂L ∂ ∂ϕ(x,y,z)∂x · α w ) · f0 z +
∂L ∂ ∂ϕ(x,y,z)∂y · β h · f0 z
+ ∂L ∂ ∂ϕ(x,y,z)∂z · (−β h · yf0 z2 − α w · xf0 z2 )
(12)
As an illustration of the derivation for Eq. 8, we can obtain the backward gradient via
∂L
∂ϕA =
∂L ∂ϕ · ∂ϕ ∂ϕA + ∂L ∂ ∂ϕ∂x · ∂ ∂ϕ∂x ∂ϕA + ∂L ∂ ∂ϕ∂y · ∂ ∂ϕ∂y ∂ϕA + ∂L ∂ ∂ϕ∂z · ∂ ∂ϕ∂z ∂ϕA
(13)
The four partial derivatives with respect to ϕA in Eq. 13 can be determined based on the sampling rules. For example, based on Eq. 3, we can quickly obtain the first derivative with respect to ϕA via
∂ϕ
∂ϕA = (1− α)(1− β) (14)
For the other three derivatives with respect to ϕA in Eq. 13, we can substitute Eq. 5, 10 into Eq. 7, and obtain them via
∂ ∂ϕ∂x ∂ϕA = −f0 z · 1− α w
∂ ∂ϕ∂y ∂ϕA = −f0 z · 1− β h
∂ ∂ϕ∂z ∂ϕA = yf0 z2 · 1− β h + xf0 z2 · 1− α w
(15)
We can then obtain Eq. 8 via substituting Eq. 14, 15 into Eq. 13.

Review summaries:
This paper presents a method for 3D scene reconstruction from a single image using implicit surface representations such as occupancy or SDF. The authors propose to incorporate loss functions on the spatial gradients to provide dense supervision in the 3D space in the case where 3D labels may be incomplete (e.g. open 3D meshes) or not well-defined everywhere. Experiments are performed on ShapeNet and ScanNet show that the proposed method can achieve competitive performance on single-image scene reconstruction tasks.

This paper describes novel loss functions for learning to predict an implicit 3D scene representation from a single image.  They argue that when working with real scan data of scenes (rather than single objects) it is difficult to generate accurate occupancy or signed distance function (SDF) ground truth as would be required for supervised learning.  Instead, they propose to only use occupancy or SDF supervision near the surfaces of objects; elsewhere, they rely on constraints on the gradient of the occupancy or SDF adapted from Gropp et al. 2020.  They perform a thorough evaluation on several benchmark datasets and compare against state-of-the-art competing methods.  They show that they outperform competing methods, even though in some cases their method has access to less supervisory data.    They also perform an ablation study to show the importance of various parts of the loss function.  

This paper presents a new method to learn implicit 3D scene reconstructions from single image input. The main improvement is a closed-form Differentiable Gradient Sampling.  By taking spatial gradient into  consideration, the proposed method can apply back-propagation of the  loss on spatial gradients to feature maps and allow the training for the case of without dense 3D supervision.   

In this paper, the authors propose a new method for single view 3D reconstruction.  A conditional (image feature prior) implicit representation framework is proposed to reconstruct 3D scene from a single view. In this paper, the authors propose that feature gradient is essential for watertight reconstruction and propose a differentiable gradient sampling method for the formulation. Experiments have been performed on both synthetic and real datasets. Superior results have been presented.