Differentiable Gradient Sampling for Learning Implicit 3D Scene Reconstructions from a Single Image
Authors: Shizhan Zhu
Sayna Ebrahimi
Angjoo Kanazawa
Trevor Darrell
Venue: ICLR 2022 Poster (2022)

Abstract: Implicit shape models are promising 3D representations for modeling arbitrary locations, with Signed Distance Functions (SDFs) particularly suitable for clear mesh surface reconstruction. Existing approaches for single object reconstruction impose supervision signals based on the loss of the signed distance value from all locations in a scene, posing difficulties when extending to real-world scenarios. The spatial gradient of the signed distance field, rather than the SDF value itself, has not been typically employed as a source of supervision for single-view reconstruction, in part due to the difficulties of differentiable sampling a spatial gradient from the feature map. In this study, we derive a novel closed-form gradient sampling solution for Differentialble Gradient Sampling (DGS) that enables backpropagation of the loss of the spatial gradient back to the feature map pixels, thus allowing the imposition of the loss efficiently on the spatial gradient. As a result, we achieve high-quality single view indoor scene reconstruction results learning directly from a real-world scanned dataset (e.g. ScannetV2). Our model also performs well when generalizing to unseen images downloaded directly from the internet (Fig. 1). We comfortably advanced the state-of-the-art results with several established datasets including ShapeNet and ScannetV2; extensive quantitative analysis confirmed that our proposed DGS module plays an essential role in achieving this performance improvement. Full codes are available in MaskedURL.

Published as a conference paper at ICLR 2022
DIFFERENTIABLE GRADIENT SAMPLING FOR LEARNING IMPLICIT 3D SCENE RECONSTRUCTIONS FROM A SINGLE IMAGE
Shizhan Zhu, Sayna Ebrahimi, Angjoo Kanazawa, Trevor Darrell UC Berkeley
ABSTRACT
Implicit neural shape functions, c.g. occupancy fields signed distance func- tions, are promising 3D representations for modeling arbitrary 3D surfaces. How- ever, existing approaches that use these representations for single-view 3D re- construction require 3D supervision signals at every location in the scene, posing difficulties when extending to real-world scenes where ideal watertight geome- try necessary to compute dense supervision is difficult to obtain. In such cases, constraints on the spatial gradient of the implicit field, rather than the value it- self, can provide training signal, but this has not been employed as a source of supervision for single-view reconstruction in part due to the difficulties of differ- entiably sampling spatial gradient from a feature map. In this paper, we derive a novel closed-form Differentiable Gradient Sampling (DGS) solution that enables backpropagation of the loss spatial gradients to the feature maps, thus allow- ing training on large-scale scenes without dense 3D supervision. As a result, we demonstrate single view implicit 3D surface reconstructions real-world scenes via learning directly from scanned dataset. Our model performs well when gen- eralizing unseen images from Pix3D downloaded directly from the Internet (Fig. 1). Extensive quantitative analysis confirms that our proposed DGS module plays an essential role in our learning framework. Video and code available httis://github.com/ htt.l/.cco//uuuuaaaaae wwtp://www/..//2/......
Figure 1: Given a single RGB image as the input (the first column), our model can predict its 3D implicit surface reconstruction (shown six novel views in the last six columns). The test images for the first 3 rows are downloaded from the Internet and the last 2 rows are from the pix3d dataset.

---

Published as a conference paper at ICLR 2022
INTRODUCTION
Recent studies have shown promising learning capabilities of implicit models for 3D geometry rep- resentations (Park et al. 2019 Mescheder et al. 2019 Mildenhall et al. 2020 SStzzannn et al., 2019
Gropp al.) 2020 Saito al. 2019: Yu et al. 2020p. Rather than explicitly representing 3D geome- try with a textured mesh (ato et al. 2018) or point cloud (Fan et al. 2017). implicit representations
express a function of points in the space, Among many recent promising implicit models, the oc- cupancy field (Mescheder et al. Distance Field (SDF) (Park al.
2019 Popov et al. 2020, Chen & Zhang 2019b) and the Signed
2019 Xu et al. 2019 & Zhang 2021 are particularly suitable
for high resolution topology free surface reconstruction. Given a query point (a. y, =) E R in the space, the occupancy function returns the occupancy status of the point o(2r, y, 2) E {++ }} while
the SDF value returns the closest distance from a point to the surface of the given 3D geometry, as well as its sign for its occupancy. The surface defined by the classification boundary of the occupancy field or the zero level-set of the SDF (Lorensen & Cline 1987).
Despite the strong representational power of these models and their success on predicting 3D objects from single image, learning to predict complex 3D geometry, such as scenes, from a real world image remains challenge for this class of methods. Obtaining the occupancy label, or the sign, from noisy non-watertigh meshes is non-trivial (Atzmon & Lipman| 2020 (Fig. 2 Furthermore, producing the label on the fly from a large number of triangle meshes, as encountered in typical real-world scenes with complex geometry, can be computationally prohibitive (Straub et al., 2019; Chang et al. 2017: Dai et al., 2017: Song et al. 2017). Storing dense query points sufficiently with pre-computed occupancies or SDF for complex geometries, meta data of meshes like octrees or hashing, is a possibilityy, but incurs severe engineering and computational burdens on storage and runtime loading (Yu et al.] 2021]. The challenge is exacerbated for predicting scenes in particular, since unlike objects scenes in the view frustrum typically cannot be enclosed by pre-defined range of space and do not have a categorical canonical coordinate system (Popov al. 2020).
Recent studies on fitting a single scene geometry to partial 3D observations (Gropp et al. 2020 Sitzmann al. 2020) have pointed out the value of the 3D spatial gradient of the implicit field. For instance, the gradients of the SDF on surface points are actually the surface normals and the 2-norm of the spatial gradients for any point in the space is 1, known as the Eikonal equation (Crandall & Lions, 1983). Similarly, in this work, we propose to regularize the gradient of the occupancy field to be zero away from the surface (Fig. (()) Thus, one can train the full occupancy field or SDF with only the surface point clouds, thanks to these constraints on the gradients of the implicit shape models, even when the value of the occupancy or the SDF itself is not available everywhere. However, existing approaches (Gropp et al. 2020 Sitzmann et al. 2020 have only demonstrated this benefit in the cases of geometry inputs only, where a network Is fit to a particular scene per model without any conditioning on input images. When locally sampled image features used in the feed-forward prediction, it is necessary to derive a differentiable gradient sampling solution over the feature pixel map.
In this paper, we tackle this challenge by deriving a closed-form differentiable gradient sampling (DGS) solution for learning a single-view 3D implicit reconstruction model (Fig. Our novel propagation of the loss gradient of the spatial gradient back to the feature maps (Eq. 6 serves in addition to the existing spatial gradient sampling (Eq. (5 and loss gradient back propaproong (Eq. 4 used in existing deep learning frameworks (Paszke et al. 2019: Abadi et al.) 2016). The resulting end-to-end learning scheme with supervision over the spatial gradients opens the potential to train a
model generalizable to unseen test cases of complex scenes with only surface point cloud supervision
a typical setting for real-world data- without requiring ground truth per-query-pon poontt labels.
Our contributions include a framework to propagate spatial gradients through spatial feature sam- pling procedure, and a novel closed-form DGS solution. Experiments on real-scanned data (Scan- netV2 (Dai et al. (0017) shows that DGS enables training a saingle-image 3D implicit reconstruction model that can generalize to unseen scenes, with only imperfect surface annotations as the supervi- sion (e.g. non-watee watertight meshes). Experiments on synthetic data (ShapeNet [hang al. (2015) indicate that our learning framework without using dense per-query-point training labels demon- strates competitive performance compared the oracle scenario where dense occupancy labels are available. To the best of our knowledge, DGS-enabled shape inference provides the first single- view implicit shape reconstruction on real scene datasets which can generalize accurately unseen scenes from different datasets or domains (see Figure
2

---

Published as a conference paper at ICLR 2022
(a)
(b)
View
Input Image
View
View
Input Image
View
Ground Truth TSDF Mesh
Voxelization
Ground Truth Mesh
TSDF Voxelization
DISN TSDF Vox DGS
Figure 2: (a. b) Truncated SDF (TSDF) (Curless & Levoy 1996) Voxelization results of the non-
watertight ground truth meshes (each shown in two views). (a) simple sphere and (b) is a real scene from the ScannetV2 training data. After depth-fusion and internal-filling (Popov et al. 2020), the inside space of both geometries (a, b) remains empty (red arrows), causing severe noise for training the occupancy field or the SDF prediction model. This type of noise particularly affects the single-view prediction problem, as no additional predicted depth surface from other views are available. (c) As result of learning the implicit prediction directly from the inaccurate and low- resolution TSDF voxels (due to engineering constraints on runtime loading and memory bottleneck for the sufficiently dense pre-computed query point occupancy labels), the prediction result (DISN TSDFVox) is clearly inferior compared to our results (DGS). The surface color denotes the evalu- ation of the "precision", with the larger blue region, the higher "precision".
RELATED WORKS
3D Implicit Representations Among all the 3D representations, implicit models are advan- tageous for arbitrarily high resolution model- ing (unlike voxels with fixed resolution and no detailed surface modeling) and easy learn- ing (unlike meshes which assume a fixed topol- ogy). Implicit models typically learn estab- lish a mapping between query point and the prediction of the point. Groueix et al.) pro- posed to Iearn mapping from the uv tex- ture map to the 3D surface point. More re- cent works (Park et al. 2019 Mescheder et al. 2019 Xu al. 2019 Popov al. [2200] Chen & Zhang (019a) focus on mapping the query point coordinates to the signed distance field or the occupancy field. In addition to these pure geometry modeling, recent works like (Saito al. 2019) also model the surface texture via learning mapping from the surface point to the RGB value, use the RGB loss as the supervi- sion signal (Niemeyer et al. 2020 Yariv et al., 2020). In contrast to defining textures explicitly on the surface, Mildenhall al. (2020) proposed the volumetric rendering representation which maps the point coordinate and the viewing angle to volumetric textures. Oechsle et al. (202T): Wang et al. (2021); Yariv al. (202T) use volume renderings for fitting high quality geometry single scenes, but these are not generalizable to unseen scenes. In multi-view stereo setting, Murez et al. (2020); Sun al. (20T);; Božič et al. (202T) proposed accumulate the multi-view predictions in a TSDF voxel volume. We believe proposed DGS module is necessary when extending most of the exist- ing implicit surface representations into feed-forward models that are conditioned on input image(s), and training these models with a loss the gradient of the predicted field. In this work, focus on 3D geometry reconstruction reconstructions from single view, as an example of the application of the proposed DGS module.
Illustration:
Occupied
Unoccupied
Learning with ideal meshes
(b) Learning with the mun-walertigh meshes
(6) Leaming with the proposed DGS
TSDF voxelization
Figure 3: Illustration of the loss imposition for the occupancy prediction scenario. (a) When learning from the ideal mesh for ShapeNet objects, we can directly supervise the training with the accurate occupancy labels. (b) On scans of real scenes with imperfections (Fig. bb)),, the TSDF voxelization
produces severe noise for training. Specifically, a considerable fraction of the objects "empty" inside. (c) Our learning scheme with DGS allevi- ates these issues via enabling imposition of losses on the gradients all the way back to image fea- tures.
Differenniable Operations End-to-end deep learning requires all modules in a computation path to be differentiable. Many differentiable modules proposed to serve particular functionality. The
3

---

Published as a conference paper at ICLR 2022
spatial transformer layer (Jaderberg al., 2015) is one of the early approaches for differentiable sampling of the feature map. Kato et al. (2018) proposed to render the mesh into image in a dif- ferentiable manner to optimize the mesh or the camera parameters to fit to particular image. A
similar idea has also been applied for rendering point clouds (Wiles et al. 2020; Yifan et al. 2019). an
Tulsiani et al. (2017) and Mildenhall al. (2020) proposed to compute the ray rendering cumulating manner. One common feature of these differentiable operations is a strategy to soften the existing explicit modeling (Liu et al./ 2019), and replace the gradient of the loss with approxi- mate signals, making sure those signals provide guidance in the right direction. Our differentiable gradient sampler also seeks to soften the spatial gradient between the adjacent pixels.
LEARNING FRAMEWORK
Problem definition and notations. We aim to train feed-forward deep model for predicting the 3D implicit surface reconstruction conditioned on a single image. We denote the input RGB image as E m Like most feed-forward models, our model employs an image encoder. We denote the extracted 2D features as Φ. Our model (-), parameterized by O, takes in the image feature &i and predicts for each 3D location (z; y, 2) the implicit value fe(:; z. 2). We denote the ground truth value as f(0; x, y: z). For the occupancy field, the implicit field value f represents the occupancy probability. We denote the predicted occupancy probability truth binary occupancy label as o( x, y: 2) E {"-" " " + "), where "_" represents unoccupied space and "+" vice versa. For SDF, the implicit field value f represents the signed distance between the query point and its projection on the surface. We denote the predicted and the ground truth signed distance (d; y; =) E IR and s(d; y. 2) E R respectively.
0g (d;; x, y, z) E [0, 1] and the ground
Background. Typical fully supervised training (Xu et al., 2019) Popov et al., 2020 imposes the loss to each individual sampled query points by comparing the predicted field value (e) (d;; v y, z) with the ground truth value f x, y. 2). For occupancy predictions, we compute the loss as (IPP))ED (e.,,)) BCE( (é; 2, x, y, 2), oe (g; x, y. 2)) (Fig. a)), where "BCE" represents the binary cross entropy loss, 'D represents the whole bingleview image training set, and PP repre- sents the set of all the possible query point sampling locations (x, y, 2) within the view frustum. For x.
SDE, we compute the loss as E(IDTi) L(san))
s(6; 2)]]]
3.1 PROPOSED FRAMEWORK
As obtaining the full labels o (d; zr y, 2) for query points (I. y. from every location in Py is non- trivial for complex geometry from real-world (Fig. (2 and due to additional engineering constraints on runtime loading and memory bottleneck for the sufficiently dense pre-computed query point occupancy labels, we propose to incorporate the loss w.r.t. the spatial gradients. For occupancy of the implicit field,
C= c=EC EOX U.PPEED (PNN)))
Σ
Aor NorlI 0e
Σ BCE(oo((; "+))+
Σ
BC -
(x.Y.))EP(t
(..).)))P)))
where Vz.g.=0((::2; v, 2) = [o 2), V Vyōe(ó;2,y, Vzó0(á; x. y, TT denotes the spatial gradient of the occupancy prediction, pu+ and pp- denote inward and outward near-surface query points, and Aor represents the loss weight of the occupancy geometric regularization. The facing direction of the mesh surface (determining inward or outward) can be determined by normals (if available), or via rendering the surface in all views in the RGBD captures (Dai et al., (017) (with the surface facing toward camera as the "outward" side). The three terms in Eq. [ correspond to the three types of losses in Fig, (c) ("V", "+". "-") "") For SDF, set the loss similar to Gropp et al. (2020):
L == Σ Σ
Asr S 1|+
Σ
1 v,
(I,P))ED (E.0,))EP,
(n.g.))EPP
(2)
Σ (I.))))P))
Aen V Ve.g. ))
4

---

Published as a conference paper at ICLR 2022
Input Image
Feature Maps
Dig. (1 Plu
Image Encoder
40
(1 403
(b) 2D Feature Map Pixcls Φ
dc (c) Zoom-in
Vaya PxyaPwt
Eillloeeatibble Gradian Sampling
P
Virb.
Kybty
Query
Location
Robjec
Loss
(d) Existing differeniable sampling (bq, 3,
(e) Proposed differentiable gradient sampling (Eq. 5.6)
(o) Framowork of our training pipcline
Figure 4: Overview of our learning framework (a) and differentiable gradient sampling (b, d, e).
where Pf denotes the query points on the ground truth surface only, and Xgy and Xan represents the loss weight for the signed distance geometric regularization term and normal term respectively. The three terms are the Eikonal regularization (Crandall & Lions] 1983), surface zero-SDF loss, and the surface normal loss respectively, where the last ferm is optional (Gropp et al. 2020). Note now the loss functions in Eq. 2 no longer require the implicit ground truth label for query points far away from the ground truth surface, enabling training with the real-world imperfect scanned data. In both cases, the availability of surface normals is optional.
Practically, we found only the proposed loss for the occupancy model (Fig. C),, Eq. T to be effec- tive in the real-world single-viev view ferd-forward scenario, and use Eq. rather than Eq. 2 in all of our experiments. There are a few reasons. First, learning with the Eikonal term (Eq. 2 suffers severely from its sensitivity of model initialization. We found either the SDF or the Truncated SDF (TSDF) representation model cannot easily converge during training. Specifically, we found it poses numer- ical difficulties when the model learned to predict large distance value or a constant truncation value for the majority of the query points in the air that is far from any surfaces. This becomes major problem when we depart from the single objects scenario (Mescheder et al. 2019 Popov et al., (020)) single scene fitting (Gropp et al. (2200) to large schale scenes, which is our focus. Second, our loss function (Eq, [T can significantly save memory footprint during training and enable large batch size training, which is crucial in our learning framework. Unlike in Eq. 2 where all the query points require spatial gradient computation, which leads to 3x higher of memory footprint, in our loss function (Eq. [T], only the non-surface query points do. We set the query point batch size of the non-surface points (512 in practice) to be much smaller than the critical near-surface points (4096 in practice), enabling learning with the image batch size of 32 in our real-scene training.
3.2 SAMPLING WITH DIFFRENNIABLE GRADIENTS
A distinct difference compared to existing works is that our spatial gradient Va fo x, y, 2) is also conditioned on the pixels in d. Gropp al. (020) devised the model fe to fit to single scene, and the spatial gradient Vz.y.= (ar, y, =) be conveniently computed because it does not involve the image sampling procedure. Jiang et al. (2020); Lin et al. (2020) used non-spatial global feature for inference and hence bypassed 2D sampling. In our learning framework, the spatial gradient computation must undergo the sampling procedure.
We name our gradient computation involving the sampling operation as the Pixel Conditioned Gra- dients and derive a closed-form solution, Differentiable Gradient Sampling (DGS), for handling forward and backward propagation. Figure 4{a) provides an illustration of our training pipeline. Each layer in our network tracks both the response of the layer as well as its spatial gradient w.r.t. (x,y. 2). While it is well established track the layer-wise spatial gradient for fully-connected (FC) layers or convolutions in existing works (Gropp et al., 2020). tracking the spatial gradients and back- bock-proaagating the loss to the feature map pixels & through the sampling module has not been studied. To this end, we derive the closed-form sampling scheme for tracking and propagating the spatial gradients V V = [V Vy _ through the sampling layer.
Background 2D Differentiable Sampling. Differentiably sampling pixel values from grid of 2D feature map with the given pixel locations (i.j) is a common operation. Throughout our paper, define the pixel coordinates in the normalized coordinate system that ranges from -I to 1. As
5

---

Published as a conference paper at ICLR 2022
illustrated in Fig. d), given the feature map & and the sampling locations (i.f), the resulting sampled value ( is
Φ(i,) = (1 - (1-a)(1111))))))))))))) ()0111--- + 36b) + a((1 - 3)0c + BOD)
(3)
Please refer to Fig. cc) for the definitions of a, 3 and A ΦB, ΦC, ØD. Without loss of generality use bilinear interpolation. During training, the gradient from the loss be back- propppropggtted via
0L (1 a) 3) di(i,
aL
aΦa
Equation 4 is for Pixel A and similarly to the other 3 pixels (please refer Eq. 9
2D Differentiable Gradient Sampling. Our learning framework (See. 3.1 Fig. aa) requires the extension of the sampling capability from just the feature value response Φ1, to its spatial gradient
Vi,o(i. [V) ( During the forward and the backward propagation, both the sampled feature response di. and its spatial gradient Vi, are recorded for further propagation (Fig. e)):
V (i. (1 - ()cc 2A) + B -g)
(5)
h
Please refer to Eq. 10 for V (o(i,j). Hence, during the forward pass, we compute the spatial gradient via Eq. ii addition to the existing value sampling (Eq. 3). During the backward pass, compute the loss gradient over the spatial gradient via
0L
OL Oo(i,1) + (v
OL ov.0(i,))
OL
J
OΦs
OΦs
OV (V,0(i,)
(6)
AL
+ (- 1-8 OL
(i,)) + 1
ÖL
= =(1 (1- a1- D5(i,j)
w
oV vVb((,
where w and l are the width and height of a pixel in normalized coordinate system, s.t. w 2/W
and h = 2/H for the feature map with the size W x H. Please refer to Eq. for 8L aL and
B0g Uec
Don
3D Differentiable Gradient Sampling. We now extend the sampling to 3D. We model the camera as the pin-hole camera. For any point (x, y, 2) in the camera space, we seek for its projected 2D locations (i.j) based on the focal length fo via i uff . æfo The feed forward pass is
di
Oj
fo z
Ved(e,y,)) Vi)(i,))
;(((i
dx
dr
0i
0j
2)0(2,,,))=VV) = (i)(i
v))(i.))
vid(i,j)
dy
Oy
Øj
yfo (- z²
2f0). 22
V=0(2,8,2) = V=)(2,,.2-VVvv(t))) vi
= v.o(i,))
(,((t,))
0z
az
Note that here, notation of the sampled 2D image feature ((I, y, 2) refers the same as p(i, ( j),
as (i, j) is the projected pixel coordinates of (x, y, 2) that represent the exact projected 2D locations when extracting the 2D features.
During the backward propagation procedure, the DGS accumulates the gradient via
ØL
ØL
d0(x, N.
ØL
ov V(z0(1,0
dd
døla, U.
OV
OL
oV, (V
OL
OV
V.0(I,M W. 2)
+ 0V (V, 2,2
GΦA
0V VV. 2)
(8)
ØL (1 -11 -B)
0L
fo
0øl2, Us
2V ov.((a, y, 2)
w
z
ØL
1 1- 3 fo
ØL
- 3
yfo
zfo
0V NV,0(I,8,)
h
OV .VV.(....
2
22
Please refer to Eq.
12 for
81. aL. and Dog
8L dep

---

Published as a conference paper at ICLR 2022
Table 1: Intersection over Union % (IoU f) benchmarking result on the high-realism ShapeNet. Our approach (the last 4 rows) demonstrates competitive performance compared to state-of-the-an approaches (the top rows) even trained without the dense occupancy labels as used in the oracle settings of these existing works. Our comparisons with the state-of-the-an approaches are direct ab- lations as we maintain exactly the same experimental setups except for the loss function. In addition,
our approach also comfortably outperforms the ablation baselines (the middle 5 approaches).
Caregory
Craft
Rifle
Disp.
Lamp
Spk.
Box
Chair
Bench
Car
Flanc
Sofa
Table
Phone
Mean
OcdNet
49.6
39.7
49.7
33.3
49.3
42.1
42.8
30.9
57.2
41.7
60.7
42.4
64.8
46.5
DISN
54.5
52.5
50.2
39.2
53.3
46.0
50.6
37.1
58.6
48.5
64.9
48.4
67.6
51.6
CoReNet
60.5
61.0
56.8
51.3
47.1
58.4
68.7
77.3
59.5
DISN(Re500)+DV
61.1
64.5
61.9
46.9
58.2
54.4
59.6
48.0
59.4
58.4
69.5
57.2
78.7
59.8
Abl.-NoGrad
11.9
4.9
23.6
15.4
31.4
30.6
25.1
10.1
20.8
7.7
28.3
20.1
21.2
19.3
Abl--Nomm (10 %)
23.7
19.9
33.1
26.5
40.0
15.2
35.6
18.6
33.0
15.7
38.8
30.7
41.0
30.1
Abl--Norrad (30 %)
26.4
15.5
36.8
23.6
40.6
36.0
36.8
19.5
38.5
19.1
47.8
30.9
39.4
31.6
AbL-NoGrad (50
39.7
39.7
37.5
53.5
38.6
FixeeE
49.6
52.5
44.4
33.0
46.3
39.9
43.4
27.5
57.5
46.9
58.8
39.1
68.0
46.7
OceNet w/ Eq
50.7
42.6
50.4
53.0
50.1
43.7
44.4
32.9
56.8
41.6
60.9
44.0
68.4
47.7
DISN Eq
53.3
51.2
51.9
38.7
52.1
43.8
50.8
36.2
58.8
47.4
64.6
47.4
66.6
$1.0
CoReNet w/ Eq D (GGS)
61.1
67,5
62.7
44.2
54.8
49.6
59.5
45.4
59.4
59.9
69.8
55.1
78.0
59.0
(aSN(Ra500HDDV w/ Eq. (DGS Best)
60.8
62.6
62.3
47.1
57.7
53.5
59.8
47.4
59.2
58.6
70.7
57.4
77.0
59.6
EXPERIMENTS
4.1 LEARNING FROM SYNTHETIC DATA (SHAPENET)
Dataset. Following Popoy al. 2020), conduct experiments on ShapeNet with the low-realism and the high-realsim renderings. For low-realism, we use the renderings of various models from Choy al. 2016). Following Mescheder et al. (2019), we split the dataset into the training set of 30661 models, the validation set of 4371 models, and the test set of 8751 models. During testing, we follow Mescheder et al. (2019) to only test the first rendering of each model. Similar to all the prior works, we use the same 13 classes to report any performance via grouping all the test cases into the same class. For high-realism, we follow the split and evaluation protocol of the single-object scenes Popov et al. (2020) we only use the 13 categories used by Choy et al. (2016), filter out repeated samples, and finally construct the data with 666,565 models from the training set, 96,084 from the validation set, and 189,74 ,48 from the test set. Following Popov et al. (2020), we only evaluate the first 1% test cases of the test set (1898 samples).
Metrics. We use the Intersection-of- Uniiion (IoU) for evaluating the model performances. For low- realism renderings, we follow the protocol of IoU benchmarking from Mescheder et al. (2019) that evaluating occupancy of the specified 100K query points in the space. For high-realism renderings, we follow the protocol from Popoo et al. 2020) to evaluate occupancy prediction of the 128 x 128 x 128 grid. Following Popov et al. (2020). the grid cube is unit cube that spans from - 0.5 to 0.5 for r and y, and from //2 to 1 //2 for in the camera coordinate system.
Comparison with state-of-the-art approaches. On high-realism ShapeNet, we build our model on top of the representative state-of-the-art models- OceNet (Mescheder al. 2019), DISN (Xu al.) 2019) CoReNet (Popov et al. (2200) for their 3D volumetric implicit modeling capability. We also incorporate a strong baseline named "DISN (ResNet50) + DVR (Niemeyer et al.] (020)", where we replace the originally used VGG (Simonyan & Zisserman, 2014) with ResNet-50 as the image encoder as used in CoReNet (Popov et al.] al.]||200) (0200) and devised the 5-layer fully connected ResNet as the decoder as used in DVR (Niemeyer et al. (020). In each DGS experiment, we maintain exactly the same encoder and decoder architecture, optimization parameters (c.g. the learning rate and the epsilon of Adam Kingma & Ba 2014h) and the prediction format (per-query occupancy probability). We set Aor to be 0.01 in our loss function (Eq. []. Note that the DGS experiments can only access nearsurface training signals, in contrast to the oracle learning setting in the state- of-the-art approaches where models were trained with dense occupancy labels. We name the DGS version of the CoReNet model as the DGS model, and the DISN+DVR counterpart as the DGS- Best model. Quantitative results are reported in Tab, X for the low-realism evaluation setting, and in Tab. for the high-realisn setting Our competitive experimental results indicate that our learning
'e noticed that in our original submission version of the paper, our experiments were conducted with evaluating the 64 x 64 x 64 volume grids due to an error. We thus revised our results in Tab. with the evaluation results returned from the 128 128 x 128 volume grid, and attach our previous results Tab. Please refer to Sec. for detailed explanation.
7

---

Published as a conference paper at ICLR 2022
View O
View
View 2
View 3
Input
OceNet
DISN
CoReNet
MiDaS
AdelaiDepth
DGS
GeoReg
TSDF
challenging test case ScannetV2. For each predicted surface with red and sky-blue colors, sky-blue indicates "positive precision" for that surface region, while red indicates "negative precision". The ground truth surface is shown the top-right corner of each prediction with gold and navy-blue colors, navy-blue indicates "positive recall", while gold indicates "negative recall", The larger the blue region is, the higher the Fl score would be.
Figure 5: Qualitative comparisons
Table 2:
Benchmarking results of single view 3D surface reconstruction on ScannetV2 test set.
Acc (1)
Compl (V)
Chamfer (L)
Prec (1)
Recall (f)
F1 Fc-sc (1)
OceNet + GeoReg
18.3
18.5
18.4
30.4
28.5
26.8
DISN + TSDFVox
13.0
53.3
33.1
25.3
10.2
12.4
CoReNet
19.5
15.6
17.6
30,9
36.3
29.9
MiDaS
13.9
18.5
16.2
41.9
30.1
34.4
AdelaiDepth
9.7
18.8
14.2
46.4
33.2
37.9
Dolation-NoGrall
17.0
11.7
14.3
34.1
47.0
36.5
Ablation FixedE
15.0
20.3
17.7
31.6
29.8
27.3
DGS
14.3
11.5
12,9
39.7
49.4
41.6
framework along with the differentiable gradient sampling layer implementation plays the critical role when learning with only the near-surface training labels, and can achieve similar performance without the dense occupancy training labels.
Please refer Sec. Aor ablation baselines details and low-realism ShapeNet experimental settings.
4.2 LEARNING FROM REAL SCANNED DATASETS (SCANNETV2)
Dataset. We use ScannetV2 (Dai et al. 2017) for training and evaluating the performance of the models on the real images. We follow the standard training testing split as used in [Sun et al. (202T) and Murez et al. (2020), where 1513 scenes are for trainval (with 1201 for training and 312 for validation), and 100 for testing. Each scene is provided with multiple image capture as well as the associated camera pose. We train the models with all the views given in the training / validation set (2423872 frames in total, after filtering out frames with invalid extrinsic poses), while for testing, we select 10 frames with different extrinsic poses for each test scene. Practically, since all the frames of the scenes are the form of video clips, with adjacent frames associated with similar extrinsic cameras poses, we select the 10 frames for each frame via extracting every 100 frames from each scene video (e.g. Frame 1, 101, 201, .... 901), resulting in 1000 frames in total in our test set (10 frames per scene, with 100 test scenes).
Metrics. We use the same evaluation metrics following Sun et al. 202 and Murez et al. 2020). Since we are the first, to our knowledge, evaluate single view 3D implicit reconstruction on the ScannetV2 benchmark, and here we only evaluate the geometries within the camera view frustum rather than the whole scene geometries as in [Sun et al. (202T): Murez al.] (2020). addition, we also only evaluate the geometries in front of the amodal depth for each pixel ray. where only the space front of the wall, bounded within the ceiling and the floor, is evaluated. We define the amodal depth for pixel ray to be delineated by the minimum between the closest structure-category surface (e.g. walls and doors, etc) and the farthest surface. In practice, in order to accommodate the evaluation of the surfaces right on the amodal depth, slack the evaluation scope with factor (1.05 in our case) multiplied with the amodal depth. This evaluation protocol would be equivalent to the "single-layered" protocol used in Sun et al. (202)) within our single-view scenario.
8

---

Published as a conference paper at ICLR 2022
View
View
View
Dech Figure 6: Qualitative comparisons unseen test image downloaded from the internet.
Inpul
OceNet GeoReg CoReNet
NoGrad
FixedE
DGS
V
Figure 7: Two representative failure cases of our approach.
Due to the inherent ambiguity of the scaling and shifting of the predicted 3D single view geometry (Ranfil 2019 Yin et al. 202T), follow them by computing the best scale and shift com- paring the predicted depth with the ground truth depth. For approaches that predict 3D surfaces, the rendered depth map from the predicted mesh would be used for calculating the scale and shift.
Please refer to Sec. B for details on baselines.
Results. We provide quantitative comparison in Tab. and qualitative comparison in Fig. re- spectively. Our model outperforms all state-of-the-- art approaches as well as the ablation models. Compared to the synthetic data scenario in Tab. [ and 4 our model demonstrates large advantages when compared to existing state-of-the-ar approaches, as our motivation stems from addressing learning from imperfect 3D labels directly from the real scan data. Compared single-view depth prediction approaches trained on massive data (Lasinger et al., 2019 Yin et al. 2021). our approach does not prevail "Acc" and "Prec". This is due the fact that these metrics only project the predicted surface to the ground truth surface, giving advantages to approaches that only predict the visible surface. Our approach still prevails for other metrics (Chamfer Distance and FI), which are considered as the most important metrics (Božić et al. 2021 Sun et al. 2021 Murez et al. 2020).
Generalization to Unseen Scenes (Pix3d and Open- Domain Images). We further test our model without further finetuning directly on unseen scenes for evaluating the generalizability of the learned model. We run our model on PSx al. (2018) as well as test images downloaded directly from the internet. We provide detailed qualitative comparison in Fig. [] more results in Fig. The results further indicate our learning framework exhibits promise for unseen scene generalization.
Failure Cases. We provide two representative failure cases of our approach in Fig. 7 The first case (the first row) demonstrates difficulties in predicting the floor occupancy as a result of the noisy and non-wer watertight mesh during training. The second case (the second row) shows that model cannot identify small objects (c.g. chairs) and not predict the invisible partition of these objects.
5 CONCLUSIONS
We have presented our learning framework for real-world 3D implicit surface reconstruction from single view image. Owing to our unique learning framework that directly trains the model from the raw scan data and our novel occupancy loss function over the gradients, we able go beyond the existing works for single-objects reconstruction or single view fitting. Thanks to our differentiable gradient sampling module enable efficient and memory-efficiee end-to-end training from images and demonstrated single view 3D surface reconstruction results on scenes for the first time, to our knowledge.

---

Published as a conference paper at ICLR 2022
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: system for large- scale machine learning. In /2th {USENIX} symposium on operating systems design and imple- mentation ((OSDI) 16), pp. 265-283, 2016.
Matan Atzmon and Yaron Lipman. Sal: Sign agnostic learning of shapes from raw data. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2565-2574, 2020.
Aljaž Božič, Pablo Palafox, Justus Thies, Angela Dai, and Matthias Nießner. Traasformerfusioon Monocular rgb scene reconstruction using transformers. arXiv preprint arXiv:2107.02191, 2021.
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint fttd:::7090655555 2017.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: 2015.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2019a.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In IEEE Computer Vision and Pattern Recognition (CVPR), 2019b.
Christopher B Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In European conference on computer vision, pp. 228-64.. Springer, 2016.
Michael Crandall and Pierre-Louis Lions. Viscosity solutions of hamilton-jacobi equations.
Transactions of the American mathematical society, 277(1):1-42, 1983.
Brian Curless and Mare Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd annual conference Computer graphics and interactive techniques, pp. 303-312, 1996.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5828-5839, 2017.
Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object recon- struction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 605-613, 2017.
Amos Gropp. Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regu- larization for learning shapes. arXiv preprint arXiv:2002. 10099, 2020.
Groueix, M Fisher, VG Kim, BC Russell, and M Aubry. Atlasnet: a papier-måché approach to learning 3d surface generation (2018). arXiv preprint arXiv:1802.055:
11.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et Spatial transformer networks. Advances in neural information processing systems, 28:2017-2025, 2015.
Yue Jiang, Dantong Ji, Zhizhong Han, and Matthias Zwicker. Sdfdiff: Differentiable rendering of signed distance fields for 3d shape optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1251-1261, 2020.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3907-3916, 2018.
10

---

Published as a conference paper at ICLR 2022
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Katrin Lasinger, René Ranftl, Konrad Schindler, and Vladlen Koltun. Towards robust monoc- ular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. arXiv preprint
arXiv:1907.01341, 2019.
Manyi Li and Hao Zhang. D2im-net: Learning detail disentangled implicit fields from single images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 10246-10255, 2021.
Chen-Hsuan Lin, Chaoyang Wang, and Simon Lucey. Sdf-sm: Learning signed distance 3d object reconstruction from static images. arXiv preprint arXiv:2010. 10505, 2020.
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. The IEEE International Conference on Computer Vision (ICCV), Oct 2019.
William Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics. 21(4):163-169, 1987.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sehastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4460-4470, 2019.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis, 2020.
Zak Murez, Tarrence van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Rabinovich. Atlas: End-to-end 3d scene reconstruction from posed images. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part Springer, 2020.
VII 16, pp.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumet- ric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3504-350 -515,, 2020.
Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. arXiv preprint arXiv:2104. 10078, 2021.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 165-174, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances neural information processing systems, pp. 0026-8037, 2019.
Stefan Popov, Pablo Bauszat, and Vittorio Ferrari. Corenet: Coherent scene reconstruction from single rgb image. In European Conference on Computer Vision, pp. 366-3633 Springer, 2020.
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. arXiv preprint 1rXiv:1907.01341 2019.
Shunsuke Saito. Zeng Huang. Ryota Natsume. Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-al Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceed- ings of the IEEE International Conference Computer Vision, pp. 2304-20420 2314, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.

---

Published as a conference paper at ICLR 2022
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Con- tinuous astructure-awar neural scene representations. arXiv preprint 1r.xiv:996..om 01668, 2019.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from single depth image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1746-1754, 2017.
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. The Replica digital replica of indoor spaces. arXiv preprint arxiv: 00997777777 2019.
dataset:
Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao. Neuralrecon: Real-time coherent 3d reconstruction from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15598-15607 2021.
Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang. Chengkai Zhang, Tianfan Xue, Joshua B Tenenbaum, and William T Freeman. Pix3d: Dataset and methods for single-image 3d shape modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9974-2983. 2018.
Shubham Tulsiani, Tinghui Zhou, Alexei A Efros, and Jitendra Malik. Multi-view supervision for single-view reconstruction differentiable ray consistency. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 2626-2634, 2017.
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2 106. 10689, 2021.
Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. Synsin: End-to-end view synthesis from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7467-7477, 2020.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Qiangeng Xu, Weiyue Wang. Duygu Ceylan, Radomir Mech, and Ulrich Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. In Advances in Neural Information Processing Systems, pp. 490-500, 2019.
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Ronen Basri, and Yaron Lip- man. Multiview neural surface reconstruction by disentangling geometry and appearance. arXiv preprint ar arXiv:2003.0995
2020.
Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. arXiv preprint arXiv:2106. 1et::2200..22222 /2052, 2021.
Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine Hornung Differentiable surface splatting for pasnt-based geometry processing. ACM Transactions Graphics (TOG), 38(6):1-14, 2019.
Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Long Mai, Simon Chen, and Chunhua Shen. Learning to recover 3d scene shape from single image. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 203- 2021.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one few images. arXiv preprint arXiv:2012.0211 2020.
12

---

Published as a conference paper at ICLR 2022
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. arXiv preprint 1xXI::2103.1402: 2021.
13

---

Published as a conference paper at ICLR 2022
APPENDIX
In this document, we provide additional experimental details and results on ShapeNet in Sec. A additional qualitative results on ScannetV2 in Sec. B. analysis of numerical gradient approximation
in Sec. C additional results generalizing external unseen scenes in Sec. D and B as well as the details of the formulation and derivations of DGS in Sec.[F)
A ADDITIONAL DETAILS AND RESULTS ON SHAPENET
Additional Quantitative Results. We provide our original results on high-realism ShapeNet in Tab. [] These results are evaluated with the 64 x 64 x 64 volume grids due an error in our prior code base. Based on the results in both Tab. I and Tab. we found our proposed learning framework performs competitively with the oracle training setting, even though our approach does not utilize dense occupancy labels as used in the oracle state-of-the-an approaches (OccNet, DISN, CoReNet and DISN (ResNet50) + DVR). On the other hand, we observe that our model demonstrates slight improvements when evaluated with the resolution of 64 x 64 x 64, while evaluating the same models with the resolution of 128 128 x 128 would reverse the course. particular, we found categories like "car" cxhibited major performance decrease on all the approaches with the higher resolution evaluating setting. This might be due to the fact that these categories demonstrate empty ground truth inside the objects, and and at a higher resolution, the metrics are biased to penalize models both resolution
that predict "occupie" inside the object. However, the competitive performance settings indicates that our learning with only the surface labels does not deteriorate the performance, and can be faithfully applied toward the real-scene settings where the dense occupancy labels are truly unavailable.
Table 3: Intersection over Union % (loU 1) benchmarking result on the high-realism ShapeNet with the resolution of 64 x 64 x 64. Please refer to Tab. (the resolution of 128 x 128 x 128) for details.
Calegory
Crail
Rille
Disp.
Lamp
Spk.
Box
Chair
Bench
Car
Plane
Sola
Table
Phone
Mean
OceNet
$4.7
48.7
55.3
38.6
57.1
52.0
48.9
39.7
70.6
48.9
63.6
49.2
71.7
33.8
DISN
60.9
56.9
61.5
55.2
57.2
68.2
74.5
59.7
CoReNet
62.5
65.5
63.2
48.2
63.0
56.7
60.6
48.3
73.7
58.1
69.8
55.0
75.1
61.5
D0SN(R
66.9
74.4
67.3
54.5
66.1
63.2
66.2
58.6
71.2
67.2
72.5
63.1
82.6
67.2
Abl--NoGrad
14.1
6.1
27.3
18.1
37.3
40.4
28.7
13.0
26.2
30.3
23.4
25.1
23.1
AbL-NoGrad (10 %)
28.2
25,8
39.3
32.4
48.2
46.5
42.0
24.6
42.4
20.0
42.7
37.1
49.4
36.8
Abl-NoGrad (30
34.1
47.9
26.5
54.9
52.7
40.8
Nol.- (50% $5)
47.8
41.5
47.6
35.1
54.1
49.2
46.8
32.1
66.0
37.9
59.6
44.4
60.2
47.9
Abl--FixdE
52.6
57.0
47.8
37.4
52.6
49,7
47.4
33.0
69.2
51.0
60.2
42.9
09.2
51.5
OceNer Eq
56.8
52.6
56.8
39.3
58.1
53.6
42.4
71.7
49.7
63.5
76.0
55.7
DISN w/ Ea
60.4
63.9
59.6
46.2
61.1
55.8
58.3
46.6
73.2
57.3
68.7
55.5
76.7
60.2
CoReNet w/ Eq (DG))
63.3
65.7
49.0
57.0
70.3
78.3
62.9
Re55( FDV
w/ Eq.
DGS Best)
66.8
75.1
68.7
55.1
65.7
63.7
66.5
58.2
71.5
67.4
72.9
63.5
84.2
67.6
Ablation Study Details. We conduct ablation studies to further validate the importance of derived DGS module, via attempting work-around training methods without DGS.
i) NoGrad To test the performance of the baseline when only the surface data provided, train this ablation model in exactly the same way compared to its original model, with the only exceptions that only the near-surface points are equipped with training labels, and we do not use any training labels from the non-surface query points (where it is not necessary to backpropagate the loss gradient of the spatial gradient using DGS). To further evaluate how the of known voxels affects the learning performance, we enlarge the near-surface region and evaluate when the rate is 10%, 30% and 50%. An increase the performances among these baselines would indicate the importance of knowing more voxel labels if our proposed gradient loss (Eq. I is not imposed.
ii) FixedE We train with both the near-surface well as the spatial gradient supervision, without DGS meaning the loss gradient of the spatial gradient would not back-propagate to any module before the sampling module in our case, the feature encoder network. Note all the other losses without gradient sampling can still back back-propagated to the feature maps.
We report the ablation results in Tab. I Both experiments conducted with the high-realism ShapeNet data. We comfortably outpertorm all the in-house ablation baselines, validating the essen- tial roles of DGS in leamingg framework.
14

---

Published as a conference paper at ICLR 2022
Experiments on low-realism ShapeNet. For our low-realism evaluation setting, all the baseline approaches reported their results in the papers. For OccNet and DISN, the reported results are based knowing the category canonical view prior, which demonstrates considerable privilege with respect to accuracy Popoy et al. (2020). Hence, we mark the results from the literature as OccNet-Privilege and DISN-Privilege (OccNet-Priv. and DISN-Priv. for short). Compared their privileged setting, our results demonstrate superior results even without the category canonical view prior privilege. To further provide the the baseline results where these two approaches are without the category canonical view prior, we retrain their models with the released codes, and report the results in the "OceNet" and "DISN" row respectively. The results further provide evidence that the category canonical view prior demonstrates privilege on accuracy, as observed in Popov et al. 2020).
as the additional illustration of the Qualitative Results. We provide qualitative results in Fig. 9 13 performance of all the approaches on ShapeNet.
Table 4: Intersection over Union % (loU f) benchmarking result on the low-realism ShapeNet. Our proposed DGS learning advances state-of-the-an approaches (OccNet, DISN, CoReNet and D'IM-
Net) compared to the reported performance from the literatures.
Caneoor
Caft
Rits
Disp.
Lamp
Spk.
Box
Chair
Bench
Car
Plane
Sofa
Tablc
Phons
Mean
Priv.
53.0
47.4
47.1
37.1
64.7
73.3
50.1
48.5
73.7
57.1
68.0
50.6
72.0
57.1
DISN-Priv.
60.2
68.0
57.7
39.7
55.9
53.1
54.9
54.2
77.0
61.7
67.1
48.9
73.6
59.4
OceNet
55.4
39.5
45.1
45.8
50.7
68.3
52.6
DISN
49.0
44.4
55.5
39.0
67.3
71.7
49.0
41.2
64.7
45,0
66.4
50.7
70,5
55.0
CoReNet
54.0
64.6
57.2
42.1
60.8
50.9
63.0
50.8
57.3
53.0
70.6
55.5
73.1
$7.9
DMIM-Net
52.7
42.1
55.0
79.8
55.8
53.7
76.2
59.1
DGS
57.0
65.7
58.6
45.5
38.6
55.2
39.8
48.2
71.6
56.8
68.1
55.6
78,4
60.0
B ADDITIONAL RESULTS ON SCANNETV2
Baselines. Since we demonstrate the first attempt to predict 3D implicit surface of scenes from single image, very few exiting works provide direct baseline performance to our task. For Occ- Net (Mescheder et al. 2019), since its image feature extraction is not local, and its gradient propa- gation does not require sampling, we use its direct application with Gropp et al. (2020) "OccNet GeoReg" as one baseline. We train DISN (Xu al. 2019) with the TSDF voxel izaaion labels (Murez) cet al.) 2220 (Snn et al. 2021; Božič et al. 2021). For CoReNet (Popov et al. 2020), we stick to its own voxelization and internal filling toolbox for obtaining the training labels. Since CoReNet can only predict geometries within the fixed range of space, tried our best to pick the best cube loca- tion based on the dataset statistics. We also incorporate depth approaches (Ranftl et al. 2019 Yin al. I for comparison by finetuning their weights on ScannetV2. Lastly, compare with the two ablation models NoGrad and FixedE as introduced in Sec.
Implementation details for DGS. Since the DISN variant architectures (Xu et al. 2019 Niemeyer
eet al. 2020) Yu et al.) 2020) demonstrate higher degress of flcxibility of representing any point the space, in contrast 3D convolutions that are fixed for particular range of space (Popov ct al., (020). we build our model based on the former architectures. Similar to Niemeyer et al. (2020): Yu et al. 2020). we use residual blocks the fully connected part with the first 3 blocks receiving the 2D features and its spatial gradients. Our image encoder uses the same RexNext101 architecture (Xie et al. 2017) as in Yin et al. (202T) and starts the training with the pre-trained weights. We use a batch size of 32 images with 2048 near-surface points and 512 non-surface points per image. Similar to the ShapeNet experiments, we set oor to be 0.01. We found in the real-world setting, this parameter setting is crucial for the convergence of the learning procedure.
Quantitative Results for 2.5D evaluation metries. We provide the quantitative 2.5D evaluation results in Tab. 5 Our approach constantly outperforms all other implicit based baselines. While our approach falls short slightly when compared to the existing depth- approaches, we claim that our approach is not directly trained with the massive depth training data. Our results indicate that our depth performance is still on par with the state-of-the-art depth prediciton approaches.
Additional qualitative results. We provide additional qualitative results ScannetV2 Fig.
14 18
15

---

Published as a conference paper at ICLR 2022
Table 5: Benchmarking results of the depth metrics on ScannetV2.
AbsRel
AbsDiff (I)
SqReI (C)
RMSE (J)
LogRMSE (1)
81 (f)
82 (1)
03 (1)
OceNet + GeoReg
13.5
23.7
7.2
33.2
17.3
82.2
95.5
99.0
DISN ToDFVox
15.8
26.9
9.2
34.9
18.8
77.3
94.1
98.5
CoReNet
6.6
30.4
84.5
96.0
MiDaS
9.3
15.8
3.1
21.2
11.7
91.4
98.7
99.8
AdelaiDepth
6.1
10.5
1.9
15.7
8.6
95.3
99.1
99.8
Ablation-Fair
10.5
17.6
5.3
26.6
14.9
88.2
96.4
98.9
Ablation-FixedF
14.4
7.6
32.9
17.9
81.4
95.4
98.9
DGS
8.9
14.7
4.1
22.7
12.7
91.1
97.5
99.3
C ANALYSIS WITH NUMERICAL GRADIENT APPROXIMATION
As part of proposed learning framework with the loss imposed the spatial gradient, the dif- ferentiable gradient sampling module (Sec. (.2) has a numerical alternative where we can perturb the query points get its simulated numerical gradients. We further compare the closed-fore per- formance with the numerical counterpart in Tab, Please find qualitative comparison in Fig.
We provide closer look experiments that the simulated gradient baseline demonstrates relatively severe convergence diffi- culties. As shown Fig. 8 the simulated gradient model (red) does not observe loss drop in the first 10k training iterations. Despite its subsequent loss drop. which indicates the simulated gradient model can still learn to predict the geometry in moderate accuracy, its converged training loss is still higher than our DGS (blue). This aligns with our benchmarking evaluations in Tab. 6 that the closed-form variant (DGS) achieve better performance.
comparison the simulated gradient baseline. We found in our
Table 6: Intersection over Union % (IoU 1) benchmarking comparison between the closed-form solution and the numerical gradients on the high-realism ShapeNet. We report the performance comparison with both the resolution settings of 64 x 64 x 64 and 128 128 x 128 (Please refer to
Sec. Aor for details).
Category
Craft
Rife
Disp.
Lamp
Spk.
Box
Chair
Bench
Car
Plane
Sofa
Table
Phone
Mcan
Numerical (64)
62.4
68.3
63.7
46.2
60.6
54.9
61.0
49.0
70.7
61.1
69.2
53.6
77.2
61.3
Cloved-
61.2
37.0
Numerical (128)
39.7
66.9
61.3
41.8
54.1
48.5
58.2
43.8
59.8
39.0
68.5
54.3
77.0
57.9
Closed-form (128)
61.1
67.5
62.7
44.2
54.8
49.6
59.5
45.4
59.4
59.9
69.8
55.1
78.0
59.0
0.8
Numerical
The Total Loss
0.7
Closed-Form
Iterations
x10
Figure 8: Convergence Analysis for the comparison between the dooed-form (blue) and numer- ical counterpart (red). Notably, the numerical counterpart does not observe loss drop in the first 10k iterations.
D ADDITIONAL GENERALIZABLLTTY QUALITATIVE RESULTS
We provide more qualitative results on our model generalizing to unseen images downloaded from the Internet in Fig. 19 Each result is visualized in 6 views. Once again, these additional visual results further indicate that our model demonstrates good generalizability capability for handling unseen indoor scene images.
16

---

Published as a conference paper at ICLR 2022
Table 7: Benchmarking results of single view 3D surface reconstruction Matterport3D
(trained by the Scannet V2 dataset).
Acc (4)
Compl (4)
Chamfer ())
Prec (?)
Recall (1)
Fi-score (1)
OccNet + GeoReg
24,8
33.3
29.0
21.9
27.7
23,4
DISN Tox
32.7
25.6
29.1
19.1
33.3
22.9
CoReNet
35.2
23.3
29.3
20.8
35.5
24.9
AdelaiDepth
14.1
32.9
23.5
33.7
29.9
Oolaton-NNorral
39,5
20.7
30.1
22.9
43.6
28.7
Ablation-Fixeda FixedE
22.7
34.7
28.7
19.5
24.0
20.2
DGS
24.4
22.1
23.2
25.7
41.8
30.5
E EVALUATION THE GENERALIZABILITY TO MATTERPORTSH DATASET
To quantatively evaluate our generaizability new datasets, we provide our results on the Matter- port3D dataset (Chang et al. 2017) using our model trained from ScannetV2 as used in Sec. 4.2
The Matterport3d dataset (Chang et al. 2017) collects indoor scenes 90 full houses. Textured meshes are provided along with the scanned real images. Since the Matterport3D dataset does not provide an official train/test split, and our model is not trained on Matterport3d, we evaluate our model over all the 90 scenes in the dataset. In particular, we use the 129600 images (without the elevated pitch views where the camera is looking up at the ceiling) from the dataset, use the 10 first images from each scene (resulting in 900 test images in total), and test their reconstruction performance in the same way as we tested on ScannetV2. We provide the quantative evaluation result in Tab. 7 We can see from the table that the results align with our evaluation in Tab, 2 that our model demonstrates clear advantages. We also noticed that the AdelaiDepth Yin et al. (2021) baseline (finetuned ScannetV2) performance is closer to results (F1) score compared to S2 evaluation. This is probably due to the fact that AdelaiDepth was directly pre-trained from massive number of training images and potentially gives some privilege when generalize to new dataset.
F DETAILS OF THE FORMULATION AND DERIVATION OF DGS
In this section, we provide the full formulation as well as the derivation procedure of our DGS in this section as the supplementary to Sec, 3.2
In Eq. we provided the backward gradient for differentiable sampling for Pixel A. The backward gradient for Pixel B, and D are
0L a)
OL
dgg
(p(i.))
ØL.
ÔL.
a(1 B) (o(i,))
(9)
dec
aL.
aL
= aB
aép
Oo(i.j)
In Eq. we provided the forward gradient in the vertical pixel direction. The forward gradient of the horizontal pixel direction
ao(i.)) Øj
(1-a)(óe 0A) + aldd dc)
(10)
w
In Eq. 6 we provided the backward gradient for only Pixel A. The backward gradient for Pixel B, and D are
17

---

Published as a conference paper at ICLR 2022
0L (1 a-83
ØL
+ (-
ØL
1- a
+
dΦn
d
OV DVi((i,))
oV (((ji
0L a( B)
0L
3
ØL
a
ØL
+
+
(11)
doc
()
h
0V (Qi
w
0V )o(ūjj
OL
0L
0L
0L
aB (
+
+ w
doD
"V,((i,)
(i
In Eq. provided the backward gradient for only Pixel A in the 3D setting. The backward gradient for Pixel B, C and D are
0L
AL
0l
1 -
fo
aL
3 fu (-
(1 (1-01) a)s+ (1-
dΦB
dœ(I, y, 2)
OV (x, y, 2)
W
(z y, 2)
0L
3
v
a
5/0
0V. y,
h
x²
22
aL
OL
fo
aL
1 - B fo
a(11 - (8) +
+
Dec
d0(r, 2)
0V
d x. y, 2)
ov
h
dL
1- 3
yfo
a
x
ff
0V VV= 1,
h
22
z2
0L
OL
0L
So
3 fo y0 y, 2)
0L
0xB3 +
(t y, 2)
8V VV, U, 2)
W
oV
2
0L
yfo
oV VV
2
22
(12)
As an illustration of the derivation for Eq. 8
we can obtain the backward gradient via
AL
AL
06
OL
0V.0
AL
OV aΦ
OL
ov
(13)
0Φ
OV
a
OV
ov.0
The four partial derivatives with respect to o in Eq. 13 can be determined based on the sampling rules. For example, based on Eq
3 we can quickly obtain the first derivative with respect Φ via
dΦ = (1 - (1-a)(11 ()
(14)
0Φ
For the other three derivatives with respect to 6 in Eq. and obtain them via
13 we can substitute Eq. 5 10 into Eq.
7
0426
fo
1
OΦa
fo
(15)
z
0V_
ufo
1
8
I.fo
1
+
d
22
h
22
w
We can then obtain Eq. via substituting Eq.
14
15
into
Eq.
13
18

---

Published as a conference paper at ICLR 2022
View 2
View
E 
View
View O
View
View
View
Input
OceNet
DISN
CoReNet NoGrad
FixedE
DGS
DGS
GT
Namurical Ginalies Closed-for form Gradient
Figure 9: Quantitative comparison on the high-realism ShapeNet (without handpick: test case num- ber 0 and 100). The reconstruction result of each approach is visualized in six different views, with the first view the same as the camera view, the first three views the same elevation as the camera view, and the last three view horizontal view.
19

---

Published as a conference paper at ICLR 2022
View 2
View 3
View 4
View 9
View
View
Input
OceNet
DISN
CoReNet NoGrad FixedE
DGS
DGS
GT
Namerical Giudien Closed- form Gradient
Figure 10: Quantitative comparison on the high-realism ShapeNet (without handpick: test case number 200 and 300). The reconstruction result of each approach is visualized in six different views, with the first view the same as the camera view, the first three views the same elevation as the camera view, and the last three view horizontal view.
20

---

Published as a conference paper at ICLR 2022
View
View
View
View o
View
View
Input
OceNet
DISN CoReNet NoGrad FixedE
DGS
DGS
GT
Nameriaal Ginalias Closed-fore Gradient
Quantitative comparison on the high-realism ShapeNet (without handpick: test case number 400 and 500). The reconstruction result of each approach is visualized in six different views, with the first view the same as the camera view, the first three views the same elevation as the camera view, and the last three view horizontal view.
Figure 11:
21

---

Published as a conference paper at ICLR 2022
View 2
E
View 1
View
View 9
View
View
View
Input
OceNet
DISN
CoReNet NoGrad FixedE Namerial Gradier Closed-foree Gradient
DGS
DGS
GT
Quantitative comparison on the high-realism ShapeNet (without handpick: test case number 600 and 700). The reconstruction result of each approach is visualized in six different views, with the first view the same as the camera view, the first three views the same elevation as the camera view, and the last three view horizontal view.
Figure 12:
22

---

Published as a conference paper at ICLR 2022
View
View
View 4
View 9
View
View
View
Input
OceNet
DISN
CoReNet NoGrad FixedE
DGS
DGS
GT
Namerial Grudien Closed-for Gradient
Figure 13: Quantitative comparison on the high-realism ShapeNet (without handpick: test case number 800 and 900). The reconstruction result of each approach is visualized in six different views, with the first view the same as the camera view, the first three views the same elevation as the camera view, and the last three view horizontal view.
23

---

Published as a conference paper at ICLR 2022
View
View
View 3
View 5
View
View
View
Input
OceNet GeoReg DISN TSDFVox Ablation Ablation-NoGradd NoGrad
Ablation FixedE
DGS
GT
Figure 14: Quantitative comparison on the ScannetV2 (without handpick: the first frame of the 1st and 2nd test scene ScannetV2). The reconstruction result of each approach is visualized in six different views, with the first view the same as the camera view, the first three views the same elevation as thecamera view, and the last three view elevated view.
24

---

Published as a conference paper at ICLR 2022
View
View
View 3
View S
View
View 3
View 5
Input
OceNet GeoReg
CoReNet
MiDaS
AdelaiDepth
DGS
GT
Figure 15: Quantitative comparison on the ScannetV2 (without handpick: the first frame of the 3rd and 4th test scene ScannetV2). The reconstruction result of each approach is visualized in six different views, with the first view the same as the camera view, the first three views the same elevation as thecamera view, and the last three view elevated view.
25

---

Published as a conference paper at ICLR 2022
View
View
View 3
View 5
View
View 3
View 5
Input
OceNet
GeoReg DISN TSDFVox Ablation NoGrad
Ablation FixedE
DGS
GT
Figure 16: Quantitative comparison on the ScannetV2 (without handpick: the first frame of the 5th and 6th test scene ScannetV2). The reconstruction result each approach visualized in six different views, with the first view the same as the camera view, the first three views the same elevation as thecamera view, and the last three view elevated view.
26

---

Published as a conference paper at ICLR 2022
View
View 3
View 5
View
View
View
Input
OceNet GeoReg
CoReNet
MiDaS
AdelaiDepth
DGS
GT
Figure 17: Quantitative comparison on the ScannetV2 (without handpick: the first frame of the 7th and 8th test scene ScannetV2). The reconstruction result each approach visualized in six different views, with the first view the same as the camera view, the first three views the same elevation as thecamera view, and the last three view elevated view.
27

---

Published as a conference paper at ICLR 2022
View
View 3
View 5
View
View 3
View
Input
OceNet
GeoReg DISN TSDFVox Ablation NoGrad
Ablation FixedE
DGS
GT
Figure 18: Quantitative comparison on the ScannetV2 (without handpick: the first frame of the 9th and 10th test scene ScannetV2). The reconstruction result of each approach is visualized in six different views, with the first view the same as the camera view, the first three views the same elevation as thecamera view, and the last three view elevated view.
28

---

Published as a conference paper at ICLR 2022
Figure 19: Additional Qualitative results of our model generalizing to unseen test images down- loaded from the Internet.
29


Reviews:
[{"review_id": "MnFuOD91e9t", "reviewer": "Reviewer_ZTzy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- I like the motivation of learning from 3D meshes that are not necessarily closed. This seems to allow 3D reconstruction neural networks to learn from a wider range of dataset resources, such as 3D scans or mesh reconstructions.\n- The experimental settings are described in a very detailed manner with justifications. The results from the proposed method seem to show some improvements upon baseline methods, more notably in scene reconstructions.\n\nWeaknesses:\n- Although learning from either closed or open 3D meshes is an interesting motivation, this only allows one to learn SDF. One cannot define occupancy for open 3D meshes, and the example in Fig 3 is misleading. How would you define occupancy near open wholes? This part is unclear.\n- The contribution is unclear and at most incremental. Two main components are described in the paper:\n  1. Loss functions for learning occupancy or SDF (Sec 3.1). In my understanding, the only novel term is the spatial gradient penalty for occupancy. The rest of the two terms are standard loss functions for learning occupancy, so the additional first term is proposed. The \"spatial gradient\" term for SDF is precisely the eikonal regularization (Crandall & Lions, 1983), and the remaining terms are standard losses as well. (It is also unclear why the loss in the background paragraph is not incorporated).  \n  The authors argue that conditioning the spatial gradients on pixels is novel. I think the authors should clearly state why conditioning on pixels is novel enough to be a standalone paper itself, as this to me is an overstatement and I don't see how simply conditioning for a different task is novel. Penalty on the spatial gradients have also been previously adopted for other single-image 3D SDF reconstruction tasks [A,B]. Also, the authors mentioned that \"the spatial gradient $\\nabla_{x,y,z}\\hat{f}_\\Theta(x,y,z)$ can be conveniently computed without the sampling procedure\" -- how?\n  2. The gradient expression of \"spatial gradients\" (Sec 3.2). My understanding is that this is basically treating the finite pixel differencing operation as a network op, which in nature is differentiable. I think this part is confusing in many aspects. First, I don't see why the authors emphasize that the formulation is closed-form, as the spatial gradient expression is clearly taken numerically and its derivative can be computed via automatic differentiation. If the authors meant to present the gradient of gradient expressions, please use second-order derivatives (e.g. $\\frac{\\partial^2 f(x)}{\\partial x^2}$). It is also unclear how the 3D case (Eq 7) is derived; the authors merely presented an equation without elaborating its meaning. It is unclear what exactly $h$ and $w$ are; the authors referred to Fig 4(c) but there are no explanations in the captions either, which prevents a total understanding. Finally, a very important reference of Spatial Transformer Networks [C] is missing, as it was the first to advocate differentiable sampling.\n- It is unclear what \"engineering constraints\" refer to (Fig 2 caption and before Eq 1).\n- What is the relationship between $x,y,z$ and $i,j$? These notations are cross-referenced throughout the paper but their distinction was never explained. In Fig 4, are the feature maps sampled according to $x,y,z$ or $i,j$? What does $\\nabla_{x,y,z}\\phi_{i,j}$ in Fig 4(a) mean?\n- Experiments:\n  - The results from the proposed method has its own training recipe (architecture, optimization etc), and thus it is unclear where the better performance is coming from. (It could not be about the losses or spatial gradients at all, but a better design of architecture.) I think it is essential to see results where the baseline methods (e.g. OccNet, DISN) are retrained with the proposed losses incorporated. Would these methods yield a boost of performance?\n  - Why are different baseline methods compared for different versions (low-res and high-res) of ShapeNet? I think it's sufficient to present just the high-res version, and have a more complete comparison with the baselines in the low-res table.\n  - Fig 5: it is undefined what \"positive/negative precision/recall\" mean. How close is a surface/surfel prediction to the ground truth is considered a \"positive precision\"? The red/blue figures are meaningless without precise definitions.\n  - Could the authors elaborate more what \"amodal depth\" means, why the surface are evaluated in terms of this new metric, and how they are visualized? Why is not the naive depth definition used to evaluate? In addition, if only depth were used to evaluate quantitatively, why would one care about reconstructing the entire scene, as one could alternatively go for a scene depth prediction task which yields better quality?\n- There is no conclusion section. What have we learned about this paper?\n\nOther minor problems:\n- Please use only a set of notations for occupancy labels, not both {0,1} and {+,-}. Also, please do not mix the use of $\\phi(I)$ and $\\phi$, where the former should be a function and the latter a variable.\n- Fig 6: why is the bed in AdelaiDepth only visible in view 0?\n\n[A] Jiang et al. \"SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization.\" CVPR 2020.  \n[B] Lin et al. \"SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images\". NeurIPS 2020.  \n[C] Jaderberg et al. \"Spatial Transformer Networks.\" NeurIPS 2015.\n", "summary_of_the_paper": "This paper presents a method for 3D scene reconstruction from a single image using implicit surface representations such as occupancy or SDF. The authors propose to incorporate loss functions on the spatial gradients to provide dense supervision in the 3D space in the case where 3D labels may be incomplete (e.g. open 3D meshes) or not well-defined everywhere. Experiments are performed on ShapeNet and ScanNet show that the proposed method can achieve competitive performance on single-image scene reconstruction tasks.", "summary_of_the_review": "I think this paper has an interesting motivation of learning 3D reconstruction from incomplete raw 3D scans / open meshes, but there are major flaws in the method description and the experiments (detailed above). I also don't think there are either sufficient novelty or insights in the paper. I think the submission needs much major revisions with additional experiments to validate the effectiveness of the proposed DSG.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AbMMujEzOi", "reviewer": "Reviewer_SoBW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Their proposed loss functions are novel as far as I know.  The idea makes sense, to only apply supervision near surface boundaries where the labels can be reliably produced.  They perform a thorough set of experiments including comparisons and an ablation study.  They also derive the closed-form gradients of their loss function and show the importance of using them over numerical derivatives.  Both the quantitative and qualitative results are convincing.\n\nOne question I had was why there were no coefficients to balance the strength of the regularization terms in equations (1) and (2).", "summary_of_the_paper": "This paper describes novel loss functions for learning to predict an implicit 3D scene representation from a single image.  They argue that when working with real scan data of scenes (rather than single objects) it is difficult to generate accurate occupancy or signed distance function (SDF) ground truth as would be required for supervised learning.  Instead, they propose to only use occupancy or SDF supervision near the surfaces of objects; elsewhere, they rely on constraints on the gradient of the occupancy or SDF adapted from Gropp et al. 2020.\n\nThey perform a thorough evaluation on several benchmark datasets and compare against state-of-the-art competing methods.  They show that they outperform competing methods, even though in some cases their method has access to less supervisory data.    They also perform an ablation study to show the importance of various parts of the loss function.\n\n", "summary_of_the_review": "This paper has novel and interesting contributions to the field of single-image 3D reconstruction.  They provide convincing experiments to validate their contributions.  The paper is also well-written and nicely presented.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "5-APjoRJSUq", "reviewer": "Reviewer_BwV5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "pros: \n1. the formation of DGS is novel and interesting, with promising performance. \n2. detailed experiments on both shapenet and scan data\n3. detailed ablation study \n4. better performance compared with pervious methods.\n\ncons:\n1. the overall learning loss is not novel (Eq 1 & 2), for the Eikonal regularization part.\n2. there are some camera  parameters are involved in DGS, which is a bit hard to get in general.\n3. it is not clear what is the percentage of the know voxel occupancy/SDF,   and how much would the rate effect the learning. It ill be better to have some discussion and  ablations. IMHO, generally if the value of voxels closed to the surface is known, it might be ok to learn with other regions are missing.\n4. from the results (Fig9 &10), DGS  seems to be more likely to generate floating points, some discussion would be better to have here. ", "summary_of_the_paper": "This paper presents a new method to learn implicit 3D scene reconstructions from single image input. The main improvement is a closed-form Differentiable Gradient Sampling.  By taking spatial gradient into  consideration, the proposed method can apply back-propagation of the  loss on spatial gradients to feature maps and allow the training for the case of without dense 3D supervision.   ", "summary_of_the_review": "Overall, I think the proposed method is novel and with reasonable performance.  I am in favor for acceptance if the authors can provide some discussion about the cons listed above.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0EktWet713X", "reviewer": "Reviewer_BEgr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strength\n# Interesting and novel idea with the use of implicit representation\nImplicit representation has been extensively explored in 3D object reconstruction and novel view synthesis recently. It is interesting to see how we can use implicit representation in various applications, e.g. single-view reconstruction.\n\n# New sampling scheme for gradient\nThe authors have proposed a new sampling scheme for computing spatial gradient and thus a closed-form solution for loss propagation.\nAlthough the approach is new, I still have some concerns listed below.\n\n# Good result\nThe authors have shown good results in a variety of datasets, quantitatively and qualitatively.\n\n# Adequate ablation study\nAn ablation study is also provided to support the effectiveness of the proposed method.\n\n\n## Weakness\n# Generalization evaluation\nThe authors provide a qualitative evaluation (single sample) on an unseen test image. It will be much better if more examples can be provided. Moreover, a quantitative evaluation will be more appreciated to show the generalization ability of the trained model.\n\n# necessity of the gradient sampling scheme.\nThough the proposed sampling scheme is new, I don't understand why it is necessary (i.e. eq 5). What will be the difference between this approach and a naive approach that compute a feature gradient map first, followed by simple differentiable sampling (eq.4).\nCould authors shed some light on the difference? If the naive approach is a reasonable approach, why the proposed method is essential in this case?", "summary_of_the_paper": "In this paper, the authors propose a new method for single view 3D reconstruction. \nA conditional (image feature prior) implicit representation framework is proposed to reconstruct 3D scene from a single view.\nIn this paper, the authors propose that feature gradient is essential for watertight reconstruction and propose a differentiable gradient sampling method for the formulation.\nExperiments have been performed on both synthetic and real datasets.\nSuperior results have been presented.", "summary_of_the_review": "The paper is well written and presented overall.\nThe essential experiments are performed and the results are well presented.\nHowever, I have a question regarding the major contribution (see weakness).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new way to train the prediction of implicit 3D scene representations from a single view. The main innovations are a novel numerically stable and memory efficient formulation of the derivatives of a loss function based on the spatial gradients of the implicit field, and focusing the training on regions near the surfaces of objects. The method leads to good performance, especially when training on imperfect ground truth scan data.

Concerns were raised about the novelty of the approach and its significance. These were adequately addressed in the author response and revisions. The experiments were found to be well described and executed, which increases the confidence in the approach and its potential impact. I recommend acceptance.
Decision: Accept (Poster)