Differentiable Volumetric Rendering:
Learning Implicit 3D Representations without 3D Supervision
Michael Niemeyer1,2 Lars Mescheder1,2,3† Michael Oechsle1,2,4 Andreas Geiger1,2
1
2
Max Planck Institute for Intelligent Systems, Tübingen
University of Tübingen
3
4
Amazon, Tübingen
ETAS GmbH, Bosch Group, Stuttgart{firstname.lastname}@tue.mpg.de

Abstract
Learning-based 3D reconstruction methods have shown
impressive results. However, most methods require 3D
supervision which is often hard to obtain for real-world
datasets. Recently, several works have proposed differentiable rendering techniques to train reconstruction models
from RGB images. Unfortunately, these approaches are
currently restricted to voxel- and mesh-based representations, suffering from discretization or low resolution. In
this work, we propose a differentiable rendering formulation for implicit shape and texture representations. Implicit representations have recently gained popularity as
they represent shape and texture continuously. Our key insight is that depth gradients can be derived analytically using the concept of implicit differentiation. This allows us
to learn implicit shape and texture representations directlyfrom RGB images. We experimentally show that our singleview reconstructions rival those learned with full 3D supervision. Moreover, we find that our method can be used for
multi-view 3D reconstruction, directly resulting in watertight meshes.

Figure 1: Overview. We show that volumetric rendering isinherently differentiable for implicit shape and texture representations. Using an analytic expression for the gradientˆ
of the depth ∂∂θd wrt. the network parameters θ, we learn implicit 3D representations fθ from 2D images.
to make it differentiable [4, 11, 15, 21, 33, 36, 43, 44, 47, 50,
58,59,62,75,76,79,88]. While yielding compelling results,
they are restricted to specific 3D representations (e.g. voxels
or meshes) that suffer from discretization artifacts and thecomputational cost limits them to small resolutions or deforming a fixed template mesh. At the same time, implicit
representations [12, 48, 56] for shape and texture [54, 66]
have been proposed which do not require discretization during training and have a constant memory footprint. However, existing approaches using implicit representations require 3D ground truth for training and it remains unclear
how to learn implicit representations from image data alone.

1. Introduction
In recent years, learning-based 3D reconstruction approaches have achieved impressive results [12, 13, 17, 24,
41, 48, 49, 56, 64, 80]. By using rich prior knowledge obtained during the training process, they are able to infer a 3D model from as little as a single image. However, most learning-based methods are restricted to synthetic data, mainly because they require accurate 3D groundtruth models as supervision for training.
To overcome this barrier, recent works have investigatedapproaches that require only 2D supervision in the formof depth maps or multi-view images. Most existing approaches achieve this by modifying the rendering process

Contribution: In this work, we introduce Differentiable
Volumetric Rendering (DVR). Our key insight is that we canderive analytic gradients for the predicted depth map withrespect to the network parameters of the implicit shape andtexture representation (see Fig. 1). This insight enables us
to design a differentiable renderer for implicit shape andtexture representations and allows us to learn these representations solely from multi-view images and object masks.

† This work was done prior to joining Amazon.

1

Since our method does not have to store volumetric data in
the forward pass, its memory footprint is independent of
the sampling accuracy of the depth prediction step. We
show that our formulation can be used for various tasks
such as single- and multi-view reconstruction, and works
with synthetic and real data. In contrast to [54], we do
not need to condition the texture representation on the geometry, but learn a single model with shared parametersthat represents both geometry and texture. Our code anddata are provided at https://github.com/autonomousvision/differentiable volumetric rendering.

2. Related Work
3D Representations: Learning-based 3D reconstruction
approaches can be categorized wrt. the representation they
use as voxel-based [8, 13, 19, 61, 64, 73, 82, 83], pointbased [2, 17, 31, 40, 77, 85], mesh-based [24, 32, 41, 55, 80],or implicit representations [3, 12, 22, 30, 48, 49, 56, 66, 81].
Voxels can be easily processed by standard deep learning architectures, but even when operating on sparse data
structures [23, 64, 74], they are limited to relatively small
resolution. While point-based approaches [2, 17, 40, 77, 85]
are more memory-efficient, they require intensive postprocessing because of missing connectivity information.
Most mesh-based methods do not perform post-processing,but they often require a deformable template mesh [80] or
represent geometry as a collection of 3D patches [24] whichleads to self-intersections and non-watertight meshes.
To mitigate these problems, implicit representations havegained popularity [3, 12, 22, 30, 48, 49, 53, 54, 56, 66, 81]. By
describing 3D geometry and texture implicitly, e.g., as thedecision boundary of a binary classifier [12,48], they do notdiscretize space and have a fixed memory footprint.
In this work, we show that the volumetric rendering stepfor implicit representations is inherently differentiable. In
contrast to previous works, this allows us to learn implicit
3D shape and texture representations using 2D supervision.
3D Reconstruction: Recovering 3D information which islost during the image capturing process is one of the longstanding goals of computer vision [25]. Classic multi-view
stereo (MVS) methods [5–7,20,37,60,68–70] usually matchfeatures between neighboring views [5, 20, 68] or reconstruct the 3D shape in a voxel grid [6, 7, 37, 60, 70]. While
the former methods produce depth maps as output whichhave to be fused in a lossy post-processing step, e.g., using
volumetric fusion [14], the latter approaches are limited by
the excessive memory requirements of 3D voxel grids. In
contrast to these highly engineered approaches, our genericmethod directly outputs a consistent representation in 3D
space which can be easily converted into a watertight meshwhile having a constant memory footprint.
Recently, learning-based approaches [16, 29, 39, 58, 63,

86, 87] have been proposed that either learn to match image features [39], refine or fuse depth maps [16, 63], optimize parts of the classical MVS pipeline [57], or replace theentire MVS pipeline with neural networks that are trained
end-to-end [29, 86, 87]. In contrast to these learning-based
approaches, our method can be supervised from 2D images
alone and outputs a consistent 3D representation.
Differentiable Rendering: We focus on methods that learn
3D geometry via differentiable rendering in contrast to recent neural rendering approaches [42,51,52,71] which synthesize high-quality novel views but do not infer the 3D object. They can again be categorized by the underlying representation of 3D geometry that they use.
Loper et al. [47] propose OpenDR which approximates
the backward pass of the traditional mesh-based graphics
pipeline and has inspired several follow-up works [11, 21,
27, 28, 33, 44, 88]. Liu et al. [44] replace the rasterization
step with a soft version to make it differentiable. Whileyielding compelling results in reconstruction tasks, theseapproaches require a deformable template mesh for training, restricting the topology of the output.
Another line of work operates on voxel grids [46, 50, 57,79]. Paschalidou et al. [57] and Tulsiani et al. [79] propose
a probabilistic ray potential formulation. While providing asolid mathematical framework, all intermediate evaluations
need to be saved for backpropagation, restricting these approaches to relatively small-resolution voxel grids.
Liu et al. [45] propose to infer implicit representations
from multi-view silhouettes by performing max-poolingover the intersections of rays with a sparse number of supporting regions. In contrast, we use texture information enabling us to improve over the visual hull and to reconstructconcave shapes. Sitzmann et al. [72] infer implicit scene
representations from RGB images via an LSTM-based differentiable renderer. While producing high-quality renderings, the geometry cannot be extracted directly and intermediate results need to be stored for computing gradients.
In contrast, we show that volumetric rendering is inherently
differentiable for implicit representations. Thus, no intermediate results need to be saved for the backward pass.

3. Method
In this section, we describe our Differentiable Volumetric Rendering (DVR) approach. We first define the implicit neural representation which we use for representing
3D shape and texture. Next, we provide a formal description of DVR and all relevant implementation details. An
overview of our approach is provided in Fig. 2.

3.1. Shape and Texture Representation
Shape: In contrast to discrete voxel- and point-based representations, we represent the 3D shape of an object implicitlyFigure 2: Differentiable Volumetric Rendering. We first predict the surface depth dˆ by performing occupancy evaluations
for a given camera matrix. To this end, we project sampled pixel u to 3D and evaluate the occupancy network at fixed steps
on the ray cast from the camera origin towards this point. We then unproject the surface depth into 3D and evaluate thetexture field at the given 3D location. The resulting 2D rendering Î can be compared to the ground truth image. When we
also have access to ground truth depth maps, we can define a loss directly on the predicted surface depth. We can make ourmodel conditional by incorporating an additional image encoder that predicts a global descriptor z of both shape and texture.

using the occupancy network introduced in [48]:fθ : R3 × Z → [0, 1](1)

An occupancy network fθ (p, z) assigns a probability of occupancy to every point p ∈ R3 in 3D space. For the taskof single-view reconstruction, we process the input image
with an encoder network gθ (·) and use the output z ∈ Z tocondition fθ . The 3D surface of an object is implicitly determined by the level set fθ = τ for a threshold parameterτ ∈ [0, 1] and can be extracted at arbitrary resolution using
isosurface extraction techniques.1
Texture: Similarly, we can describe the texture of a 3D
object using a texture field [54]t θ : R3 × Z → R3

(2)

which regresses an RGB color value for every point p ∈ R3
in 3D space. Again, tθ can be conditioned on a latent embedding z of the object. The texture of an object is givenby the values of tθ on the object’s surface (fθ = τ ). In thiswork, we implement fθ and tθ as a single neural networkwith two shallow heads.
Supervision: Recent works [12, 48, 54, 56, 66] have shownthat it is possible to learn fθ and tθ with 3D supervision(i.e., ground truth 3D models). However, ground truth 3D
data is often very expensive or even impossible to obtain forreal-world datasets. In the next section, we introduce DVR,an alternative approach that enables us to learn both fθ andtθ from 2D images alone. For clarity, we drop the condition
variable z in the following.
1 See Mescheder et al. [48] for details.

Figure 3: Notation. To render an object from the occupancy
network fθ and texture field tθ , we cast a ray with direction
w through a pixel u and determine the intersection point p̂
with the isosurface fθ (p) = τ . Afterwards, we evaluate thetexture field tθ at p̂ to obtain the color prediction Îu at u.

3.2. Differentiable Volumetric Rendering
Our goal is to learn fθ and tθ from 2D image observations. Consider a single image observation. We define aphotometric reconstruction lossX
L(Î, I) =kÎu − Iu k
(3)uwhich we aim to optimize. Here, I denotes the observed
image and Î is the image rendered by our implicit model.2
Moreover, Iu denotes the RGB value of the observation I
2 Note that the rendered image Î depends on θ through f and t . Weθθhave dropped this dependency here to avoid clutter in the notation.

at pixel u and k · k is a (robust) photo-consistency measuresuch as the `1 -norm. To minimize the reconstruction loss
L wrt. the network parameters θ using gradient-based optimization techniques, we must be able to (i) render Î givenfθ and tθ and (ii) compute gradients of L wrt. the network
parameters θ. Our core contribution is to provide solutions
to both problems, leading to an efficient algorithm for learning implicit 3D representations from 2D images.

the gradient of fθ at p̂ wrt. the network parameters θ andthe surface point p̂. Thus, in contrast to voxel-based approaches [58, 79], we do not have to store intermediate results (e.g., volumetric data) for computing the gradient of
the loss wrt. the parameters, resulting in a memory-efficientalgorithm. In the next section, we describe our implementation of DVR which makes use of reverse-mode automaticdifferentiation to compute the full gradient (4).

Rendering: For a camera located at r0 we can predict thecolor Îu at pixel u by casting a ray from r0 through u anddetermining the first point of intersection p̂ with the isosurface {p ∈ R3 |fθ (p) = τ } as illustrated in Fig. 3. The color
value Îu is then given by Îu = tθ (p̂). We refer the reader
to Section 3.3 for details on the ray casting process.

3.3. Implementation

Gradients: To obtain gradients of L with respect to θ, we
first use the multivariate chain rule:∂L X ∂L ∂ Îu
=
·
∂θ
∂θ
u ∂ Îu(4)∂gdenotes the Jacobian matrix for a vector-valued
Here, ∂xfunction g with vector-valued argument x and · indicatesmatrix multiplication. By exploiting Îu = tθ (p̂), we obtain∂ Îudtθ (p̂)∂tθ (p̂) ∂tθ (p̂) ∂ p̂
=
=
+
·
∂θ
dθ
∂θ
∂ p̂
∂θ

(5)

since both tθ as well as p̂ depend on θ. Because p̂ is defined
implicitly, calculating ∂∂θp̂ is non-trivial. We first exploit thatp̂ lies on the ray from r0 through u. For any pixel u, thisray can be described by r(d) = r0 + dw where w is thevector connecting r0 and u (see Fig. 3). Since p̂ must lie onˆ such that p̂ = r(d).
ˆ We callr, there exists a depth value d,∂ p̂ˆd the surface depth. This enables us to rewrite ∂θ as
ˆ
∂r(d)
∂ dˆ
∂ p̂
=
=w
∂θ
∂θ
∂θ

(6)

For computing the gradient of the surface depth dˆ with respect to θ we exploit implicit differentiation [3, 65]. Differentiating fθ (p̂) = τ on both sides wrt. θ, we obtain:∂fθ (p̂) ∂fθ (p̂) ∂ p̂
+
·
=0
∂θ
∂ p̂
∂θ
∂fθ (p̂) ∂fθ (p̂)
∂ dˆ
⇐
⇒
+
·w
=0
∂θ
∂ p̂
∂θ

(7)

Rearranging (7), we arrive at the following closed form exˆpression for the gradient of the surface depth d:

−1
∂ dˆ
∂fθ (p̂)∂fθ (p̂)
=−
·w
(8)
∂θ
∂ p̂
∂θ
We remark that calculating the gradient of the surface depthdˆ wrt. the network parameters θ only involves calculating

To use automatic differentiation, we have to implement
the forward and backward pass for the surface depth preˆ In the following, we describe how
diction step θ → d.
both passes are implemented. For more details, we refer thereader to the supplementary material.
Forward Pass: As visualized in Fig. 3, we can determine dˆby finding the first occupancy change on the ray r. To detect
an occupancy change, we evaluate the occupancy networknfθ (·) at n equally-spaced samples on the ray {prayj }j=1 . Using a step size of ∆s, we can express the coordinates ofthese point in world-coordinates as
pray
j = r(j∆s + s0 )(9)where s0 determines the closest possible surface point. We
first find the smallest j for which fθ changes from free space(fθ < τ ) to occupied space (fθ ≥ τ ):rayj = argmin fθ (pray
)
≥
τ
>
f
(p
)
(10)
0
0
θ
j +1
j
j0

We obtain an approximation to the surface depth dˆ by applying the iterative secant method to the interval [j∆s +s0 , (j + 1)∆s + s0 ]. In practice, we compute the surface
depth for a batch of Np points in parallel. It is important tonote that we do not need to unroll the forward pass or store
any intermediate results as we exploit implicit differentiation to directly obtain the gradient of dˆ wrt. θ.
Backward Pass: The input to the backward pass is the graof the loss wrt. a single surface depth predicdient λ = ∂L
∂ dˆ
ˆtion. The output of the backward pass is λ ∂∂θd , which can becomputed using (8). In practice, however, we would like toimplement the backward pass not only for a single surface
depth dˆ but for a whole batch of depth values.
ˆ
We can implement this efficiently by rewriting λ ∂∂θd as−1∂fθ (p̂)∂fθ (p̂)µwith µ = −
·w
λ
(11)
∂θ
∂ p̂
Importantly, the left term in (11) corresponds to a normal
backward operation applied to the neural network fθ andthe right term in (11) is just an (element-wise) scalar multiplication for all elements in the batch. We can hence conveniently compute the backward pass of the operator θ → dˆby first multiplying the incoming gradient λ element-wisewith a factor and then backpropagating the result throughthe operator θ → fθ (p̂). Both operations can be efficiently
parallelized in common deep learning frameworks.

3.4. Training
During training, we assume that we are given N images{Ik }N
k=1 together with corresponding camera intrinsics, extrinsics, and object masks {Mk }N
k=1 . As our experiments
show, our method works with as little as one image per object. In addition, our method can also incorporate depth
information {Dk }N
k=1 , if available.
For training fθ and tθ , we randomly sample an image
Ik and Np points u on the image plane. We distinguish
the following three cases: First, let P0 denote the set ofpoints u that lie inside the object mask Mk and for which
the occupancy network predicts a finite surface depth dˆ as
described in Section 3.3. For these points we can define aloss Lrgb (θ) directly on the predicted image Îk . Moreover,let P1 denote the points u which lie outside the object mask
Mk . While we cannot define a photometric loss for thesepoints, we can define a loss Lfreespace (θ) that encourages the
network to remove spurious geometry along corresponding
rays. Finally, let P2 denote the set of points u which lie
inside the object mask Mk , but for which the occupancyˆ Again,network does not predict a finite surface depth d.
we cannot use a photometric loss for these points, but wecan define a loss Loccupancy (θ) that encourages the network
to produce a finite surface depth.
RGB Loss: For each point in P0 , we detect the predictedsurface depth dˆ as described in Section 3.3. We define aphoto-consistency loss for the points as
X
Lrgb (θ) =kξ(I)u − ξ(Î)u k(12)

where BCE is the binary cross entropy. When no surface
depth is predicted, we apply the freespace loss to a randomly sampled point on the ray.
Occupancy Loss: If a point u lies inside the object mask
but the predicted surface depth dˆ is infinite, the network
falsely predicts no surface points on ray r. To encourage
predicting occupied space on this ray, we uniformly sample
depth values drandom and define
Loccupancy (θ) =X

BCE(fθ (r(drandom )), 1)

(15)

u∈P2

In the single-view reconstruction experiments, we instead
use the first point on the ray which lies inside all objectmasks (depth of the visual hull). If we have additional depth
supervision, we use the ground truth depth for the occupancy loss. Intuitively, Loccupancy encourages the network to
occupy space along the respective rays which can then be
used by Lrgb in (12) and Ldepth in (13) to refine the initial
occupancy.
Normal Loss: Optionally, our representation allows us toincorporate a smoothness prior by regularizing surface normals. This is useful especially for real-world data as training with 2D or 2.5D supervision includes unconstrained areas where this prior enforces more natural shapes. We define this loss as
X
Lnormal (θ) =||n(pˆu ) − n(qu )||2
(16)
u∈P0

where n(·) denotes the normal vector, pˆu the predicted surface point and qu a randomly sampled neighbor of pˆu .3

3.5. Implementation Detailsu∈P0

where ξ(·) computes image features and k·k defines a robust
error metric. In practice, we use RGB-values and (optionally) image gradients as features and an `1 -loss for k · k.
Depth Loss: When the depth is also given, we can directly
incorporate an `1 loss on the predicted surface depth asX
ˆ1
Ldepth (θ) =|d − d|(13)
u∈P0

where d indicates the ground truth depth value of the sampled image point u and dˆ denotes the predicted surface
depth for pixel u.
Freespace Loss: If a point u lies outside the object mask
but the predicted surface depth dˆ is finite, the network
ˆ Therefore, we pefalsely predicts surface point p̂ = r(d).
nalize this occupancy withX
Lfreespace (θ) =BCE(fθ (p̂), 0)
(14)
u∈P1

We implement the combined network with 5 fullyconnected ResNet [26] blocks and ReLU activation. The
output dimension of the last layer is 4, one dimension forthe occupancy probability and three dimensions for the texture. For the single-view reconstruction experiments, we
encode the input image with an ResNet-18 [26] encoder
network gφ which outputs a 256-dimensional latent codez. To facilitate training, we start with a ray sampling accuracy of n = 16 which we iteratively increase to n = 128 bydoubling n after 50, 150, and 250 thousand iterations. We
choose the sampling interval [s0 , n∆s+s0 ] such that it covers the volume of interest for each object. We set τ = 0.5
for all experiments. We train on a single NVIDIA V100
GPU with a batch size of 64 images with 1024 random pixels each. We use the Adam optimizer [35] with learning rate
γ = 10−4 which we decrease by a factor of 5 after 750 and1000 epochs, respectively.
3 See supplementary for details.

4. Experiments

Input

SoftRas Ours (LRGB ) Pixel2Mesh Ours (LDepth )We conduct two different types of experiments to validate our approach. First, we investigate how well our approach reconstructs 3D shape and texture from a single
RGB image when trained on a large collection of RGB or
RGB-D images. Here, we consider both the case where wehave access to multi-view supervision and the case where
we use only a single RGB-D image per object during training. Next, we apply our approach to the challenging task ofmulti-view reconstruction, where the goal is to reconstructcomplex 3D objects from real-world multi-view imagery.

4.1. Single-View Reconstruction
First, we investigate to which degree our method can infer a 3D shape and texture representation from single-views.
We train a single model jointly on all categories.
Datasets: To adhere to community standards [13, 48, 80],
we use the Choy et al. [13] subset (13 classes) of the
ShapeNet dataset [10] for 2.5D and 3D supervised methods
with training, validation, and test splits from [48]. While
we use the renderings from Choy et al. [13] as input, we
additionally render 24 images of resolution 2562 with depth
maps and object masks per object which we use for supervision. We randomly sample the viewpoint on the northern hemisphere as well as the distance of the camera to theobject to get diverse supervision data. For 2D supervised
methods, we adhere to community standards [33, 44, 84]
and use the renderings and splits from [33]. Similar
to [13, 33, 48], we train with objects in canonical pose.
Baselines: We compare against the following methods
which all produce watertight meshes as output: 3DR2N2 [13] (voxel-based), Pixel2Mesh [80] (mesh-based),
and ONet [48] (implicit representation). We further compare against both the 2D and the 2.5D supervised version ofDifferentiable Ray Consistency (DRC) [79] (voxel-based)
and the 2D supervised Soft Rasterizer (SoftRas) [44] (meshbased). For 3D-R2N2, we use the pre-trained model from[48] which was shown to produce better results than theoriginal model from [13]. For the other baselines, we use
the pre-trained models4 from the authors.

Figure 4: Single-View Reconstruction. We show the input
renderings from [13] and the output of our 2D supervised(LRGB ) and 2.5D supervised (LDepth ) model, Soft Rasterizer [44] and Pixel2Mesh [80]. For 2D supervised methods
we use a corresponding view from [33] as input.
Input

Prediction

Input

Prediction

Figure 5: Single-View Reconstruction with Single-View
Supervision. While only trained with a single-view per object, our model predicts accurate 3D geometry and texture.

We first consider the case where we have access to multiview supervision with N = 24 images and corresponding
object masks. In addition, we also investigate the case whenground truth depth maps are given.

79], we compare directly wrt. to the ground truth shapemodels, not the voxelized or watertight versions.
In Table 1 and Fig. 4 we show quantitative and qualitative results for our method and various baselines. We
can see that our method is able to infer accurate 3D shapeand texture representations from single-view images when
only trained on multi-view images and object masks as
supervision signal. Quantitatively (Table 1), our method
performs best among the approaches with 2D supervisionand rivals the quality of methods with full 3D supervision.
When trained with depth, our method performs comparably to the methods which use full 3D information. Qualitatively (Fig. 4), we see that in contrast to the mesh-based
approaches, our method is not restricted to certain topologies. When trained with the photo-consistency loss LRGB ,we see that our approach is able to predict accurate texture
information in addition to the 3D shape.

Results: We evaluate the results using the Chamfer-L1 distance from [48]. In contrast to previous works [13, 44, 48,

4.1.2

4 Unfortunately, we cannot show texture results for DRC and SoftRas
as texture prediction is not part of the official code repositories.

The previous experiment indicates that our model is able
to infer accurate shape and texture information without 3D

4.1.1

Multi-View Supervision

Single-View Supervision

2D Supervision
DRC (Mask) [79] SoftRas [44]Ours (LRGB )2.5D Supervision
DRC (Depth) [79] Ours (LDepth )3D R2N2 [13]3D Supervision
ONet [48] Pixel2Mesh [80]category
airplanebenchcabinetcar
chairdisplay
lamp
loudspeaker
rifle
sofa
table
telephonevessel

0.659
0.340
0.660
-

0.149
0.241
0.231
0.221
0.338
0.284
0.381
0.320
0.155
0.407
0.374
0.131
0.233

0.190
0.210
0.220
0.196
0.264
0.255
0.413
0.289
0.175
0.224
0.280
0.148
0.245

0.377
0.316
0.510
-

0.143
0.165
0.183
0.179
0.226
0.246
0.362
0.295
0.143
0.221
0.180
0.130
0.206

0.215
0.210
0.246
0.250
0.282
0.323
0.566
0.333
0.199
0.264
0.247
0.221
0.248

0.151
0.171
0.189
0.181
0.224
0.275
0.380
0.290
0.160
0.217
0.185
0.155
0.220

0.183
0.191
0.194
0.154
0.259
0.231
0.309
0.284
0.151
0.211
0.215
0.145
0.201

mean

0.553

0.266

0.239

0.401

0.206

0.277

0.215

0.210

Table 1: Single-View Reconstruction. We report Chamfer-L1 distances wrt. the ground truth meshes for the single-viewexperiment. We compare against Differentiable Ray Consistency (DRC) [79] (2D and 2.5D supervision), Soft Rasterizer [44](2D supervision), 3D-R2N2 [13], Occupancy Networks (ONet) [48], and Pixel2Mesh [80] (all 3D supervision).
supervision. A natural question to ask is how many images
are required during training. To this end, we investigate
the case when only a single image with depth and camera
information is available. Since we represent the 3D shapein a canonical object coordinate system, the hypothesis isthat the model can aggregate the information over multiple
training instances, although it sees every object only fromone perspective. As the same image is used both as input
and supervision signal, we now condition on our renderings
instead of the ones provided by Choy et al. [13].
Results: Surprisingly, Fig. 5 shows that our method can infer appropriate 3D shape and texture when only a singleview is available per object, confirming our hypothesis.
Quantitatively, the Chamfer distance of the model trainedwith LRGB and LDepth with only a single view (0.410) is
comparable to the model trained with LDepth with 24 views(0.383). The reason for the numbers being worse thanin Section 4.1 is that for our renderings, we do not only
sample the viewpoint, but also the distance to the object resulting in a much harder task (see Fig. 5).

4.2. Multi-View Reconstruction
Finally, we investigate if our method is also applicable
to multi-view reconstruction in real-world scenarios. We
investigate two cases: First, when multi-view images andobject masks are given. Second, when additional sparse
depth maps are given which can be obtained from classic
multi-view stereo algorithms [67]. For this experiment, we
do not condition our model and train one model per object.
Dataset: We conduct this experiment on scans 65, 106, and
118 from the challenging real-world DTU dataset [1]. The
dataset contains 49 or 65 images with camera informationfor each object and baseline and structured light groundtruth data. The presented objects are challenging as theirappearance changes in different viewpoints due to specularities. Our sampling-based approach allows us to train on thefull image resolution of 1200 × 1600. We label the objectmasks ourselves and always remove the same images withprofound changes in lighting conditions, e.g., caused by theappearance of scanner parts in the background.
Baselines: We compare against classical approaches thathave 3D meshes as output. To this end, we run screened
Poisson surface reconstruction (sPSR) [34] on the outputof the classical MVS algorithms Campbell et al. [9], Furukawa et al. [18], Tola et al. [78], and Colmap [67]. We
find that the results on the DTU benchmark for the baselines are highly sensitive to the trim parameter of sPSR and
therefore report results for the trim parameters 0 (watertightoutput), 5 (good qualitative results) and 7 (good quantitative
results). For a fair comparison, we use the object masks toremove all points which lie outside the visual hull from thepredictions of the baselines before running sPSR.5 We use
the official DTU evaluation script in “surface mode”.
Results: We show qualitative and quantitative resultsin Fig. 6 and Table 2. Qualitatively, we find that our method
can be used for multi-view 3D reconstruction, directly resulting in watertight meshes. The ability to accurately
model cavities of the objects shows that our model uses texture information to improve over the visual hull (Fig. 7).
Quantitatively, Table 2 shows that our approach rivals theresults from highly tuned MVS algorithms. We note thatthe DTU ground truth is itself sparse (Fig. 7c) and methods
are therefore rewarded for trading off completeness for accuracy, which explains the better quantitative performanceof the baselines for higher trim parameters (Fig. 8).
5 See supplementary material for details.

(a) Shape(b) Normals(c) Texture

Figure 6: Multi-View Stereo. We show the shape, normals, and the textured shape for our method trained with 2D images
and sparse depth maps for scan 106 of the DTU dataset [1].

(a) Visual Hull [38](b) Ours (LRGB )(c) Ground Truth

Figure 7: Comparison against Visual Hull. We show thevisual hull, the shape prediction of our model trained withLRGB , and the ground truth for scan 118 of the DTU dataset.
Our method uses RGB cues to improve over the visual hulland predicts parts that are missing in the ground truth.

(a) Colmap 5(b) Colmap 7(c) Ours

Figure 8: Effect of Trim Parameter. We show screened
Poisson surface reconstructions [34] with trim parameters
5 and 7 for Colmap [67] and the prediction of our modeltrained with LRGB + LDepth for scan 106 of the DTU dataset.

5. Conclusion and Future WorkIn this work, we have presented Differentiable Volumetric Rendering (DVR). Observing that volumetric rendering
is inherently differentiable for implicit representations allows us to formulate an analytic expression for the gradientsof the depth with respect to the network parameters. Our
experiments show that DVR enables us to learn implicit 3D
shape representations from multi-view imagery without 3D

Trim Param.

Accuracy

CompletenessChamfer-L1

Tola [78] + sPSR
Furu [18] + sPSR
Colmap [67] + sPSRCamp [9] + sPSR

0
0
0
0

2.409
2.146
1.881
2.213

1.242
0.888
0.726
0.670

1.826
1.517
1.303
1.441

Tola [78] + sPSR
Furu [18] + sPSR
Colmap [67] + sPSRCamp [9] + sPSR

5
5
5
5

1.531
1.733
1.400
1.991

1.267
0.888
0.782
0.670

1.399
1.311
1.091
1.331

Tola [78] + sPSR
Furu [18] + sPSR
Colmap [67] + sPSRCamp [9] + sPSR

7
7
7
7

0.396
0.723
0.446
1.466

1.424
0.955
1.020
0.719

0.910
0.839
0.733
1.092

Ours (LRGB )Ours (LRGB + LDepth )

-

1.054
0.789

0.760
0.775

0.907
0.782

Table 2: Multi-View Stereo. We show quantitative results
for scans 65, 106, and 118 on the DTU dataset. For thebaselines, we perform screened Poisson surface reconstruction (sPSR) [34] with trim parameters 0, 5, and 7 to obtainthe final output. It shows that our generic method achieves
results comparable to the highly optimized MVS methods.

supervision, rivaling models that are learned with full 3D
supervision. Moreover, we found that our model can also
be used for multi-view 3D reconstruction. We believe that
DVR is a useful technique that broadens the scope of applications of implicit shape and texture representations.
In the future, we plan to investigate how to circumventthe need for object masks and camera information, e.g., by
predicting soft masks and how to estimate not only texture
but also more complex material properties.

Acknowledgments
This work was supported by an NVIDIA research gift.
The authors thank the International Max Planck Research
School for Intelligent Systems (IMPRS-IS) for supporting
Michael Niemeyer.

References[1] Henrik Aanæs, Rasmus R. Jensen, George Vogiatzis, Engin
Tola, and Anders B. Dahl. Large-scale data for multiple-view
stereopsis. International Journal of Computer Vision (IJCV),
120(2):153–168, 2016. 7, 8
[2] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, andLeonidas J. Guibas. Learning representations and generativemodels for 3D point clouds. In Proc. of the International
Conf. on Machine learning (ICML), 2018. 2[3] Matan Atzmon, Niv Haim, Lior Yariv, Ofer Israelov, HaggaiMaron, and Yaron Lipman. Controlling neural level sets. In
Advances in Neural Information Processing Systems (NIPS),
2019. 2, 4
[4] Bruce G. Baumgart. Geometric Modeling for Computer Vision. Stanford University, 1974. 1[5] Michael Bleyer, Christoph Rhemann, and Carsten Rother.
Patchmatch stereo - stereo matching with slanted support
windows. In Proc. of the British Machine Vision Conf.
(BMVC), 2011. 2[6] Jeremy S. De Bonet and Paul Viola. Poxels: Probabilistic
voxelized volume reconstruction. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 1999. 2[7] Adrian Broadhurst, Tom W. Drummond, and Roberto
Cipolla. A probabilistic framework for space carving. In
Proc. of the IEEE International Conf. on Computer Vision(ICCV), 2001. 2[8] André Brock, Theodore Lim, James M. Ritchie, and Nick
Weston. Generative and discriminative voxel modeling withconvolutional neural networks. arXiv.org, 1608.04236, 2016.
2[9] Neill D.F. Campbell, George Vogiatzis, Carlos Hernández,
and Roberto Cipolla. Using multiple hypotheses to improve
depth-maps for multi-view stereo. In Proc. of the European
Conf. on Computer Vision (ECCV), 2008. 7, 8
[10] Angel X. Chang, Thomas A. Funkhouser, Leonidas J.
Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio
Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong
Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich
3d model repository. arXiv.org, 1512.03012, 2015. 6[11] Wenzheng Chen, Huan Ling, Jun Gao, Edward Smith, Jaako
Lehtinen, Alec Jacobson, and Sanja Fidler. Learning to predict 3d objects with an interpolation-based differentiable renderer. In Advances in Neural Information Processing Systems (NIPS), 2019. 1, 2
[12] Zhiqin Chen and Hao Zhang. Learning implicit fields forgenerative shape modeling. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019. 1, 2,
3[13] Christopher B. Choy, Danfei Xu, JunYoung Gwak, Kevin
Chen, and Silvio Savarese. 3d-r2n2: A unified approach forsingle and multi-view 3d object reconstruction. In Proc. of
the European Conf. on Computer Vision (ECCV), 2016. 1, 2,
6, 7[14] Brian Curless and Marc Levoy. A volumetric method forbuilding complex models from range images. In ACM Trans.
on Graphics, 1996. 2[15] Valentin Deschaintre, Miika Aittala, Frédo Durand, George
Drettakis, and Adrien Bousseau. Single-image SVBRDF
capture with a rendering-aware deep network. In ACM Trans.
on Graphics, 2018. 1[16] Simon Donné and Andreas Geiger. Learning non-volumetric
depth fusion using successive reprojections. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2019. 2[17] Haoqiang Fan, Hao Su, and Leonidas J. Guibas. A point
set generation network for 3d object reconstruction from asingle image. Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2017. 1, 2
[18] Yasutaka Furukawa and Jean Ponce. Accurate, dense, androbust multiview stereopsis. IEEE Trans. on Pattern Analysis
and Machine Intelligence (PAMI), 32(8):1362–1376, 2010.
7, 8[19] Matheus Gadelha, Subhransu Maji, and Rui Wang. 3d shapeinduction from 2d views of multiple objects. In Proc. of theInternational Conf. on 3D Vision (3DV), 2017. 2[20] Silvano Galliani, Katrin Lasinger, and Konrad Schindler.
Gipuma: Massively parallel multi-view stereo reconstruction. Publikationen der Deutschen Gesellschaft für Photogrammetrie, Fernerkundung und Geoinformation e. V,
25:361–369, 2016. 2[21] Kyle Genova, Forrester Cole, Aaron Maschinot, Aaron
Sarna, Daniel Vlasic, and William T. Freeman. Unsupervised
training for 3d morphable model regression. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2018. 1, 2
[22] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna,
William T. Freeman, and Thomas Funkhouser. Learning
shape templates with structured implicit functions. In Proc.
of the IEEE International Conf. on Computer Vision (ICCV),
2019. 2[23] Ben Graham. Sparse 3d convolutional neural networks. In
Proc. of the British Machine Vision Conf. (BMVC), 2015. 2[24] Thibault Groueix, Matthew Fisher, Vladimir G. Kim,
Bryan C. Russell, and Mathieu Aubry. AtlasNet: A papiermâché approach to learning 3d surface generation. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), 2018. 1, 2
[25] Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press,
2003. 2[26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2016. 5[27] Paul Henderson and Vittorio Ferrari. Learning to generate
and reconstruct 3d meshes with only 2d supervision. In Proc.
of the British Machine Vision Conf. (BMVC), 2018. 2[28] Paul Henderson and Vittorio Ferrari. Learning single-image
3d reconstruction by generative modelling of shape, pose andshading. International Journal of Computer Vision (IJCV),2019. 2[29] Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra
Ahuja, and Jia-Bin Huang. Deepmvs: Learning multi-view

stereopsis. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018. 2[30] Zeng Huang, Tianye Li, Weikai Chen, Yajie Zhao, Jun Xing,
Chloe LeGendre, Linjie Luo, Chongyang Ma, and Hao Li.
Deep volumetric video from very sparse multi-view performance capture. In Proc. of the European Conf. on Computer
Vision (ECCV), 2018. 2[31] Li Jiang, Shaoshuai Shi, Xiaojuan Qi, and Jiaya Jia. GAL:
geometric adversarial loss for single-view 3d-object reconstruction. In Proc. of the European Conf. on Computer Vision(ECCV), 2018. 2[32] Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros, andJitendra Malik. Learning category-specific mesh reconstruction from image collections. In Proc. of the European Conf.
on Computer Vision (ECCV), 2018. 2[33] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018. 1, 2, 6[34] Michael M. Kazhdan and Hugues Hoppe. Screened poisson
surface reconstruction. ACM Trans. on Graphics, 32(3):29,
2013. 7, 8[35] Diederik P. Kingma and Jimmy Ba. Adam: A method forstochastic optimization. In Proc. of the International Conf.
on Learning Representations (ICLR), 2015. 5[36] Abhijit Kundu, Yin Li, and James M. Rehg. 3d-rcnn:
Instance-level 3d object reconstruction via render-andcompare. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018. 1[37] Kiriakos N. Kutulakos and Steven M. Seitz. A theory ofshape by space carving. International Journal of Computer
Vision (IJCV), 38(3):199–218, 2000. 2[38] Aldo Laurentini. The visual hull concept for silhouette-based
image understanding. IEEE Trans. on Pattern Analysis andMachine Intelligence (PAMI), 16(2):150–162, 1994. 8[39] Vincent Leroy, Jean-Sébastien Franco, and Edmond Boyer.
Shape reconstruction using volume sweeping and learned
photoconsistency. In Proc. of the European Conf. on Computer Vision (ECCV), 2018. 2[40] Kejie Li, Trung Pham, Huangying Zhan, and Ian D. Reid.
Efficient dense point cloud object reconstruction using deformation vector fields. In Proc. of the European Conf. on
Computer Vision (ECCV), 2018. 2[41] Yiyi Liao, Simon Donné, and Andreas Geiger. Deep marching cubes: Learning explicit surface representations. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), 2018. 1, 2[42] Yiyi Liao, Katja Schwarz, Lars Mescheder, and Andreas
Geiger. Towards unsupervised learning of generative models
for 3d controllable image synthesis. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2020. 2[43] Guilin Liu, Duygu Ceylan, Ersin Yumer, Jimei Yang, andJyh-Ming Lien. Material editing using a physically based
rendering network. In Proc. of the IEEE International Conf.
on Computer Vision (ICCV), 2017. 1[44] Shichen Liu, Weikai Chen, Tianye Li, and Hao Li. Softrasterizer: Differentiable rendering for unsupervised singleview mesh reconstruction. In Proc. of the IEEE International
Conf. on Computer Vision (ICCV), 2019. 1, 2, 6, 7

[45] Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li.
Learning to infer implicit surfaces without 3d supervision. In
Advances in Neural Information Processing Systems (NIPS),
2019. 2[46] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel
Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images.
In ACM Trans. on Graphics, 2019. 2[47] Matthew M. Loper and Michael J. Black. Opendr: An approximate differentiable renderer. In Proc. of the European
Conf. on Computer Vision (ECCV), 2014. 1, 2
[48] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks:
Learning 3d reconstruction in function space. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2019. 1, 2, 3, 6, 7
[49] Mateusz Michalkiewicz, Jhony K. Pontes, Dominic Jack,
Mahsa Baktashmotlagh, and Anders Eriksson. Implicit surface representations as layers in neural networks. In Proc.
of the IEEE International Conf. on Computer Vision (ICCV),
2019. 1, 2[50] Thu Nguyen-Phuoc, Chuan Li, Stephen Balaban, and YongLiang Yang. Rendernet: A deep convolutional network for
differentiable rendering from 3d shapes. In Advances in Neural Information Processing Systems (NIPS), 2018. 1, 2[51] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, ChristianRichardt, and Yong-Liang Yang. Hologan: Unsupervised
learning of 3d representations from natural images. In Proc.
of the IEEE International Conf. on Computer Vision (ICCV),
2019. 2[52] Thu Nguyen-Phuoc, Christian Richardt, Long Mai, YongLiang Yang, and Niloy Mitra. Blockgan: Learning 3d
object-aware scene representations from unlabelled images.
arXiv.org, 2020. 2[53] Michael Niemeyer, Lars Mescheder, Michael Oechsle, andAndreas Geiger. Occupancy flow: 4d reconstruction bylearning particle dynamics. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019. 2[54] Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo
Strauss, and Andreas Geiger. Texture fields: Learning texture representations in function space. In Proc. of the IEEE
International Conf. on Computer Vision (ICCV), 2019. 1, 2,
3[55] Junyi Pan, Xiaoguang Han, Weikai Chen, Jiapeng Tang, andKui Jia. Deep mesh reconstruction from single rgb images
via topology modification networks. In Proc. of the IEEE
International Conf. on Computer Vision (ICCV), 2019. 2[56] Jeong J. Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019. 1, 2, 3
[57] Despoina Paschalidou, Ali Osman Ulusoy, and Andreas
Geiger. Superquadrics revisited: Learning 3d shape parsing
beyond cuboids. In Proc. IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), 2019. 2

[58] Despoina Paschalidou, Ali Osman Ulusoy, Carolin Schmitt,
Luc van Gool, and Andreas Geiger. Raynet: Learning volumetric 3d reconstruction with ray potentials. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2018. 1, 2, 4[59] Felix Petersen, Amit H. Bermano, Oliver Deussen, andDaniel Cohen-Or. Pix2vex: Image-to-geometry reconstruction using a smooth differentiable renderer. arXiv.org, 2019.
1[60] Andrew Prock and Chuck Dyer. Towards real-time voxel coloring. In Proceedings of the DARPA Image Understanding
Workshop, 1998. 2[61] Danilo Jimenez Rezende, S. M. Ali Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas Heess.
Unsupervised learning of 3d structure from images. In Advances in Neural Information Processing Systems (NIPS),
2016. 2[62] Elad Richardson, Matan Sela, Roy Or-El, and Ron Kimmel.
Learning detailed face reconstruction from a single image.
In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017. 1[63] Gernot Riegler, Ali Osman Ulusoy, Horst Bischof, and Andreas Geiger. OctNetFusion: Learning depth fusion from
data. In Proc. of the International Conf. on 3D Vision (3DV),2017. 2[64] Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger.
Octnet: Learning deep 3d representations at high resolutions.
In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017. 1, 2
[65] Walter Rudin et al. Principles of mathematical analysis, volume 3. McGraw-hill New York, 1964. 4[66] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned
implicit function for high-resolution clothed human digitization. Proc. of the IEEE International Conf. on Computer
Vision (ICCV), 2019. 1, 2, 3
[67] Johannes L. Schönberger and Jan-Michael Frahm. Structurefrom-motion revisited. In Proc. IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR), 2016. 7, 8
[68] Johannes L. Schönberger, Enliang Zheng, Marc Pollefeys,
and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In Proc. of the European Conf.
on Computer Vision (ECCV), 2016. 2[69] Steven M. Seitz, Brian Curless, James Diebel, Daniel
Scharstein, and Richard Szeliski. A comparison and evaluation of multi-view stereo reconstruction algorithms. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2006. 2[70] Steven M. Seitz and Charles R. Dyer. Photorealistic scenereconstruction by voxel coloring. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 1997. 2[71] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias
Nießner, Gordon Wetzstein, and Michael Zollhöfer. Deepvoxels: Learning persistent 3d feature embeddings. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), pages 2437–2446, 2019. 2[72] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3dstructure-aware neural scene representations. In Advancesin Neural Information Processing Systems (NIPS), 2019. 2[73] David Stutz and Andreas Geiger. Learning 3d shape completion from laser scan data with weak supervision. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), 2018. 2[74] Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox.
Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs. In Proc. of the IEEE
International Conf. on Computer Vision (ICCV), 2017. 2[75] Ayush Tewari, Michael Zollhöfer, Pablo Garrido, Florian
Bernard, Hyeongwoo Kim, Patrick Pérez, and Christian
Theobalt. Self-supervised multi-level face model learning
for monocular reconstruction at over 250 hz. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2018. 1[76] Ayush Tewari, Michael Zollhöfer, Hyeongwoo Kim, Pablo
Garrido, Florian Bernard, Patrick Pérez, and Christian
Theobalt. Mofa: Model-based deep convolutional face autoencoder for unsupervised monocular reconstruction. In
Proc. of the IEEE International Conf. on Computer Vision(ICCV), 2017. 1[77] Hugues Thomas, Charles R. Qi, Jean-Emmanuel Deschaud,
Beatriz Marcotegui, François Goulette, and Leonidas J.
Guibas. Kpconv: Flexible and deformable convolution forpoint clouds. In Proc. of the IEEE International Conf. on
Computer Vision (ICCV), 2019. 2[78] Engin Tola, Christoph Strecha, and Pascal Fua. Efficientlarge-scale multi-view stereo for ultra high-resolution image
sets. Machine Vision and Applications (MVA), 23(5):903–
920, 2012. 7, 8[79] Shubham Tulsiani, Tinghui Zhou, Alexei A. Efros, and Jitendra Malik. Multi-view supervision for single-view reconstruction via differentiable ray consistency. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2017. 1, 2, 4, 6, 7[80] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei
Liu, and Yu-Gang Jiang. Pixel2Mesh: Generating 3D meshmodels from single RGB images. In Proc. of the European
Conf. on Computer Vision (ECCV), 2018. 1, 2, 6, 7
[81] Weiyue Wang, Xu Qiangeng, Duygu Ceylan, RadomirMech, and Ulrich Neumann. DISN: Deep implicit surface
network for high-quality single-view 3d reconstruction. Advances in Neural Information Processing Systems (NIPS),
2019. 2[82] Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, andJosh Tenenbaum. Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. In Advances in Neural Information Processing Systems (NIPS),
2016. 2[83] Haozhe Xie, Hongxun Yao, Xiaoshuai Sun, Shangchen
Zhou, and Shengping Zhang. Pix2vox: Context-aware 3dreconstruction from single and multi-view images. In Proc.
of the IEEE International Conf. on Computer Vision (ICCV),
2019. 2

[84] Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, andHonglak Lee. Perspective transformer nets: Learning singleview 3d object reconstruction without 3d supervision. In
Advances in Neural Information Processing Systems (NIPS),
2016. 6[85] Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge
Belongie, and Bharath Hariharan. Pointflow: 3d point cloud
generation with continuous normalizing flows. In Proc. of
the IEEE International Conf. on Computer Vision (ICCV),
2019. 2[86] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan.
Mvsnet: Depth inference for unstructured multi-view stereo.
In Proc. of the European Conf. on Computer Vision (ECCV),
2018. 2[87] Yao Yao, Zixin Luo, Shiwei Li, Tianwei Shen, Tian Fang,and Long Quan. Recurrent mvsnet for high-resolution multiview stereo depth inference. Proc. IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR), 2019. 2[88] Jacek Zienkiewicz, Andrew J. Davison, and Stefan
Leutenegger. Real-time height map fusion using differentiable rendering. In Proc. IEEE International Conf. on Intelligent Robots and Systems (IROS), 2016. 1, 2