RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials
Despoina Paschalidou1,5 Ali Osman Ulusoy2 Carolin Schmitt1 Luc van Gool3 Andreas Geiger1,41Autonomous Vision Group, MPI for Intelligent Systems and University of Tübingen2
Microsoft 3 Computer Vision Lab, ETH Zürich & KU Leuven
4
Computer Vision and Geometry Group, ETH Zürich
5
Max Planck ETH Center for Learning Systems{firstname.lastname}@tue.mpg.devangool@vision.ee.ethz.ch

Abstract
In this paper, we consider the problem of reconstructing a dense 3D model using images captured from different views. Recent methods based on convolutional neural networks (CNN) allow learning the entire task from
data. However, they do not incorporate the physics ofimage formation such as perspective geometry and occlusion. Instead, classical approaches based on Markov Random Fields (MRF) with ray-potentials explicitly model these
physical processes, but they cannot cope with large surface
appearance variations across different viewpoints. In thispaper, we propose RayNet, which combines the strengths
of both frameworks. RayNet integrates a CNN that learns
view-invariant feature representations with an MRF that explicitly encodes the physics of perspective projection andocclusion. We train RayNet end-to-end using empirical risk
minimization. We thoroughly evaluate our approach on
challenging real-world datasets and demonstrate its benefits over a piece-wise trained baseline, hand-crafted models
as well as other learning-based approaches.

(a) Input Image(b) Ground-truth(c) Ulusoy et al. [35](d) Hartmann et al. [14](e) RayNet

1. Introduction
Passive 3D reconstruction is the task of estimating a 3Dmodel from a collection of 2D images taken from different
viewpoints. This is a highly ill-posed problem due to large
ambiguities introduced by occlusions and surface appearance variations across different views.
Several recent works have approached this problem byformulating the task as inference in a Markov random field(MRF) with high-order ray potentials that explicitly model
the physics of the image formation process along each viewing ray [19, 33, 35]. The ray potential encourages consistency between the pixel recorded by the camera and thecolor of the first visible surface along the ray. By accumulating these constrains from each input camera ray, these

Figure 1: Multi-View 3D Reconstruction. By combining
representation learning with explicit physical constraints
about perspective geometry and multi-view occlusion relationships, our approach (e) produces more accurate results
than entirely model-based (c) or learning-based methods
that ignore such physical constraints (d).
approaches estimate a 3D model that is globally consistent1

in terms of occlusion relationships.
While this formulation correctly models occlusion, thecomplex nature of inference in ray potentials restricts thesemodels to pixel-wise color comparisons, which leads tolarge ambiguities in the reconstruction [35]. Instead of using images as input, Savinov et al. [28] utilize pre-computed
depth maps using zero-mean normalized cross-correlation
in a small image neighborhood. In this case, the ray potentials encourage consistency between the input depth map
and the depth of the first visible voxel along the ray. Whileconsidering a large image neighborhood improves uponpixel-wise comparisons, our experiments show that suchhand-crafted image similarity measures cannot handle complex variations of surface appearance.
In contrast, recent learning-based solutions to motion
estimation [10, 15, 24], stereo matching [20, 21, 42] and3D reconstruction [5, 6, 9, 16, 37] have demonstrated impressive results by learning feature representations that aremuch more robust to local viewpoint and lighting changes.
However, existing methods exploit neither the physical constraints of perspective geometry nor the resulting occlusion relationships across viewpoints, and therefore requirea large model capacity as well as an enormous amount oflabelled training data.
This work aims at combining the benefits of a learningbased approach with the strengths of a model that incorporates the physical process of perspective projection andocclusion relationships. Towards this goal, we propose an
end-to-end trainable architecture called RayNet which integrates a convolutional neural network (CNN) that learns
surface appearance variations (e.g. across different viewpoints and lighting conditions) with an MRF that explicitly encodes the physics of perspective projection and occlusion. More specifically, RayNet uses a learned feature
representation that is correlated with nearby images to estimate surface probabilities along each ray of the input image set. These surface probabilities are then fused using an
MRF with high-order ray potentials that aggregates occlusion constraints across all viewpoints. RayNet is learned
end-to-end using empirical risk minimization. In particular,
errors are backpropagated to the CNN based on the outputof the MRF. This allows the CNN to specialize its representation to the joint task while explicitly considering the 3D
fusion process.
Unfortunately, naı̈ve backpropagation through the unrolled MRF is intractable due to the large number of messages that need to be stored during training. We propose
a stochastic ray sampling approach which allows efficient
backpropagation of errors to the CNN. We show that the
MRF acts as an effective regularizer and improves both theoutput of the CNN as well as the output of the joint model
for challenging real-world reconstruction problems. Compared to existing MRF-based [35] or learning-based meth-ods [14, 16], RayNet improves the accuracy of the 3D reconstruction by taking into consideration both local information around every pixel (via the CNN) as well as global
information about the entire scene (via the MRF).
Our code and data is available on the project website1 .

2. Related Work
3D reconstruction methods can be roughly categorized
into model-based and learning-based approaches, whichlearn the task from data. As a thorough survey on 3D
reconstruction techniques is beyond the scope of this paper, we discuss only the most related approaches and refer
to [7, 13, 29] for a more thorough review.
Ray-based 3D Reconstruction: Pollard and Mundy [23]propose a volumetric reconstruction method that updates
the occupancy and color of each voxel sequentially for every image. However, their method lacks a global probabilistic formulation. To address this limitation, a number
of approaches have phrased 3D reconstruction as inference
in a Markov random field (MRF) by exploiting the special
characteristics of high-order ray potentials [19, 28, 33, 35].
Ray potentials allow for accurately describing the image
formation process, yielding 3D reconstructions consistent
with the input images. Recently, Ulusoy et al. [34] integrated scene specific 3D shape knowledge to further improve the quality of the 3D reconstructions. A drawback ofthese techniques is that very simplistic photometric terms
are needed to keep inference tractable, e.g., pixel-wise color
consistency, limiting their performance.
In this work, we integrate such a ray-based MRF with a
CNN that learns multi-view patch similarity. This results inan end-to-end trainable model that is more robust to appearance changes due to viewpoint variations, while tightly integrating perspective geometry and occlusion relationships
across viewpoints.
Learning-based 3D Reconstruction: The development oflarge 3D shape databases [4, 38] has fostered the development of learning based solutions [3, 5, 8, 37] to 3D reconstruction. Choy et al. [5] propose a unified framework forsingle and multi-view reconstruction by using a 3D recurrent neural network (3D-R2N2) based on long-short-term
memory (LSTM). Girdhar et al. [8] propose a network thatembeds image and shape together for single view 3D volumetric shape generation. Hierarchical space partitioning
structures (e.g., octrees) have been proposed to increase theoutput resolution beyond 323 voxels [12, 27, 31].
As most of the aforementioned methods solve the 3D
reconstruction problem via recognizing the scene content,
they are only applicable to object reconstruction and do not
generalize well to novel object categories or full scenes.
1 https://avg.is.tue.mpg.de/research projects/raynet

Vi
ew
ce
nt

Expected
Loss(

SharedSurface
Probabilites

F

(

Ad
ja

D

Ground Truth
Depth Map

2D CNN(

Shared

tV
iew

Re
fe
re
n

ce

Vi
ew

2D CNND

Reference
Camera

Ad
ja

ce
n

Depth
Distribution

FeatureMap

2D CNN
Unrolled Message Passing
Multi-View CNN (Section 3.1)

Depth Prediction & Loss

Markov Random Field (Section 3.2)(a) RayNet Architecture in Plate Notation with Plates denoting Copies(b) Multi-View Projection

Figure 2: RayNet. Given a reference view and its adjacent views, we extract features via a 2D CNN (blue). Features
corresponding to the projection of the same voxel along ray r (see (b) for an illustration) are aggregated via the average innerproduct into per pixel depth distributions. The average runs over all pairs of views and σ(·) denotes the softmax operator.
The depth distributions for all rays (i.e., all pixels in all views) are passed to the unrolled MRF. The final depth predictions
dr are passed to the loss function. The forward pass is illustrated in green. The backpropagation pass is highlighted in red.
C = {Ci } is the set of all cameras, W × H are the image dimensions and D is the max. number of voxels along each ray.
Towards a more general learning based model for 3D reconstruction, [16, 17] propose to unproject the input images
into 3D voxel space and process the concatenated unprojected volumes using a 3D convolutional neural network.
While these approaches take projective geometry into account, they do not explicitly exploit occlusion relationships
across viewpoints, as proposed in this paper. Instead, they
rely on a generic 3D CNN to learn this mapping from data.
We compare to [16] in our experimental evaluation and obtain more accurate 3D reconstructions and significantly better runtime performance. In addition, the lightweight natureof our model’s forward inference allows for reconstructing
scenes up to 2563 voxels resolution in a single pass. In contrast, [17] is limited to 323 voxels and [16] requires processing large volumes using a sliding window, thereby losing
global spatial relationships.
A major limitation of all aforementioned approaches isthat they require full 3D supervision for training, which isquite restrictive. Tulsiani et al. [32] relax these assumptions
by formulating a differentiable view consistency loss thatmeasures the inconsistency between the predicted 3D shapeand its observation. Similarly, Rezende et al. [26] propose
a neural projection layer and a black box renderer for supervising the learning process. Yan et al. [39] and Gwak
et al. [9] use 2D silhouettes as supervision for 3D reconstruction from a single image. While all these methods exploit ray constraints inside the loss function, our goal is to
directly integrate the physical properties of the image formation process into the model via unrolled MRF inferencewith ray potentials. Thus, we are able to significantly reduce
the number of parameters in the network and our network
does not need to acquire these first principles from data.

3. ModelThe input to our approach is a set of images and theircorresponding camera poses, which are obtained using
structure-from-motion [36]. Our goal is to model the known
physical processes of perspective projection and occlusion,
while learning the parameters that are difficult to model,e.g., those describing surface appearance variations acrossdifferent views. Our architecture utilizes a CNN to learna feature representation for image patches that are compared across nearby views to estimate a depth distribution
for each ray in the input images. For all our experiments,
we use 4 nearby views. Due to the small size of each image
patch, these distributions are typically noisy. We pass thesenoisy distributions to our MRF which aggregates them intoa occlusion-consistent 3D reconstruction. We formulate inference in the MRF as a differentiable function, hence allowing end-to-end training using backpropagation.
We first specify our CNN architecture which predicts
depth distributions for each input pixel/ray. We then detailthe MRF for fusing these noisy measurements. Finally, we
discuss appropriate loss functions and show how our model
can be efficiently trained using stochastic ray sampling. The
overall architecture is illustrated in Fig. 2.

3.1. Multi-View CNN
CNNs have been proven successful for learning similarity measures between two or more image patches forstereo [20, 40, 41] as well as multi-view stereo [14]. Similar to these works, we design a network architecture that
estimates a depth distribution for every pixel in each view.
The network used is illustrated in Fig. 2a (left). The

network first extracts a 32-dimensional feature per pixelin each input image using a fully convolutional network.
The weights of this network are shared across all images.
Each layer comprises convolution, spatial batch normalization and a ReLU non-linearity. We follow common practiceand remove the ReLU from the last layer in order to retain information encoded both in the negative and positive
range [20].
We then use these features to calculate per ray depth distributions for all input images. More formally, let X denotea voxel grid which discretizes the 3D space. Let R denotethe complete set of rays in the input views. In particular,
we assume one ray per image pixel. Thus, the cardinalityof R equals the number of input images times the number of pixels per image. For every voxel i along each rayr ∈ R, we compute the surface probability sri by projecting
the voxel center location into the reference view (i.e., theimage which comprises pixel r) and into all adjacent views(4 views in our experiments) as illustrated in Fig. 2b. For
clarity, Fig. 2b shows only 2 out of 4 adjacent cameras considered in our experimental evaluation. We obtain sri as theaverage inner product between all pairs of views. Note that
sri is high if all views agree in terms of the learned feature
representation. Thus, sri reflects the probability of a surface being located at voxel i along ray r. We abbreviate thesurface probabilities for all rays of a single image with S.

Due to the local nature of the extracted features as wellas occlusions, the depth distributions computed by the CNN
are typically noisy. Our MRF aggregates these depth distributions by exploiting occlusion relationships across all
viewpoints, yielding significantly improved depth predictions. Occlusion constraints are encoded using high-order
ray potentials [19, 28, 35]. We differ from [19, 35] in that
we do not reason about surface appearance within the MRF
but instead incorporate depth distributions estimated by our
CNN. This allows for a more accurate depth signal and also
avoids the costly sampling-based discrete-continuous inference in [35]. We compare against [35] in our experimental
evaluation and demonstrate improved performance.
We associate each voxel i ∈ X with a binary occupancy
variable oi ∈ {0, 1}, indicating whether the voxel is occupied (oi = 1) or free (oi = 0). For a single ray r ∈ R, letor = (or1 , or2 , . . . , orNr ) denote the ordered set of occupancy
variables associated with the voxels which intersect ray r.
The order is defined by the distance to the camera.
The joint distribution over all occupancy variables o ={oi |i ∈ X } factorizes into unary and ray potentials
Y
1 Y
ϕi (oi )ψr (or )| {z }| {z }
Z
i∈X

unary

where Z denotes the partition function. The corresponding
factor graph is illustrated in Fig. 3.
Unary Potentials: The unary potentials encode our priorbelief about the state of the occupancy variables. We model
ϕi (oi ) using a Bernoulli distribution
ϕi (oi ) = γ oi (1 − γ)1−oi

3.2. Markov Random Fieldp(o) =

Figure 3: Factor Graph of the MRF. We illustrate a 2D
slice through the 3D volume and two cameras with one ray
each: r, r0 ∈ R. Unary factors ϕi (oi ) are defined for every voxel i ∈ X of the voxel grid. Ray factors ψr (or ) aredefined for every ray r ∈ R and connect to the variables
along the ray or = (or1 , . . . , orNr ). dri denotes the Euclidean
distance of the ith voxel along ray r to its corresponding
camera center.

r∈R

ray

(1)

(2)

where γ is the probability that the ith voxel is occupied.
Ray Potentials: The ray potentials encourage the predicted depth at pixel/ray r to coincide with the first occupiedvoxel along the ray. More formally, we haveψr (or ) =

Nr
X
i=1

ori

Y

(1 − orj )sri

(3)

j<i

where sri is the probability that the visible surface along
ray r is located at voxel i. This probability is predicted by
the neural network described in Section 3.1. Note that the
product over the occupancy variables in Eq. (3) equals 1 if
and only if i is the first occupied voxel (i.e., if ori = 1 andorj = 0 for j < i). Thus ψr (·) is large if the surface ispredicted at the first occupied voxel in the model.
Inference: Having specified our MRF, we detail our inference approach in this model. Provided noisy depth measurements from the CNN (sri ), the goal of the inference
algorithm is to aggregate these measurements using raypotentials into a 3D reconstruction and to estimate globallyconsistent depth distributions at every pixel.
Let dri denote the distance of the ith voxel along ray r to
the respective camera center as illustrated in Fig. 3. Let further dr ∈ {dr1 , . . . , drNr } be a random variable representingthe depth along ray r. In other words, dr = dri denotes thatthe depth at ray r is the distance to the ith voxel along rayr. We associate the occupancy and depth variables along a
ray using the following equation:
dr =

Nr
X

ori

i=1

Y

(1 − ori )dri

(4)

j<i

Our inference procedure estimates a probability distributionp(dr = dri ) for each pixel/ray r in the input views. Unfortunately computing the exact solution in a loopy high-ordermodel such as our MRF is NP-hard [22]. We use loopy
sum-product belief propagation for approximate inference.
As demonstrated by Ulusoy et al. [35], belief propagationin models involving high-order ray potentials is tractable as
the factor-to-variable messages can be computed in linear
time due to the special structure of the ray potentials. In
practice, we run a fixed number of iterations interleaving
factor-to-variable and variable-to-factor message updates.
We observed that convergence typically occurs after 3 iterations and thus fix the iteration number to 3 when unrollingthe message passing algorithm. We refer to the supplementary material for message equation derivations.

3.3. Loss Function
We utilize empirical risk minimization to train RayNet.
Let ∆(dr , d?r ) denote a loss function which measures thediscrepancy between the predicted depth dr and the groundtruth depth d?r at pixel r. The most commonly used metricfor evaluating depth maps is the absolute depth error. We
therefore use the `1 loss ∆(dr , d?r ) = |dr − d?r | to train
RayNet. In particular, we seek to minimize the expected
loss, also referred to as the empirical risk R(θ):R(θ) =X

Ep(dr ) ∆(dr , d?r )

(5)

r∈R?

=

Nr
X X

p(dr = dri ) ∆(dri , d?r )r∈R? i=1

with respect to the model parameters θ. Here, R? denotes
the set of ground truth pixels/rays in all images of all training scenes and p(dr ) is the depth distribution predicted by
the model for ray r of the respective image in the respective
training scene. The parameters θ comprises the occupancy
prior γ as well as parameters of the neural network.

3.4. Training
In order to train RayNet in an end-to-end fashion, we
need to backpropagate the gradients through the unrolled
MRF message passing algorithm to the CNN. However,a naı̈ve implementation of backpropagation is not feasible
due to memory limitations. In particular, backpropagation

requires storing all intermediate messages from all beliefpropagation iterations in memory. For a modest dataset of≈ 50 images with 360 × 640 pixel resolution and a voxel
grid of size 2563 , this would require > 100GB GPU memory, which is intractable using current hardware.
To tackle this problem, we perform backpropagation using mini-batches where each mini-batch is a stochasticallysampled subset of the input rays. In particular, each minibatch consists of 2000 rays randomly sampled from a subset
of 10 consecutive input images. Our experiments show thatlearning with rays from neighboring views leads to fasterconvergence, as the network can focus on small portion of
the scene at a time. After backpropagation, we update themodel parameters and randomly select a new set of rays forthe next iteration. This approximates the true gradient of themini-batch. The gradients are obtained using TensorFlow’s
AutoDiff functionality [2].
While training RayNet in an end-to-end fashion is feasible, we further speed it up by pretraining the CNN followedby fine-tuning the entire RayNet architecture. For pretraining, we randomly pick a set of pixels from a randomly chosen reference view for each mini-batch. We discretize theray corresponding to each pixel according to the voxel gridand project all intersected voxel centers into the adjacentviews as illustrated in Fig. 2b. For backpropagation we use
the same loss function as during end-to-end training, cf. Eq.
(5).

4. Experimental Evaluation
In this Section, we present experiments evaluating our
method on two challenging datasets. The first dataset consists of two scenes, BARUS&HOLLEY and DOWNTOWN, both of which were captured in urban environments from an aerial platform. The images, camera posesand LIDAR point cloud are provided by Restrepo et al. [25].
Ulusoy et al. triangulated the LIDAR point cloud to achievea dense ground truth mesh [35]. In total the dataset consists of roughly 200 views with an image resolution of1280 × 720 pixels. We use the BARUS&HOLLEY scene
as the training set and reserve DOWNTOWN for testing.
Our second dataset is the widely used DTU multi-view
stereo benchmark [1], which comprises 124 indoor scenesof various objects captured from 49 camera views underseven different lighting conditions. We evaluate RayNet
on two objects from this dataset: BIRD and BUDDHA.
For all datasets, we down-sample the images such that thelargest dimension is 640 pixels.
We compare RayNet to various baselines both qualitatively and quantitatively. For the quantitative evaluation, we
compute accuracy, completeness and per pixel mean depth
error, and report the mean and the median for each. The
first two metrics are estimated in 3D space, while the latter is defined in image space and averaged over all groundtruth depth maps. In addition, we also report the Chamferdistance, which is a metric that expresses jointly the accuracy and the completeness. Accuracy is measured as the
distance from a point in the reconstructed point cloud toits closest neighbor in the ground truth point cloud, while
completeness is calculated as the distance from a point in
the ground truth point cloud to its closest neighbor in thepredicted point cloud. For additional information on these
metrics we refer the reader to [1]. We generate the point
clouds for the accuracy and completeness computation by
projecting every pixel from every view into the 3D spaceaccording to the predicted depth and the provided camera
poses.

4.1. Aerial Dataset
In this Section, we validate our technique on the aerial
dataset of Restrepo et al. [25] and compare it to various
model-based and learning-based baselines.
Quantitative evaluation: We pretrain the neural network
using the Adam optimizer [18] with a learning rate of 10−3
and a batch size of 32 for 100K iterations. Subsequently,
for end-to-end training, we use a voxel grid of size 643 , thesame optimizer with learning rate 10−4 and batches of 2000
randomly sampled rays from consecutive images. RayNet
is trained for 20K iterations. Note that due to the use of
the larger batch size, RayNet is trained on approximately
10 times more rays compared to pretraining. During training we use a voxel grid of size 643 to reduce computation.
However, we run the forward pass with a voxel grid of size
2563 for the final evaluation.
We compare RayNet with two commonly used patch
comparison measures: the sum of absolute differences(SAD) and the zero-mean normalized cross correlation(ZNCC). In particular, we use SAD or ZNCC to computea cost volume per pixel and then choose the depth withthe lowest cost to produce a depthmap. We use the publicly available code distributed by Haene et al. [11] for these
two baselines. We follow [11] and compute the max scoreamong all pairs of patch combinations, which allows somerobustness to occlusions. In addition, we also compare ourmethod with the probabilistic 3D reconstruction method
by Ulusoy et al. [35], again using their publicly available
implementation. We further compare against the learning
based approach of Hartmann et al. [14], which we reimplemented and trained based on the original paper [14].
Table 1 summarizes accuracy, completeness and per
pixel mean depth error for all implemented baselines. In
addition to the aforementioned baselines, we also compare
our full RayNet approach Ours (CNN+MRF) with our CNN
frontend in isolation, denoted Ours (CNN). We observe thatjoint optimization of the full model improves upon the CNN
frontend. Furthermore, RayNet outperforms both the classic as well as learning-based baselines in terms of accuracy,

mean depth error and Chamfer distance while performingon par with most baselines in terms of completeness.
Qualitative Evaluation:
We visualize the results of
RayNet and the baselines in Fig. 4. Both the ZNCC baseline and the approach of Hartmann et al. [14] require alarge receptive field size for optimal performance, yieldingsmooth depth maps, However, this large receptive field also
causes bleeding artefacts at object boundaries. In contrast,
our baseline CNN and the approach of Ulusoy et al. [35]yield sharper boundaries, while exhibiting a larger level ofnoise. By combining the advantages of learning-based descriptors with a small receptive field and MRF inference,
our full RayNet approach (CNN+MRF) results in significantly smoother reconstructions while retaining sharp object boundaries. Additional results are provided in the supplementary material.

4.2. DTU Dataset
In this section, we provide results on the BIRD and
BUDDHA scenes of the DTU dataset. We use the provided
splits to train and test RayNet. We evaluate SurfaceNet [16]at two different resolutions: the original high resolution
variant, which requires more than 4 hours to reconstruct asingle scene, and a faster variant that uses approximately
the same resolution (2563 ) as our approach. We refer to thefirst one as SurfaceNet (HD) and to the latter as SurfaceNet(LR). We test SurfaceNet with the pretrained models provided by [16].
We evaluate both methods in terms of accuracy and completeness and report the mean and the median in Table 2.
Our full RayNet model outperforms nearly all baselines interms of completeness, while it performs worse in terms ofaccuracy. We believe this is due to the fact that both [16] and[14] utilize the original high resolution images, while ourapproach operates on downsampled versions of the images.
We observe that some of the fine textural details that arepresent in the original images are lost in the down-sampled
versions. Besides, our current implementation of the 2Dfeature extractor uses a smaller receptive field size (11 pixels) compared to SurfaceNet (64 × 64) and [14] (32 × 32).
Finally, while our method aims at predicting a complete 3Dreconstruction inside the evaluation volume (resulting in occasional outliers), [16] and [14] prune unreliable predictions
from the output and return an incomplete reconstruction (resulting in higher accuracy and lower completeness).
Fig. 5 shows a qualitative comparison between SurfaceNet (LR) and RayNet. It can be clearly observed that
SurfaceNet often cannot reconstruct large parts of the object (e.g., the wings in the BIRD scene or part of the headin the BUDDHA scene) which it considers as unreliable. In
contrast, RayNet generates more complete 3D reconstructions. RayNet is also significantly better at capturing object
boundaries compared to SurfaceNet.

(a) Image(b) Ours (CNN)(c) Ours (CNN+MRF)(d) ZNCC(e) Ulusoy et al. [35](f) Hartmann et al. [14](g) Image(h) Ours (CNN)(i) Ours (CNN+MRF)(j) ZNCC(k) Ulusoy et al. [35](l) Hartmann et al. [14]Figure 4: Qualitative Results on Aerial Dataset. We show the depth maps predicted by our method (b+c,i+h) as well as
three baselines (d-f,j-l) for two input images from different viewpoints (a,g). Darker colors correspond to closer regions.
Methods
ZNCC
SAD
Ulusoy et al. [35]Hartmann et al. [14]Ours (CNN)
Ours (CNN+MRF)

Accuracy
MeanMedian0.07530.03840.0746
0.03730.07900.01670.09070.02850.08040.02200.06110.0160Completeness
MeanMedian0.01110.00410.01200.00420.00880.00650.02090.02090.01540.01320.01250.0085

Mean Depth Error
MeanMedian0.13450.13510.12330.12350.11430.10500.16480.1222
0.0977
0.09160.07440.0728Chamfer
0.04320.04330.04390.05580.04790.0368

Table 1: Quantitative Results on Aerial Dataset. We show the mean and median accuracy, completeness, per pixel error
and Chamfer distance for various baselines and our method. Lower values indicate better results. See text for details.

4.3. Computational requirements
The runtime and memory requirements of RayNet depend mainly on three factors: the number of voxels, thenumber of input images and the number of pixels/rays per
image, which is typically equal to the image resolution. All
our experiments were computed on an Intel i7 computerMethods
BUDDHA
ZNCC
SAD
SurfaceNet (HD) [16]SurfaceNet (LR) [16]Hartmann et al. [14]Ulusoy et al. [35]RayNet
BIRD
ZNCC
SAD
SurfaceNet (HD) [16]SurfaceNet (LR) [16]Hartmann et al. [14]Ulusoy et al. [35]RayNetAccuracy
MeanMedianCompleteness
MeanMedian

6.107
6.683
0.738
2.034
0.637
4.784
1.993

4.613
5.270
0.574
1.676
0.206
3.522
1.119

0.646
0.753
0.677
1.453
1.057
0.953
0.481

0.466
0.510
0.505
1.141
0.475
0.402
0.357

6.882
7.138
1.493
2.887
1.881
6.024
2.618

5.553
5.750
1.249
2.468
0.271
4.623
1.680

0.918
0.916
1.612
2.330
4.167
2.966
0.983

0.662
0.646
0.888
1.556
1.044
0.898
0.668

Table 2: Quantitative Results on DTU Dataset. We
present the mean and the median of accuracy and completeness measures for RayNet and various baselines. All results
are reported in millimeters, the default unit for DTU.

with an Nvidia GTX Titan X GPU. We train RayNet endto-end, which takes roughly 1 day and requires 7 GB permini-batch update for the DTU dataset. Once the network
is trained, it takes approximately 25 minutes to obtain a fullreconstruction of a typical scene from the DTU dataset. In
contrast, SurfaceNet (HD) [16] requires more than 4 hours
for this task. SurfaceNet (LR), which operates at the samevoxel resolution as RayNet, requires 3 hours.

(a) Ground Truth(b) SurfaceNet [16](c) RayNet(d) Ground Truth(e) SurfaceNet [16](f) RayNet(g) Ground Truth(h) SurfaceNet [16](i) RayNet(j) Ground Truth(k) SurfaceNet [16](l) RayNet(m) Ground Truth(n) SurfaceNet [16](o) RayNet(p) Ground Truth(q) SurfaceNet [16](r) RayNet

5. Conclusion
We propose RayNet, which is an end-to-end trained network that incorporates a CNN that learns multi-view image
similarity with an MRF with ray potentials that explicitly
models perspective projection and enforces occlusion constraints across viewpoints. We directly embed the physics
of multi-view geometry into RayNet. Hence, RayNet is not
required to learn these complex relationships from data. Instead, the network can focus on learning view-invariant feature representations that are very difficult to model. Our
experiments indicate that RayNet improves over learningbased approaches that do not incorporate multi-view geometry constraints and over model-based approaches that do
not learn multi-view image matching.
Our current implementation precludes training models
with a higher resolution than 2563 voxels. While this resolution is finer than most existing learning-based approaches,it does not allow capturing high resolution details in large
scenes. In future work, we plan to adapt our method
to higher resolution outputs using octree-based representations [12, 31, 35]. We also would like to extend RayNet to

Figure 5: Qualitative Results on DTU Dataset. We visualize the ground-truth (a,d,g,j,m) and the depth reconstuctions
using SurfaceNet (LR) (b,e,h,k,n) and RayNet (c,f,i,l,o).
predict a semantic label per voxel in addition to occupancy.
Recent works show such a joint prediction improves overreconstruction or segmentation in isolation [28, 30].

Acknowledgments
This research was supported by the Max Planck ETHCenter for Learning Systems.

References[1] H. Aanæs, R. R. Jensen, G. Vogiatzis, E. Tola, and A. B.
Dahl. Large-scale data for multiple-view stereopsis. International Journal of Computer Vision (IJCV), 120(2):153–168,
2016. 5, 6

[2] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean,
M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur,
J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner,
P. A. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, and
X. Zhang. Tensorflow: A system for large-scale machinelearning. arXiv.org, 1605.08695, 2016. 5[3] P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. J. Guibas.
Representation learning and adversarial generation of 3d
point clouds. arXiv.org, 1707.02392, 2017. 2[4] A. X. Chang, T. A. Funkhouser, L. J. Guibas, P. Hanrahan,
Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su,
J. Xiao, L. Yi, and F. Yu. Shapenet: An information-rich 3dmodel repository. arXiv.org, 1512.03012, 2015. 2[5] C. B. Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese. 3dr2n2: A unified approach for single and multi-view 3d object
reconstruction. In Proc. of the European Conf. on Computer
Vision (ECCV), 2016. 2[6] H. Fan, H. Su, and L. J. Guibas. A point set generation
network for 3d object reconstruction from a single image.
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017. 2[7] Y. Furukawa, B. Curless, S. M. Seitz, and R. Szeliski. Reconstructing building interiors from images. In Proc. of theIEEE International Conf. on Computer Vision (ICCV), 2009.
2[8] R. Girdhar, D. F. Fouhey, M. Rodriguez, and A. Gupta.
Learning a predictable and generative vector representationfor objects. In Proc. of the European Conf. on Computer
Vision (ECCV), 2016. 2[9] J. Gwak, C. B. Choy, A. Garg, M. Chandraker, and
S. Savarese. Weakly supervised generative adversarial networks for 3d reconstruction. arXiv.org, 1705.10904, 2017.
2, 3[10] F. Güney and A. Geiger. Deep discrete flow. In Proc. of the
Asian Conf. on Computer Vision (ACCV), 2016. 2[11] C. Häne, L. Heng, G. H. Lee, A. Sizov, and M. Pollefeys.
Real-time direct dense matching on fisheye images using
plane-sweeping stereo. In Proc. of the International Conf.
on 3D Vision (3DV), 2014. 6[12] C. Häne, S. Tulsiani, and J. Malik. Hierarchical surface prediction for 3d object reconstruction. arXiv.org, 1704.00710,
2017. 2, 8
[13] R. I. Hartley and A. Zisserman. Multiple View Geometryin Computer Vision. Cambridge University Press, second
edition, 2004. 2[14] W. Hartmann, S. Galliani, M. Havlena, L. Van Gool, and
K. Schindler. Learned multi-patch similarity. In Proc. of theIEEE International Conf. on Computer Vision (ICCV), 2017.
1, 2, 3, 6, 7, 8
[15] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and
T. Brox. Flownet 2.0: Evolution of optical flow estimation
with deep networks. Proc. IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), 2017. 2[16] M. Ji, J. Gall, H. Zheng, Y. Liu, and L. Fang. SurfaceNet:an end-to-end 3d neural network for multiview stereopsis. In
Proc. of the IEEE International Conf. on Computer Vision(ICCV), 2017. 2, 3, 6, 8

[17] A. Kar, C. Häne, and J. Malik. Learning a multi-view stereo
machine. In Advances in Neural Information Processing Systems (NIPS), 2017. 3[18] D. P. Kingma and J. Ba. Adam: A method for stochastic
optimization. In Proc. of the International Conf. on Learning
Representations (ICLR), 2015. 6[19] S. Liu and D. Cooper. Statistical inverse ray tracing forimage-based 3d modeling. IEEE Trans. on Pattern Analysis
and Machine Intelligence (PAMI), 36(10):2074–2088, 2014.
1, 2, 4[20] W. Luo, A. Schwing, and R. Urtasun. Efficient deep learningfor stereo matching. In Proc. IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), 2016. 2, 3, 4[21] N. Mayer, E. Ilg, P. Haeusser, P. Fischer, D. Cremers,
A. Dosovitskiy, and T. Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow
estimation. In Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2016. 2[22] S. Nowozin and C. H. Lampert. Structured learning and prediction in computer vision. Foundations and Trends in Computer Graphics and Vision, 2011. 5[23] T. Pollard and J. L. Mundy. Change detection in a 3-d world.
In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2007. 2[24] A. Ranjan and M. Black. Optical flow estimation using aspatial pyramid network. In Proc. IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR), 2017. 2[25] M. I. Restrepo, A. O. Ulusoy, and J. L. Mundy. Evaluationof feature-based 3-d registration of probabilistic volumetricscenes. ISPRS Journal of Photogrammetry and Remote Sensing (JPRS), 2014. 5, 6
[26] D. J. Rezende, S. M. A. Eslami, S. Mohamed, P. Battaglia,
M. Jaderberg, and N. Heess. Unsupervised learning of 3d
structure from images. In Advances in Neural Information
Processing Systems (NIPS), 2016. 3[27] G. Riegler, A. O. Ulusoy, H. Bischof, and A. Geiger. OctNetFusion: Learning depth fusion from data. In Proc. of theInternational Conf. on 3D Vision (3DV), 2017. 2[28] N. Savinov, L. Ladicky, C. Häne, and M. Pollefeys. Discrete
optimization of ray potentials for semantic 3d reconstruction.
In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2015. 2, 4, 8
[29] S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, andR. Szeliski. A comparison and evaluation of multi-view
stereo reconstruction algorithms. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2006. 2[30] S. Song, F. Yu, A. Zeng, A. X. Chang, M. Savva, andT. Funkhouser. Semantic scene completion from a single
depth image. In Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2017. 8[31] M. Tatarchenko, A. Dosovitskiy, and T. Brox. Octree generating networks: Efficient convolutional architectures forhigh-resolution 3d outputs. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2017. 2, 8[32] S. Tulsiani, T. Zhou, A. A. Efros, and J. Malik. Multi-view
supervision for single-view reconstruction via differentiable
ray consistency. In Proc. IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), 2017. 3

[33] A. O. Ulusoy, M. Black, and A. Geiger. Patches, planes andprobabilities: A non-local prior for volumetric 3d reconstruction. In Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), 2016. 1, 2
[34] A. O. Ulusoy, M. Black, and A. Geiger. Semantic multi-view
stereo: Jointly estimating objects and voxels. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2017. 2[35] A. O. Ulusoy, A. Geiger, and M. J. Black. Towards probabilistic volumetric reconstruction using ray potentials. In
Proc. of the International Conf. on 3D Vision (3DV), 2015.
1, 2, 4, 5, 6, 7, 8[36] C. Wu, S. Agarwal, B. Curless, and S. Seitz. Multicore bundle adjustment. In Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2011. 3[37] J. Wu, C. Zhang, T. Xue, B. Freeman, and J. Tenenbaum.
Learning a probabilistic latent space of object shapes via 3d
generative-adversarial modeling. In Advances in Neural Information Processing Systems (NIPS), 2016. 2[38] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, andJ. Xiao. 3d shapenets: A deep representation for volumetricshapes. In Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), 2015. 2[39] X. Yan, J. Yang, E. Yumer, Y. Guo, and H. Lee. Perspective
transformer nets: Learning single-view 3d object reconstruction without 3d supervision. In Advances in Neural Information Processing Systems (NIPS), 2016. 3[40] S. Zagoruyko and N. Komodakis. Learning to compare image patches via convolutional neural networks. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), 2015. 3[41] J. Zbontar and Y. LeCun. Computing the stereo matching cost with a convolutional neural network. arXiv.org,
1409.4326, 2014. 3[42] J. Žbontar and Y. LeCun. Stereo matching by training a convolutional neural network to compare image patches. Journal of Machine Learning Research (JMLR), 17(65):1–32,
2016. 2