Supplementary Material for
RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials
Despoina Paschalidou1,5 Ali Osman Ulusoy2 Carolin Schmitt1 Luc van Gool3 Andreas Geiger1,41Autonomous Vision Group, MPI for Intelligent Systems and University of Tübingen2
Microsoft 3 Computer Vision Lab, ETH Zürich & KU Leuven
4
Computer Vision and Geometry Group, ETH Zürich
5
Max Planck ETH Center for Learning Systems{firstname.lastname}@tue.mpg.devangool@vision.ee.ethz.chAbstract
In this supplementary document, we present our inference algorithm as well as additional qualitative results. We startby deriving the message expressions for the sum-product belief propagation algorithm used for approximate inference in
our Markov Random Field. Subsequently, we show additional results from the aerial dataset from different viewpoints
and we compare RayNet with commonly used disparity methods like ZNCC, with a probabilistic method [5] and with alearning-based approach [2]. In all cases, our method outperforms the baseline methods, thus resulting in significantly
smoother reconstructions, while retaining sharp object boundaries. Finally, we present visualizations of the completeness ofthe predicted reconstructions on the DTU dataset and we compare against SurfaceNet [3].

1. Markov Random Field Inference
Our inference algorithm is based on the sum-product belief propagation [4]. The detailed derivations of the belief propagation equations is presented in Section 1.1.

1.1. Sum-product Message Derivations
The sum-product algorithm for factor graphs works by passing messages between factors and variables. The general formof messages is given byXY
µf →x (x) =φf (Xf )µy→f (y)(1)
Xf \xy∈Xf \xµx→f (x) =Yµg→x (x)(2)

g∈Fx \fwhere Xf denotes all variables associated with factor f and Fx is the set of factors to which variable x connects. Upon
termination (namely either at convergence or until a maximum number of iterations has been reached), the approximate
marginal distribution of each variable can be computed as the product of messages from all neighbouring factors according
to Equation 3.
Yp(x) ∝
µg→x (x)(3)
g∈Fx

In our case, the factor graph of the MRF consists of the occupancy variables o = {oi |i ∈ X }, where X is the set of voxels
in the voxel grid, the unary factors ϕi (oi ), defined for every voxel i ∈ X and the ray factors ψr (or ), defined for every rayr ∈ R, where R is the set of all rays and or = (or1 , or2 , . . . , orNr ) denotes the ordered set of occupancy variables associatedwith the voxels which intersect ray r.
1

Figure 1: Factor Graph of the MRF for a single ray. Unary factors ϕi (oi ) are defined for every voxel i ∈ X of the voxelgrid. Ray factor ψr (or ) is defined for the query ray r ∈ R and connects to the variables along the ray or = (or1 , . . . , orNr ).
Consider a single ray r associated with a ray factor ψr (or ). The ray factor ψr is associated with Nr occupancy variables,
namely this ray passes through Nr voxels. The corresponding factor graph is illustrated in Fig. 1.
1.1.1

Ray factor to occupancy variables messages µψr →ori

The ray potentials encourage the predicted depth at pixel/ray r to coincide with the first occupied voxel along the ray, thus
we can rewrite the expression for the ray potentials as follows
 r
s1 ,if or1 = 1Nrsr ,XY
if or1 = 0, or2 = 12(4)ori(1 − ori )sri =ψr (or ) =...

i=1
j<i
 r
sNr , if or1 = 0, or2 = 0, . . . , orNr −1 = 0, orNr = 1
where sri is the probability that the surface intersects ray r at voxel i as predicted by the neural network described in the main
submission.
We first derive the message equations from the ray factor to the ith occupancy variable, when it is occupied (ori = 1).
Eq. 1 becomesµψr →ori (ori = 1) =X···

or1

XX···

ori−1 ori+1X

ψr (or )

orNr

NrY

µorj →ψr (orj )

(5)

j=1
j6=i

We carry out the summations over the occupancy variables one by one. We begin the summation over or1 according toEquation 5(♦)

z"µψr →ori (ori = 1) = µor1 →ψr (or1 = 1)

X···

XX···}|Xor2

ori−1 ori+1orNr

XXXX

ψr (or1 = 1, . . . , orNr )or2···

ori−1 ori+1···ψr (or1 = 0, . . . , orNr )orNr{z|#{µorj →ψr (orj ) +

j=2
j6=i"
µor1 →ψr (or1 = 0)

NrY

(†)

NrY

#(6)

µorj →ψr (orj )

j=2
j6=i}

In the first term of Eq. 6, marked with (♦) we can replace the ray potential ψr (or1 = 1, . . . orNr = 0) = sr1 from Eq. 4.
Since sr1 only depends on the or1 , it can be brought out of the summations over the other occupancy variables. Furthermore,NrX
XXXY
the remaining term······
µorj →ψr (orj ) evaluates to 1 because we assume that all incoming messages areor2

ori−1 ori+1

orNr j=2
j6=inormalized such that they sum to 1. The last term, marked with (†), does not simplify.

Subsequently, we expand the summation over or2"µψr →ori (ori = 1) =µor1 →ψr (or1 = 1)sr1 + µor1 →ψr (or1 = 0)"X

µor2 →ψr (or2 = 1)

···

XX···

X

or3

ori−1 ori+1orNrXXXX

ψr (or1 = 0, or2 = 1, . . . , orNr )······ψr (or1 = 0, or2 = 0, . . . , orNr )orNr

ori−1 ori+1

or3#µorj →ψr (orj )

+

j=3
j6=i

"
µor2 →ψr (or2 = 0)

NrY

NrY

(7)

##
µorj →ψr (orj )

j=3
j6=i

The second term of Eq. 7 can be further simplified by replacing the ray potential from Eq. 4 and by assuming normalized
assuming messages with µor1 →ψr (or1 = 0)µor2 →ψr (or2 = 1)sr2 .
We continue carrying out the summations until (including) the ori−1 occupancy variables and we getµψr →ori (ori = 1) =i−1
Xµorj →ψr (orj = 1)

j=1

j−1
Y

µork →ψr (ork = 0)srj +i−1Yµorj →ψr (orj = 0)×

j=1

k=1

"X···

ori+1

X

ψ(or1 = 0, . . . , ori−1 = 0, ori = 1, . . . , orNr )orNrNrY#µorj →ψr (orj )(8)

j=i+1|{z}(‡)

In the last term of Eq. 8, marked with (‡), we can substitute the ray potential ψ(or1 = 0, . . . , ori−1 = 0, ori = 1, . . . , orNr )with sri and we get the final form of the message when the ith voxel is occupied as followsµψr →ori (ori = 1) =i−1
Xµorj →ψr (orj = 1)

j=1

j−1
Y

µork →ψr (ork = 0)srj +i−1Yµorj →ψr (orj = 0)sri(9)

j=1

k=1

Following a similar technique we derive the message expression when the ith voxel is free, namely (ori = 0) as followsµψr →ori (ori = 0) =i−1
Xµorj →ψr (orj = 1)

j=1

j−1
Y

µork →ψr (ork = 0)srj +k=1i−1Yµork →ψr (ork = 0)×

k=1"X
ori+1···

X

ψ(or1 = 0, . . . , ori−1 = 0, ori = 0, . . . , orNr )

orNr

NrY

#µorj →ψr (orj )(10)

j=i+1{z|}(q)The last term of Eq. 10, marked with (q) can be simplified by summing out all occupancy variables one by one as followsµori+1 →ψr (ori+1 = 1)sri+1 + · · · + µori+1 →ψr (ori+1 = 0) . . . µorNr −1 →ψr (orNr −1 = 0)µorNr →ψr (orNr = 1)srNr(11)

The resulting expression when the ith voxel is free becomesµψr →ori(ori = 0) =i−1X
j=1µorj →ψr(orj = 1)

j−1
Y
k=1

µork →ψr(ork = 0)srj +NrX
j=i+1

µorj →ψr(orj = 1)

j−1
Y
k=1
k6=i

µork →ψr (ork = 0)srj(12)

1.1.2

Occupancy variable to ray factor messages µori →ψr

Using Equation 2, we derive the message from the ith occupancy variable to the ray factor ψr . This message can be definedas the product of all messages from the set of factors Fori to which the occupancy variable ori is connected (except for theψr factor). The Fori set consists of the unary to occupancy variable message, namely µϕi →ori (ori ) as well as all the factor-tooccupancy messages for all the k ∈ L rays, except for the r ray.
Y(13)µψk →ori (ori )µori →ψr (ori ) = ϕi (ori )ψk ∈L\ψr

1.1.3

Messages from and to the unary factors µϕi →ori and µori →ϕiThe messages from and to the unary factors are straightforward to compute using Equations 1 and 2. The message from theoccupancy variable ori to the unary factor φi is defined as the product of all messages from the set of factors Fori to which theoccupancy variable ori is connected (except for the ϕi factor). The set of factors Fori includes all the ψr (.) factors, one foreach ray that might pass through the ith voxel. Let L be the set of rays that pass through the ith voxel, the µori →ϕi becomesYYµori →ϕi (ori ) =µg→ori (ori ) =µψr →ori (ori )(14)
ψr ∈Lg∈For \ϕiiRegarding the message from unary factor ϕi to the occupancy variable ori , each unary factor is associated with the occupancy variable of the ith voxel ori , thus the unary factor to occupancy variable message is expressed asrrµϕi →ori (ori ) = ϕi (ori ) = γ oi (1 − γ)1−oi(15)1.2. Depth Estimation Derivation
In this section, we provide details for our depth estimation procedure. As described in Section 3.2 of the submission, weassociate the occupancy and depth variables along a ray using the Eq. 16.
dr =NrX
i=1oriY(1 − ori )dri(16)j<iUsing the sum-product belief propagation and the message expressions derived in Section 1.1 we estimate a probability
distribution p(dr = dri ) for each depth variable dr .
Eq. 16 encourages that the predicted depth at pixel/ray r coincide with the first occupied voxel along the ray, thus it canbe rewritten as follows
 rd1 ,if or1 = 1


Nr
dr ,
XY
if or1 = 0, or2 = 12
dr =ori(1 − orj )dri =(17)
...

i=1
j<i rdNr , if or1 = 0, or2 = 0, . . . , orNr −1 = 0, orNr = 1
Taking into account the above expression for the depth formation process and the fact that we always assume that there
exists at least one occupied voxel along the ray, the probability distribution p(dr = dri ) becomesp(dr = dri ) = p(or1 = 0, . . . , ori−1 = 0, ori = 1)(18)

We can easily estimate the marginal distribution of the occupancy variables from Eq. 3 as followsp(or1 , . . . , orNr ) ∝ ψr (o)NrYµori →ψr (ori )

(19)

i=1

However, we only want to estimate the p(or1 = 0, . . . , ori−1 = 0, ori = 1) ∀i = 1, . . . , Nr . This can be done by simply
summing out all the other occupancy variables from Eq. 19

p(or1 = 0, . . . , ori−1 = 0, ori = 1) =X···

ori+1

∝

X

Xp(or1 = 0, . . . , ori−1 = 0, ori = 1, ori+1 , . . . , orNr )orNr···ori+1X

ψ(or1 = 0, . . . , ori−1 = 0, oi = 1, ori+1 , . . . , orNr )orNr(20)"× µori →ψr (ori = 1)i−1Y#µorj →ψr (orj = 0)j=1µorj →ψr (orj )j=i+1{z|NrY}()

Note that the term marked with () does not depend on the occupancy variables summed, hence it can be taken out of thesummations as followsp(or1 = 0, . . . , ori−1 = 0, ori = 1) ∝ µori →ψr (ori = 1)i−1
Yµorj →ψr (orj = 0)×j=1

X
ori+1···X

ψ(or1 = 0, . . . , ori−1 = 0, ori = 1, ori+1 , . . . , orNr )

orNr

NrY

µorj →ψr (orj )

(21)

j=i+1

{z|}

(†)

The ray potential ψ(or1 = 0, . . . , ori = 1, ori+1 , . . . orNr ) term in the expression marked with (†) can be replaced with srifrom Eq. 4, hence it can also been taking out of the summations, resulting inp(or1 = 0, . . . , ori−1 = 0, ori = 1) ∝ µori →ψr (ori = 1)i−1Yµorj →ψr (orj = 0)sri ×X···

ori+1

j=1|NrX Yµorj →ψr (orj )orNr j=i+1{zthis term evaluates to 1(22)}However, as we have already mentioned before all incoming messages are normalized such that the sum to 1, thus thejoint probability simplifies top(or1 = 0, . . . , ori−1 = 0, ori = 1) ∝ µori →ψr (ori = 1)i−1Yµorj →ψr (orj = 0)sri

(23)

j=1

By substituting Eq. 23 in Eq. 18, we get the final expression for the probability distributionp(dr = dri ) =i−1Y1µori →ψr (ori = 1)
µorj →ψr (orj = 0)sri
Z
j=1(24)

where Z is a normalization constant such that the depth distribution sums to 1.

2. Aerial Dataset
In this Section, we provide additional qualitative results on the aerial dataset. We compare RayNet with the zero-mean
normalized cross correlation (ZNCC), which is a commonly used disparity method, calculated over multiple input views.
In addition, we also compare our method with the probabilistic 3D reconstruction method by Ulusoy et al. [5]. We further
compare against the learning based approach of Hartmann et al. [2]. For more detailed analysis regarding the implementationsof these baselines please refer to Section 4.1 of our submission.
Fig. 2 and Fig. 3 depict the predicted depth maps for all methods for a set of images from different viewpoints. In all
images, both ZNCC and the Hartmann et al. [2] baselines result in smoother depth maps with artefacts around the image
boundaries. In contrast, our CNN baseline as well as the probabilistic approach by Ulusoy et al [5] yield sharped boundaries
with significant noise. In general, facades of the buildings are considered to be problematic areas, because they lack anytexture. It is important to point out, that also in this case (Fig. 3m), our method results in more accurate reconstructions
compared to the other baselines.

(a) Image(b) Ours (CNN)(c) Ours (CNN+MRF)(d) ZNCC(e) Ulusoy et al. [5](f) Hartmann et al. [2](g) Image(h) Ours (CNN)(i) Ours (CNN+MRF)(j) ZNCC(k) Ulusoy et al. [5](l) Hartmann et al. [2](m) Image(n) Ours (CNN)(o) Ours (CNN+MRF)(p) ZNCC(q) Ulusoy et al. [5](r) Hartmann et al. [2]Figure 2: Qualitative Results on Aerial Dataset for images from different viewpoints. We show the depth maps predictedby our method (b+c,i+h, n+o) as well as three baselines (d-f,j-l, p-r) for three input images from different viewpoints (a,g,m). Darker colours correspond to closer regions.

(a) Image(b) Ours (CNN)(c) Ours (CNN+MRF)(d) ZNCC(e) Ulusoy et al. [5](f) Hartmann et al. [2](g) Image(h) Ours (CNN)(i) Ours (CNN+MRF)(j) ZNCC(k) Ulusoy et al. [5](l) Hartmann et al. [2](m) Image(n) Ours (CNN)(o) Ours (CNN+MRF)(p) ZNCC(q) Ulusoy et al. [5](r) Hartmann et al. [2]Figure 3: Qualitative Results on Aerial Dataset for images from different viewpoints. We show the depth maps predictedby our method (b+c,i+h, n+o) as well as three baselines (d-f,j-l,p-r) for two input images from different viewpoints (a,g,m).
Darker colours correspond to closer regions.

3. DTU Dataset
In this Section, we show additional qualitative results on the DTU dataset. We evaluate RayNet on six objects from thisdataset: BUNNY, STONE EDGE, BIRD, SNOWMEN, BRICKS and BUDDHA and we compare the predicted reconstructions against SurfaceNet [3] on the same resolution, namely 2563 . Fig. 4 shows the six objects.

(a) BUNNY(b) STONEHENGE(c) BIRD(d) SNOWMEN(e) BRICKS(f) BUDDHAFigure 4: DTU dataset. Sample images from DTU [1].
In all experiments, we observe that RayNet leads always to complete reconstructions, while SurfaceNet results in missingparts. It is also important to note that in the case of SurfaceNet we use the images in their original size, namely 1400 ×
1600, because when we tried to use the downsampled images the reconstruction failed completely. However, this exposes
SurfaceNet to more than 6 times more information/pixels compared to RayNet, which uses the downsampled images.

(a) Ground Truth(b) SurfaceNet [3](c) RayNet(d) Ground Truth(e) SurfaceNet [3](f) RayNet

Figure 5: Qualitative Results on the BIRD. We visualize the ground-truth (a,d) and the depth reconstuctions using SurfaceNet (LR) (b,e) and RayNet (c,f)(a) Ground Truth(b) SurfaceNet [3](c) RayNet(d) Ground Truth(e) SurfaceNet [3](f) RayNet

Figure 6: Qualitative Results on the BUDDHA. We visualize the ground-truth (a,d) and the depth reconstuctions using
SurfaceNet (LR) (b,e) and RayNet (c,f)

(a) Ground Truth(b) SurfaceNet [3](c) RayNet(d) Ground Truth(e) SurfaceNet [3](f) RayNet(g) Ground Truth(h) SurfaceNet [3](i) RayNet(j) Ground Truth(k) SurfaceNet [3](l) RayNet(m) Ground Truth(n) SurfaceNet [3](o) RayNet

Figure 7: Qualitative Results on the STONEHENGE. We visualize the ground-truth (a,d,g,j,m) and the depth reconstuctions using SurfaceNet (LR) (b,e,h,k,n) and RayNet (c,f,i,l,o)(a) Ground Truth(b) SurfaceNet [3](c) RayNet(d) Ground Truth(e) SurfaceNet [3](f) RayNet(g) Ground Truth(h) SurfaceNet [3](i) RayNet(j) Ground Truth(k) SurfaceNet [3](l) RayNet(m) Ground Truth(n) SurfaceNet [3](o) RayNet

Figure 8: Qualitative Results on the BRICKS. We visualize the ground-truth (a,d,g,j,m) and the depth reconstuctions using
SurfaceNet (LR) (b,e,h,k,n) and RayNet (c,f,i,l,o)(a) Ground Truth(b) SurfaceNet [3](c) RayNet(d) Ground Truth(e) SurfaceNet [3](f) RayNet(g) Ground Truth(h) SurfaceNet [3](i) RayNet(j) Ground Truth(k) SurfaceNet [3](l) RayNet(m) Ground Truth(n) SurfaceNet [3](o) RayNet

Figure 9: Qualitative Results on the BUNNY. We visualize the ground-truth (a,d) and the depth reconstuctions usingSurfaceNet (LR) (b,e) and RayNet (c,f)

(a) Ground Truth(b) SurfaceNet [3](c) RayNet(d) Ground Truth(e) SurfaceNet [3](f) RayNet(g) Ground Truth(h) SurfaceNet [3](i) RayNet(j) Ground Truth(k) SurfaceNet [3](l) RayNet(m) Ground Truth(n) SurfaceNet [3](o) RayNet

Figure 10: Qualitative Results on the SNOWMEN. We visualize the ground-truth (a,d) and the depth reconstuctions using
SurfaceNet (LR) (b,e) and RayNet (c,f)

References[1] C. Häne, L. Heng, G. H. Lee, A. Sizov, and M. Pollefeys. Real-time direct dense matching on fisheye images using plane-sweeping
stereo. In Proc. of the International Conf. on 3D Vision (3DV), 2014. 8[2] W. Hartmann, S. Galliani, M. Havlena, L. Van Gool, and K. Schindler. Learned multi-patch similarity. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2017. 1, 5, 6, 7
[3] M. Ji, J. Gall, H. Zheng, Y. Liu, and L. Fang. SurfaceNet: an end-to-end 3d neural network for multiview stereopsis. In Proc. of theIEEE International Conf. on Computer Vision (ICCV), 2017. 1, 8, 9, 10, 11, 12, 13[4] S. Nowozin and C. H. Lampert. Structured learning and prediction in computer vision. Foundations and Trends in Computer Graphicsand Vision, 2011. 1[5] A. O. Ulusoy, A. Geiger, and M. J. Black. Towards probabilistic volumetric reconstruction using ray potentials. In Proc. of theInternational Conf. on 3D Vision (3DV), 2015. 1, 5, 6, 7