Omnidirectional 3D Reconstruction in Augmented Manhattan Worlds
Miriam Schönbein1 and Andreas Geiger2

Abstract— This paper proposes a method for high-quality
omnidirectional 3D reconstruction of augmented Manhattan
worlds from catadioptric stereo video sequences. In contrast to
existing works we do not rely on constructing virtual perspective
views, but instead propose to optimize depth jointly in a unified
omnidirectional space. Furthermore, we show that plane-basedprior models can be applied even though planes in 3D do
not project to planes in the omnidirectional domain. Towards
this goal, we propose an omnidirectional slanted-plane Markov
random field model which relies on plane hypotheses extractedusing a novel voting scheme for 3D planes in omnidirectional
space. To quantitatively evaluate our method we introduce adataset which we have captured using our autonomous driving
platform AnnieWAY which we equipped with two horizontallyaligned catadioptric cameras and a Velodyne HDL-64E laser
scanner for precise ground truth depth measurements. As
evidenced by our experiments, the proposed method clearly
benefits from the unified view and significantly outperforms
existing stereo matching techniques both quantitatively andqualitatively. Furthermore, our method is able to reduce noise
and the obtained depth maps can be represented very compactly
by a small number of image segments and plane parameters.

I. I NTRODUCTION
3D perception is an important prerequisite for many tasks
in robotics. For instance, consider self-driving vehicles [1]which need to accurately sense their environment in orderto plan the next maneuver. Clearly, a 360◦ field of view isdesirable. During the DARPA Urban Challenge [2] laserbased solutions have been popularized for that purpose.
However, they provide only very sparse point clouds or are
extremely expensive like the Velodyne HDL-64E. Furthermore, they suffer from rolling shutter effects and a separate
video sensor is required to provide color information foreach laser point. Instead, in this paper we advocate theuse of catadioptric cameras [3], [4] for 3D reconstruction.
Combining a traditional (perspective) camera with a mirror
coated surface, they are cheap to produce and provide a 360◦
view of the scene which, in contrast to fisheye cameras canbe parameterized by the specific choice of the mirror shape.
Our setup is illustrated in Fig. 1(a).
We tackle the problem of reconstructing the static partsof 3D scenes which follow the augmented manhattan world
assumption [5], i.e., scenes which can be described by vertical and horizontal planes in 3D. Note that this assumption
does not require vertical planes to be orthogonal with respect
to each other as in [6], [7], but only with respect to the1 Miriam Schönbein is with the Institute of Measurement and Control
Systems, Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany.

miriam.schoenbein@kit.edu
2 Andreas Geiger is with Max Planck Institute for Intelligent Systems, Perceiving Systems Department, 72076 Tübingen, Germany.

andreas.geiger@tue.mpg.de(a) Stereo Camera Setup(b) 3D Reconstruction

Fig. 1.
Omnidirectional 3D Reconstruction. Figure (a) illustrates
our catadioptric stereo camera setup. Figure (b) shows the result of theomnidirectional 3D reconstruction obtained by our method.

horizontal planes. As illustrated in Fig. 1(b), many urban
scenes closely follow this assumption and also indoor scenes
are often composed of mainly horizontal or vertical surfaces[8], [9]. In this work, we show that incorporating such priorknowledge into the model can greatly benefit 3D reconstruction, in particular when dealing with omnidirectional images
that often suffer from blur and low contrast.
While planarity priors for stereo matching have been proposed in the context of traditional perspective cameras [10],[11], 3D planes do not project to planes in omnidirectional
space, thereby preventing the use of classical prior models.
We tackle this problem by proposing a slanted surface
Markov random field (MRF) model based on superpixels
in a virtual omnidirectional view. We start by sphericallyrectifying adjacent camera views and obtain initial depth
estimates by matching each pair of views. Next, we aggregate
all depth measurements in one common omnidirectional
space and propose a Hough voting scheme which yields theset of dominant 3D planes in the scene. Subsequently, each
3D plane hypothesis is mapped to a non-linear surface inthe omnidirectional image space, from which we computepotentials for all superpixels in the image. Plane optimization
is formulated as a discrete labeling problem and carried out
using loopy belief propagation which amounts to finding
the best plane hypothesis for each superpixel under theassumption that nearby superpixels are likely belonging to
the same surface.
Furthermore, we introduce a novel dataset of 152 diverseand challenging urban scenes for which we provide omnidirectional imagery as well as laser-based ground truth
depth maps. We quantitatively show that our model outperforms state-of-the-art stereo matching techniques [12], [10]which have demonstrated superior performance in related
evaluations such as the KITTI stereo benchmark [13]. We
also show that our results are qualitatively more pleasing asthey are less susceptible to noise and allow for identifying
the dominant planes which can be useful input information
to subsequent higher-level reasoning stages such as scene ϕs
understanding [14]. Our code, dataset and ground truth depth
maps are publicly available1 .

θs1

θs2

II. R ELATED W ORK
While there exists a large body of literature on omnidirectional camera calibration [15], [16], [17], [18], localization[19], [20] and sparse structure-from-motion / SLAM [21],[22], [23], surprisingly little research has been carried out
towards dense 3D reconstruction with catadioptric cameras.
In [24], Svoboda and Pajdla investigate the epipolar geometry of central catadioptric systems. They show that the
epipolar lines correspond to general conics in the omnidirectional image which reduce to radial lines for verticallyaligned catadioptric cameras. As the latter allows for simple
rectification, several methods take advantage of this setup
by either mounting two catadioptric cameras on top of eachother [25] or by using a double mirror design which allows
for stereo matching with a single camera only [26]. Unfortunately, this configuration has a fixed and short baseline andonly allows for accurate reconstructions in the very close
range.
For general camera motion, [27], [28] propose to reproject the omnidirectional image to a panoramic image ona virtual cylinder. Stereo correspondences are establishedby searching along sinosoidal shaped epipolar curves [27],[29], [30]. Gonzalez and Lacroix [28] overcome this problem
by rectifying the epipolar curves in panoramic images tostraight lines. Similarly, Geyer and Daniilidis [31] present
a conformal rectification method for parabolic images bymapping from bipolar coordinates to a rectangular grid. In
this paper, we take advantage of spherical rectification [32],[33], [34] which is more flexible, can handle the existence ofmore than one epipole and does not depend on a particular
projection model.
Towards dense 3D reconstruction, Arican and Frossard[34] obtain disparity maps from two omnidirectional viewsby optimizing a pixel-wise energy using graph cuts similar
to the work of Fleck et al. [30]. Lhuiller [35] reconstructs thescene from three consecutive omnidirectional images which
are projected onto the six faces of a virtual cube in order toallow for traditional stereo matching techniques. The local
results are fused into a global model by selecting the mostreliable viewpoints for each scene point and merging the 3D
points using their median. This approach has been extended
in [36] towards reconstruction of larger models from video
sequences.
In contrast to the presented works that either consider
3D reconstruction from only two views or fuse depth maps
in a rather ad-hoc manner, here we present a direct approach to 360◦ depth map optimization based on disparity
estimates from two temporally and two spatially adjacentomnidirectional views. Note that due to the diverse spatial
1 http://www.mrt.kit.edu/software/

Fig. 2. Rectified Catadioptric Stereo Pair. The horizontal lines depict
points with the same azimuth angle ϕs in the left and right image.

distribution of baselines this setup eliminates depth ’blind
spots’ which occur when reconstructing from two views
only. Furthermore, we show how planarity priors can beincorporated directly in the omnidirectional domain, leading
to clean low-noise 3D reconstructions. Accommodating thefact of limited public omnidirectional datasets, we contributeour data and the corresponding 3D ground truth to therobotics community.
III. O MNIDIRECTIONAL M ULTI -V IEW S TEREO
To ease the formulation of the 3D reconstruction problem
we first compute a virtual 360◦ disparity image from four
omnidirectional views captured by a catadioptric stereo pairat two consecutive time steps. Through combination of
depth information in one unified view we enable efficient
inference and overcome the problem of blind spots near the
epipoles [37] or occluded regions in some of the images.
We calibrate our omnidirectional stereo camera rig using themethod of [18] which optimizes for the best single viewpoint
approximation even in cases where the cameras are slightly
non-central, e.g., due to inaccuracies in the manufacturing
process. Next, we estimate camera motion between two
consecutive frames. We rectify temporal and spatial adjacentomnidirectional input pairs and combine their disparity maps
in a single unified 360◦ inverse depth image which forms thebasis for the plane-based inference discussed in Sec. IV. We
discuss these steps in the following.
A. Motion Estimation
To estimate motion between two consecutive frames captured by the catadioptric stereo camera rig, we match sparsefeatures between all views of both consecutive stereo pairs.
We employ the FAST corner detector [38] in combination
with the BRIEF descriptor [39], which empirically led tothe best results for our images which suffer from blur at theimage boundaries and incidental noise due to small scratches
in the mirror surface. In practice, we obtain around 1300
correspondences in temporal and spatial direction. Using theextrinsic calibration of the stereo camera rig, we triangulate

xs
x s1
ϕs
e12

z

xs2
ϕs

γ
ρs

θ s2

θs1
e11 zs1

ktk

e21

e22 zs2

x

ρ

dh

x

(a) Horizontal Plane (Side View)

Fig. 3. Spherical rectification. After applying the rectifying rotation a 3D
point x lies on the plane Π with the same azimuth angle ϕs in both (rotated)spherical coordinate systems. The rotated coordinate system depends on therelative position of both cameras determined by extrinsic calibration (stereo)
or motion estimation (motion stereo), respectively.

feature points in the previous frame t − 1 and estimate
motion by minimizing the reprojection error with respect
to the observations in the current frame t similar to theStereoScan system [40] for perspective cameras. Robustness
against outliers is achieved by using RANSAC with 150
iterations. To balance the spatial distribution of feature points
we employ bucketing using 16 cells with a maximum of 12
features per cell.
B. Rectification
To allow for efficient stereo matching, we rectify all
four omnidirectional stereo pairs using spherical rectification
similar to [32], [34]. This is illustrated in Fig. 3. We rotate
an image pair in the spherical domain such that the epipoles
coincide with the coordinate poles (z-axis). The remaining
degree of freedom is chosen to minimize the relative rotationof the y-axis with respect to the camera coordinate system.
Thus epipolar great circles coincide with the longitudes
and disparity estimation reduces to a onedimensional search
problem with constant azimuth angle ϕs . Fig. 2 depicts theresult of the spherical rectification process. For further details
we refer the reader to the appendix.
C. 360◦ Disparity Image
Given the rectified image pairs, we obtain disparity maps
using semi-global matching [12] which has shown excellent
performance in state-of-the-art perspective stereo benchmarks such as the KITTI stereo evaluation [13]. In thespherical domain the angular disparity γ = θs2 − θs1 isdefined as the difference between the angles θs1 and θs2of the two viewing rays imaging the same 3D world point
xs . The depth ρs of xs is then given as
ktk · sin θs2
ρs =
sin γ

y r
x
dv

r

y s2

ys1

ϕ
α

x, y

θ

(1)

where ktk denotes the baseline between the cameras. Dueto the fact, that the images are highly distorted near the
epipoles (leading to increased reconstruction error) as well as
occlusions by the recording platform itself, we extract stereo
depth estimates only from the front- (120◦ ) and backward(120◦ ) parts of the ego-vehicle while motion disparity is
extracted only from the corresponding side (each 120◦ ).

(b) Vertical Plane (Top View)

Fig. 4. Plane hypotheses. This figure shows the relationship between apoint x described by the spherical parameters ϕ, θ and its depth r, andthe plane parameters dh , dv and α for horizontal and vertical planes in thecoordinate system of the virtual camera.

After triangulating all points in the four spherical image
pairs Isj (θs , ϕs ) (with j denoting the image number), we
project them into a new virtual 360◦ image I(ϕ, θ) with thecamera coordinate system located in the center of the four
original views. We choose the center as origin for the virtual
coordinate system to minimize the relative displacementof all reflected rays. Furthermore, all points are rotatedsuch that the x-y-plane of the new coordinate system isparallel to the groundplane. We estimate this transformationby computing the dominant plane below the camera using
RANSAC plane fitting of the 3D points. Depth values foroverlapping regions are merged by computing the mean value
for each of these pixels. Fig. 5 (left) illustrates the resulting
virtual 360◦ intensity image I(ϕ, θ) and inverse depth imageD(ϕ,pθ), where we define inverse depth by D = 1/r withr = x2 + y 2 independent of the z-component of each 3D
point to ease the representation of planes as will be discussed
in Sec. IV. Note that working with inverse depth instead ofdepth implicitly accounts for the error characteristics of theunderlying stereo measurements.
IV. S LANTED -P LANE MRF
For efficient inference and to propagate information overlarger distances, we first partition the image into ∼ 1000
superpixels using the StereoSLIC algorithm [10] applied tothe 360◦ inverse depth image D(ϕ, θ) from the previous
section. Next, we extract the set of dominant plane hypotheses for each scene and the problem of finding the bestplane per superpixel is cast as a discrete labeling problem.
The estimation of the plane hypotheses and the energy
formulation are presented in the following.
A. Plane Hypotheses
Based on the fact that our coordinate system is parallelto the groundplane (x-y), we are able to describe verticalplanes using two variables (angle α and distance dv ) andhorizontal planes with a single variable only (distance dh )as illustrated in Fig. 4. Since the depth r is independent fromthe z-component, the relationship between a 3D point x andthe distance of a plane passing through x is given byr
dh (r, θ) =
(2)
tan θ
dv (r, ϕ, α) = r · cos (ϕ − α)(3)where the variables denote angles and distances as defined inFig. 4. This suggests a simple hough voting scheme: We accumulate the votes of all pixels in the virtual omnidirectional
image in a 1-dimensional horizontal plane accumulator array
H(dh ) and in a 2-dimensional vertical plane accumulator
array H(dv , α) as illustrated in Fig. 5 (middle). To make thevotes more discriminative, we disambiguate pixels belonging
to horizontal and vertical surfaces by casting each vote withan additional weight which corresponds to the likelihoodof a pixel belonging to a horizontal (or vertical) surface.
This likelihood is modeled by logistic regression using thevertical inverse depth gradients as input. We estimate theparameters of the sigmoid function using a held out training
set for which all horizontal and vertical surfaces have been
manually labeled. The maxima of the voting accumulators
H(dh ) and H(dv , α) are computed using an efficient nonmaxima suppression implementation [41] which we havemodified to handle cyclic image panoramas.
B. Energy Formulation

The second unary term models the prior probability forsurfaces to be horizontal or vertical and is given by
2 ph (s) − 1 if s ∈ H
ψu2 (s) = wu2 ×(6)
1 − 2 ph (s) otherwise
where H is the set of horizontal planes and
1 X 0
ph (p) ∈ [0, 1]
ph (s) =
|Ps |

(7)

p∈Ps

is the prior probability of superpixel s being horizontal. Here,p0h (p) is simply the probability of pixel p being horizontal
which we compute from our held-out training set augmented
with manually labeled polygons of vertical and horizontal
surfaces. Note how (6) assigns a positive score to plane
hypotheses that agree with the expected plane type andnegative scores otherwise.
Our pairwise model encourages neighboring superpixels
to agree at their boundariesXψp (s1 , s2 ) = wp
ρp (D̂(p, s1 ) − D̂(p, s2 ))(8)p∈Bs1 ,s2

Given the plane hypotheses from the previous section, we
formulate the problem of assigning each superpixel to oneof the planes as a discrete energy minimization problem.
More formally, let S = {s1 , . . . , sM } denote the variables ofinterest, each corresponding to one of the superpixels, wheres takes a discrete plane index s ∈ {1, . . . , N } as value. Here,M denotes the total number of superpixels in the image andN is the number of plane hypotheses. We define the energyfunction to be minimized asXXψp (s1 , s2 ) (4)Ψ(S) =[ψu1 (s) + ψu2 (s)] +s∈S(s1 ,s2 )∈NSwith unary terms ψu and pairwise terms ψp where NS denotes the set of neighboring superpixels, i.e., all superpixels
that share a common boundary.
The first unary term models the inverse depth fidelityiXhψu1 (s) = wu1 a(s)ρu (D̂(p, s) − D(p))
(5)
p∈Ps

with weight parameter wu1 . Here, D̂(p, s) is the inverse
depth at pixel p = (ϕ, θ)T predicted from the plane withindex s, D(p) is the inverse depth estimate at pixel p(see Sec. III-C) and ρu (x) = min(|x|, τu ) is a robust l1
penalty function with truncation parameter τu . Furthermore,
Ps denotes the set of all pixels with valid inverse depth
hypothesis D(p) which are covered by superpixel s anda(s) ∈ [0, 1] is a function that predicts the accuracy of
the inverse depth map D averaged over superpixel s fromtraining data. The latter has been introduced as we found thereliability of SGM to correlate strongly with image blur andhence also image location when dealing with omnidirectional
images. In practice, we take a(s) as the average ratio ofcorrectly predicted depth values computed from a held-out
training set.

where wp is a smoothness parameter and Bs1 ,s2 is the set ofboundary pixels that are shared between s1 and s2 . Similar
to the depth fidelity term, we take ρu (x) = min(|x|, τp ) as
the robust l1 penalty with truncation parameter τp .
C. Learning and Inference
For inferring S we make use of min-sum loopy belief
propagation to approximately minimize the energy specified
in (4). The parameters of our model are estimated from aseparate training set consisting of 80 images. As (4) depends
nonlinearly on τu and τp , traditional CRF learning algorithms[42] are not feasible and we resort to Bayesian optimization[43] for estimating the parameters, yielding wu1 = 1.2,
wu2 = 1.0, wp = 1.0, τu = 0.05 and τp = 0.08.
V. E VALUATION
We evaluate our approach using stereo sequences captured with our autonomous driving platform AnnieWAY. We
equipped the vehicle with two horizontally aligned hypercatadioptric cameras on top of the roof of the vehicle, a highprecision GPS/IMU system that delivers groundtruth motion
and a Velodyne laser scanner that provides 360◦ laser scanswith a vertical resolution of 64 laser beams.
A. Ground truth
We use the Velodyne laser scanner as reference sensor
for our quantitative evaluation. As we focus on static scenes
only, we are able to accumulate the laser point clouds(+/- 5 frames) using ICP point-to-plane fitting which yields
relatively dense ground truth depth maps (see Fig. 6(a)(top) for an illustration). The calibration between the catadioptric camera and the Velodyne laser scanner is obtainedby minimizing the reprojection error in the image frommanually selected correspondences. To evaluate the qualityof depth information depending on surface inclination, we
also labeled all horizontal and vertical planes and obtain the

0m
5m
dv

10m
15m

0◦

α

360◦

Fig. 5. Plane Hypotheses. This figure shows the virtual omnidirectional intensity image (left top) and the corresponding false color depth map from
SGBM (left bottom), the Hough space for vertical planes (middle) and the intensity and inverse depth image with three randomly selected planes (right)
corresponding to the colored maxima in the Hough space. Note how the cyan maximum describes a plane that is closer to the camera center (smaller dv )than the planes corresponding to the green and purple maxima. This can also be verified by looking at the plane visualizations on the right.

plane parameters using the Hough transformation presented
in Sec. IV-A with the ground truth depth maps as input(in contrast to the estimated ones used in our method). Our
dataset comprises 80 training and 72 test scenes in total.
B. Quantitative Results
We evaluate the proposed method against state-of-the-art
stereo vision algorithms. Our baselines include simple Block
Matching (BM), Semi-Global Matching (SGBM) [12] (inboth cases we made use of the OpenCV implementations), as
well as the more recently developed StereoSLIC algorithm[10]. To investigate the importance of the proposed planebased prior, we also implement a winner takes all (WTA)
plane selection strategy, which selects the best plane independently for each superpixel. Note that this corresponds tominimizing (4) while ignoring pairwise potentials ψp (s1 , s2 )and the horizontal prior ψu2 (s).
We compute the inverse depth error e = |Dgt − Dest | forevery pixel for which groundtruth is available. To guarantee
a fair comparison, we fill in missing values in the resulting
inverse depth images using background interpolation [12],[13]. We report the mean number of bad pixels and themean end-point error averaged over the full test set. A pixel
which has an inverse depth error e larger than 0.05 1/m isregarded as a bad pixel. Tab. I shows the mean percentage
of bad pixels and the mean end-point error for all algorithms
averaged over all 72 test images. The first row depicts theerrors for all pixels where depth ground truth is available,
while the other rows consider planar regions only. For WTA
we vary the threshold of our non-maxima suppression stage
between 50 and 500 (WTA 50 / WTA 500 in Tab. I), yielding
about 5 to 150 planes on average. For our method, we set
this threshold to a constant value of 150.
Our experiments show that the proposed method significantly outperforms the baselines. The difference is especiallypronounced for horizontal planes, but our method also decreases the number of bad pixels for vertical planes withrespect to all baseline methods.
C. Qualitative Results
Fig. 6 and 7 depict the inverse depth images for the
analyzed algorithms (b-f) and the inverse depth groundtruth
obtained from the Velodyne laser scanner (a). Colors represent distance, where green is close and blue denotes distantpoints. Alongside, we show the 3D reconstructions obtained
when reprojecting all pixels of the corresponding inverse
depth maps back into 3D (b-f). Note how our algorithm isable to produce much cleaner depth images and smoother
3D reconstructions. A random selection of challenging 3D
scenes reconstructed using our method is given in Fig. 8.
VI. C ONCLUSIONS AND F UTURE W ORK
In this paper we presented a method for high-quality omnidirectional 3D reconstruction from a single virtual inverse
depth image. We showed how efficient inference with planebased prior models is possible and leads to clean and easy
to interpret depth maps that outperform state-of-the art depth
estimation techniques in terms of 3D reconstruction error. In
the future, we plan to investigate possible extensions towards
integrating depth information from more than four views toallow for example for urban reconstructions at larger scales.
A PPENDIX
This appendix provides details of the spherical rectification
outlined in Sec. III-B. For clarity we only illustrate theprocess for the first (reference) camera. The mapping forthe second camera is obtained in a similar manner.
Let Io (u, v) denote the omnidirectional input image withpixel coordinates (u, v)T and let Is (θs , ϕs ) denote the rectified spherical image which depends on the azimuth angle
ϕs ∈ [0, 2π] and inclination angle θs ∈ [0, π] as illustrated
in Fig. 3. We obtainp
x2r + yr2
yrθs = arctan(9)ϕs = arctanxr
zr
where (xr , yr , zr )T = R · (x, y, z)T , R is a rotation matrix and the ray (x, y, z)T corresponds to pixel (u, v)T in
Io (u, v). The rectifying rotation matrix R is computed suchthat the epipoles coincide with the coordinate poles, i.e., allepipoles lie on the line connecting both camera centers. Thisis achieved by lettingR =[r1 , r2 , e11 ]r2=

yo − (eT11 y0 ) e11

r1=

r2 × r3

where yo denotes the y-axis of the original omnidirectional
camera system (before rotation) and e11 is the first epipolar
point as illustrated in Fig. 3. Note that this definition removes

Bad Pixels (%)
All Pixel
All Planes
Horizontal Planes
Vertical Planes

SGBM
11.89
13.41
17.45
2.52

BM
9.52
7.27
6.75
5.81

StereoSLIC
8.95
9.50
12.24
1.85

WTA 50 WTA 100
11.62
11.63
13.22
13.16
17.48
17.40
2.11
2.10

WTA 150 WTA 200
11.59
11.62
12.85
12.28
17.04
16.33
2.20
2.33

WTA 300
12.63
11.96
15.29
6.64

WTA 500
14.66
11.98
13.28
14.10

Ours
4.04
1.24
1.03
1.51

Mean Error (1/r) SGBM BM
All Pixel
0.026 0.022
All Planes
0.029 0.022
Horizontal Planes
0.034 0.023
0.008 0.013
Vertical Planes

StereoSLIC
0.021
0.022
0.026
0.008

WTA 50 WTA 100
0.029
0.029
0.033
0.033
0.038
0.038
0.008
0.008

WTA 150 WTA 200
0.029
0.029
0.032
0.031
0.037
0.036
0.009
0.009

WTA 300
0.030
0.030
0.034
0.016

WTA 500
0.031
0.030
0.032
0.023

Ours
0.013
0.009
0.010
0.008

TABLE I
Quantitative Analysis. T HIS TABLE SHOWS THE MEAN PERCENTAGE OF BAD PIXELS AND THE MEAN INVERSE DEPTH ERROR FOR ALL BASELINES
AND THE PROPOSED METHOD AVERAGED OVER ALL 72 TEST IMAGES . T HE FIRST ROW DEPICTS THE ERRORS FOR ALL PIXELS WHERE DEPTH GROUND
TRUTH IS AVAILABLE , WHILE THE OTHER ROWS CONSIDER PLANAR REGIONS ( OF A SPECIFIC TYPE ) ONLY.

(a) Groundtruth / Intensity Image(b) BM(c) SGBM(d) StereoSLIC(e) WTA(f) Ours

Fig. 6. Inverse Depth Maps and 3D Reconstructions. The figures show the inverse depth images and the resulting 3D reconstruction for the same
scene for the baseline algorithms (BM, SGM, StereoSLIC), for the best WTA result with threshold 150 and our MRF based plane estimation.

the remaining degree of freedom by ensuring that the rotatedy-axis is similar to the original one. The epipoles are
obtained from the essential matrix E = [t]× R which is
specified by the rigid motion [R|t] between both cameras.

R EFERENCES[1] A. Geiger, M. Lauer, F. Moosmann, B. Ranft, H. Rapp, C. Stiller, andJ. Ziegler, “Team annieway’s entry to the grand cooperative driving
challenge 2011,” TITS, 2012.
[2] M. Buehler, K. Iagnemma, and S. Singh, Eds., The DARPA Urban
Challenge, ser. Advanced Robotics, vol. 56, 2009.
[3] S. Baker and S. K. Nayar, “A theory of single-viewpoint catadioptric
image formation,” IJCV, 1999.

[4] C. Geyer and K. Daniilidis, “A unifying theory for central panoramicsystems and practical implications,” in ECCV, 2000.
[5] G. Schindler and F. Dellaert, “Atlanta world: An expectation maximization framework for simultaneous low-level edge grouping andcamera calibration in complex man-made environments,” in CVPR,
2004.
[6] Y. Furukawa, B. Curless, S. M. Seitz, and R. Szeliski, “Manhattanworld stereo,” in CVPR, 2009.
[7] ——, “Reconstructing building interiors from images,” in ICCV, 2009.
[8] A. Schwing and R. Urtasun, “Efficient exact inference for 3d indoor
scene understanding,” in ECCV, 2012.
[9] B. Zeisl, C. Zach, and M. Pollefeys, “Stereo reconstruction of building
interiors with a vertical structure prior,” in THREEDIMPVT, 2011.
[10] K. Yamaguchi, D. McAllester, and R. Urtasun, “Robust monocularepipolar flow estimation,” CVPR, 2013.
[11] B. Mičušı́k and J. Košecká, “Multi-view superpixel stereo in urban

(a) Groundtruth / Velodyne(b) BM(c) SGBM(d) StereoSLIC(e) WTA(f) Ours

Fig. 7. Inverse Depth Maps and 3D Reconstructions. The figures show the inverse depth images and the resulting 3D reconstruction for the same
scene for the baseline algorithms (BM, SGM, StereoSLIC), for the best WTA result with threshold 150 and our MRF based plane estimation.

environments,” IJCV, 2010.
[12] H. Hirschmüller, “Stereo processing by semiglobal matching andmutual information,” PAMI, 2008.
[13] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous
driving? the kitti vision benchmark suite,” in CVPR, 2012.
[14] A. Geiger, M. Lauer, C. Wojek, C. Stiller, and R. Urtasun, “3d traffic
scene understanding from movable platforms,” PAMI, 2014.
[15] D. Scaramuzza and A. Martinelli, “A toolbox for easily calibrating
omnidirectional cameras,” in IROS, 2006.
[16] C. Mei and P. Rives, “Single view point omnidirectional camera
calibration from planar grids,” in ICRA, 2007.
[17] L. Puig, J. Bermúdez, P. Sturm, and J. J. Guerrero, “Calibrationof omnidirectional cameras in practice: A comparison of methods,”
CVIU, 2012.
[18] M. Schönbein, T. Strauss, and A. Geiger, “Calibrating and centering
quasi-central catadioptric cameras,” in ICRA, 2014.
[19] A. Murillo, J. Guerrero, and C. Sagues, “Surf features for efficientrobot localization with omnidirectional images,” in ICRA, 2007.
[20] C. Valgren and A. J. Lilienthal, “Sift, surf and seasons: Long-term
outdoor localization using local features.” in EMCR, 2007.
[21] A. Pagani and D. Stricker, “Structure from motion using full spherical
panoramic cameras,” in ICCV Workshops, 2011.
[22] J.-P. Tardif, Y. Pavlidis, and K. Daniilidis, “Monocular visual odometry
in urban environments using an omnidirectional camera,” in IROS,
2008.
[23] A. Rituerto, L. Puig, and J. J. Guerrero, “Visual slam with anomnidirectional camera,” in ICPR, 2010.
[24] T. Svoboda and T. Pajdla, “Epipolar geometry for central catadioptriccameras,” IJCV, 2002.
[25] J. Gluckman, S. K. Nayar, and K. J. Thoresz, “Real-time omnidirectional and panoramic stereo,” in In DARPA Image Understanding
Workshop, 1998.
[26] S. Yi and N. Ahuja, “An omnidirectional stereo vision system using
a single camera,” in ICPR, 2006.
[27] R. Bunschoten, B. J. A. Krse, and N. A. Vlassis, “Robust scene reconstruction from an omnidirectional vision system.” IEEE T. Robotics
and Automation, vol. 19, no. 2, pp. 351–357, 2003.

[28] J.-J. Gonzalez-Barbosa and S. Lacroix, “Fast dense panoramic stereovision.” in ICRA, 2005.
[29] S. B. Kang and R. Szeliski, “3-d scene data recovery using omnidirectional multibaseline stereo,” IJCV, 1995.
[30] S. Fleck, F. Busch, P. Biber, W. Strasser, and H. Andreasson, “Omnidirectional 3d modeling on a mobile robot using graph cuts,” in ICRA,
2005.
[31] C. G. Kostas and K. Daniilidis, “Conformal rectification of omnidirectional stereo pairs,” in Omnivis, 2003.
[32] J. Fujiki, A. Torii, and S. Akaho, “Epipolar geometry via rectificationof spherical images,” in Computer Vision/Computer Graphics Collaboration Techniques, 2007.
[33] S. Li, “Real-time spherical stereo,” in ICPR, 2006.
[34] Z. Arican and P. Frossard, “Dense disparity estimation from omnidirectional images.” in AVSS, 2007.
[35] M. Lhuillier, “Toward flexible 3d modeling using a catadioptriccamera,” in CVPR, 2007.
[36] S. Yu and M. Lhuillier, “Surface reconstruction of scenes using acatadioptric camera,” in Computer Vision/Computer Graphics Collaboration Techniques, 2011.
[37] M. Schönbein, H. Rapp, and M. Lauer, “Panoramic 3d reconstruction
with three catadioptric cameras,” in IAS, 2013.
[38] E. Rosten and T. Drummond, “Machine learning for high-speed cornerdetection.” Springer Berlin Heidelberg, 2006, pp. 430–443.
[39] M. Calonder, V. Lepetit, M. Ozuysal, T. Trzcinski, C. Strecha, andP. Fua, “Brief: Computing a local binary descriptor very fast,” PAMI,
2012.
[40] A. Geiger, J. Ziegler, and C. Stiller, “StereoScan: Dense 3d reconstruction in real-time,” in IV, 2011.
[41] A. Neubeck and L. V. Gool, “Efficient non-maximum suppression,” in
ICPR, 2006.
[42] T. Hazan and R. Urtasun, “A primal-dual message-passing algorithm
for approximated large scale structured prediction,” in NIPS, 2010.
[43] H. L. Jasper Snoek and R. P. Adams, “Practical bayesian optimizationof machine learning algorithms,” in NIPS, 2012.

Fig. 8. 3D Reconstruction: This figure shows 3D reconstructions for different urban scenarios obtained when reprojecting the inverse depth maps
produced by our method into 3D. Note that the viewpoint of the rendered 3D point clouds deviates significantly from the viewpoint of the four cameras.