1

3D Traffic Scene Understanding
from Movable Platforms
Andreas Geiger

Martin Lauer

Christian Wojek

Christoph Stiller

Raquel Urtasun

Abstract—In this paper, we present a novel probabilistic generative model for multi-object traffic scene understanding from movable
platforms which reasons jointly about the 3D scene layout as well as the location and orientation of objects in the scene. In particular, thescene topology, geometry and traffic activities are inferred from short video sequences. Inspired by the impressive driving capabilitiesof humans, our model does not rely on GPS, lidar or map knowledge. Instead, it takes advantage of a diverse set of visual cues in theform of vehicle tracklets, vanishing points, semantic scene labels, scene flow and occupancy grids. For each of these cues we propose
likelihood functions that are integrated into a probabilistic generative model. We learn all model parameters from training data using
contrastive divergence. Experiments conducted on videos of 113 representative intersections show that our approach successfully
infers the correct layout in a variety of very challenging scenarios. To evaluate the importance of each feature cue, experiments using
different feature combinations are conducted. Furthermore, we show how by employing context derived from the proposed method weare able to improve over the state-of-the-art in terms of object detection and object orientation estimation in challenging and clutteredurban environments.
Index Terms—3D Scene Understanding, Autonomous Driving, 3D Scene Layout Estimation

F

1

I NTRODUCTION

R

ECENT

scientific and engineering progress in advanced driver assistance systems and autonomous
mobile robots make us believe that cars will be able
to drive without human intervention in the near future [31]. While autonomous driving on highways has
been successfully demonstrated for several decades [21],navigating urban environments remains an unsolved
problem. This is mainly due to the complexity of urban
traffic situations and the limited capabilities of current
onboard sensors and image understanding algorithms.
In recent years, the need for solving the onboard scene
understanding problem has been bypassed thanks to
the use of manually labeled maps and GPS localization
systems, e.g., competitors at the DARPA Urban Challenge [13] or Google’s autonomous car. While leading toimpressive solutions, this approach suffers from the factthat up-to-date sub-meter accurate maps will most likely
be impossible to acquire in practice. Furthermore, GPSsignals are not always available, and localization canbecome imprecise in the presence of skyscrapers, tunnels
or jammed signals. We believe that safe operation of anautonomous car can only be guaranteed if the car caninterpret its environment reliably from its own sensory• A. Geiger is with the MPI for Intelligent Systems in Tübingen and with
the Institute of Measurement and Control, KIT, Germany.
E-mail: andreas.geiger@tue.mpg.de• M. Lauer and C. Stiller are with the Institute of Measurement and Control,Karlsruhe Institute of Technology, Germany.
E-mail: {lauer,stiller}@kit.edu• C. Wojek is with Carl Zeiss Corporate Research, Germany.
E-mail: christian.wojek@zeiss.com• R. Urtasun is with the Toyota Technological Institute at Chicago, USA.
E-mail: rurtasun@ttic.edu

input just like a human driver can navigate unknown
environments even without maps [12]. Video cameras
are particularly appealing sensors in this context as
they are cheap and often readily available in modern
vehicles. But what makes the interpretation of urban
traffic scenarios so challenging compared to highways
and rural roads?
First, the traffic situations are very complex. Manydifferent traffic participants may be present and thegeometric layout of roads and crossroads is more variable than the geometry of highways. Unfortunately, theavailability of unambiguous visual features is limited
since road markings and curb stones are often missing or
occluded. Difficult illumination conditions such as castshadows caused by vegetation or infrastructure easily
confuse image processing algorithms. Furthermore, thelimited aperture angle of onboard cameras, their low
mount point and the limited depth perception of stereo
make the inference problem very difficult. As a consequence, only objects close to the observer can be locatedreliably.
In this paper, we present a method that is able torobustly deal with the inherent difficulties in onboard
recognition of urban intersections. Our approach reasons
jointly about the 3D scene layout of intersections aswell as the location and orientation of objects in thescene. We refer the reader to Fig. 1 for an illustration.
Towards this goal we exploit a larger number of visualcues than existing approaches, which typically rely onroad markings that are often unavailable. In particular,
we propose five different visual cues, which describe
the static environment encoded in terms of occupancy
grids, semantic labels and vanishing points, as well as
the dynamic information of vehicles encoded by scene

2
Vehicle Tracklets

Unobserved

Vanishing Points

Occupied
Freespace

Scene Labels

Sky
Background
Foreground

3D Scene Flow

Observer

Occupancy Grid

Monocular Features

Stereo Features (Bird's eye view)

Inference Result

Fig. 1: 3D Intersection Understanding. Our system makes use of monocular (left) and stereo (middle) feature cues to infer the road layout andthe location of traffic participants in the scene (right) from short video sequences. The observer is depicted in black.

flow and vehicle tracklets. As each of these features
might be ambiguous and lead to wrong interpretationsof the scene, we introduce a probabilistic model that integrates all cues in a principled manner. Our approach iscomposed of a scalable geometry model for intersections
and a proabilistic model that relates the geometry of anintersection to the visual features. We derive learningand inference procedures for this model and demonstrate its performance on video sequences from realworld intersections. Our dataset and a MATLAB/C++
reference implementation is made publicly available.1
A preliminary version of this paper appeared in [30],[33]. Herein, we unify both models and perform a greatly
enriched analysis on a variety of different feature combinations, yielding novel insights into the importanceof the individual feature cues. Moreover, we propose aprincipled way to learn the parameters of our model.
As the partition function of our joint distribution is
intractable, we re-formulate the problem in terms ofa Gibbs random field and derive all required potentials. We show how the gradients of this function canbe approximated by drawing samples from the model
distribution, and learning can be simply performed bygradient ascent on the log-likelihood.
This paper is organized as follows. Section 2 gives an
overview over related work. In Section 3 we present our
probabilistic intersection model as well as our learningand inference procedures. Section 4 gives details on
the computation of the visual features. Finally, Section
5 describes the data collection process as well as our
experiments. Conclusions are drawn in Section 6.

2

R ELATED W ORK

In 1986, Ernst Dickmanns and his team equipped a van
with cameras and demonstrated the first self-driving car
on well-marked streets (i.e., highways) without traffic[21]. Motivated by this success, several efforts have been
conducted towards autonomous driving in real traffic[10], [29], [51] triggering research on lane estimation [3],1. http://www.cvlibs.net/projects/intersection[18], [21], [41], [49] and road detection [1], [2], [17] algorithms. However, simplifying assumptions were madesuch as low-traffic environments, manual longitudinal
control or human intervention for lane changes andhighway exits. While the DARPA Urban Challenge [13]came closer to challenging urban traffic situations, thestreets were wider than usual, the field of view was unobstructed and only a very limited number of traffic participants were present. Furthermore, GPS in combination
with manually annotated maps was neccessary for localization and navigation. Subsequently, Google gathereda team of engineers to equip a Toyota Prius with selfdriving capabilities. Similarly to the participants of the
Urban Challenge, Google’s driver-less car is equippedwith a Velodyne 3D laser scanner for perception andrequires manually annotated maps at lane-level accuracy
for path planning. In contrast, the approach presented
in this paper aims at analyzing complex and clutteredscenes in the absence of maps or 3D point clouds.
For a long time intersection understanding has beenrecognized as a difficult problem [16], [22], [24], [35], [53].
For instance, Luetzeler and Dickmanns [48] extract local
image features and match these to a T-shaped intersection model that involves several parameters. They use
a multifocal active camera system that allows to detect
missing lane boundary segments as an indicator forintersections. Subsequently, the segments are matchedwith a road map and tracked over time using a Kalman
Filter. Zhu et al. [64] use a laser scanner for intersection
recognition by searching for positions in front of thevehicle that may serve as a source for long virtual beams
that stay collision free in a 2D static grid map. While all
existing approaches focus on a small number of simple
features such as road markings or 2D grids, in thispaper we argue that a bigger picture of the scene that
integrates several sources of information in a robust way
is able to handle challenging real-world scenarios. In
contrast to [64], our method is also able to deal withstereo information, which is much noisier than laserbased measurements.
In computer vision, a large body of work has focused

3

on estimating 3D from single images [36], [39], [50],[54], [55]. Often, a Manhattan world [7], [43] is assumed
to infer vanishing points from line segments. In addition, several methods try to infer the 3D locations ofobjects in outdoor scenarios [6], [20], [38]. In [14], [23],[60] the camera pose and the location of objects areinferred jointly. Structure-from-motion point clouds areleveraged in [11] to segment and semantically classify
the environment. Recently, Singh and Kosecka [56] haveproposed a semantic model for urban scene recognition
segmenting buildings, road, sky, cars and trees. They
employ a trained classifier for pixelwise scene classification in a panoramic image. Unfortunately, most of
the existing 3D layout estimation techniques are mainly
qualitative, do not model object dynamics, suffer fromclutter and lack the level of accuracy neccessary for realworld applications such as autonomous driving. Existing
methods that take objects into account usually model thescene in terms of a simple ground plane and thus are
not able to draw conclusions from the complex interplayof the objects with the larger scene layout. In contrast,
we propose a method that is able to extract accurate
geometric information by reasoning jointly about staticand dynamic elements as well as their interplay.
For a long time, dynamic objects have been consideredeither in isolation [5], [26], [27], [52] or using simple
motion [9], [23], [42], [47], [63] or social interaction [15],[62] models. For example, Choi et al. [15] introduce ahierarchy of activities, modeling the behavior of groups.
Methods for unsupervised activity recognition and abnormality detection [44], [59] are able to recover spatiotemporal dependencies using video sequences from astatic camera mounted on top of a building. While
promising results have been shown, the interplay ofobjects with their environment is neglected and the focus
is put on surveillance scenarios with a fixed camera
viewpoint, limiting their applicability. In contrast, themethod developed in this paper infers semantics at
a higher level such as multi-object traffic patterns atintersections, in order to improve layout and object
estimation. Importantly, we do inference at scenes that
we have never seen before and our viewpoint is substantially lower compared to the surveillance scenario,
which renders the problem very challenging.

3

U RBAN S CENE U NDERSTANDING

In this paper, we tackle the problem of understanding
complex 3D traffic scenes from short video sequences.
In particular, we are interested in inferring the scenelayout (e.g., number and location of streets) as well as the
3D location and orientation of traffic participants (e.g.,cars) from short video sequences captured onboard adriving vehicle. We assume a flat road surface and model
the scene layout and all objects in the road coordinate
system, which is located directly below the left camerain the last frame of the video using the yaw angle andcoordinate axis illustrated in Fig. 2 (right) and Fig. 7.

1

2

3

4

5

6

7

Fig. 2: Topology Model (left) and Geometry Model (right) in bird’s
eye perspective. The gray shaded areas (left) illustrate the flexibility of
the crossing street (α).

The number and location of streets and vehicles are related to the image observations through our probabilistic
model. In this section, we introduce all relevant notation,
develop the geometric and probabilistic models and describe our inference and parameter learning procedure.
3.1

Geometric Model

Our geometric model is inspired by typical traffic scenes.
We assume that the layout of the scene is dominatedby up to four roads intersecting at a single location,
the intersection center. All vehicles are either parked
at the side of the road or drive on lanes and respectsome basic rules such as right-handed traffic. Lanes aremodeled using B-splines. All inbound and outboundstreets are connected with a lane, except for U-turns.
Road boundaries determine the border between drivable
regions and areas that are likely to contain buildings andinfrastructure.
Let R = {κ, c, w, r, α} be the set of random variables describing the road layout, where κ ∈ {1, . . . , 7}
denotes the topology of the intersection (see Fig. 2)
and c = (x, z)T ∈ R2 is the intersection center. Letw ∈ R+ be the street width and r ∈ [− π4 , + π4 ] theglobal rotation, i.e., the observer’s yaw orientation withrespect to the incoming street. We define α ∈ [− π4 , + π4 ]as the crossing angle, i.e., the relative orientation of thecrossing street with respect to the incoming street. For
simplicity, we assume that all opposing intersection arms
are collinear and all streets share the same width. Allvariables are illustrated in Fig. 2. Note that our model
is able to express straight roads, turns, three-armed andfour-armed intersections.
For simplicity, we restrict our focus to two lanes per
street, one incoming and one outgoing lane for eachintersection arm. As vehicles are allowed to cross theintersection in any possible direction, we have K(K − 1)lanes for a K-armed intersection. For each street wemodel two parking areas, one at the left side and one
at the right side, yielding 2K parking areas in total. Two(out of six) lanes of a 3-armed intersection as well as one
parking area are illustrated in Fig. 3 (left).
Lane centerlines are modeled using quadratic Bsplines [19] governed by five control points {q1 , . . . , q5 },4

Fig. 3: Lane/Parking Areas (left) and Lane B-splines (right). Lane centerlines are defined via B-splines and parking areas are located at eachroad side. For tractability of our tracklet likelihood, all lanes/parking
areas are discretized at 1 meter intervals (see Section 3.3.2).

which are located at the center of the lane as illustrated inFig. 3 (right). The knot vector controlling the shape of the
B-splines was chosen to be (0 0 0 0.1 0.9 1 1 1)T , forcing
the spline to interpolate all but the central control point.
Empirically this resulted in realistic curvatures. Given
all lane splines and all parking areas, we equidistantly
define discrete vehicle locations s at 1m intervals as
illustrated in Fig. 3 (left) to allow for efficient inferenceof the scene layout and the vehicle locations.

Fig. 4: Probabilistic Graphical Model corresponding to the joint
distribution over image evidence E and road layout R in Eq. (1).

model does not explicitly reason about vehicle velocities.
The occupancy grid O = {ρ1 , . . . , ρNo } is represented by
No cells of size no × no meters. Each cell ρi ∈ {−1, 0, +1}
can be either free (−1), occupied (+1) or unobserved (0). We
postpone the discussion on feature extraction to Section
4 and describe our probabilistic model in this section.
3.3

3.2

Image Evidence

Besides the geometric model, we define a probabilistic
model to explain the image evidence E = {T , V, S, F, O}
with vehicle tracklets T , vanishing points V, semantic
scene labels S, scene flow F and occupancy grid O. Allfeatures are mapped into the last frame of each sequence
using visual odometry.
Let T = {t1 , . . . , tNt } be the set of vehicle tracklets
that have been detected in the sequence. A vehicle
tracklet t is defined as a sequence of object detections
projected into bird’s eye perspective t = {d1 , . . . , dMd }.
Here, d = (fd , md , Sd , od ) denotes a single object detection with fd ∈ N the frame number, md ∈ R2 , Sd ∈ R2×2
the mean and covariance describing the object locationin road coordinates and od ∈ R8 the probability of theobject facing into each of eight possible directions.
Let V = {v1 , . . . , vNv } be the set of vanishing points.
We detect up to two vanishing points (Nv ∈ {0, 1, 2})
and represent each vanishing point by a single rotation
angle around the yaw axis of the road coordinate system,
i.e. vi ∈ [0, π). The vertical vanishing point is noninformative for our task and not considered here. Let
S = {s1 , . . . , sNs } be the set of semantic labels. Here,Ns is the number of image patches of size ns × ns pixels
on a regular grid and si ∈ ∆2 is a discrete probability
distribution over the semantic categories road, backgroundand sky. Both, vanishing points and scene flow, are computed in the last frame of each sequence as this yielded agood compromise between the quality of the results andcomputation time. Further, denote F = {f1 , . . . , fNf } theset of scene flow vectors capturing the 3D motion of thescene not explained by the observer’s egomotion. Eachflow vector f = (pf , qf ) is defined by its location pf ∈ R2
and velocity qf ∈ R2 on the road plane. All velocity
vectors are normalized to kqf k2 = 1 as our scene flow

Probabilistic Model

We assume that all observations E = {T , V, S, F, O}
are conditionally independent given the road layout R.
As a consequence, the joint distribution over the image
evidence E and the road parameters R factorizes asNt
Yp(E, R|Θ) = p(R|Θ)

Nv
Yp(ti |R, Θ)

i=1| {z } |
Prior

×

Ns
Y

p(si |R, Θ)

i=1|

p(fi |R, Θ)

{z

} |

{z

}

Vanishing PointsNoYp(ρi |R, Θ)

(1)

i=1

i=1

Scene Labels

} |

{z

Vehicle Tracklets

NfY

p(vi |R, Θ)

i=1

{z

Scene Flow

} |

{z

}

Occupancy Grid

where Θ denotes the set of all parameters in our model.
This is illustrated in the graphical model of Fig. 4.
3.3.1 Prior
We define the prior on road parameters R asp(R|Θ) = p(κ|Θ) p(c, r, w|κ, Θ) p(α|κ, Θ)

(2)

with
κ ∼ Cat(ξ p )−1µp(κ) , Λ(κ)
p(c, r, log w)T |κ ∼ N

α|κ ∼ fκ (α, σα )λp

(3)
(4)
(5)

where Cat(·) denotes the categorical distribution andc, r and w are modeled jointly to capture correlations
between the variables. The width w is modeled using alog-Normal distribution to enforce positivity. Empirically
we found α to be highly multi-modal. We thus model α
using kernel density estimation with a Gaussian kerneland bandwidth σα . The scalar λp controls the importanceof the crossing street prior. All parameters are learned
from training data as detailed in Section 3.5.

5

3.3.2

Vehicle Tracklets

Assuming a uniform prior on all lanes and parking areas
we model vehicle tracklets as
LX

p(t, l|R)(6)

p(t, l|R) = p(l|R) p(t|l, R) ∝ p(t|l, R)(7)

p(t|R, Θ) =Fig. 5: Semantic Scene Label Likelihood. Intuitively, the semantic
likelihood measures the ’overlap’ of the rendered model hypothesis(left) with the semantic labels estimated by a classifier (right).

l=1

where the tracklet index i and the dependency on theparameters Θ have been dropped for clarity and the latent lane (or parking spot) index l has been marginalized.
To evaluate the tracklet posterior for lanes pl (t|l, R), we
discretize the lane spline at 1 meter intervals and augment the observation model with an additional discretelatent variable s per object detection d, which indexes
the location on the lane, as illustrated in Fig. 3. We
employ a simple left-to-right Hidden Markov Modelto model the dynamics. Marginalizing over all hidden
states s1 , . . . , sMd yields
X
pl (t|l, R) =p(s1 ) pl (d1 |s1 , l, R)

For the object orientation likelihood, we impose acategorical distribution over 8 object orientations odp(od |s, l, R) =×p(sj |sj−1 ) pl (dj |sj , l, R)

(8)

j=2

where Md denotes the number of object detections in the
tracklet. We allow tracklets to start anywhere on the lane
with equal probability. Our motion model is simple, yeteffective. By constraining all tracklets to move forwardwith uniform probability, i.e.,1
if sj ≥ sj−1
(9)p(sj |sj−1 ) = Ml −sj−1 +10otherwise
lanes on crossing streets can be distinguished purely
based on the vehicle’s motion. Here, Ml is the numberof spline points on lane l. The emission probability forlanes pl (d|s, l, R) is factorized into the object’s locationand the object’s orientationpl (d|s, l, R) = p(md |s, l, R, Sd ) p(od |s, l, R)

(10)

(13)

i=1

where ϕt (s, l, R) ∈ {1, . . . , 8} selects the orientation bincorresponding to the viewpoint relative to the observer:
The relative viewing direction is computed from thetangent of lane l at spline point s. Intuitively, Eq. (13)encourages lane associations such that the estimatedvehicle orientation and the direction of the lane coincide.
As parking cars are static, the tracklet likelihood forparking areas reduces to

s1 ,...,sMd
Md
Y

8
Y
[ϕ (s,l,R)=i]od,itpp (t|l, R) =

Md
XY
s

p(s) pp (dj |s, l, R)

(14)

j=1

assuming a uniform prior on the location s. Note that
we do not make any assumptions about the orientationof a parked car. Thus the emission probability becomes1
p(µd |s, l, R, Sd )8with p(µd |s, l, R, Sd ) defined in Eq. (11).
pp (d|s, l, R) =(15)

3.3.3 Vanishing Points
We model the vanishing likelihood over v ∈ [0, π) as
p(v|R, Θ) ∝ ζv + (1 − ζv ) exp (−λv φv (v, R, Θ))(16)with orientation error
φv (v, R, Θ) = 1 − cos(2v − 2ϕv (R))

(17)

(11)

derived from the von Mises distribution, which is acontinuous probability distribution on the circle. Here,ζv is a small constant capturing outlier detections, ϕv (R)is the orientation of the closest street, based on thecurrent road model configuration R, and λv is a precisionparameter that controls the importance of this term.

with inlier and outlier distributions defined by
(φt − md )T S−1
d (φt − md )pin (md |s, l, R, Sd ) ∝ exp −
2


T
m md
pout (md |s, l, R) ∝ exp − d 2
(12)
2σout

3.3.4 Semantic Scene Labels
The semantic scene label likelihood [33] for each ns × ns
pixels image patch s is modeled as
λs
ws,φs (R) · sφs (R)(18)
p(s|R, Θ) ∝ exp
Ns

respectively. Here, φt (s, l, R) ∈ R2 denotes the 2D location of spline point s on lane l according to the B-spline
model presented in Section 3.1, ζt is the outlier probability and σout is a parameter controlling the ’spread’ ofthe outlier distribution.

where Ns is the total number of image patches (seeSection 4.3) and λs is a parameter controlling the importance of this cue. φs (R) ∈ {1, 2, 3} selects the class labelof s according to a ’virtual’ segmentation of the scene
which depends on the reprojection of the road layout

The object location is modeled as a Gaussian mixturep(md |s, l, R, Sd ) = (1 − ζt ) pin (md |s, l, R, Sd )+ ζt pout (md |s, l, R)

6Fig. 6: Scene Flow (left) and Occupancy Grid Likelihood (right).
The proposed scene flow likelihood encourages flow vectors to agreewith the lane geometry. The geometric occupancy prior (right) can beenvisioned as a ’template’ of freespace and occupied areas.

R such that sφs (R) returns the probability of the classlabel given a semantic classifier. This is illustrated inFig. 5. Additionally, the weight vector ws ∈ R3 controls
the importance of each semantic class. Our model uses
three classes: foreground (e.g., road), background (e.g.,buildings, vegetation) and sky. To simplify matters, we
assume that the background (i.e., buildings, trees) starts
directly behind the curb of the road and buildings reacha height of four stories on average, thereby defining thebackground area which separates the sky from the roadregion. Facades adjacent to the observer’s own street
are not considered. Despite these approximations, wehave observed that many inner-city scenes in our dataset
follow this scheme closely.
3.3.5

Scene Flow

In contrast to the tracklet observations, the 3D scene
flow likelihood directly explains all moving objects in thescene given the road layout R. Thus, moving objects that
do not fit the appearance model of the car detector (e.g.,trucks, tractors, quad bikes, motorbikes) are consideredhere as well. The probability of a scene flow vector
depends on its proximity to the closest lane and on how
well its velocity vector aligns with the tangent of therespective B-spline at the corresponding foot pointp(f |R, Θ) ∝ φf (f , R, Θ)

1
Nf

(19)

where



kpf k22φf (·) = ζf exp − 2+ (1 − ζf ) exp −φ̃f (f , R, Θ)
2σout
φ̃f (·) = −λf 1 kpf − ϕf (pf , R)k22 − λf 2 (1 − qTf ϕ̃f (pf , R))with parameters ζf , λf 1 , λf 2 , σout . Here, ζf accounts foroutliers and λf 1 and λf 2 control the importance of thelocation and the orientation agreement. Nf normalizes
for the number of scene flow vectors and, similar tothe vehicle tracklet model from Section 3.3.2, σout denotes the width of the outlier distribution. The functions
ϕf (pf , R) ∈ R2 and ϕ̃f (pf , R) ∈ R2 return the splinefoot point and tangent vector at the location closest
to pf , respectively. This is illustrated in Fig. 6. The
dependencies are modeled as a hard mixture, i.e. for eachflow vector we select the spline l that maximizes Eq. (19).

3.3.6 Occupancy Grid
We assume that the road area should coincide with freespace while non-road areas may be covered by buildings
or vegetation. Thus, we use stereo information to compute an occupancy grid, which represents occupied andfree-space in bird’s eye perspective, see Fig. 1 (middle)
for an illustration. The occupancy likelihood of each cell
ρ in the grid is modeled as followsλoρ · φo (ρ, R)(20)
p(ρ|R, Θ) ∝ exp
No
where φo (ρ, R) ∈ {wo,1 , wo,2 , wo,3 } is a mapping that for
any cell ρ ∈ {−1, 0, +1} returns the value (or weight) ofa model-dependent geometric prior expressing the belief
on the location of free space (i.e., road) and buildings
alongside the road. The geometric prior is illustrated
in Fig. 6 for the case of a right turn. Intuitively, it
encourages free space where the road is located and obstacles elsewhere, with a preference towards the roadsideregion. λo controls the strength of this term.
3.4

Inference

As inference is a key component for learning we startwith a discussion on how to obtain samples from thejoint distribution using Markov Chain Monte Carlo techniques. Given the image evidence E, we are interested
in determining the underlying road layout R and thelocation of cars C = {(l, s)} in the scene, where l denotes
the lane index and s contains the spline points of all
detections in a tracklet. Unfortunately, the posteriors
involved in this computation have no analytical solution and cannot be computed in closed form. Thus weapproximate them using Metropolis-Hastings sampling[4]. To keep computations tractable, we split the problem into two sub-problems: We first estimate R whilemarginalizing C as
X
R̂ = argmax p(R|E, Θ) = argmaxp(R, C|E, Θ) (21)R

R

C

through Eq. (1), Eq. (8) and Eq. (14). Given an estimateof R, we infer the object locations C asCˆ = argmax p(C|E, R, Θ)

(22)

C

Both steps are detailed in the following.
3.4.1 Inferring the Road Layout
For maximizing Eq. (21), we draw ninf samples using
Markov Chain Monte Carlo and select the one withthe highest probability as our MAP estimate R̂. We
exploit a combination of local, inter-topology and global
moves to obtain a well-mixing Markov chain. While local
moves modify R only slightly, global moves sample R
directly from the prior. This ensures a quick traversalof the search space, while still exploring local modes.
For local moves we choose symmetric proposals in theform of Gaussians centered on the previous state. Table

7

Local Metropolis Proposals (33%)
1. Vary center of crossroads c (σc )2. Vary width of all roads w (σw )3. Vary angle of crossing street α (σα )4. Vary overall orientation r (σr )5. Vary center c and width w jointly
6. Vary center c, width w, angle α and rotation r jointlyInter-Topology Metropolis Proposals (33%)
7. Re-sample κ uniformly
Global Metropolis-Hastings Proposals (33%)
8. Re-sample all parameters R = {κ, c, w, r, α} from the priorwhere Ψ(Ed , Rd , Θ) is the sum of a set of potential functions {ψi } corresponding to the probability distributionsin Section 3.3 which are described at the end of thissection. The log-likelihood function is given byL(E, R|Θ) =3.4.2 Inferring the Location of Objects
Given the road model R, we are now interested in recovering the location of cars C = {(l1 , s1 ), . . . , (lNt , sNt )}.
Conditioned on R, all tracklets become independent andthe inference problem decomposes into sub-problems.
Neglecting the tracklet index i and the dependency onΘ for clarity, and observing that p(t) is constant, ˆl can beinferred by computing the marginal over object locations
ˆl = argmax p(l|t, R) = argmax p(t, l|R)
l

(23)

l

with p(t, l|R) defined by Eq. (7). Given l, we haveŝ1 , . . . , ŝMd = argmax pl (t, s1 , . . . , sMd |l, R)(24)

s1 ,...,sMd

which can be easily inferred using Viterbi decoding for
hidden Markov models [8].
3.5

Learning

The parameters Θ = {λp , ξ p , λt , λv , λs , λf 1 , λf 2 , λo } ofour model are learned from held-out training data using
maximum likelihood and contrastive divergence [37].
Let (E, R) be a training set with E = {E1 , . . . , ED }denoting the image evidence and R = {R1 , . . . , RD } theannotated road layout of each sequence. The parameterset Θ̂ maximizing the likelihood of the data is given byΘ̂ = argmax p(E, R|Θ)

(25)

Θ

with
p(E, R|Θ) =D
Yp(Ed , Rd |Θ)

(26)

=−

Taking the partial derivative of L(E, R|Θ) with respect
to a parameter θi ∈ Θ, we obtainD∂∂Ψ(Ed , Rd , Θ) +log Zd (Θ)
∂θi
∂θi
d=1
(29)
While the first term is easy to evaluate, the second term
can be rewritten asZ∂
1∂
log Zd (Θ) =exp(−Ψ(Ed , R, Θ))dR∂θi
Zd (Θ)∂θi∂=−
Ψ(Ed , R, Θ)(30)∂θip(Ed ,R|Θ)X∂L(E, R|Θ)
=−
∂θi



where h·ip(·) denotes the expected value with respect top(·). Note that in contrast to [37] the potentials Ψ additionally depend on Ed in our case. While it is impossible
to evaluate this expression exactly, it can be approximated by drawing samples using MCMC as describedin Section 3.4. Sampling exhaustively from the model
distribution is computationally prohibitive. We therefore
make use of contrastive divergence [37], and instead of
running the Markov chain until convergence, we only
run it for a small number of nlearn iterations per gradient
update. By starting the chain at the training data weobtain samples in all places where we want the model
distribution to be accurate. Running the MCMC sampler
for a couple of iterations already draws the samples
closer to the model distribution. For more details andderivations we refer the interested reader to [28], [37].
Energy Potentials: The joint potential Ψ(E, R, Θ) in Eq.
(27) decomposes asΨ(E, R, Θ) = ψp (R, Θ) + ψt (T , R, Θ) + ψv (V, R, Θ)+ ψs (S, R, Θ) + ψf (F, R, Θ) + ψo (O, R, Θ)
using the same subscript notation as in Eq. (1). Taking
the logarithm of Eq. (2) we obtainψp (R, Θ) = −λp log fκ (α) −7X[κ = i] log ξp,ii=1

Unfortunately, maximizing Eq. (26) directly for Θ isintractable due to the partition function. Instead, werewrite p(Ed , Rd |Θ) in terms of a Gibbs random field
1
exp (−Ψ(Ed , Rd , Θ))
Zd (Θ)(Ψ(Ed , Rd |Θ) + log Zd (Θ)) (28)

d=1

d=1

p(Ed , Rd |Θ) =log p(Ed , Rd |Θ)

d=1
D
X

TABLE 1: Metropolis-Hastings-Proposals for Inference. We randomly
chose a local, inter-topology or global move with uniform probability.

1 gives an overview of the move categories which are
selected at random. Each sample requires the evaluationof p(R|E, Θ) up to a normalizing constant. The marginalization in Eq. (8) can be carried out efficiently using theforward algorithm for hidden Markov models. To avoid
trans-dimensional moves, we include α in all models.

D
X

(27)

1
(κ)
T (κ)
+ φp (R, µ(κ)
p ) Λp φp (R, µp ) (31)
2
(κ)

(κ)

with φp (R, µp ) = (c, r, log w)T − µp . We set µp ∈ R4
and Λp ∈ R4×4 to their empiric marginals and learn
ξ p , λp ∈ Θ using the approach described in Section 3.5.

8The derivatives are readily derived from Eq. (31). Taking
the logarithm of Eq. (6) - (20) we obtain:Nψt = −

L

t
X
λt X
log
p(ti , l|R)
Nt i=1l=1

ψv = −Nv
X

log [ζv + (1 − ζv ) exp (−λv φv (vi , R, Θ))]i=1

ψs = −

λs
Ns

Ns
X

ws,φs (R) si,φs (R)

i=1
Nf

No
1 X
λo X
ψf = −
log φf (fi , R, Θ), ψo = −ρi φo (R)Nf i=1
No i=1

Here, we have added an additional degree of freedom
λt to the tracklet potential ψt , which accommodates forviolations of the naı̈ve Bayesian observation model andcontrols the relative strength of the tracklet feature withrespect to the prior and all other features.

4

I MAGE E VIDENCE

This section describes the feature cues used by our
probabilistic model described in Section 3.3. Using motion information from visual odometry, we represent
all features in the reference coordinate system which
is located on the road surface below the left camera
coordinate system in the last frame of each sequence,
as illustrated in Fig. 7. While vehicle tracklets, vanishingpoints and semantic scene labels can be computed froma monocular video stream, the scene flow and occupancy
grid features require a stereo setup. The importance ofeach of these cues is investigated in our experiments.
4.1

Fig. 7: Projection of 2D Object Detections into 3D. By learning thestatstics of the vehicle dimensions, we are able to relate the 2D object
bounding box to the location of the object in 3D.

Vehicle Tracklets

We define a tracklet as a set of object detections, projected into bird’s eye perspective t = {d1 , . . . , dMd } withd = (fd , md , Sd , od ). Here, fd ∈ N is the frame number
and md ∈ R2 , Sd ∈ R2×2 are the mean and covariancematrix of the object location. od ∈ R8 describes theprobability of the vehicle being viewed from any of
8 possible orientations. Our goal now is to associate
object detections to tracklets and project them into 3D
using cues such as the object size or the bounding box
ground contact point in combination with the height andthe pitch angle of the camera. Association of detections
to tracklets is performed in image-scale space to betteraccount for uncertainties of the object detector.
For detection, we train the part-based object detectorof [25] on a large set of manually annotated images.
As our training set comprises ground truth orientation
labels, we modify the latent SVM [25] such that the latentcomponent variables are fixed to the orientations. We
normalize o to one using the softmax transformation,
applied to the scores of all detections that overlap the
non-maximum-suppressed ones.
Tracking is performed in two simple stages. First, weassociate all object detections frame-by-frame using the

Hungarian method [45]. The affinity matrix is computed
using geometry and appearance cues of the object. As
geometry cue we employ the bounding box intersectionover-union score. The appearance cue is computed bycorrelating the bounding box region in the previous
frame with the bounding box region in the current frame,using a small margin (20%) to account for the localization
uncertainty of the object detector. In a second stage weassociate tracklets to each other. We allow for bridging
occlusions of up to 20 frames and make use of the
Hungarian algorithm for optimal tracklet association.
Towards this goal, we employ a geometry cue [28] thatpredicts the object location and size from the boundingboxes in the other tracklet. Similar to above, object appearance is compared via normalized cross-correlation.
Given the associated bounding boxes, we estimate the3D location of the vehicles relative to the camera. Let(u, v)T be the image coordinates of the object’s boundingbox bottom-center and let w, h be its width and height.
Let (x, 0, z)T be the 3D location of an object in groundplane coordinates (y = 0) as illustrated in Fig. 7. Further,let ∆x, ∆y be the object width and height in meters, measured via parallel-projection to the z = 0 plane. Finally,
let o denote the MAP orientation of the vehicle as returned by the object detector. Assuming a uniform priorover x and z, the posterior on the object’s 3D location
can be factorized as p(x, z|u, v)p(z|w, ∆x, o)p(z|h, ∆y).
Here, the first term relates the bounding box position(u, v)T to the 3D location (x, 0, z)T and the second andlast term model the relationship between the distancez and the width w and height h of the bounding box.
As the bounding box width varies with the orientationof a vehicle, p(z|w, ∆x, o) depends on o. We learn aseparate set of parameters for each o, but will drop thisdependency in the following for clarity of presentation.
−2
Let x, z|u, v ∼ N (µ1 , Λ−11 ), z|w, ∆x ∼ N (µ2 , λ2 ) and−2
z|h, ∆y ∼ N (µ3 , λ3 ). Then, x, z|u, v, w, h, ∆x, ∆y ∼N (µ, Σ) with

T

T
µ = ΣΛ1 µ1 + ΣΛ2 0 µ2 + ΣΛ3 0 µ3−1

Σ = (Λ1 + Λ2 + Λ3 )(32)

where Λ1 has full rank and Λ2 , Λ3 are singular matricesof the form Λ2 = diag(0, λ2 ) and Λ3 = diag(0, λ3 ).
Assuming a standard pinhole camera model we have
u v 1 T = P3×4 x 0 z 1 T
(33)

9

µ
σ

∆x1
1.86
0.28

∆x2
4.37
0.64

∆x3
2.82
0.63

∆y
1.59
0.22

Fig. 8: Object Size Statistics. ∆x and ∆y are the width and heightafter parallel projection to z = 0. Cars in our dataset are ∼ 1.9m wideand ∼ 4.4m long.

where P = K T R is the product of a calibration matrix
K3×3 , the transformation from ground plane coordinates
to camera coordinates T3×4 and an additional pitch error
4×4byθ, parameterizedT theTrotation matrix R (θ). Givenuv
we obtain x z
by solving the linear system TA x z = b with

uP31 − P11 uP33 − P13
P14 − uP34
A=, b=
vP31 − P21 vP33 − P23
P24 − vP34
where we have made use of hθip(θ) = 0. The covariance Tof x z can be approximated using error propagation
Λ1 = Σ−11 ,

Σ1 = J diag(σu2 , σv2 , σθ2 ) JT

(34)

where the Jacobian J ∈ R3×3 is given by
J=

∂
∂
∂
−1
b ∂v
A−1 b ∂θ
A−1 b
∂u A



(35)


with ∂(A−1 b) = A−1 ∂b − ∂AA−1 b . Furthermore, fora pinhole camera with focal length f we have µ2 = z =f ∆x
w . We obtain the precision in z as
λ2 = σ2−2 ,σ22 = J

 2



σw 0
f
JT , J = − fw∆x
(36)
2
2
w
0 σ∆x

Similarly, we have µ3 = z = f ∆yh with precision
λ3 = σ3−2 ,σ32 = J

 2



σh 0
f
JT , J = − fh∆y
2
2
h
0 σ∆y

(37)

The unknown parameters of the proposed projection
model σu , σv , σw , σh , ∆x, ∆y, σ∆x and σ∆y are determined empirically from a held out training set composedof 1020 stereo images with 3634 annotated vehicle 2D
bounding boxes, including orientation labels (8 bins).
The object depth is computed from the median disparity
within the respective bounding box. and computed thecorresponding disparity maps. Due to the characteristics
of sliding-window detectors, we expect the noise to be
dependent on the object scale. We model this relationshipusing linear regression with respect to the bounding box
height h. We learn a separate ∆x for each of three carorientation classes illustrated in Fig. 8.
As the raw 3D location estimates {(m, S)} are noisy
due to the low camera viewpoint, the uncertainties in theobject detector and the ground plane estimation process,we temporally integrate detections within a tracklet t using a Kalman smoother [40] assuming constant velocity.

true positive rate

1
0.8
0.6
0.4
0.2
0
0

Learning based
Kosecka et al.
0.5
false positive rate

1

Fig. 9: Structured Line Segments. ROC classification results into
structure and clutter (left). Qualitative results from Kosecka et al. [43](top-right) vs. our method (bottom-right). Red corresponds to structure.

4.2

Vanishing Points

We detect up to two vanishing points V = {v1 , . . . , vNv }in the last frame of each sequence, where a vanishingpoint is defined by a rotation angle around the y-axis inroad coordinates, i.e. we assume that all vanishing lines
are collinear with the ground plane and vi represents
the yaw angle. All 3D lines that are collinear with avanishing line intersect at the same vanishing point. For
typical scenarios, two vanishing points are dominant:
One which is collinear with the forward facing street
and one which is collinear with the crossing street. Thisis illustrated in Fig. 1.
In order to detect vanishing points we make use of
the method described by Kosecka et al. [43], which wehave modified slightly for taking into account the known
camera calibration information. We restrict the search
space such that all vanishing lines are collinear withthe ground plane. Additionally, we relax the model toalso allow for non-orthogonal vanishing points as thisis required by the intersection types encountered in ourdataset. Unfortunately, traditional vanishing point detection methods [7], [43] require relatively clean scenarios
and tend to fail in the presence of clutter such as castshadows or defects in the road surface. We learn a knearest-neighbor classifier based on a held-out annotated
set of 185 images, in which all detected line segments
have been manually labeled as either structure (e.g. roadmarkings, building facades) or clutter. The classifier’s
confidence on structure is used as a weight in the vanishing point voting process. Details on the feature set which
we use can be found in [33].
Fig. 9 (left) shows the ROC curve for classifying lines
into structure and clutter. The curves have been obtainedby adjusting k for the k-nn classifier in the learning
based method and by varying the inlier threshold for[43]. Fig. 9 (right) compares the classification results for aparticular scene: While the cast shadows in the lower-leftpart of the image causes wrong evidence for traditional
detectors (top), the proposed classification step is able toreject most of these line segments (bottom).
4.3

Semantic Scene Labels

For extracting semantic information in the form of scenelabels we use the joint boosting framework proposedin [58] to learn a strong classifier. Following [61], we10divide the last image of each sequence into patches ofsize ns × ns pixels and classify them into the categories
road, background and sky. Details on the feature set canbe found in [33]. In order to avoid hard decisions andto interpret the boosting confidences as probabilities weapply the softmax transformation [46] to the resultingscores. The semantic label of a single patch is defined ass ∈ ∆2 , where ∆2 is the unit 2-simplex, i.e., the space ofpossible categorical distributions. For training, we use
a held out dataset of 200 hand-labeled images. Aftersoftmax normalization we obtain a label distributions for each image patch s ∈ S, which is used in our
semantic scene label likelihood described in Section 3.3.4.
4.4

Scene Flow and Egomotion

We compute the vehicle’s egomotion and 3D scene flowusing the method presented in [34]. Towards this goal we
accumulate all vectors in the reference (road) coordinate
system, compensate the egomotion and threshold themby their length, i.e., we remove short vectors that arelikely to belong to the static environment. As the 3D
scene flow likelihood doesn’t account for object velocities, we normalize all flow vectors to unit length andproject them onto the road plane as illustrated in Fig. 1(red arrows in occupancy grid).
4.5

Occupancy Grid

Buildings represent obstacles in the scene and thus
should never coincide with drivable regions (road). This
assumption is incorporated into the occupancy grid feature. We construct a 2D grid in road plane coordinatesfrom disparity measurements which classifies the area infront of the vehicle into the categories obstacle, free space
and unobserved cells as illustrated in Fig. 1.
We make use of the efficient large-scale stereo matching algorithm ELAS [32] to compute disparity maps
for each frame in each sequence. Given the disparity
maps and the motion estimates from Section 4.4, we
compute a 2D occupancy grid of the environment inreference road coordinates, representing obstacles anddrivable (road) areas. The occupancy grid is obtained byassuming spatial independency amongst cells, applying
the discrete Bayes filter [57] and ignoring moving objects(tracklets, scene flow).

5

We then map the annotated geometry into the road coordinate system using the GPS coordinates of the vehicle.
Note that the GPS is only used for annotation. Additionally, for all vehicle tracklets that have been detectedby the approach described in Section 4.1, we annotate
their lane index l, including parking. For vehicles thathave been associated to a lane, the tangent at the closest
foot point of lane l is used as object orientation groundtruth. Furthermore, we manually annotate all lanes ineach scenario with a binary label indicating whether thelane is active, i.e., whether vehicles move on that lane.
5.1

Settings

Our experiments evaluate the performance of the proposed system as well as the importance of each individual feature cue, abbreviated as
P = Prior (see Section 3.3.1)
T = Tracklets (see Section 3.3.2)
V = Vanishing Lines (see Section 3.3.3)
S = Semantic labels (see Section 3.3.4)
F = Scene Flow (see Section 3.3.5)
O = Occupancy Grid (see Section 3.3.6)
To gain insights into the strengths and weaknesses ofeach cue, we evaluate the prior alone, all terms individually (PT, PV, PS, PF, PO), all terms but one (PVSFO,PTSFO, PTVFO, PTVSO, PTVSF) and the full model(PTVSFO). For each setting we learn a separate set ofparameters.
For evaluating the proposed model, we leverage 10fold cross-validation and learn the model parametersΘ = {λp , ξ p , λt , λv , λs , λf 1 , λf 2 , λo } in each fold using
contrastive divergence, as described in Section 3.5. All
parameters are initialized to θi = 1. Convergence typically occurred after 150-250 gradient ascent steps. We
excluded ζt , ζv and ζf from the maximum likelihood
estimation process described in Section 3.5 as we foundthem hard to optimize. Instead, we estimated them using
cross-validation. All parameters which are not part of Θ
have been determined empirically and are summarizedin Table 2. Inference is performed by drawing ninf =10, 000 samples and computing the MAP solution. Our
mixed C++/MATLAB implementation requires ∼ 8 seconds per frame for inference and about an hour forlearning the model parameters on a standard PC using
8 CPU cores. Most time is spent on feature extraction,
especially object detection.

E XPERIMENTAL E VALUATION

We recorded a dataset of 113 video sequences of real
traffic with a duration of 5 to 30 seconds each. This corresponds to a total of 9438 frames or ∼ 944 seconds. All
sequences end when entering the intersection, i.e., when
the autonomous system would need to take a decision.
Fig. 11 shows the large variability in appearance. We
annotate the data using GoogleMaps aerial images. For
each intersection in the database we labeled the centerof the intersection as well as the number, orientation andwidth of the intersecting streets in bird’s eye perspective.

5.2

Topology and Geometry

To judge the performance of the proposed model, we
evaluate the estimation results of each setting against
several metrics. First, we measure the accuracy in topology estimation, which is the percentage of all 113 cases
in which the correct topology κ has been recovered.
We propose three geometric metrics: average Euclidean
error when estimating the intersection center, average
street orientation error and the road area overlap. Regarding the street orientation, we assign each street to11Prior: (Section 3.3.1)
σα = 0.1 rad
KDE kernel bandwidth
Vehicle Tracklets: (Section 3.3.2 and Section 4.1)
τd = 0.2
NMS overlap threshold (object detection)
τt1 = 0.5
Gating threshold (tracking stage 1)
τt2 = 0.3
Gating threshold (tracking stage 2)−20
ζt = 10
Outlier threshold
σout = 70 m
Std. deviation of outlier distribution
Vanishing Points: (Section 3.3.3)
ζv = 10−10
Outlier threshold
Semantic Scene Labels: (Section 3.3.4 and Section 4.3)
ns = 4 Px
Image patch (superpixel) size
ws = (1, 1, 4)
Scene label weights
Scene Flow: (Section 3.3.5 and Section 4.4)
nf = 50
Number of RANSAC samples
ζf = 10−15
Outlier threshold
σout = 70 m
Std. deviation of outlier distribution
Occupancy Grid: (Section 3.3.6 and Section 4.5)
no = 1 m
Occupancy grid cell size
wo = (−1, 4, 1) Weights of geometric prior∆o = (2, 20) m
Margins of geometric prior
Inference and Learning: (Section 3.4 and Section 3.5)
ninf = 10, 000
Number of samples at inference
nlearn = 10
Number of samples per learning iteration
niter = 500
Number of learning iterations

5.3

Tracklet Associations and Semantic Activities

Besides the geometric reasoning discussed so far, an
important aspect in real-world applications is to understand the scene at a higher level. This includes theassociation of vehicle tracklets to lanes (’Tracklet Accuracy’ in Table 3) as well as the detection of active
lanes (’Lane Accuracy’ in Table 3), where we assume asingle lane connecting each inbound with each outboundstreet, as described in Section 3.1. With active we referto lanes on which at least one vehicle is moving. For
evaluating the above mentioned metrics we extract all
unique tracklets, defined by a minimum length of 10
meters. We define a lane as active if at least one tracklet
has been uniquely assigned to it. The tracklet and lane
accuracies for all settings are depicted in Table 3 (rows 5
and 6). As expected, the best results are obtained wheneither the 3D scene flow or the vehicle tracklet features
are present, with 80% accuracy in tracklet associations
and 90% accuracy in detecting active lanes.
5.4

Object Detection and Orientation Estimation

As evidenced by Table 3, each feature is able toimprove results compared to the prior (column 1-6). The
strongest cues in our framework are tracklets, 3D scene
flow and the occupancy grid features. This indicatesthat despite its noisy nature, depth information is very
important. The smallest gain in performance is observed
for the vanishing point feature as it only improves
performance in combination with other cues.

As we have shown in Section 5.2, objects help in estimating the layout and geometry of the scene. On the other
hand, knowledge about the road layout should also help
in improving the performance of object detectors, bothin accuracy and object orientation estimation.
First, we re-estimate the orientations of all objects
that are used as input to our model. For associating
the tracklets to lanes and the detections to lane spline
points, we employ the inference procedure described in
Section 3.4.2. We select the tangent angle at the associated spline’s foot point s on the inferred lane l as ournovel orientation estimate. Table 3 (row 7) shows that weare able to significantly reduce the orientation error ofmoving vehicles from 32.6 degrees, corresponding to theorientation error of the raw detections (not depicted in
the table), down to 14.0 degrees when using our modelin combination with vehicle tracklets or 3D scene flow.
We also manually annotated all cars in the last frameof each sequence using 2D bounding boxes, yielding 355
labeled car instances in total. Given the object detections
and the inferred road geometry from Section 5.2, we rescore each object detection by adding the following term
to the scores of [25]
"
 X

#

3
(xi − µi )2
∆2l
exp −
−1
0.5 max exp − 2 +
l
2w
2σi2
i=1

Performance improves further when combining features. In terms of topology estimation, the best results
have been obtained by making use of all information.
Regarding the geometric error measures all settings that
include the occupancy grid feature perform comparably
well. This leads us to the conclusion that occupancy
information is complementary information, while othercues such as 3D scene flow and vehicle tracklets canpartly replace each other.

Here ∆l is the distance of a car detection to lane splinel, w is the estimated street width and {µi , σi } are meanand standard deviation of the object width, height andposition, respectively. These estimates are obtained froma held-out training set using maximum likelihood estimation. Fig. 10 depicts the precision-recall curves for the
L-SVM baseline [25] and our approach. Note that ourgeometric and topological constraints increase detection
performance significantly, improving average precision

TABLE 2: Setting of Constants in the Model. For reproducibility of
our results, we specify all constants in our model. On acceptance of
this paper, we will also release the data and MATLAB/C++ code.

its (rotationally) closest counterpart in the ground truth
layout in order to decouple the orientation measure fromthe estimated topology κ. More precisely, we take thelayout with the smaller number of streets and assign
all streets to their closest counterparts in the layout
with the larger number of streets. Finally, the road area
overlap measures to which extend the estimated roadlayout overlaps with the ground truth by computing
the intersection-over-union of both road areas. Here, theroad area is defined as the concave hull enclosing all
streets up to a distance of three times the average groundtruth street width.

12Topology Accuracy (%, ↑)Location Error (m, ↓)Street Orient. Err. (deg, ↓)Road Area Overlap (%, ↑)Tracklet Accuracy (%, ↑)Lane Accuracy (%, ↑)Object Orient. Err. (deg, ↓)Object Detection AP (%, ↑)

P

PT

PV

PS

PF

PO

19.5
6.8
8.7
33.1
28.1
77.3
53.0
73.6

64.6
5.1
5.4
51.0
78.7
90.3
17.7
73.7

20.3
6.7
8.7
32.9
30.0
78.0
54.4
73.7

55.8 66.4
7.9 4.5
7.3 5.2
41.2 53.6
35.8 79.2
80.2 90.5
45.1 17.5
73.1 73.6

83.2
4.0
5.1
63.4
65.1
85.2
24.2
74.1

PVS
FO
91.1
3.0
3.6
69.3
80.7
89.7
14.0
74.1

PTS PTV PTV PTV PTV
SO
SF
SFO
FO FO
89.4 91.1 90.3 72.6 92.0
2.9
2.7
3.1
4.8
3.0
3.6
3.6
3.6
4.7
3.6
69.7 71.2 68.9 57.5 69.9
80.3 79.8 77.6 81.6 82.0
89.4 89.8 88.3 90.2 89.7
15.1 14.9 15.4 15.1 14.3
74.1 74.2 74.1 73.7 74.0

P
T
V
S
F
O

=
=
=
=
=
=

Prior
Tracklets
Vanishing Lines
Semantic labels
Scene Flow
Occupancy Grid

TABLE 3: Quantitative Results. ↑ (↓) means higher (lower) is better. All numbers represent averages over all 113 sequences.

Precision (%)

100

50

0
0

L−SVM
PTVSFO
50
Recall (%)

100

Fig. 10: Improving Object Detection. Precision-recall curves (left) andan example where our algorithm (bottom-right) is able to eliminate
false positives of a state-of-the-art object detector [25] (top-right).

from 69.9% to 74.2%. The benefits of including thisknowledge into the detection process are also illustrated
in Fig. 10 (right). In order to include the partly occluded
car to the right into the detection result, the thresholdof the baseline has to be lowered to a value which
produces two false positives (top). In contrast, our rescored ranking is able to handle this case (bottom).
5.5

Qualitative Results

Fig. 11 illustrates our inference results for the setting’PTVSFO’, with the most likely lanes for each uniquetracklet, indicated by an arrow. The ego-vehicle (observer) is depicted in black. For most sequences the roadlayout has been estimated correctly and the vehicles havebeen assigned to the correct lanes. Only vehicles that arevery far away or visible only for a couple of frames pose
problems in terms of their lane associations. However,note that in many cases this didn’t affect the estimatedlayout. More results can be found in our supplementary
video which is available from our project website.2

6

often unavailable in urban scenarios. While we foundperformance improvements for all features, occupancy
grids as well as vehicle tracklets and 3D scene flow havebeen identified as the strongest and most important cues.
To accommodate for the ambiguities in the visual
information we developed a probabilistic model to describe the appearance of the proposed features relative
to the intersection geometry. We utilized Markov Chain
Monte Carlo sampling for inference to cope with thecomplex relationship between the crossroad layout andthe appearance of the features. As evidenced by theexperiments, our approach is able to recognize urban
intersections reliably with an accuracy of up to 90% on
a data set of 113 realistic real-world intersections. For
most of these intersections, traditional approaches basedon lane markings and curb stones would have failed
due to the absence of these feature cues. Moreover, wefound that context from our model helps to improve
the performance of state-of-the-art object detectors interms of detecting objects as well as estimating theirorientation.
Our approach serves as basis for future improvements.
Currently, the assumption that tracklets are independent can lead to implausible configurations such as cars
colliding with each other. Including higher-level background knowledge on how cars can pass an intersection
will help to reduce ambiguities and increase robustness.
Furthermore, improved sensor observations and morecomputing power will allow for more accurate motion
models and lane representations. Another interesting
direction of research will be to integrate informationfrom other traffic participants (e.g., pedestrians) into themodel to perform collaborative inference.

C ONCLUSIONS AND F UTURE D IRECTIONS

We have developed an approach that allows autonomousvehicles to estimate the layout of urban intersections
based on onboard stereo imagery alone. Our model does
not rely on strong prior knowledge such as intersection
maps, but infers all information from different types ofvisual features that describe the static environment of
the crossroads (i.e., facades of houses) and the motionsof objects (i.e., cars) in the scene. Different from previous
approaches, our method does not rely on traditionalfeatures like lane markings or curb stones which are
2. http://www.cvlibs.net/projects/intersection

Andreas Geiger received his Diploma in computer science and his Ph.D. degree from Karlsruhe Institute of Technology in 2008 and 2013.
Currently, he is a research scientist in the Perceiving Systems group at the Max Planck Institute for Intelligent Systems in Tübingen. His
research interests include computer vision, machine learning and scene understanding.

13Fig. 11: Inference Results. For each sequence we show the input image (top) and the inference result (bottom). Arrows indicate the predicted
driving direction(s). The last row shows four cases where topology estimation has failed. However, note that in many cases the lanes are still
associated correctly.

Martin Lauer studied computer science at
Karlsruhe University and received his Ph.D. incomputer science from University of Osnabrück.
As research group leader he works in the fields
of machine vision, robotics, and machine learning at Karlsruhe Institute of Technology.

Christian Wojek received his Diplom degree in
Computer Science from the University of Karlsruhe in 2006 and his PhD from TU Darmstadt
in 2010. He was awarded a DAAD scholarshipto visit McGill University from 2004 to 2005. He
was with MPI Informatics Saarbruecken as a
postdoctoral fellow from 2010 to 2011 and in2011 joined Carl Zeiss Corporate Research. His
research interests are object detection, scene
understanding and activity recognition in particular with application to real world scenarios.

Christoph Stiller is professor at the Karlsruhe
Institute of Technology and head of the Instituteof Measurement and Control. At the same time
he is director at the FZI Research Center for
Information Technology heading the mobile perception research group. His research focus is instereo vision, camera based scene understanding, driver assistance systems, and autonomousvehicles.

Raquel Urtasun Raquel Urtasun is an Assistant Professor at TTI-Chicago. Previously, she
was a postdoctoral research scientist at UC
Berkeley and ICSI and a postdoctoral associate
at the Computer Science and Artificial Intelligence Laboratory (CSAIL) at MIT. Raquel Urtasun completed her PhD at the Computer Vision Laboratory at EPFL, Switzerland. Her majorinterests are statistical machine learning andcomputer vision.

14

R EFERENCES[1]

Y. Alon, A. Ferencz, and A. Shashua. Off-road path following
using region classification and geometric projection constraints.
In CVPR, 2006.
[2] J. M. Alvarez, T. Gevers, and A. M. Lopez. 3d scene priors forroad detection. In CVPR, 2010.
[3] M. Aly. Real time detection of lane markers in urban streets. In
IV, 2008.
[4] C. Andrieu, N. de Freitas, A. Doucet, and M. I. Jordan. An
introduction to MCMC for machine learning. Machine Learning,
50(1-2):5–43, 2003.
[5] M. Andriluka, S. Roth, and B. Schiele. Monocular 3d pose
estimation and tracking by detection. In CVPR, 2010.
[6] S. Bao, M. Sun, and S. Savarese. Toward coherent object detection
and scene layout understanding. In CVPR, 2010.
[7] O. Barinova, V. Lempitsky, E. Tretyak, and P. Kohli. Geometricimage parsing in man-made environments. In ECCV, 2010.
[8] C. M. Bishop. Pattern Recognition and Machine Learning. Springer,
1st ed. 2006 edition, October 2006.
[9] M. D. Breitenstein, F. Reichlin, B. Leibe, E. Koller-Meier, andL. Van Gool. Robust tracking-by-detection using a detector
confidence particle filter. In ICCV, 2009.
[10] A. Broggi, P. Medici, P. Zani, A. Coati, and M. Panciroli. Autonomous vehicles control in the VisLab Intercontinental Autonomous Challenge. ARC, 36(1):161–171, 2012.
[11] G. Brostow, J. Shotton, J. Fauqueur, and R. Cipolla. Segmentation
and recognition using SfM point clouds. In ECCV, 2008.
[12] M. A. Brubaker, A. Geiger, and R. Urtasun. Lost! leveraging thecrowd for probabilistic visual self-localization. In CVPR, 2013.
[13] M. Buehler, K. Iagnemma, and S. Singh, editors. The DARPA Urban
Challenge, volume 56 of Advanced Robotics, 2009.
[14] W. Choi and S. Savarese. Multiple target tracking in world
coordinate with single, minimally calibrated camera. In ECCV,
2010.
[15] W. Choi and S. Savarese. A unified framework for multi-target
tracking and collective activity recognition. In ECCV, 2012.
[16] J. Crisman and C. Thorpe. Scarf: A color vision system that tracks
roads and intersections. TRA, 9(1):49–58, February 1993.
[17] H. Dahlkamp, A. Kaehler, D. Stavens, S. Thrun, and G. R. Bradski.
Self-supervised monocular road detection in desert terrain. In
RSS, 2006.
[18] R. Danescu and S. Nedevschi. Probabilistic lane tracking indifficult road scenarios using stereovision. TITS, 10(2):272–282,
2009.
[19] C. De Boor. A Practical Guide to Splines. Number 27 in Applied
Mathematical Sciences. Springer-Verlag, 1978.
[20] C. Desai, D. Ramanan, and C. Fowlkes. Discriminative models
for multi-class object layout. In ICCV, 2009.
[21] E. D. Dickmanns and B. D. Mysliwetz. Recursive 3-d road andrelative ego-state recognition. PAMI, 14(2):199–213, Feb 1992.
[22] W. Enkelmann, G. Struck, and J. Geisler. Roma - a system formodel-based analysis of road markings. In IV, 1995.
[23] A. Ess, B. Leibe, K. Schindler, and L. V. Gool. Robust multi-person
tracking from a mobile platform. PAMI, 31:1831–1846, 2009.
[24] A. Ess, T. Mueller, H. Grabner, and L. van Gool. Segmentationbased urban traffic scene understanding. In BMVC, 2009.
[25] P. Felzenszwalb, R. Girshick, D. McAllester, and D. Ramanan.
Object detection with discriminatively trained part based models.
PAMI, 32:1627–1645, 2010.
[26] D. M. Gavrila. A bayesian, exemplar-based approach to hierarchical shape matching. PAMI, 29:1408–1421, 2007.
[27] D. M. Gavrila and S. Munder. Multi-cue pedestrian detection andtracking from a moving vehicle. IJCV, 73:41–59, 2007.
[28] A. Geiger. Probabilistic Models for 3D Urban Scene Understanding
from Movable Platforms. PhD thesis, KIT, 2013.
[29] A. Geiger, M. Lauer, F. Moosmann, B. Ranft, H. Rapp, C. Stiller,
and J. Ziegler. Team annieway’s entry to the grand cooperative
driving challenge 2011. TITS, 2012.
[30] A. Geiger, M. Lauer, and R. Urtasun. A generative model for 3d
urban scene understanding from movable platforms. In CVPR,
2011.
[31] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous
driving? The KITTI vision benchmark suite. In CVPR, 2012.
[32] A. Geiger, M. Roser, and R. Urtasun. Efficient large-scale stereo
matching. In ACCV, 2010.

[33] A. Geiger, C. Wojek, and R. Urtasun. Joint 3d estimation of objects
and scene layout. In NIPS, 2011.
[34] A. Geiger, J. Ziegler, and C. Stiller. StereoScan: Dense 3d reconstruction in real-time. In IV, 2011.
[35] V. Gengenbach, H. H. Nagel, F. Heimes, G. Struck, and H. Kollnig.
Model-based recognition of intersections and lane structures. In
IV, 1995.
[36] A. Gupta, A. A. Efros, and M. Hebert. Blocks world revisited:
Image understanding using qualitative geometry and mechanics.
In ECCV, 2010.
[37] G. Hinton. Training products of experts by minimizing contrastive
divergence. NC, 14:2002, 2000.
[38] D. Hoiem, A. Efros, and M. Hebert. Putting objects in perspective.
IJCV, 80:3–15, 2008.
[39] D. Hoiem, A. A. Efros, and M. Hebert. Recovering surface layout
from an image. IJCV, 75(1):151–172, October 2007.
[40] R. E. Kalman. A new approach to linear filtering and prediction
problems. JBE, 1(82):35–45, 1960.
[41] V. Kastrinaki. A survey of video processing techniques for traffic
applications. IVC, 21(4):359, 2003.
[42] Z. Khan, T. Balch, and F. Dellaert. Mcmc-based particle filtering
for tracking a variable number of interacting targets. PAMI,
27(11):1805–1819, 2005.
[43] J. Kosecka and W. Zhang. Video compass. In ECCV, 2002.
[44] D. Kuettel, M. D. Breitenstein, L. V. Gool, and V. Ferrari. What’s
going on?: Discovering spatio-temporal dependencies in dynamic
scenes. In CVPR, 2010.
[45] H. W. Kuhn. The Hungarian method for the assignment problem.
NRLQ, 2:83–97, 1955.
[46] S. Kumar and M. Hebert. A hierarchical field framework for
unified context-based classification. In ICCV, 2005.
[47] B. Leibe, K. Schindler, N. Cornelis, and L. Van Gool. Coupled
detection and tracking from static cameras and moving vehicles.
PAMI, 30(10):1683–1698, 2008.
[48] M. Lutzeler and E. D. Dickmanns. Ems-vision: recognition ofintersections on unmarked road networks. In IV, 2000.
[49] J. C. McCall and M. M. Trivedi. Video-based lane estimationand tracking for driver assistance: survey, system, and evaluation.
TITS, 7(1):20–37, March 2006.
[50] V. Nedovic, A. W. M. Smeulders, A. Redert, and J. M. Geusebroek.
Stages as models of scene geometry. PAMI, 32:1673–1687, 2010.
[51] D. Pomerleau. Ralph: Rapidly adapting lateral position handler.
In IV, pages 506–511, 1995.
[52] D. Ramanan and D. Forsyth. Finding and tracking people fromthe bottom up. In CVPR, 2003.
[53] C. Rasmussen. Road shape classification for detecting and negotiating intersections. In IV, 2003.
[54] A. Saxena, S. H. Chung, and A. Y. Ng. 3-D depth reconstruction
from a single still image. IJCV, 76:53–69, 2008.
[55] A. Saxena, M. Sun, and A. Y. Ng. Make3d: Learning 3d scene
structure from a single still image. PAMI, 31:824–840, 2009.
[56] G. Singh and J. Kosecka. Acquiring semantics induced topology
in urban environments. In ICRA, 2012.
[57] S. Thrun, W. Burgard, and D. Fox. Probabilistic Robotics. The MIT
Press, 2005.
[58] A. Torralba, K. P. Murphy, and W. T. Freeman. Sharing features:
efficient boosting procedures for multiclass object detection. In
CVPR, 2004.
[59] X. Wang, X. Ma, and W. Grimson. Unsupervised activity perception in crowded and complicated scenes using hierarchicalbayesian models. PAMI, 31:539–555, 2009.
[60] C. Wojek, S. Walk, S. Roth, K. Schindler, and B. Schiele. Monocularvisual scene understanding: Understanding multi-object traffic
scenes. PAMI, 2012.
[61] C. Wojek and B. Schiele. A dynamic conditional random fieldmodel for joint labeling of object and scene classes. In ECCV,
2008.
[62] K. Yamaguchi, A. C. Berg, L. E. Ortiz, and T. L. Berg. Who are
you with and where are you going? In CVPR, 2011.
[63] Q. Yu and G. Medioni. Multiple-target tracking by spatiotemporalmonte carlo markov chain data association. PAMI, 31(12):2196–
2210, 2009.
[64] Q. Zhu, L. Chen, Q. Li, M. Li, A. Nuchter, and J. Wang. 3d
lidar point cloud based intersection recognition for autonomous
driving. In IV, 2012.