J OINT 3D E STIMATION OFO BJECTS AND S CENE L AYOUTA NDREAS G EIGERC HRISTIAN W OJEKR AQUEL U RTASUNKITMPI S AARBRÜCKENTTI C HICAGOP ROBABILISTIC M ODELO VERVIEWVehicle Tracklets

Vanishing Points⇒Our goal is to estimate the most likely configuration R = (θ, c, w, r, α) given the image evidence
E = {T, V, S}, which comprises vehicle tracklets T = {t1 , .., tN }, vanishing points V = {vf , vc } andsemantic labels S. Given R we assume all observations to be independent:

Scene Labelsp(E, R) = p(R)| {z }

Vehicle Tracklets

P RIOR• Urban video sequences (∼10 s)• Monocular, Grayscale @ 10 Hz

V EHICLE T RACKLETSp(R)=p(θ)p(c, w)p(r)p(α)θ∼∼δ(θM AP )N (µcw , Σcw )r∼

N (µr , σr )

α

∼

p(α)

(c, log w)

Features:• Vehicle tracklets (tracking-by-detection)• Semantic scene labels (joint boosting)• Vanishing points (long lines)
Inference:• Metropolis-Hastings sampling• Dynamic programmingT• Road parameters: R = (θ, c, w, r, α)• Scene layout: θ ∈ {1, .., 7}• Intersection center: c ∈ R2
• Street width: w ∈ R+π π• Scene rotation: r ∈ [− 4 , 4 ]• Crossing street angle: α ∈ [− π4 , π4 ]Output:• Street layout (road size, orientation)• Object locations and orientations• Intersection crossing activities

T OPOLOGY AND G EOMETRY
123

4

56

#p(tn , ln |R) p(vf |R) p(vc |R)
p(S|R)|{z}|{z}
n=1 lnPrior |Semantic Labels{z} Vanishing Points

We propose a novel generative model thatis able to reason jointly about the 3D scenelayout and the location / orientation of objects.
Input:"NYX

VANISHING P OINTS
p(vf |R) ∝δf N (vf |µf , σf ) + const

p(vc |R) ∝δc N (vc |µc , σc ) + const• Forward vanishing point: vf ∈ R• Crossing vanishing point: vc ∈ [− π4 , π4 ]• Existance variables: δf , δc ∈ {0, 1}p(t|l, R) =X

p(s1 )p(d1 |s1 , l, R)

s1 ,..,sMMYp(sm |sm−1 )p(dm |sm , l, R)

m=2p(d|s, l, R) =p(o|s, l, R)p(b|s, l, R)p(o|s, l, R) =Mult(o|π o )p(b|s, l, R) ∝ N (π(b)|µb , Σb ) + const• Tracklet: t = {d1 , .., dM }• Lane: l ∈ {1, .., K(K − 1) + 2K}• Location on lane: s ∈ N• Detection: d = (o ∈ {1, .., 8}, b ∈ R4 )S EMANTIC L ABELS
p(S|R) ∝ exp(γXXc(u,v)∈Sc(c)Su,v)• Class: c ∈ {road, building, sky}
• Pixels of reprojected model: Sc• Scene label probability: Su,v

7

We model street scenes in bird’s eye perspective using 7 scene layouts θ (left) and the following geometry parameters R (right):R ESULTS• Center of intersection c• Street width w• Scene rotation r• Crossing street angle α
We model the set of possible vehicle locations
with lanes connecting the streets and parking
spots at the road side:
We have:• K streets• K(K − 1) lanes• 2K parking spots

F UTURE W ORK• Find more discriminative cues
• Extend inference (e.g., pedestrians)• Holistic scene interpretation incorporating buildings, infrastructure, vegetationand other types of vehicles

Automatically inferred scene descriptions: Tracklets from all frames superimposed (left). Inference result with θ known (middle) and θ unknown (right). The inferred intersection layout isshown in gray, ground truth labels are given in blue and detected activities are depicted in red.
Topology Location Orientation Overlap Activity
Topology
GP MKL–
6.0 m
9.6 deg
44.9 %
18.4 %
knownOurs–
5.8 m
5.9 deg
53.0 %
11.5 %
Topology
GP MKL
27.4 %
6.2 m21.7 deg
39.3 %
28.1 %
unknown
70.8 %
6.6 m
7.2 deg
48.1 %
16.6 %
Ours
OracleStereo92.9 %
4.4 m
6.6 deg
62.7 %
8.0 %Topology and Geometry Estimation: Errors / Accuracies with respect to GP-Baseline and Stereo
Error
Felzenszwalb et al. (raw)32.6 ◦
Felzenszwalb et al. (smoothed) 31.2 ◦Our method (θ unknown)
15.7 ◦
Our method (θ known)
13.7 ◦
Orientation Estimation: Compared to state-of-the-art object detection we decrease angular orientation errors of moving objects by using context from our model. Measured in bird’s eye view.