Understanding High-Level Semantics by Modeling Traffic Patterns∗†‡∗{ Hongyi Zhang , Andreas Geiger and Raquel Urtasun }I NTRODUCTIONP ROBABILISTIC M ODEL
gi1hi1‡

Peking University, MPI Tübingen, TTI Chicago

VANISH . P OINTS / S EM . L ABELSR ESULTS
Case study:

gihi

†aln

Red:
Inferred pattern
Green:
True pattern

Sdiframe i 1 frame iGiven a short monocular video sequence froma movable platform we propose a joint probabilistic model for estimating:• The 3D urban scene layout• The objects (e.g., cars) in the scene
Contributions with respect to [1]:• Model for Traffic patterns• Interactions between tracklets• Novel dynamical modelNVI NFERENCE

Variables:• R: Road parameters (width, rotation, etc.)• a: Traffic patterns (i.e. traffic signal phase)• ln : Lane n-th tracklet is driving on• (gi , hi ): Vehicle dynamics• di : Tracklet detection (location, heading)• S: Scene label evidence• V: Vanishing point evidence

2• Simulated annealing (MH sampling)• Mixture of local and global moves

Qualitative Results:

Inferring traffic patterns:
N X
Yp(a|T, R) ∝p(tn |a, ln , R)
Inferring car-to-lane associations:p(ln |a, tn , R) ∝ p(tn |a, ln , R).

Detection likelihood:

where a tracklet t = {d1 , . . . , dM } is represented
by the set of its detections and p(tn |a, ln , R) canbe approximated by Expectation Propagation.

We model street scenes in bird’s eye perspective
using 7 scene layouts θ (left) and the geometry
parameters R (right):
1

Inferring road geometry:

n=1 lnT RACKLETST OPOLOGY AND G EOMETRY

Left: traffic pattern disambiguates lane association of the static car (rightmost).
Right: Correct inference result for scene from theI NTRODUCTION. [1] infers colliding vehicles.

3L EARNING
456p(di |gi , hi ) = N ((si , hi ), (ξΛdi )−1γ) × pheading• g = (s, b), s: spline point, b ∈ {stop, go}• Λdi : tracklet precision matrix• pheading : heading probability• ξ, γ: model parameters

7We model the set of possible vehicle locations
with lanes connecting the streets and parking
spots at the road side:
We have:• K streets• K(K − 1) lanes• 2K parking spots

Forward dynamics: p(bi |bi−1 )π(·)
p(bi |bi−1 )p(gi |gi−1 ) =0

if bi = goif bi = stop ∧ si = si−1
if bi = stop ∧ si 6= si−1

where the transition probability p(bi |·) also depends on a, l which decide if the lane is active,and π(·) models the driving speed.
Lateral dynamics:• hi = hi−1 + ∆σh2 (Gaussian noise)

Pattern and car-to-lane association error:Learning traffic patterns:• Enumerate all combinations of K patterns• Score them by number of correct tracklets• 4 patterns explain most scenarios

T-L error (all)
Method 3-arm 4-arm[1]
46.7% 49.9%
Ours 15.2% 30.1%

T-L error (>10m) Pattern error
3-arm
4-arm
3-arm 4-arm
17.9%
30.1%
––
3.6%
14.0%
18.2% 19.4%

Road geometry estimation:
3-arm patterns (blue: learned patterns):
Pattern 1

Pattern 2

Pattern 3

Pattern 4

Pattern 5

Pattern 6

Pattern 7

Pattern 8

Pattern 9

Pattern 11

Pattern 12

Pattern 13Pattern 14

Pattern 15

Pattern 16

Pattern 17

Pattern 18

Pattern 19

Pattern 10

Location
Orientation
Overlap
Method 3-arm 4-arm 3-arm 4-arm 3-arm 4-arm[1]
4.3 m 5.4 m 3.3 deg 8.0 deg 58.7% 56.0%
Ours 5.7 m 4.9 m 2.4 deg 4.3 deg 61.5% 61.3%

4-arm patterns (blue: learned patterns):
Pattern 1

Pattern 2

Pattern 3

Pattern 4

Pattern 5

Pattern 6

Pattern 7

Pattern 8

Pattern 9

Pattern 10

Pattern 11

Learning forward dynamics:
Estimate p(bi−1 , bi ) on active/inactive lane separately (S: stop states, G: go states)
Lane State S→S G→S S→G G→GInactive 0.888 0.017 0.015 0.080
Active0.027 0.010 0.005 0.958

R EFERENCES[1] A. Geiger, C. Wojek, and R. Urtasun. Joint 3d
estimation of objects and scene layout. In NIPS,Granada, Spain, December 2011.