A Generative Model for 3D Urban Scene
Understanding from Movable Platforms
Andreas Geiger*, Martin Lauer*, Raquel Urtasun***K ARLSRUHE I NSTITUTE OF T ECHNOLOGYKIT – University of the State of Baden-Wuerttemberg andNational Laboratory of the Helmholtz Association**TOYOTA T ECHNOLOGICAL I NSTITUTE AT C HICAGO

www.kit.edu

3D Urban Scene Understanding

A Generative Model for 3D Urban Scene Understanding

2/18

Related WorkUrban Scene Segmentation
Semantic categories: road, vehicles, building, sky
Joint detection and segmentation [Wojek 2008]
Appearance combined with SfM [Sturgess 2009]
Geometric Methods
3D from single images [Hoiem 2007, Saxena 2009]
Incorporating laws of physics [Gupta 2010]
Activity Recognition
Extraction of typical activity patterns in 2D optical flow
Non-parametric clustering [Wang 2009, Kuettel 2010]

A Generative Model for 3D Urban Scene Understanding

3/18

Our Approach

Goal: Infer from short video sequences (moving observer)
Topology and geometry of the scene
Semantic information (e.g., traffic situation)

Probabilistic generative model of 3D urban scenes
Static features: Building facades
Dynamic features: Moving vehicles
A Generative Model for 3D Urban Scene Understanding

4/18

Our Approach

Goal: Infer from short video sequences (moving observer)
Topology and geometry of the scene
Semantic information (e.g., traffic situation)

Probabilistic generative model of 3D urban scenes
Static features: Building facades
Dynamic features: Moving vehicles
A Generative Model for 3D Urban Scene Understanding

4/18

Our Approach

Goal: Infer from short video sequences (moving observer)
Topology and geometry of the scene
Semantic information (e.g., traffic situation)

Probabilistic generative model of 3D urban scenes
Static features: Building facades
Dynamic features: Moving vehicles
A Generative Model for 3D Urban Scene Understanding

4/18

Our Approach

Goal: Infer from short video sequences (moving observer)
Topology and geometry of the scene
Semantic information (e.g., traffic situation)

Probabilistic generative model of 3D urban scenes
Static features: Building facades
Dynamic features: Moving vehicles
A Generative Model for 3D Urban Scene Understanding

4/18

Our Approach

Goal: Infer from short video sequences (moving observer)
Topology and geometry of the scene
Semantic information (e.g., traffic situation)

Probabilistic generative model of 3D urban scenes
Static features: Building facades
Dynamic features: Moving vehicles
A Generative Model for 3D Urban Scene Understanding

4/18

Our Approach

Goal: Infer from short video sequences (moving observer)
Topology and geometry of the scene
Semantic information (e.g., traffic situation)

Probabilistic generative model of 3D urban scenes
Static features: Building facades
Dynamic features: Moving vehicles
A Generative Model for 3D Urban Scene Understanding

4/18

Topology Model

Reasoning in bird’s eye perspectiveA Generative Model for 3D Urban Scene Understanding

5/18

Geometry Model
k=3 arms

street 3street 2street 1

k ... number of intersection armsc ... center of intersection
w ... street widthr ... global rotationo ... relative street orientation

A Generative Model for 3D Urban Scene Understanding

6/18

Geometry Modelcenterstreet 3street 2street 1

k ... number of intersection armsc ... center of intersection
w ... street widthr ... global rotationo ... relative street orientation

A Generative Model for 3D Urban Scene Understanding

6/18

Geometry Modelwidthcenterstreet 3

street 2
street 1

k ... number of intersection armsc ... center of intersection
w ... street widthr ... global rotationo ... relative street orientation

A Generative Model for 3D Urban Scene Understanding

6/18

Geometry Modelwidthcenterstreet 3

rotationstreet 2
street 1

k ... number of intersection armsc ... center of intersection
w ... street widthr ... global rotationo ... relative street orientation

A Generative Model for 3D Urban Scene Understanding

6/18

Geometry Model
relativeorientation
width
centerstreet 3

rotationstreet 2
street 1

k ... number of intersection armsc ... center of intersection
w ... street widthr ... global rotationo ... relative street orientation

A Generative Model for 3D Urban Scene Understanding

6/18

Geometry Model
relativeorientation
width
centerstreet 3

rotationstreet 2
street 1

k ... number of intersection armsc ... center of intersection

w ... street widthr ... global rotationo ... relative street orientation

A Generative Model for 3D Urban Scene Understanding

R
6/18

Observations

Static features: Occupancy grids. For each cell, a variableindicates if it is occupied (+1), unobserved (0) or free (-1).
Dynamic features: Sparse 3D scene flow
Registration using stereo visual odometry

A Generative Model for 3D Urban Scene Understanding

7/18

Probabilistic Modelpriorsparameters

A Generative Model for 3D Urban Scene Understanding

observations

8/18

Probabilistic Modelpriorsparameters

observations

all roadparameters(k,c,w,r,o)

A Generative Model for 3D Urban Scene Understanding

8/18

Probabilistic Modelpriorsparameters

A Generative Model for 3D Urban Scene Understanding

observations

8/18

Probabilistic Modelpriorsparameters

A Generative Model for 3D Urban Scene Understanding

observations

8/18

Probabilistic Modelpriorsparameters

A Generative Model for 3D Urban Scene Understanding

observations

8/18

Probabilistic Modelpriorsparameters

observations

relativeorientation

A Generative Model for 3D Urban Scene Understanding

8/18

Probabilistic Modelpriorsparameters

observations

relativeorientation

numberof arms

A Generative Model for 3D Urban Scene Understanding

8/18

Probabilistic Modelpriorsparameters

observations

relativeorientation

numberof arms
center cwidth w
rotation r

A Generative Model for 3D Urban Scene Understanding

8/18

Probabilistic Modelpriorsparameters

A Generative Model for 3D Urban Scene Understanding

observations

8/18

Probabilistic Model

A Generative Model for 3D Urban Scene Understanding

8/18

Probabilistic Model

A Generative Model for 3D Urban Scene Understanding

8/18

Likelihood: Static Features
Static observation likelihoodp(xs |o, θ, ρs ) ∝ exp{βf (xs , o, θ, ρs )}A Generative Model for 3D Urban Scene Understanding

9/18

Likelihood: Dynamic Features
Flow observation likelihood(kv − ti k22d2p(xf |o, θ, ρf ) ∝ max exp − i 2 −i2ρf 1
2ρ2f 2A Generative Model for 3D Urban Scene Understanding)

10/18

Likelihood: Dynamic Features
Flow observation likelihood(kv − ti k22
d2p(xf |o, θ, ρf ) ∝ max exp − i 2 −i2ρf 12ρ2f 2

A Generative Model for 3D Urban Scene Understanding)

10/18

Learning & Inference
Learning:
Orientation:
Gibbs sampling with Dirichlet
Process Mixture Model (MAP)
Center/rotation/width:
Maximum Likelihood
Observation model:
MH sampling (MAP)
Inference:
Reversible Jump MCMC:
Local MH moves
Global MH moves
Reversible jumps
A Generative Model for 3D Urban Scene Understanding

11/18

Learning & Inference
Learning:
Orientation:
Gibbs sampling with Dirichlet
Process Mixture Model (MAP)
Center/rotation/width:
Maximum Likelihood
Observation model:
MH sampling (MAP)
Inference:
Reversible Jump MCMC:
Local MH moves
Global MH moves
Reversible jumps
A Generative Model for 3D Urban Scene Understanding

11/18

Learning & Inference
Learning:
Orientation:
Gibbs sampling with Dirichlet
Process Mixture Model (MAP)
Center/rotation/width:
Maximum Likelihood
Observation model:
MH sampling (MAP)
Inference:
Reversible Jump MCMC:
Local MH moves
Global MH moves
Reversible jumps
A Generative Model for 3D Urban Scene Understanding

11/18

Learning & Inference
Learning:
Orientation:
Gibbs sampling with Dirichlet
Process Mixture Model (MAP)
Center/rotation/width:
Maximum Likelihood
Observation model:
MH sampling (MAP)
Inference:
Reversible Jump MCMC:
Local MH moves
Global MH moves
Reversible jumps
A Generative Model for 3D Urban Scene Understanding

11/18

Sampling the Model
Samples from the prior:
6060

60

505050404040303030202020101010000−10−10−10−20−4060−30−20−100102030−2040 −40
60

−30

−20−100102030−2040 −406050505040

4040303030

202020101010000−10−10−10−20−40

−30

−20

−100

1020

30

−2040 −40

−30

−20−10010A Generative Model for 3D Urban Scene Understanding

2030−2040 −40

−30

−20−100102030

40

−30

−20−100102030

4012/18

Estimating Geometry and Topology
Accuracy in estimating kLocation error
Orientation error
Overlap measure

GP Regression
61.1 %
5.4 m
14.1 deg
49.3 %A Generative Model for 3D Urban Scene Understanding

Ours
92.9 %
4.4 m
6.6 deg
62.7 %

13/18

Improving Object Detection
Re-weight scores of [Felzenszwalb08] using spatial prior

Increase in average precision from 71.3 % to 74.9 %precision

10.5
our method
baseline
00

0.2

0.4

0.6

0.81

recallA Generative Model for 3D Urban Scene Understanding

14/18

Extracting Semantic Activities

Activity: k(k − 1) dimensional binary vector aError measure: Normalized Hamming distance
Results:
Hamming distanceGP regression
0.16

A Generative Model for 3D Urban Scene Understanding

Ours
0.08

15/18

Extracting Semantic Activities

Activity: k(k − 1) dimensional binary vector aError measure: Normalized Hamming distance
Results:
Hamming distanceGP regression
0.16

A Generative Model for 3D Urban Scene Understanding

Ours
0.08

15/18

Extracting Semantic Activities

Activity: k(k − 1) dimensional binary vector aError measure: Normalized Hamming distance
Results:
Hamming distanceGP regression
0.16

A Generative Model for 3D Urban Scene Understanding

Ours
0.08

15/18

Extracting Semantic Activities

Activity: k(k − 1) dimensional binary vector aError measure: Normalized Hamming distance
Results:Hamming distanceGP regression
0.16

A Generative Model for 3D Urban Scene Understanding

Ours
0.08

15/18

Extracting Semantic Activities

Activity: k(k − 1) dimensional binary vector aError measure: Normalized Hamming distance
Results:Hamming distanceGP regression
0.16

A Generative Model for 3D Urban Scene Understanding

Ours
0.08

15/18

Extracting Semantic ActivitiesActivity: k(k − 1) dimensional binary vector aError measure: Normalized Hamming distance
Results:
Hamming distanceGP regression
0.16

A Generative Model for 3D Urban Scene Understanding

Ours
0.08

15/18

Extracting Semantic Activities

Activity: k(k − 1) dimensional binary vector aError measure: Normalized Hamming distance
Results:
Hamming distanceGP regression
0.16

A Generative Model for 3D Urban Scene Understanding

Ours
0.08

15/18

Extracting Semantic Activities

Activity: k(k − 1) dimensional binary vector aError measure: Normalized Hamming distance
Results:
Hamming distanceGP regression
0.16

A Generative Model for 3D Urban Scene Understanding

Ours
0.08

15/18

Extracting Semantic Activities

Activity: k(k − 1) dimensional binary vector aError measure: Normalized Hamming distance
Results:
Hamming distanceGP regression
0.16

A Generative Model for 3D Urban Scene Understanding

Ours
0.08

15/18

Inference Results

A Generative Model for 3D Urban Scene Understanding

16/18

Summary and Future Work

Conclusion:
Generative model of 3D urban scenes
Static and dynamic features
Improved object detection & activity recognition
Future work:
Features: Vanishing points, scene labels, ...
Joint object and scene layout reasoning
Scene understanding with a single camera
A Generative Model for 3D Urban Scene Understanding

17/18

Summary and Future Work

Conclusion:
Generative model of 3D urban scenes
Static and dynamic features
Improved object detection & activity recognition
Future work:
Features: Vanishing points, scene labels, ...
Joint object and scene layout reasoning
Scene understanding with a single camera
A Generative Model for 3D Urban Scene Understanding

17/18

Summary and Future WorkConclusion:
Generative model of 3D urban scenes
Static and dynamic features
Improved object detection & activity recognition
Future work:
Features: Vanishing points, scene labels, ...
Joint object and scene layout reasoning
Scene understanding with a single camera
A Generative Model for 3D Urban Scene Understanding

17/18

Summary and Future Work

Conclusion:
Generative model of 3D urban scenes
Static and dynamic features
Improved object detection & activity recognition
Future work:
Features: Vanishing points, scene labels, ...
Joint object and scene layout reasoning
Scene understanding with a single camera
A Generative Model for 3D Urban Scene Understanding

17/18

Summary and Future Work

Conclusion:
Generative model of 3D urban scenes
Static and dynamic features
Improved object detection & activity recognition
Future work:
Features: Vanishing points, scene labels, ...
Joint object and scene layout reasoning
Scene understanding with a single camera
A Generative Model for 3D Urban Scene Understanding

17/18

Summary and Future Work

Conclusion:
Generative model of 3D urban scenes
Static and dynamic features
Improved object detection & activity recognition
Future work:
Features: Vanishing points, scene labels, ...
Joint object and scene layout reasoning
Scene understanding with a single camera
A Generative Model for 3D Urban Scene Understanding

17/18

Geometry Model
Unique orientation vector o, constrained to the ∆k−1 simplex
kXoi = 1oi ≥ 0i=1

A Generative Model for 3D Urban Scene Understanding

αi = r + 2πi−1Xoj

j=1

18/18