Supplementary Material for
Joint 3D Estimation of Objects and Scene Layout

Andreas Geiger
Karlsruhe Institute of Technology

Christian Wojek
MPI Saarbrücken

Raquel Urtasun
TTI Chicago

geiger@kit.edu

cwojek@mpi-inf.mpg.derurtasun@ttic.edu

In this supplementary material, we first show how the 3D trackets can be generated from 2D observations. We then depict additional results for inference when θ is know and when it is unknown.

1

3D Tracklet Generation from 2D Observations

This section describes the mapping
ϕ : f, b, C → π, Σ(1)which takes a frame index f ∈ N, an object bounding box b ∈ R4 and the calibration parameters
C, and maps it to the object location π ∈ R2 and uncertainty Σ ∈ R2×2 in bird’s eye perspective.
As cues for this mapping we use the bounding box width and height, as well as the location of thebounding box foot point. Scene depth adaptive error propagation is employed for obtaining Σ. The
unknown parameters of the mapping are the uncertainty in bounding box location σu , σv , width σ∆uand height σ∆v as well as the real-world object dimensions ∆x , ∆y along with their uncertainties
σ∆x , σ∆y . Those parameters are learned from a separate stereo image training dataset, including
1020 images with 3634 manually labeled vehicles in total.
Let (u, v) be the image coordinate of the lower middle point of the object’s bounding box obtainedby a classical 2D object detector. Similarly, let (∆u, ∆v) be the width and height of the boundingbox. Further, let (x, y, z) be the 3D coordinates of an object (alignment: right, down, forward) andlet (∆x, ∆y) be the object width and height in meters, measured via parallel-projection to the planez = 0, which is coplanar to the image plane.
As u, v, ∆u, ∆v are observations from the object detector, we are interested inp(u, v, ∆u, ∆v|y, z, ∆x, ∆y)= p(u, v|x, z, ∆x, ∆y) p(∆u|x, z, ∆x, ∆y) p(∆v|x, z, ∆x, ∆y)= p(u, v|x, z) p(∆u|z, ∆x) p(∆v|z, ∆y)∝ p(x, z|u, v) p(z|∆u, ∆x) p(z|∆v, ∆y)(2)where in the last line we have made the assumption of a uniform prior over x and z. We will now
examine each of these terms individually.
1.1

Estimating p(x, z|u, v)

We assume p(x, z|u, v) = N (µ1 , Σ1 ) and the standard pinhole projection model !
xu
3×4  y v =Pz 11
where P = KTR is composed of a calibration matrix K3×3 , a transformation from road plane
coordinates to camera coordinates T3×4 which is estimated by RANSAC ground plane fitting and1

an additional camera pitch error θ, parameterized by the rotation matrix R4×4 (θ). Given (u, v) weobtain (x, z) by solving the linear system xA=bzwith
A(u, v)

=
θ=0



=

b(u, v)=

u(p33 cos θ − p32 sin θ) − (p13 cos θ − p12 sin θ)v(p33 cos θ − p32 sin θ) − (p23 cos θ − p22 sin θ)up33 − p13
vp33 − p23

up31 − p11
vp31 − p21
up31 − p11
vp31 − p21
p14 − up34
p24 − vp34

(3)
(4)

(5)

where pij are elements of P, with µ1 = A−1 b. Here, we make use of the fact that the road plane isknown and has been adjusted for within P, hence the pitch error θ of R is zero on average, and only
the variance of θ is considered in the following equations, which derive the covariance matrix Σ1 .
Assuming the covariance of (u, v) is known, the covariance of (x, z) can be obtained by error propagation. Since the transformation is non-linear, we linearize it by means of a first-order Taylor series
expansion. Assuming σu , σv , σθ to be given, the covariance of (x, z) can be computed as 2σu 00
Σ1 = J  0 σv2 0  JT00 σθ2with JacobianJ=∂(A−1 b)∂u∂(A−1 b)∂v∂(A−1 b)∂θwhere∂(A−1 b)= ∂A−1 b + A−1 ∂b= −A−1 ∂AA−1 b + A−1 ∂b= A−1 ∂b − ∂AA−1 bwith∂Ap31 p33=00∂u∂A00=p31 p33∂v

∂A0 p12 − up32
=0 p22 − vp32
∂v∂b−p34
=0∂u

∂b0
=−p34∂v ∂b0=0∂v

at θ = 0.
1.2

Estimating p(z|∆u, ∆x)

We assume p(z|∆u, ∆x) = N (µ2 , σ22 ). From the pinhole model we have∆u =f ∆xzor equivalently
f ∆x∆uwhich again is a non-linear function, this time in ∆u. Using the same reasoning from above, weobtain the variance in z as 2
σ0σ22 = J ∆u
JT20σ∆xwith Jacobianf ∆xfJ = − (∆u)
2∆uµ2 = z =2

1.3

Estimating p(z|∆v, ∆y)Similarly, we can writeµ3 = z =f ∆y∆v

 2σ∆v00

with variance
σ32 = Jand Jacobian

1.4

2σ∆yf ∆yJ = − (∆v)
2f∆vJTParameter Learning

The unknown parameters for our tracklet model are σu , σv , σ∆u , σ∆v as well as ∆x, ∆y, σ∆x , σ∆y .
A principled way to obtain these values is to estimate them automatically from training data. We do
this by building a dataset of 1020 images including 3634 manually annotated vehicles and disparity
maps. Our labels include a tightly fitting bounding box as well as the heading of the car, quantized
into 8 orientations.
We first estimate the parameters related to detection accuracy σu , σv , σ∆u , σ∆v by comparing theobject detections to manually labeled bounding boxes. Due to the characteristics of sliding-window
detectors, we expect the noise to be dependent on the object scale. A good approximation to object
scale is the bounding box height ∆v, since – in contrast to the bounding box width – it is largely
invariant with respect to car orientation. Furthermore, it is readily given by the object detector. Figure 1 depicts σu , σv , σ∆u , σ∆v as a function of ∆v, computed from the differences between object
detections and manually labeled bounding boxes. As expected the noise behaviour is approximately
linear in ∆v, thus we represent this dependency via a linear least-squares fit.
∆x, ∆y, σ∆x , σ∆y can be easily computed from our annotated data in conjunction with known
stereo disparity via
z∆ub∆uz∆v
b∆v
∆x ==
∆y ==
fd
fd
where b is the camera baseline and d is the median value of all valid disparities within the boundingbox. Here ∆u and ∆v are given by the manually labeled ground truth. Special care has to be takenfor ∆x, as it strongly depends on the orientation of the vehicle. We tackle this problem by learning aseparate ∆x for each of the eight canonical car orientations. Figures 2 and 3 show the results: Whenviewed from behind or frontally, a typical car appears ∼ 2 meters wide, while it appears ∼ 4.4
meters wide when viewed from the side.
1.5

Putting it together

Multiplying the terms in the last row of Eq. 2 leads tox, z|u, v, ∆u, ∆v ∼ N (µ, Σ)withΛ =Σ =µ =

Λ1 + Λ2 + Λ3
Λ−1ΣΛ1 µ1 + ΣΛ2 µ2 + ΣΛ3 µ3(6)(7)(8)

Here Λ is a precision matrix, the index denotes the corresponding term, Λ1 has full rank and Λ2 , Λ3
are singular matrices of the form
0 00 0
Λ2 =Λ3 =0 λ20 λ3
Resulting depth probabilities for 3 tracklet detections are depicted in Fig. 4. Here the colored curves
are the individual cues discussed in the previous sections and the black curves are the combinedresults.
340
35

σuσv
σ∆ uσ

30σ∆ v25σuσv

20σ∆uσ∆v

a0.0400
0.09150.14740.1111b
1.8239-1.3080
1.1972-1.2124

a0.1076
0.03960.21070.0893b3.8502
1.98485.36251.0360a0.0975
0.06350.18420.0806b3.14070.52815.26660.832315105020406080∆v

100120140(a) Heading 1,240σu
σv

35

σ∆ u
σ∆ v

σ

3025σu
σv

20

σ∆u
σ∆v

15105020

406080∆v

100120140(b) Heading 3,44035σu
σv
σ∆ u

σ

30

σ∆ v

25σu
σv

20σ∆u
σ∆v

15105020

406080∆v

100120140(c) Heading 5,..,8Figure 1: Object detection accuracies and line parameters (σ = a ∆v + b)

4

1.6

1.6

1.4

1.4

1.2

1.2
2

110.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

00p(∆ x3)p(∆ x )

2
1.8

12345
∆ x1 [meters]

6

7

008

22

1.8

1.8

1.6

1.6

1.4

1.4

1.2

1.2p(∆ y)p(∆ x1)

2
1.8

1

0.8

0.6

0.6

0.4

0.4

0.2

0.2
12

345
∆ x3 [meters]

6782345∆ x2 [meters]

6

78

1234
∆ y [meters]

6781

0.8

001005Figure 2: Object size probability density

513

µσ

∆x1
1.86
0.28

∆x2
4.37
0.64

∆x3
2.82
0.63

∆y1.590.22

Figure 3: Estimated Object Parameters (∆x = width, ∆y = height)

5

72

684

0.2
pos
width
heightcombinedp(z)

0.15
0.1
0.05
00102030
z40

50

60

0.4
pos
width
heightcombinedp(z)

0.3
0.2
0.1
00102030
z40

50

60

0.4
pos
width
heightcombinedp(z)

0.3
0.2
0.1
00102030
z40

5060403020100−20Figure 4: Example of depth probabilities estimated in this case from 3 detections

6−1001020

2

Inference Results

This section shows randomly selected inference results when θ is know and when θ is unknown on
2,3 and 4-armed intersections. Note that in some cases, even for humans, it is extremely difficult
to distinguish 3-armed from 4-armed intersections using only monocular cues. For every row, 3D
tracklets projected into the image are shown in the left, while results with θ known and θ unknown
are shown in the next two images.

Figure 5: Automatically inferred scene descriptions for 4-armed intersections. (Left) Trackets from all
frames are superimposed in the last frame of the sequence. (Middle) Inference result with θ known and (Right)
θ unknown. Detections belonging to the same tracklet are grouped by color, the observer is depicted in black.
Inferred activities are shown in red. The ground truth labels for the intersection layout are given in blue.

7

Figure 6: Automatically inferred scene descriptions for 3-armed intersections. (Left) Trackets from all
frames are superimposed in the last frame of the sequence. (Middle) Inference result with θ known and (Right)
θ unknown. Detections belonging to the same tracklet are grouped by color, the observer is depicted in black.
Inferred activities are shown in red. The ground truth labels for the intersection layout are given in blue.

8

Figure 7: Automatically inferred scene descriptions for 2-armed intersections. (Left) Trackets from all
frames are superimposed in the last frame of the sequence. (Middle) Inference result with θ known and (Right)
θ unknown. Detections belonging to the same tracklet are grouped by color, the observer is depicted in black.
Inferred activities are shown in red. The ground truth labels for the intersection layout are given in blue.

9