Supplementary Material for
Exploiting Object Similarity in 3D Reconstruction
Chen Zhou1,2
Fatma Güney3
Yizhou Wang1,2
Andreas Geiger3
1
Nat’l Engineering Laboratory for Video Technology
2Cooperative Medianet Innovation Center, Peking University, China
3
MPI for Intelligent Systems Tübingen{zhouch,yizhou.wang}@pku.edu.cn{fatma.guney,andreas.geiger}@tue.mpg.deAbstract
In this supplementary material we first provide details of our experimental setup and the inference process with respect
to the individual optimization blocks. Furthermore, we provide additional quantitative and qualitative results: First, weshow the full plots corresponding to Figure 5 in the main paper. Second, we visualize the accuracy and completeness of theindividual methods. And finally, we qualitatively compare the results of the proposed approach to the initial reconstruction
baseline on novel sequences.

1. Experimental Setup
For capturing our dataset and the 3D ground truth, we used a Velodyne HDL-64 laser scanner, running at 10 fps. For
synchronization, we attached a hardware trigger to the scanner (reed contact) which triggers all cameras at the moment thescanner is facing forward. Each scan returns the timestamp of each point which is used to unwarp the point cloud according
to the vehicle motion obtained using a high-precision OXTS 3003 RTK GPS/IMU unit (2cm accuracy). The cameras have a
global shutter and are in perfect sync. The average driving velocity was 40 km/h.

2. InferenceThis section provides details on optimizing the individual blocks in our energy function
argminNXΨi (π, φ, X, V, k)(1)

π,φ,X,V,k i=1

whereΨi (·)= ψshp (πi , φki , xi ) + λscale ψscale (si , vki )(i)+ λreg ψreg(πi )(2)

The reader is referred to section 3.3 of the paper for the definition of each term.

2.1. Block {π, V}
We optimize Eq. 1 with respect to {π, V} via gradient descent. For clarity of exposition we consider a single term Ψi ,dropping the observation index i. First, we note that the derivatives of ψscale (s, vk ) and ψreg (π) with respect to {π, V} arereadily given due to their simple quadratic form. The derivative of ψshp (π, φ, x) with respect to π is given by∂ψshp (π, φ, x)∂π=X2 ∂w

(φ(p, x) − d(π(p)))

p∈Ω1∂π− 2w(π(p))(φ(p, x) − d(π(p))∂d∂π(3)

Using pt to denote π(p) ∈ R3 (the transformed point) and the chain rule we have:∂w∂π∂d∂π==∂w ∂pt∂pt ∂π∂d ∂pt∂pt ∂π(4)(5)∂d∂wand ∂p
are gradients of the weight field and the signed distance function field which we obtain using bilinear
Where ∂pttt
interpolation. Finally, we compute ∂p∂π . According to our parametrization, the transformed point is given by pt = RSp + cos r − sin r 0 [t 0]T , where R is the rotation matrix formed by 2D rotation angle r (i.e., sin r cos r 0 ) , S is the diagonal scaling matrix
001(i.e. diag(s)) and t is the 2D translation vector in the x-y plane (i.e., ground plane).
Thus, we obtain:∂pt∂r==∂pt∂s∂(RSp + t)∂r
− sin(r)sx px − cos(r)sy py cos(r)sx px − sin(r)sy py 0∂(RSp + t)∂scos(r)px − sin(r)pycos(r)py=  sin(r)px
00=∂pt∂t
=I2×201×2(6)
(7)

(8)
00
pz

(9)

(10)

2.2. Block {φ, X}
We solve the M independent weighted PCA problems using the robust approach of Torre et al. [1].

2.3. Block {X, k}
Recall that X = {x1 , . . . , xN } are the coefficient vectors for all observations and k = {k1 , . . . , kN } denote the modelassociations with ki ∈ {1, . . . , M }. With slight change in notation, let the linear model associated with observation i bedescribed by the mean µki ∈ R|Ω| and the D-dimensional orthonomal basis ξkdi ∈ R|Ω| , d ∈ {1, . . . , D}. The coefficient
vector xi for observation i is obtained by solving the following weighted least squares estimation problem (in closed form):
BTki Wi Bki xi = Bki Wi (di − µki )(11)Here, B = {ξk1i , . . . , ξkDi } denotes the |Ω|×D basis matrix, Wi = diag(w(πi (p))) ∈ R|Ω|×|Ω| , and di = d(πi (p)) ∈ R|Ω| isthe vector of signed distance function values for observation i. We estimate the solution to Eq. 11 for each ki ∈ {1, . . . , M }in order to find the minimizer of Eq. 9 in the main paper.

3. Additional Results
In this section, we first show the full plots for all methods corresponding to Figure 5 in the main paper (Fig. 1). Next,
we visualize the accuracy (Fig. 2) and the completeness (Fig. 3) of the different methods for the sequence presented in themain paper. We also include zoomed in comparisons of our fused result (PC 0) to our initial reconstruction in terms of bothaccuracy (Fig. 4, Fig. 6) and completeness (Fig. 5, Fig. 7). Finally, we show additional qualitative results on other sequencesfrom our dataset in Fig. 8 – Fig. 12.

References[1] F. D. la Torre and M. J. Black. Robust principal component analysis for computer vision. In ICCV, pages 362–369, 2001.
2Buildings

All60
40
20

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1100

Completeness (%)

80

00.2

8060402000.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

180
60
40
2000.2

10080

80

80

40
20

0.3

0.4

0.5

0.6

0.7

0.8

0.9

60
40
20
00.2

1Accuracy (%)

10060

0.3

Threshold (meters)

0.4

0.5

0.6

0.7

0.8

0.9

00.2

10.3

0.4

0.5

0.6

0.7

0.8

0.9

1F1 score (%)

80

F1 score (%)

10080
60
40
20
00.2

Threshold (meters)

PMVS2 (default)
PMVS2 (optimized)
PMVS2 (default) + Poisson

0.3

0.4

0.5

0.6

0.7

0.8

0.9

10.30.4

0.5

0.6

0.7

0.8

0.9

10.8

0.9

1Threshold (meters)

100

00.2

0.7

2080

200.6

40

100

400.5

60

Threshold (meters)

60

0.4

Threshold (meters)

100

00.2

0.3

Threshold (meters)

Accuracy (%)

Accuracy (%)

Threshold (meters)

F1 score (%)

Cars

100

Completeness (%)

Completeness (%)

100

0.80.91Threshold (meters)

PMVS2 (optimized) + Poisson
PMVS2 (default) + SSD
PMVS2 (optimized) + SSD

60
40
20
00.2

0.3

0.4

0.5

0.6

0.7

Threshold (meters)

Ours (Initial)Ours (PC 0)
Ours (PC 1)

Figure 1: Varying the Evaluation Distance. This figure shows quantitative results in terms of completeness, accuracy andF1 score when varying the evaluation distance τ between 0.2 and 1.0 m.

Figure 2: Accuracy Visualization. From top to bottom: Point cloud from PMVS2 using the default parameter setting,point cloud from PMVS2 with optimized parameters, PMVS2 (optimized) + Poisson, PMVS2 (optimized) + SSD, our initialreconstruction, and our fused result (PC 0). Green-coloured points are reconstructed 3D points for which at least one groundtruth 3D point is within a distance of τ = 0.5 m and red-coloured points are other (falsely) reconstructed points.

Figure 3: Completeness Visualization. From top to bottom: Point cloud from PMVS2 using the default parameter setting,point cloud from PMVS2 with optimized parameters, PMVS2 (optimized) + Poisson, PMVS2 (optimized) + SSD, our initial reconstruction, and our fused result (PC 0). Green-coloured points are ground truth 3D points for which at least one
reconstructed 3D point is within a distance of τ = 0.5 m and red-coloured points are other (missed) ground truth points.

Figure 4: Accuracy Visualization. Our initial reconstruction and our fused result (PC 0). Green-coloured points are reconstructed 3D points for which at least one ground truth 3D point is within a distance of τ = 0.5 m and red-coloured points areother (falsely) reconstructed points.

Figure 5: Completeness Visualization. Our initial reconstruction and our fused result (PC 0). Green-coloured points areground truth 3D points for which at least one reconstructed 3D point is within a distance of τ = 0.5 m and red-colouredpoints are other (missed) ground truth points.

Figure 6: Accuracy Visualization. Our initial reconstruction and our fused result (PC 0). Green-coloured points are reconstructed 3D points for which at least one ground truth 3D point is within a distance of τ = 0.5 m and red-coloured points areother (falsely) reconstructed points.

Figure 7: Completeness Visualization. Our initial reconstruction and our fused result (PC 0). Green-coloured points areground truth 3D points for which at least one reconstructed 3D point is within a distance of τ = 0.5 m and red-coloured
points are other (missed) ground truth points.

4. Additional Qualitative Result

Figure 8: Reconstruction of a sequence with elongated buildings from two different viewpoints. The first and third rows
show the original (initial) reconstruction, the second and fourth row show our final result. Note how our model is able tocomplete missing walls by reasoning jointly about buildings of similar shape.

Figure 9: Reconstruction of cars. The left figure shows the original reconstruction, the right figure shows our final result.

Figure 10: Reconstruction of a third sequence. The first row shows the original reconstruction, the second row shows ourfinal result. Note how the proposed approach is to complete buildings even in the presence of very little surface informationin the initial reconstruction.

Figure 11: Comparison of reconstruction for two more sequences.

Figure 12: Comparison of reconstruction result using PC0 and PC1. From top to bottom: initial reconstruction, PC0 andPC1. Note for the righmost building, PC1 correctly captures the variation in the front wall, however noise are introduced on
the roof because of weaker regularization.