Exploiting Object Similarity in 3D Reconstruction
Chen Zhou1,2
Fatma Güney3
Yizhou Wang1,2
Andreas Geiger3
1
Nat’l Engineering Laboratory for Video Technology
2
Cooperative Medianet Innovation Center, Peking University, China
3
MPI for Intelligent Systems Tübingen{zhouch,yizhou.wang}@pku.edu.cn{fatma.guney,andreas.geiger}@tue.mpg.de

Abstract

lutions result in relatively sparse 3D point clouds. Besides,
lighting conditions are difficult, occlusions are omnipresent
and many objects are only visible in a few frames as illustrated in Fig. 1. In this work, we address these difficulties by taking advantage of a fact which hitherto has been
largely ignored: The larger the reconstructed area, the more
likely objects of similar type and shape (e.g., buildings orvehicles) will occur in the scene. Furthermore, man-madeenvironments are designed to be visually pleasing. Therefore, reoccuring structures like buildings often exhibit similar shapes in local neighborings.
Inspired by the impressive capability of toddlers to learn
about objects from very few observations [3], we ask thefollowing question: Can we exploit recurring objects ofsimilar 3D shapes for jointly learning a shape model while
reconstructing the scene? Such a system would be useful inmany ways: First, we can expect increased completeness ofthe 3D reconstruction by regularizing across shapes using
the principle of parsimony (i.e., by limiting the number ofmodels and parameters). Second, we obtain a decomposition of a scene into its constituent objects in terms of 3D
bounding boxes and segments. And finally, other tasks suchas recognition or synthesis could benefit from the learned
3D shape models as well.
Our approach is summarized as follows: We first obtainan initial 3D reconstruction by structure-from-motion andvolumetric fusion of disparity maps using a memory efficient representation based on voxel hashing [31]. Next, wetrain several generic 3D object detectors by extending exemplar SVMs [29] to truncated signed distance functions(TSDF) in 3D. Given the 3D box proposals from thesedetectors, we formulate a discrete-continuous optimization
problem which we solve using block coordinate descent.
More specifically, we minimize the difference between the
TSDF values of the initial reconstruction and the predicted
TSDF values by jointly assigning each proposal to a volumetric 3D shape model, optimizing for the pose of the
3D proposals and the latent shape variables, and finding
the shape model parameters. Furthermore, we contribute

Despite recent progress, reconstructing outdoor scenes
in 3D from movable platforms remains a highly difficult endeavour. Challenges include low frame rates, occlusions,large distortions and difficult lighting conditions. In thispaper, we leverage the fact that the larger the reconstructed
area, the more likely objects of similar type and shape will
occur in the scene. This is particularly true for outdoor
scenes where buildings and vehicles often suffer from missing texture or reflections, but share similarity in 3D shape.
We take advantage of this shape similarity by localizing objects using detectors and jointly reconstructing them whilelearning a volumetric model of their shape. This allows us
to reduce noise while completing missing surfaces as objects of similar shape benefit from all observations for therespective category. We evaluate our approach with respect
to LIDAR ground truth on a novel challenging suburban
dataset and show its advantages over the state-of-the-art.

1. Introduction
3D reconstruction is one of the fundamental problems incomputer vision and has consequently received a lot of attention over the last decades. Today’s hardware capabilities
allow for robust structure-from-motion pipelines capable ofreconstructing sparse 3D city models from millions of internet images [10] or dense models from video sequences inreal time [42]. With the advent of the Kinect sensor, depth
information has become available indoors and approaches
based on volumetric fusion [31] or mesh optimization [46]are able to produce reconstructions with fine details given alarge number of RGB-D observations.
However, obtaining accurate reconstructions outdoors
and in less constrained environments observed during urban
driving [13, 33] remains highly challenging: RGB-D sensors (e.g., Kinect) can’t be employed outdoors due to theirvery limited range and expensive LIDAR-based mobile so1

(a) Fisheye Image(b) Lighting(c) Occlusions(d) Saturation(e) Reflections(f) Appearance

Figure 1: Challenges of 3D Reconstruction from Movable Platforms. We have collected a novel suburban dataset foromnidirectional 3D reconstruction from fisheye images (a). Our dataset has been captured under normal daylight conditions
and poses a variety of challenges to current reconstruction pipelines, including uncontrolled light conditions (b), occlusions(c), sensor saturation at bright surfaces (d), reflecting surfaces (e), and large appearance changes between successive frames
when driving at regular speeds (f). We propose to overcome these challenges by regularizing across objects of similar shape.
a novel multi-view reconstruction dataset recorded from amoving platform on which we evaluate the proposed approach with respect to several state-of-the-art baselines. We
make our code, dataset and supplementary material available on our project website1 .

2. Related Work
Existing works on multi-view 3D reconstruction canbe roughly grouped into three categories according to theemployed representation: Point-based approaches [12, 37]
which produce a set of (oriented) 3D points from which
a mesh is extracted in a second step using, e.g., Poisson
surface reconstruction [23]. Methods, which directly optimize the vertices and faces of a textured 3D mesh [7,41] (orgeometric primitives [27] ) in order to minimize a photoconsistency measure. And finally, volumetric techniques[30, 39] which represent the surface implicitly as the zerocrossing of a distance function or using a probability map
defined at regular voxel locations in 3D. In this paper, wefollow the third line of methods as volumetric representations are flexible in terms of surface topology and allow for
combining different shapes via volumetric fusion [6].
To overcome outliers in range maps obtained from stereo
vision, local regularization is typically employed. This
leads to smoother reconstructions by penalizing the perimeter of level sets [43] or encouraging local planarity [18].
For some scenarios, even stronger assumptions can be leveraged, such as piecewise planarity [13] or a Manhattan world[11]. The model proposed in this paper can also be seen asa prior on the reconstruction, but taking a different viewpoint: Instead of encouraging smoothness of surfaces, we
encourage shape similarity across different instances of thesame object category. Our objective integrates observations
from several different instances which allows our model tofill-in occluded parts or textureless regions. In contrast tomodels assuming piecewise planarity or a Manhattan world,
our approach also applies to non-planar object classes such
1 http://www.cvlibs.net/projects/similarity reconstruction

cars. In this sense, our ideas are related to the depth superresolution method of Hornacek et al. [21]. However, while
they tackle single depth images using a patched-base representation, we model complete 3D scenes at the object level.
Lately, models integrating appearance information into
the reconstruction problem have gained popularity [9, 32].
Häne et al. [17] leverage the fact that semantics and surface
orientation are mutually dependent, e.g., the surface normalof the ground is more likely to face upwards than downwards. This knowledge helps in particular for object classes
with a dominant orientation (e.g., ground) but has less advantages for classes with a more uniform normal distribution (e.g., building, car). Kundu et al. [24] directly constrain
the range of possible depth values by conditioning ray potentials on the semantic class. To overcome shape ambiguities, object knowledge has been leveraged by Güney et al.
[15] for stereo matching and by Häne et al. [16] for volumetric reconstruction. Prisacariu et al. [34,35] have demonstrated impressive results by leveraging GPLVMs for learning a non-linear TSDF embedding of the object shape forsegmentation and reconstruction. While those approaches
typically rely on pre-trained appearance models of known
object classes, semantic classifiers or a dataset of 3D CAD
models, our approach learns these models “on the fly” while
reconstructing the scene. Apart from a generic object detector, no semantic annotation or 3D models are required. Furthermore, most of the approaches for object-based reconstruction focus on a single object while our method handles
several objects at the same time.
Related to our approach are also a number of techniques which consider joint image alignment and segmentation or appearance estimation [5] with applications in facerecognition [8] and medical imaging [40] . Our approachshares similarity with these methods in terms of estimating an instance specific transformation jointly with a lowdimensional representation of the object. However, our focus is on 3D reconstruction rather than 2D segmentation
and we do not assume that objects are presented in a stereotypical pose. Instead, our method localizes objects accord-

Figure 2: Spherical Rectification. Given two fisheye cameras (x1 , y1 , z1 ) and (x2 , y2 , z2 ), we rectify the images
yielding an angular representation (θ, ϕ) where pixels in thesame row (i.e., same ϕ) are located on an epipolar plane Π.
ing to a discriminatively trained object proposal generator.
Furthermore, our model handles missing information (e.g.,
unobserved voxels) and is able to deal with a broad range ofshapes by clustering them into different model components.
Unfortunately, existing datasets such as KITTI [14] orGoogle Street View [2] provide only a very limited field ofview [14] or sampling rate [2]. For evaluating our approach,we therefore recorded a novel urban dataset with 3D groundtruth. Our setup comprises a combination of front-facing
perspective cameras, side-facing Fisheye cameras, a Velodyne 3D scanner and a SICK pushbroom scanner, capturing
images and laser scans at roughly 10 fps.

3. Method
In this work, we are interested in volumetric 3D reconstruction leveraging the fact that 3D objects with similar
shapes (such as cars or buildings) appear frequently in large
environments. In particular, we aim at jointly clustering objects into categories according to their 3D shape, learning a
3D shape model for each of these categories and completing the 3D reconstruction by filling-in missing information
via restriction to the jointly estimated 3D shape models. We
start with a basic volumetric reconstruction pipeline which
recovers a volumetric representation in terms of a truncatedsigned distance function (TSDF) of the scene. Based on thisreconstruction, we apply an ensemble of exemplar-based
object detectors to find instances of objects in the scene.
The detected objects are then jointly optimized for theirshape and after convergence fused back into the original reconstruction to smoothly blend with non-object parts of thescene such as road or sidewalk.

3.1. Volumetric Fusion
This section describes our basic volumetric reconstruction pipeline which serves as input and baseline to our
method. We leverage the truncated signed distance function (TSDF) representation [6] due to its generality and as
it makes no assumptions about the 3D surface topology,
which is unknown in our case. Surfaces are implicitly represented as the 0-level set of the TSDF and can be easily recovered using, e.g., the marching cubes algorithm byLorensen et al. [28].
Given a sequence of fisheye images, we first obtain the
intrinsic camera parameters as well as the camera poses using the approach of Heng et al. [19]. We rectify the imagesof adjacent frames spherically (see Fig. 2) such that epipolarlines become horizontal and run semi-global matching [20]in order to obtain disparity maps. Taking advantage of theefficient hashmap representation of Niesner et al. [31], we
perform volumetric fusion [6], i.e., we update the weights
w(p) and the truncated signed distance values d(p) via
ˆwi (p)di (p) + ŵ(p)d(p)wi (p) + ŵ(p)wi+1 (p) = wi (p) + ŵ(p)di+1 (p) =

(1)
(2)

where p ∈ R3 denotes the location of a cell in the voluˆ
metric grid, and d(·),ŵ(·) represent the truncated signed
distance value and weight of the current observation (i.e.,
disparity map), respectively. We use a truncation thresholdof 1.2 m and a weight that slowly decays behind the surˆ byface. As we found that approximating the distances d(·)
calculating them along the viewing ray [39] results in artifacts when the ray hits the surface at highly slanted angles,
we use a different strategy which approximates the TSDF
values more faithfully: We triangulate each disparity map
and convert the resulting meshes into volumes using a 3D
signed distance transform2 . This leads to correct distanceestimates also for surfaces which are not fronto-parallel.

3.2. Object Detection
Given the volumetric 3D reconstruction from the previous section, we are interested in discovering objects of similar 3D shape to form the basis for our joint optimizationin Section 3.3. Establishing correspondences in 3D space
has several advantages over correspondences in the image
domain: Besides the fact that similar 3D shapes can bematched even if the appearance disagrees, 3D models mustnot be scale invariant, thereby gaining robustness. A variety
of methods can be leveraged for object discovery in 3D, including methods based on convexity/symmetry criteria [22],discriminative approaches [44] or graph matching [45].
In this work, we use a discriminative approach to obtain
3D box proposals: For each object category (e.g., building
or car), we train an ensemble of linear exemplar SVMs [29]directly on the TSDF volume using a small (≤ 3) number ofannotated instances. Compared to learning more complex
features [38], we found this simple approach sufficient forour needs. First, we extract a small subsequence for training in which we label the relevant objects using 3D bounding boxes. For efficient object labeling we developed a 3D
visualization tool based on WebGL, which allows to label
2 http://grail.cs.washington.edu/software-data/ply2vri/

cube [0, 1]3 via πi to coordinate system of the initial reconstruction. In this work, we focus our attention on a subset ofaffine mappings: We assume that all objects are located on
a common (and known) ground plane, and thus parametrize
each mapping πi in terms of the following 5 parameters:

Figure 3: Visualization of TSDF Model. This figure shows
a volumetric visualization of the learned TSDF model mean
for a building (left) and a car (right). Red colors indicate
voxels close to the estimated surface (i.e., 0-level set)
objects in 3D point clouds in a few seconds. Next, we define a feature vector by concatenating all voxels within the
3D box. Each voxel comprises the truncated signed distanceto the surface d(p) as well as a binary flag indicating if thevoxel has been observed or not (i.e., it is within the truncation limit) as feature. We train exemplar SVMs on thisrepresentation using several rounds of hard negative mining
and apply them in a sliding window fashion to the full 3D
reconstruction. As 3D representations offer the advantage
of scale invariance, we slide and rotate a 3D box of fixedscale over the 3D reconstruction volume. Fig. 4 (left) illustrates our detection results.

3.3. Joint Object Reconstruction
Given the initial volumetric 3D reconstruction and thebox proposals, our goal is to jointly recover objects withsimilar shape while learning an object model for each category. More formally, let d(p) ∈ R denote the truncatedsigned distance to the closest surface at point p ∈ R3 . Let
further w(p) ∈ R≥0 denote the weight at point p whichtakes w = 0 in unobserved regions or outside the truncation band, and w = 1 close to the estimated surface. As the
TSDF representation estimated by volumetric fusion (Section 3.1) yields only values at discrete voxel locations, weuse bilinear interpolation for calculating d(p) and w(p) atintermediate points. We formulate our objective as minimizing the distance between the TSDF values of the initialreconstruction specified by the mappings d(p) and w(p),and a set of jointly optimized 3D shape models. Fig. 3 illustrates two of the 3D shape models learned by our approach.
In the following, let N denote the number of observations (i.e., 3D bounding box proposals from Section 3.2)
and let M denote the number of shape models we want tolearn. Let further Ω ⊂ [0, 1]3 be the discrete domain of
our shape models. While many choices are possible, we
simply take Ω as an axis aligned 3D grid with equidistantspacing between points. Let now π = {π1 , . . . , πN } denote a set of affine mappings from the unit cube to R3 , i.e.,
πi : [0, 1]3 → R3 . We may think of these mappings as specifying the location of an observed object i in terms of its 3D
bounding box, obtained by mapping the edges of the unit

• translation in the 2D ground plane: ti ∈ R2
• rotation around the vertical axis: ri ∈ [0, 2π]• scaling in all three dimensions: si ∈ R3≥0
To model the shape of the objects, many choices are possible. For simplicity, we consider a linear embedding. Let
φ = {φ1 , . . . , φM } denote a set of D-dimensional linear
embeddings (D  |Ω|) which specify a real value for everyp ∈ Ω given a coefficient vector x ∈ RD :φj (p, x) = µj (p) +

D
X

(j)

xd ξd (p)

(3)

d=1

Here, the parameters µ ∈ R|Ω| and ξd ∈ R|Ω| specify themean as well as an orthonormal basis, respectively. Each φjrepresents the shape (or TSDF) of model j by specifying asigned distance value at each point p ∈ Ω, given an observation dependent coefficient vector x as input. We specify
one coefficient vector for each observation xi ∈ RD , X ={x1 , . . . , xN }. The complexity of the model varies withthe dimensionality D. For the simplest case (D = 0) we
obtain the mean model. Finally, we associate each modelwith its average scale vj ∈ R3≥0 , V = {v1 , . . . , vM } toencourage scale consistency between different observations
associated to a model. The assignment of models to observations is determined by an index set k = {k1 , . . . , kN }with ki ∈ {1, . . . , M }.
We are now ready to formulate our objective in terms ofa discrete-continuous optimization problem
argminNX

Ψi (π, φ, X, V, k)(4)

π,φ,X,V,k i=1

with energy function
Ψi (·) = ψshp (πi , φki , xi ) + λscale ψscale (si , vki )(i)+ λreg ψreg
(πi )

(5)

where ψshp (·) ensures that the shape of observation i fits
the associated model ki , ψscale (·) encourages agreement inscale, and ψreg (·) is a regularizer which penalizes strong
deviations from the initial detections. λscale , λreg ∈ R≥0
are parameters controlling the influence of the different
terms. We define ψshp (·) as the weighted squared TSDF
difference between the associated model φ and the observation specified via π:X2ψshp (π, φ, x) =w(π(p)) [φ(p, x) − d(π(p))] (6)p∈ΩBlock {X, k}: The final block optimizes the coefficients
X jointly with the model associations k while keeping theother variables fixed. We first note that this optimization
can be performed independently for each observation i ∈{1, . . . , N }. We thus obtain xi and ki for observation i as
Figure 4: Object Poses. This figure shows the initial 3D detections (left) and the result after joint alignment and model
learning (right) overlayed with the initial reconstruction.
Scale discrepancy is measured by the squared distance between the observation scale s and the model scale v:
ψscale (s, v) = ks − vk2

(7)

Finally, strong deviations from the initial detection i are penalized as(i)ψreg(πi ) = (ri − r̂i )2 + kti − t̂i k2 + ksi − ŝi k2(8)where (ri , ti , si ) are the transformation parameters of object i and (r̂i , t̂i , ŝi ) denote the initial transformation parameters specified by the detection.

3.4. Inference
Optimizing Eq. 4 directly is a very difficult task due tothe large dimensionality of the parameter space. We therefore partition the set of variables into tractable blocks andapply block coordinate descent (BCD) to find a local minimizer. Each block lowers the value of the objective function, thus our algorithm is guaranteed to converge. Morespecifically, we initialize π, V and k according to the detections, the mean of φ to a random observation, X = 0,
and iterate the following blocks:
Block {π, V}: The first block optimizes the object poses
π jointly with the model scales V while keeping the othervariables fixed. Due to the small number of parameters involved, we leverage gradient descent in order to find a local minimum. We first differentiate the objective function
in Eq. 4 with respect to {ti , ri , si } and V (see supplementary material for details), and then solve the non-linear leastsquares problem in Eq. 4 using the Ceres solver [1]. Fig. 4
illustrates the object poses before (left) and after (right) convergence.
Block {φ, X}: The second block optimizes the shapemodels φ jointly with the coefficients X while keeping theother variables fixed. Note that this optimization does notdepend on the scale term ψscale (·). Further, the object poses
π are fixed. Thus the objective in Eq. 4 reduces to M independent weighted PCA problems which we solve using therobust approach of Torre et al. [25]. For the case D = 0,
this becomes equivalent to computing the weighted mean.

argmin ψshp (πi , φki , xi ) + λscale ψscale (si , vki )(9)xi ,kiwhich can be found by minimization with respect to xi foreach ki ∈ {1, . . . , M }. As ψscale (·) does not depend on
xi , the minimizer of ψshp (·) with respect to xi can be identified with the solution to an ordinary linear least squares
problem. For details, we refer the reader to the supplementary material.

4. Experimental Evaluation
This section presents results of the proposed approachand compares our method to several baselines both quantitatively as well as qualitatively. As existing datasets for quantitative evaluation of multi-view reconstruction (e.g., Middlebury [36]) focus on small scenes of single objects where
our algorithm is not applicable, we have recorded a novel
suburban dataset for our purpose where objects of similar
shape such as buildings and cars occur frequently. Towards
this goal, we have equipped a station wagon with fisheye
cameras to the side as well as a Velodyne and a pushbroom
laser scanner in order to generate ground truth. All cameras
and laser scanners have been synchronized at a frame rateof roughly 10 fps, yielding about one captured frame every
meter at typical driving speeds of 25 mph. For calibration
we used the approach proposed by Heng et al. [19].
Using this setup, we recorded a sequence of 320 frames
containing several buildings and cars. As the sequences
have been recorded under regular daylight conditions, theyare (unlike Middlebury [36]) highly challenging for current
reconstruction pipelines. Some of the challenges are highlighted in Fig. 1. For evaluation, we subsample the laser
scanner point cloud and the meshes produced by the methods equidistantly at 0.1 m, extrude incomplete planar surfaces, and clip all points in the ground truth as well as all
results at 20 m distance from the closest camera center. See
supplementary for an illustration of our 3D ground truth.

4.1. Methods
As baseline, we leverage the state-of-the-art reconstruction pipeline PMVS2 from Furukawa et al. [12] in combination with two meshing alternatives, namely Poisson reconstruction [23] and smooth signed distance surface reconstruction [4]. As those surface reconstruction methods
tend to produce closed surfaces, we clip large triangles assuggested by Furukawa et al. [12]. In order to make ourdata applicable to PMVS2, we projected the fisheye images

All
Comp.
Acc.
PMVS2 (default)
12.24 % 79.26 %PMVS2 (optimized)
26.65 % 78.31 %PMVS2 (default) + Poisson20.90 % 64.12 %PMVS2 (optimized) + Poisson 27.38 % 51.93 %PMVS2 (default) + SSD
25.27 % 54.21 %PMVS2 (optimized) + SSD
39.27 % 61.51 %Ours (Initial)
60.21 % 94.15 %Ours (PC 0)76.83 % 91.35 %Ours (PC 1)
76.65 % 89.53 %F1 score
21.20 %
39.77 %
31.53 %
35.85 %
34.47 %
47.94 %
73.45 %
83.46 %
82.59 %

Buildings
Comp.
Acc.
16.35 % 85.70 %30.57 % 91.59 %25.49 % 77.05 %29.39 % 75.50 %34.12 % 66.32 %44.07 % 81.19 %52.70 % 93.84 %72.24 % 91.26 %71.55 % 89.46 %F1 score
27.46 %
45.84 %
38.30 %
42.31 %
45.05 %
57.13 %
67.50 %
80.64 %
79.51 %

Cars
Comp.
Acc.
F1 score
0%0%0%7.31 %
96.97 % 13.59 %1.58 % 100.00 % 3.12 %1.46 % 100.00 % 2.88 %0%0%0%
6.94 % 100.00 % 12.98 %85.20 % 94.35 % 89.54 %92.69 % 93.46 % 93.07 %91.53 % 92.90 % 92.21 %Table 1: Quantitative Evaluation. This figure shows the performance of the baseline methods (PMVS2 variants and initialreconstruction) and our method (PC 0/1) with respect to completeness, accuracy and F1 score using a 0.5 m detection
threshold. We separately evaluate all regions within 20 m from the camera, as well as regions representing the categories“buildings” and “cars”.
Buildings
Cars
All
80
60
40
20
0
0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

100

Completeness (%)

100

Completeness (%)

Completeness (%)

100

80
60
40
20
0
0.2

0.3

0.6

0.7

0.8

0.9

40
20
0
0.2

1

100

80

80

60
40
20

0.4

0.5

0.6

0.7

0.8

0.9

60
40
20
0
0.2

1

Accuracy (%)

100

80

0.3

0.3

Threshold (meters)

0.4

0.5

0.6

0.7

0.8

0.9

0
0.2

1

0.4

0.5

0.6

0.7

0.8

Threshold (meters)

PMVS2 (optimized)

0.9

1F1 score (%)

80

F1 score (%)

100

80

0.3

60
40
20
0
0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.8

0.9

0.8

0.9

1

60
40
20

1

Threshold (meters)

PMVS2 (optimized) + Poisson

0.7

Threshold (meters)

100

20

0.6

20

80

40

0.5

40

Threshold (meters)

60

0.4

60

100

0
0.2

0.3

Threshold (meters)

100

0
0.2

F1 score (%)

0.5

60

Threshold (meters)
Accuracy (%)

Accuracy (%)

Threshold (meters)

0.4

80

PMVS2 (optimized) + SSD

0
0.2

0.3

0.4

0.5

0.6

0.7

Threshold (meters)

Ours (Initial)

Ours (PC 0)

Figure 5: Varying the Evaluation Distance. This figure shows quantitative results in terms of completeness, accuracy andF1 score when varying the evaluation distance τ between 0.2 and 1.0 m. To avoid clutter, we only show results for a subsetof the methods. We refer the reader to the supplementary material for the full plots.
onto virtual perspective image planes using an opening angle of 120◦ , such that the images cover all objects which appear in the ground truth. We realized that the performanceof PMVS2 depends heavily on the parameter settings. We
thus optimized all parameters of the baseline with respect tothe reconstruction metrics using grid search on our compute
cluster. For completeness we show both, PMVS2 results
using the optimized parameter settings as well as PMVS2
using the default parameters. In addition to the meshed re-sults, we also directly evaluate the point cloud returned by
PMVS2. Furthermore, we compare the results of our fullmodel with respect to the initial reconstruction. We leverage
the marching cubes algorithm [28] to turn our volumetric reconstruction results into meshes. For all meshes, we remove
spurious isolated vertices created by the reconstruction algorithm in a post-processing step. Throughout all experiments, we set the parameters in our model to λsize = 1000and λreg = 50 which we have been determined empirically.

Figure 7: Model Completion. Our method fills in the occluded/unseen side of objects (left: input; right: our result).
Furthermore, as sky regions often lead to spurious matches,
we trained a sky detector using ALE [26] and removed sky
regions before processing the images for all methods.

4.2. Quantitative Experiments
For quantitative evaluation, we measure performance interms of completeness, accuracy and F1 score. We calculate
completeness as the percentage of ground truth 3D points
for which at least one reconstructed 3D point (i.e., vertex of
the subsampled mesh) is within a distance of τ = 0.5 m.
Similarly, we calculate accuracy as the percentage of reconstructed 3D points for which at least one ground truth 3D
point is within a distance of τ = 0.5 m. Furthermore, we
provide the combined F1 score:
F1 = 2 ·completeness · accuracy
completeness + accuracy(10)

Our quantitative results are shown in Table 1, evaluated atall 3D ground truth points (left column), as well as restricted
to buildings and vehicles3 (middle and right column). We
evaluate our initial reconstruction as well as our joint reconstruction results for D = 0 (PC 0) and D = 1 (PC 1).
As evidenced by our experiments, our initial reconstruction
is able to outperform all variants of PMVS2 [12] in termsof both completeness as well as accuracy in almost all regions. Note that for cars, PMVS2 recovers less than 10%of the surfaces due to the challenges in matching textureless and specular surfaces. In contrast, our joint reconstruction (PC 0 / PC 1) transfers surface information from similar shapes in the scene, boosting completeness by more than
15% with respect to to our initial reconstruction baseline
with a moderate loss in accuracy. This leads to significant
improvements in F1 score for all three categories.
It may seem surprising that our weighted mean model(PC 0) slightly outperforms the more expressive modelcomprising one principal component (PC 1). After inspection, we found this effect to be caused by systematic errors
in the initial reconstruction. This systematic noise can bepartially overcome by our mean model (PC 0), which posesa stronger regularization on our joint reconstruction. An
alternative for alleviating this effect would be to integrateclass-specific object knowledge into the process [34, 35].
Results for a subset of the methods when varying theevaluation distance threshold τ are illustrated in Fig. 5.
3 In order to evaluate buildings and vehicles separately, we have annotated them with ground truth 3D bounding boxes.

While completeness and accuracy decreases for all methods with smaller detection thresholds, the relative gain in
performance of our method with respect to the baselines increases. This indicates that our reconstructions are not only
more complete, but also metrically more accurate. The fullplot is provided in the supplementary material.

4.3. Qualitative Experiments
Our qualitative results are shown in Fig. 6-8. Fig. 6
demonstrates the variability of our learned models andFig. 7 illustrates the ability of our model to complete regions of objects which have never been observed. Fig. 8
shows (from-top-to-bottom): the point cloud created by
PMVS2 [12] using the default parameter setting, the resultwith optimized parameters, meshed results of the optimized
PMVS2 point cloud using Poisson [23] and SSD [4] surface
reconstruction, our initial reconstruction, and our final results (PC 0). The colors denote the height, normalized withrespect to the highest and the lowest point of the reconstruction. Note how our method is able to recover cars as well
as missing walls of the buildings. Furthermore, our reconstructions convey much more detail than the baseline methods. As roofs in this sequence are barely observed from theviewpoint of the vehicle, none of the algorithms was able
to reconstruct them. Additional results are provided in the
supplementary material.

5. Conclusions
We have presented a novel method for jointly reconstructing objects with similar shapes in 3D by optimizing
their pose and shape parameters using a volumetric representation. We demonstrated the value of our model withrespect to PMVS2 and our initial reconstruction on a novel
challenging suburban dataset. Our method improves in particular with respect to completeness as it transfers surface
knowledge between objects of similar shape. One weakness of our model is that it is only little robust to outliersin the detection. In the future, we thus plan to incorporate
outlier handling by optimizing a robust function. Further,
we want extend our method to more object categories andcombine it with external 3D shape knowledge by learning a
joint model from 3D CAD models as well as a collection of
automatically retrieved objects in large scenes.
Acknowledgements. We would like to express thanks for support
from the following research grants: NSFC-61231010, 61272027,
61421062, 61210005 and the Okawa Foundation Research Grant.

References[1] S. Agarwal, K. Mierle, and Others. Ceres solver. http://
ceres-solver.org. 5[2] D. Anguelov, C. Dulong, D. Filip, C. Frueh, S. Lafon,
R. Lyon, A. S. Ogale, L. Vincent, and J. Weaver. Google

(a) Model Variety.

(b) Intra-model Variation.

(c) Variation in Scale.

Figure 6: Visualization of Learned Models. Figure a shows different shape models learned by our approach from two
viewpoints. Intra-model shape variations are illustrated in b, where the rows show (top-to-bottom): the input data, the reconstruction with 0 PCs and the reconstruction with 1 PC. Note how the PC 1 model is able to capture the step in the rightmost
building while PC 0 enforces stronger regularization. Finally, we show variation in model scale (s) after optimization infigure c. Note how the width and height of the objects differs between instances.

(a) Left Side of the Street.

(b) Right Side of the Street.

Figure 8: Qualitative Results. This figure shows our reconstruction results from both sides of the same street, respectively.
From top-to-bottom: Point cloud from PMVS2 using the default parameter setting, point cloud from PMVS2 with optimized
parameters, PMVS2 (optimized) + Poisson, PMVS2 (optimized) + SSD, our initial reconstruction, and our fused result (PC0). Note how our method is able to recover cars as well as missing building walls, and handling different types of buildings.
street view: Capturing the world at street level. Computer,43(6):32–38, 2010. 3[3] I. Biederman. Recognition-by-components: a theory of human image understanding. Psychological Review, 94(2):115,
1987. 1[4] F. Calakli and G. Taubin.
SSD: smooth signed distance surface reconstruction. Computer Graphics Forum,30(7):1993–2002, 2011. 5, 7
[5] D. Cremers, M. Rousson, and R. Deriche. A review of statistical approaches to level set segmentation: Integrating color,texture, motion and shape. IJCV, 72:215, 2007. 2[6] B. Curless and M. Levoy. A volumetric method for building
complex models from range images. In SIGGRAPH, 1996.
2, 3[7] A. Delaunoy and M. Pollefeys. Photometric bundle adjustment for dense multi-view 3d modeling. In CVPR, 2014. 2[8] W. Deng, J. Hu, J. Lu, and J. Guo. Transform-invariant PCA:
A unified approach to fully automatic facealignment, representation, and recognition. PAMI, 36(6):1275–1284, 2014.
2[9] D. F. Fouhey, A. Gupta, and M. Hebert. Data-driven 3D
primitives for single image understanding. In ICCV, 2013. 2[10] J.-M. Frahm, P. Fite-Georgel, D. Gallup, T. Johnson,
R. Raguram, C. Wu, Y.-H. Jen, E. Dunn, B. Clipp, S. Lazebnik, and M. Pollefeys. Building rome on a cloudless day. In
ECCV, 2010. 1[11] Y. Furukawa, B. Curless, S. M. Seitz, and R. Szeliski.
Manhattan-world stereo. In CVPR, 2009. 2[12] Y. Furukawa and J. Ponce. Accurate, dense, and robust multiview stereopsis. PAMI, 32(8):1362–1376, 2010. 2, 5, 7
[13] D. Gallup, J.-M. Frahm, and M. Pollefeys. Piecewise planar and non-planar stereo for urban scene reconstruction. In
CVPR, 2010. 1, 2
[14] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meetsrobotics: The KITTI dataset. IJRR, 32(11):1231–1237,
2013. 3[15] F. Güney and A. Geiger. Displets: Resolving stereo ambiguities using object knowledge. In CVPR, 2015. 2[16] C. Haene, N. Savinov, and M. Pollefeys. Class specific 3d
object shape priors using surface normals. In CVPR, 2014. 2[17] C. Haene, C. Zach, A. Cohen, R. Angst, and M. Pollefeys.
Joint 3D scene reconstruction and class segmentation. In
CVPR, 2013. 2[18] C. Haene, C. Zach, B. Zeisl, and M. Pollefeys. A Patch Priorfor Dense 3D Reconstruction in Man-Made Environments.
In 3DIMPVT, 2012. 2[19] L. Heng, B. Li, and M. Pollefeys. Camodocal: Automatic intrinsic and extrinsic calibration of a rig with multiple genericcameras and odometry. In IROS, 2013. 3, 5[20] H. Hirschmüller. Stereo processing by semiglobal matching
and mutual information. PAMI, 30(2):328–341, 2008. 3[21] M. Hornacek, C. Rhemann, M. Gelautz, and C. Rother.
Depth super resolution by rigid body self-similarity in 3d.
In CVPR, 2013. 2[22] A. Karpathy, S. Miller, and L. Fei-Fei. Object discovery in
3d scenes via shape analysis. In ICRA, 2013. 3[23] M. M. Kazhdan and H. Hoppe. Screened poisson surface
reconstruction. SIGGRAPH, 32(3):29, 2013. 2, 5, 7
[24] A. Kundu, Y. Li, F. Dellaert, F. Li, and J. Rehg. Joint semantic segmentation and 3d reconstruction from monocular
video. In ECCV, 2014. 2[25] F. D. la Torre and M. J. Black. Robust principal component
analysis for computer vision. In ICCV, pages 362–369, 2001.
5[26] L. Ladicky, C. Russell, P. Kohli, and P. H. S. Torr. Associative hierarchical random fields. PAMI, 36(6):1056–1077,
2014. 7[27] F. Lafarge, R. Keriven, M. Bredif, and H.-H. Vu. A hybrid multiview stereo algorithm for modeling urban scenes.
PAMI, 35(1):5–17, 2013. 2[28] W. E. Lorensen and H. E. Cline. Marching cubes: A highresolution 3d surface construction algorithm. In SIGGRAPH,
1987. 3, 6
[29] T. Malisiewicz, A. Gupta, and A. A. Efros. Ensemble ofexemplar-svms for object detection and beyond. In ICCV,2011. 1, 3[30] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux,D. Kim, A. J. Davison, P. Kohli, J. Shotton, S. Hodges, and
A. Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In ISMAR, 2011. 2[31] M. Nießner, M. Zollhöfer, S. Izadi, and M. Stamminger.
Real-time 3d reconstruction at scale using voxel hashing. In
SIGGRAPH, 2013. 1, 3
[32] A. Owens, J. Xiao, A. Torralba, and W. T. Freeman. Shapeanchors for data-driven multi-view reconstruction. In ICCV,2013. 2[33] M. Pollefeys. Detailed real-time urban 3d reconstruction
from video. IJCV, 78(2-3):143–167, July 2008. 1[34] V. A. Prisacariu and I. Reid. Nonlinear shape manifolds as
shape priors in level set segmentation and tracking. In CVPR,
2011. 2, 7[35] V. A. Prisacariu and I. Reid. Shared shape spaces. In ICCV,2011. 2, 7
[36] S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, andR. Szeliski. A comparison and evaluation of multi-view
stereo reconstruction algorithms. In CVPR, 2006. 5[37] Q. Shan, B. Curless, Y. Furukawa, C. Hernandez, and S. M.
Seitz. Occluding contours for multi-view stereo. In CVPR,
2014. 2[38] S. Song and J. Xiao. Sliding shapes for 3D object detection
in depth images. In ECCV, 2014. 3[39] F. Steinbrucker, C. Kerl, and D. Cremers. Large-scale multiresolution surface reconstruction from rgb-d sequences. In
ICCV, 2013. 2, 3[40] A. Tsai, W. M. W. III, S. K. Warfield, and A. S. Willsky.
An em algorithm for shape classification based on level sets.
Medical Image Analysis, 9(5):491–502, 2005. 2[41] H. Vu, P. Labatut, J. Pons, and R. Keriven. High accuracy and visibility-consistent dense multiview stereo. PAMI,
34(5):889–901, 2012. 2[42] A. Wendel, M. Maurer, G. Graber, T. Pock, and H. Bischof.
Dense reconstruction on-the-fly. In CVPR, 2012. 1[43] C. Zach, T. Pock, and H. Bischof. A globally optimal algorithm for robust tv-l1 range image integration. In ICCV,2007. 2[44] Q. Zhang, X. Song, X. Shao, H. Zhao, and R. Shibasaki.
Start from minimum labeling: Learning of 3d object models
and point labeling from a large and complex environment. In
ICRA, 2014. 3[45] Q. Zhang, X. Song, X. Shao, H. Zhao, and R. Shibasaki.
When 3d reconstruction meets ubiquitous RGB-D images.
In CVPR, 2014. 3[46] Q.-Y. Zhou, S. Miller, and V. Koltun. Elastic fragments for
dense scene reconstruction. In ICCV, 2013. 1