On Joint Estimation of Pose, Geometry and svBRDF
from a Handheld Scanner
Carolin Schmitt

1,2,*Simon Donné

1,2,*

Gernot Riegler
2¹MPI for Intelligent Systems, Tübingen

3

Vladlen Koltun³ Andreas Geiger

University of Tübingen3

1,2Intel Intelligent Systems LabMotivation

Joint Formulation

Pose Ablation Study

Structured light sensing (e.g. Kinect) enabled large-scale 3D
reconstructions. Yet, the level of realism is limited since physicallight transport is not modeled. Illumination eﬀects such as
specular reﬂections or shadows are baked into the texture.

2.5D scene representation based on one reference view:

Reﬁning the initial poses is crucial for recovering ﬁne texture details.
Top: using input poses, bottom: optimizing the poses.

Photometric Test Error
Fixed Poses
Full ModelReﬂections and geometry are linked strongly. To usethat mutual information, object geometry and material properties
are best inferred jointly. We propose a novel formulation forjoint pose, geometry and svBRDF estimation.

Overall Specularity Regions Other Regions
3.349
1.210
1.151
1.138
3.243
1.081

While existing approaches use alternating optimization procedures,
we propose a single objective that can be optimized using
oﬀ-the-shelf gradient-based solvers.
Optimized variables:

Handheld Sensor

Estimation,
Geometry,
Normals

Appearance

Camera Poses

RGB input images captured underknown point light illumination.
A single objective function minimized by oﬀ-the-shelf gradient-based solvers.
Figure 1: Illustr ation. Based on images captured from ahandheld scanner with point light illumination, we jointly
Complete but coarse
initialoptimize
for the depth.
camera poses, the surface geometry andspatially varying materials using a single objective function.

ferred jointly: a good model of light transport allows for recovering geometric detail using shading cues. An accurate
shape model, in turn, facilitates the estimation of material
properties. This is particularly relevant for shiny surfaces
integrates
material assignments into the optimization process as a diﬀerentiable
where small changes in the geometry greatly
impact the appearance and location of specularregularizer.
ref ections. Yet joint
opti-naturally leads to a semantically meaningful material segmentation.
This
Traditionally accurate geometry reconstruction
mization of these quantities (shown in Fig. 1) is challenging.
requires known appearance properties. Several works have addressed this problem by assuming
multiple images from a static camera [16, 21, 23, 28, 105]
which is impractical for mobile scanning applications. Only
Likewise, accurate appearance estimationa few works consider the challenging problem of joint gerequires very well known geometry.
ometry and material estimation from a handheld device[20, 24, 57]. However, existing approaches assume known
camera poses and leverage sophisticated pipelines, decomJoint estimation of geometry and materials
posing the problem into smaller problems using multiple
requires only a rough initialization for both.
decoupled objectives and optimization algorithms that treat
geometry and materials separately. Furthermore the numberof base materials must be provided and/or pre-processing isrequired to cluster the object surface accordingly.

Material Assignments

1

Segmentation

Reconstructions

Geometry

Duck

Input RGB

Initial Depth and Appearance

Reﬁned Geometry, Normals and AppearanceGnomeHydrant

SheepVaseTeapot

GlobeRabbit

number of specular base materials; independently determined by our model.

https://github.com/autonomousvision/handheld_svbrdf_geometry

GirlAAE (◦ ) AEA (mm)+ Fully portable, no laboratory setup for material estimation
object geometry and material properties are in+ For arbitrary geometry, no prior assumptions likeIdeally,
planarity

Materials

Geometry

Normals

Figure 9: Qualitative Geometr y Compar ison. We show, forsurface normals rather than the estimated normals np) andWe observe that the photometric approaches recover far mor
structure for Higo et al. [24] is rather noisy, while the disj
disambiguating texture and geometry for very f ne details. W

Diﬀuse Specular

Custom built sensor rig.

Error

Estimation Crop
Error
Structure Normals
Figure5: Pose Optimization.
Compared to using the input
Geometric
Evaluation
poses(top),optimizing
theposes(bottom)
improves
reconQualitatively and quantitatively, the proposed method recovers
structions,
both
qualitatively.
Thebaselines.
photomore details
withquantitatively
less noise and anda lower
error than the- Proposed (disjoint): alternating instead of joint optimization of geometry and materials.
metricerror
isreportedfor
regions
withandwithout
specu- Artec Spider scans serve as ground truth.
lar highlights.
supplementfor(disjoint)
more details.
TSDFSee
Fusionthe Higoet al. Proposed
Proposed

[R|t]

,

Crop

TSDF Fusion
Higo et al.
Proposed (disjoint)
Proposed
TSDF Fusion
Higo et al.
Proposed (disjoint)
Proposed

Segmentations

duck
0.81
2.65
0.81
0.80
6.75
7.77
6.01
5.17

pineapple
1.24
1.05
1.00
1.00
12.09
10.62
9.138.98

girl
1.11
1.59
1.06
1.00
11.40
11.30
9.81
8.73

gnome
0.73
1.60
0.65
0.64
7.64
9.24
6.41
5.74

sheep
0.79
2.09
0.64
0.57
8.38
8.65
6.66
5.60

hydrant
1.35
1.81
1.18
1.16
11.67
15.27
9.59
8.50

rabbit
2.16
2.85
2.25
2.16
24.42
27.67
23.01
23.50

Test Reconstructions

The explicit shadow and occlusion modeling enables
Table
1:
Quantitative
GeometryComparison.
Wereportreconstructing strongly
non-convex scenes.
(a) Materials(b) Geometryboth the average euclidean accuracy and average angular
Figureas6:discussed
Loss Regularizer s.4.1.
Withoutregularization
error,
in Section
Pleasetherefer
to the sup(top), theis inconsistent
within homogeneous
plement
forappearance
a more detailed
table.
areas of the object. Using the regularization losses (bottom),
we are able
propagateandsuccessfullyupdates.
We to
also
evaluate thethe information
improvement
overthe initial
generalize
to using
new illumination
conditions
theettest
TSDF
fusionthe implementation
of on
Zeng
al.set.
[98].
2
Reconstruction
Geometry
Crops of Object Boundaries
Our experimental evaluation is shown in Fig. 9 and Table
1.
We
seethat
TSDF
fusion,apurelygeometric
ap{carolin.schmitt,simon.donne}@tue.mpg.de
Mater ial Segmentation: Decomposing the appearance of