Multiview Photometric Stereo using Planar Mesh Parameterization
Jaesik Park1 ∗Sudipta N. Sinha2
Yasuyuki Matsushita3
Yu-Wing Tai1
In So Kweon1
1
Korea Advanced Institute of Science and Technology, South Korea
2
Microsoft Research Redmond, USA 3 Microsoft Research Asia, China
Abstract

In this paper, we present a new multiview photometricstereo method that efﬁciently combines geometric and photometric cues. Our method takes as input, a set of images
captured from multiple viewpoints illuminated by different
light sources. It starts by recovering a coarse 3D mesh using
existing state of the art SfM and MVS techniques. The ideaof transforming this mesh into a parameterized 2D space
using a distortion minimizing piecewise continuous 3D-to2D mapping lies at the core of our method. Unlike priormethods that use explicit 3D representations [6, 15], we
use a planar parameterization of the mesh [19] and cast themesh reﬁnement problem into one of estimating a displacement map texture in the 2D parameter domain. We showthat both photometric stereo based surface normal estimation and mesh reﬁnement can be efﬁciently and accurately
performed in the parameterized 2D space.

We propose a method for accurate 3D shape reconstruction using uncalibrated multiview photometric stereo. A
coarse mesh reconstructed using multiview stereo is ﬁrst
parameterized using a planar mesh parameterization technique. Subsequently, multiview photometric stereo is performed in the 2D parameter domain of the mesh, where allgeometric and photometric cues from multiple images canbe treated uniformly. Unlike traditional methods, there isno need for merging view-dependent surface normal maps.
Our key contribution is a new photometric stereo basedmesh reﬁnement technique that can efﬁciently reconstructmeshes with extremely ﬁne geometric details by directly
estimating a displacement texture map in the 2D parameter domain. We demonstrate that intricate surface geometry can be reconstructed using several challenging datasets
containing surfaces with specular reﬂections, multiple albedos and complex topologies.

Our proposed technique has two advantages. First, images from multiple viewpoints can be naturally handled
when performing multiview photometric stereo in the 2D
parameter domain, because all the images can be registeredwithout introducing large pixel distortions. As surface normals can be directly estimated in this space using multiple
registered images captured under varying lighting, it avoids
the needs to ﬁrst estimate per-view normal maps and thenmerge normal maps obtained from multiple viewpoints. Instead, images from multiple views can be jointly handled inour representation. Second, we can efﬁciently recover an
extremely detailed 3D mesh exploiting the full resolution
available in the input images. The level of geometric detail in our representation can be easily controlled by specifying the appropriate resolution of the estimated displacement map and the optimization is more efﬁcient than direct
3D methods that must resort to subdividing the mesh andreﬁning the vertex positions.

1. Introduction
Recovering an accurate 3D shape from images is an important and challenging problem in computer vision. With
recent progress in structure from motion (SfM) [12] andmultiview stereo (MVS) [18], it is nowadays possible to reconstruct 3D models for many challenging scenes. These
geometric methods recover 3D shape by estimating pixel
correspondences in multiple views. Hence, they can suffer in accuracy when surfaces are weakly textured or cameras have wide baselines. On the other hand, photometric methods, such as shape-from-shading [9] and photometric stereo [25], use shading cues to estimate per-pixel surface normal maps but do not directly provide depth estimates. These two types of approaches have complementarystrengths and have been combined in prior work [28, 15, 6,
27, 26] in the literature.

We describe how the proposed technique is used within
a complete 3D reconstruction pipeline. We have evaluatedour method on challenging sequences involving objects thathave intricate 3D shapes or have multiple albedos, reﬂective
surfaces or complex topologies. We also perform a quantitative evaluation which demonstrates the advantage of ourmesh reﬁnement technique over existing methods [6, 15].

∗ Part of this work was done while the ﬁrst author was visiting Mi-

crosoft Research Asia as a research intern. This work was also supported
in part by the National Research Foundation of Korea(NRF) grant fundedby the Korea government(MEST) (No.2010-0028680) and by the Ministryof Trade, Industry and Energy, Korea, under the Human Resources Development Program supervised by the NIPA (H1502-13-1001). Yu-Wing Tai
was supported by NRF-2011-0013349.

1

Input image

SfM

Depth Map

Multiview stereo

Mesh parameterization

Surface normal estimation Mesh refinement

Figure 1. Overview of our method. Structure-from-motion is used to calibrate the cameras and multiview stereo is used to recover a coarse
mesh. After parameterizing the mesh, multiview photometric stereo and mesh reﬁnement are performed in the 2D parameter domain.
Base Mesh Calibration 3D Shape Representation
Optimize by
Normals Estimated in
Nehab et al. [15] 3D Scanner Manual
Regular 3D Mesh
Mesh Reﬁnement
Individual Image
Hernandez et al. [6] Visual Hull
Auto
Regular 3D Mesh
Mesh Reﬁnement Face Normals of Mesh
Our method
SfM+MVS
Auto
Mesh+Displacement Map Displacement Map 2D Parameterized Space
Table 1. A comparison of our method and prior techniques [6, 15]. We use a different 3D shape representation, which is key to the highaccuracy and efﬁciency of the normal map estimation and mesh reﬁnement steps in our method.

2. Related Work
The idea of fusing geometric and photometric cues
for high-quality 3D scanning is gaining attention due totheir complementary strengths; multi-view geometric approaches are quite robust, but the recovered shapes can becoarse whereas photometric approaches can recover ﬁne details by estimating surface normals. These methods can bebroadly classiﬁed into 2D depth map reﬁnement approaches
and the ones that perform 3D mesh reﬁnement.
Nehab et al. [15] propose an efﬁcient method for 2D
depth map reﬁnement by adjusting depth values using orthogonality between depth gradients and surface orientations. Zhang et al. [29] extend their method to better preserve depth discontinuities. Okatani and Deguchi [16] propose a probabilistic framework for shape reﬁnement using
the ﬁrst-order derivative of surface normals. While thesemethods are effective and can be used for recovering a full
3D mesh, they require additional processing for registeringand merging multiple view-dependent depth maps.
For 3D mesh reﬁnement, Rushmier and Bernardini [17]adjust local normal directions obtained using photometricstereo. Nehab et al. [15] state that their 2D depth reﬁnement
method can be extended to handle a 3D mesh. Lensch etal. [11] introduce a generalized method for modeling nonLambertian surfaces by wavelet-based BRDFs and use it
for mesh reﬁnement. Hernandez et al. [6] iteratively reﬁnemesh polygons by minimizing a quadratic energy function.
Wu et al. [26] use the spherical harmonics representation toestimate global illumination, and reﬁne a preliminary meshusing photometric stereo by minimizing 1 penalties. In
an extended approach [27], geometric details are added using shape-from-shading under natural lightings. Vlasic etal. [23] ﬁrst integrate per-view normal maps into partial
meshes, then deforms them using thin-plate offsets to improve the alignment while preserving geometric details.

These 3D mesh reﬁnement methods generally use ahigh-resolution mesh in order to enclose high frequency
details obtained by photometric methods; however, determining the appropriate mesh resolution is non-trivial dueto the view-dependent variation of effective resolutions. In
contrast, our method allows the mesh resolution to be derived directly from the normal map resolution and avoids
the problem of undersampling mesh vertices. In addition,
our 2D parameterization approach performs mesh reﬁnement efﬁciently, where only 1D vertex displacements are
optimized rather than directly working in the 3D coordinates. Table 1 summarizes how our approach related to thetwo closely related methods proposed by Nehab et al. [15]and Hernandez et al. [6].

3. Proposed Method
In this section, we describe the key elements of the proposed method. For now, let us assume that all the cameras
are calibrated and the initial base mesh is available. The
methods for calibration and obtaining the initial base mesh
are later explained in Sec. 4. After describing the mesh parameterization scheme, we explain how surface normal estimation and mesh reﬁnement is performed in the 2D parameter domain. Figure 1 shows an overview of our approach.

3.1. Mesh Parameterization
In our method, ﬁrst the triangulated base mesh denotedby M, is mapped to a planar parameterized space using apiecewise continuous function f : R3 → R2 , which is referred to as mesh parameterization [19] (see Fig. 1). While
this process is not limited to a particular mesh parameterization method, in this paper, we use the Iso-charts method
proposed by Zhou et al. [30], which minimizes non-uniform
distortions of the original mesh by ﬁnding optimal cuts thatpartition the mesh into segments. Each connected segment

Algorithm 1: Image Warping
Input: Image I, camera projection matrix P, mesh M andits face visibility
Output: Warped image I for each pixel u ∈ U do
Find triangle t ∈ U that contains u
Find barycentric coefﬁcients, wt for u in t
Find face f ∈ M that maps to t and its vertices {xt }if f is visible thenx ← Barycentric-interpolation({xt }, wt )I  (u) ← I(Px )is mapped by its own mapping function to a single chart in
the parameter domain. We denote the 2D parameter domain
as U , which contains an arbitrary arrangement of the charts.
Using Iso-charts, we obtain a one-to-one mapping fM froma 2D point u = [u, v]T in U to a 3D point x on the mesh
M. For maximally utilizing photometric stereo estimates,
the resolution of U is set proportional (0.8 times smaller) to
the input image resolution.

3.2. Image Warping−1
Using the camera calibration and inverse mapping fM,
we warp input images I to images I in the U coordinates.
The images are warped using the standard inverse mapping
technique, i.e., we begin with a pixel u in the U coordinates and determine its corresponding pixel location in the−1
input image I via the inverse mapping function fM. Sincethe forward mapping function fM is discrete, we use a−1. Specifpiece-wise linear interpolation to approximate fM
ically, our method ﬁnds the projected mesh face that encloses pixel u in the U coordinates, determines 3D positionx that corresponds to pixel u using barycentric interpolation. Finally, the intensity of pixel u in I  is determinedby mapping the pixel in image I via the 3D scene point x .
This procedure is summarized in Algorithm 1. We use kdtrees [14] to accelerate the search for the 2D triangle. The
warping function is computed once for each viewpoint andthat warp is applied to multiple images captured from thatviewpoint and illuminated by different light sources. During image warping, only visible mesh faces are considered
and z-buffering is used to ﬁnd which faces are visible to thecamera.

3.3. Surface Normal Estimation
One of the key beneﬁts of our distortion minimizing
mesh parameterization scheme [30] is that pixels in images from multiple viewpoint and different lighting are wellaligned in the 2D parameter domain of the base mesh without signiﬁcant errors caused by viewpoint variations. Unlike single-view photometric stereo, in our case, we havemore observations from different nearby viewpoints thatare reasonably well aligned using the base mesh geometry.

(a)

(b)

(c)

(d)

Figure 2. An example of surface normal map and displacement
map estimation. (a) Input image. (b) Initial normal map obtained
from the base mesh, in U. (c) Disambiguated normals from photometric stereo in U. Here, unit 3D vectors have been linearly
mapped to RGB. (d) The estimated displacement map.

Therefore, the parameterization allows images from multiple viewpoints to be used effectively for multiview photometric stereo.
In this section, we introduce our method for estimating
surface normals given warped images I  . To handle a large
amount of observations efﬁciently, we use the Lambertian
reﬂectance model at this stage. By denoting the image intensities in the form of an observation matrix I ∈ Rp×q ,where p is the number of valid pixels in U , and q is the number of all the images that are taken from varying view points
under varying lightings, the Lambertian image formation
model can be expressed in matrix form, as I = NL. Here,N ∈ Rp×3 is an albedo-scaled surface normal matrix, andL ∈ R3×q represents a lighting matrix. Unlike the singleview photometric stereo case, I has many missing elements
as most 3D points are not visible from all the viewpoints.
Therefore, we compute surface normals N using subsets of
the observations which form dense block matrices in I. In
general, ﬁnding dense block matrices IS from the matrix I
is a NP-hard problem. However, since the columns of I arearranged in the image capture sequence in our case, valid intensity observations at u tend to be in adjacent columns of
I. The problem of ﬁnding the sub-matrices then reduces toﬁnding maximum cliques in an interval graph. We use themethod proposed by [3] for ﬁnding multiple, overlapping
dense block matrices in I.
Next, given an observation matrix IS , we apply the uncalibrated photometric stereo method of Hayakawa [4] toeach IS . Each observation matrix IS can be approximatedand decomposed into a product of two rank-3 matrices as
IS ≈ U3 Σ3 V3T = ρ(NS A−1 )(ALS ),
1

1

(1)

where NS = U3 Σ32 , LS = Σ32 V3T , and A is a nonsingular 3×3 matrix that represents a linear shape-light ambiguity that exists in uncalibrated photometric stereo. Σ3 isa diagonal matrix with three singular values, and U3 andV3 are orthonormal matrices with only ﬁrst three columns
and rows, respectively. To automatically resolve the linearambiguity A, we use the mesh normals Nf ∈ Rp×3 obtained from the base mesh, which is coarse yet reasonably
close to the correct surface normal. Speciﬁcally, we regard
1
NS A−1 ≈ Nf = U3 Σ32 A−1 . Using the pseudo-inverseof Nf , we solve for A and obtain the surface normal estimate N̂S as
1−1 T
A ← (NTNf U3 Σ32 ,f Nf )(2)
1
N̂S = U3 Σ32 A−1 ,where N̂S is a disambiguated surface normal matrix for
subset S. To combine duplicate solutions from distinct submatrices IS , we apply weighted sum to consolidate the normal estimate asnp (u) =1 
(nf (u)TnS (u))nS (u),M

(3)

S∈S(u)

where S(u) denotes a set of sub-matrix indices that include
u. nT
f nS (= wS ) is a weighting factor, which is the cosineof the angle between the face normal and estimated normal
vectors, and M = ΣwS normalizes the weighted sum ofnS . This weighting is used for reducing the effect of outliers. Figure 2 shows an example of the computed surface
normal maps.

3.4. Geometry Reﬁnement
The major advantage of working in the 2D parameterdomain is that 3D mesh reﬁnement can be performed byestimating a 2D displacement map of the base mesh M.
The geometry reﬁnement problem can now be formulated
as ﬁnding the optimal displacement d ∈ R per pixel u asx∗ (u) = x(u) + d(u)nf (u),

d


u∈U

nT
p

∂x∗
∂u

2
+λ



LEDs

Light array
L

Camera #2

Rotation table

Camera #1

Target objects

CamerasRotation stage

Figure 3. Imaging setup: rotation stage, light array and two cameras. For a particular rotation angle, several images are captured
under varying lightings.

use a cross-shape operator for computing partial derivatives,∂∂, and [1, 0, −1]T for ∂v. To deﬁne pari.e., [−1, 0, 1] for ∂utial derivatives at pixels on the boundary of two charts, we
use their respective inverse mappings to look up neighboring 3D points on the mesh. This operation is important as it
prevents seams from occurring on the chart boundaries byencouraging points across seams to have similar displacement values. The second term of Eq. (5) is a regularization
term that discourages large displacements.
The problem of Eq. (5) can be formulated as a sparselinear system that can be efﬁciently solved using an offthe-shelf sparse linear solver. In our implementation, we
empirically choose λ = 0.3. An example of an estimateddisplacement map is shown in Fig. 2. In our method, thelevel of geometric detail is controlled by the resolution of U
regardless of the resolution of the base mesh. For example,
a base mesh with as few as 2K vertices with a 512×512 displacement map can generate 262K effective vertices. Since
our approach directly estimates a displacement map on acoarse mesh, our 3D models can be efﬁciently stored andrendered efﬁciently on modern graphics hardware that supports displacement mapping [21].

(4)

where nf is a unit face normal of the triangle in M to which
x is mapped, and x∗ is the reﬁned 3D position. Noticethat the reﬁnement is deﬁned in the 2D domain using u
as indices. Now, given photometric normals np obtained
via photometric stereo and the initial position x ∈ M, weestimate the displacement dˆ by minimizing the following
energy function:dˆ = argmin

Real photographTarget object

d2 (u).

(5)u∈U

The ﬁrst term of Eq. (5) is a data term that encourages thesurface gradient at x∗ to be orthogonal to the orientation ofphotometric normal np . This term is related to the one proposed by Nehab et al. [15]. However, we estimate only asingle displacement for each 3D point, optimizing a single
scalar instead of three coordinates thereby reducing meshreﬁnement to estimating the optimal displacement map. We

4. Reconstruction Pipeline
This section describes our reconstruction pipeline – theimaging setup and the SfM and MVS pre-processing stages.

4.1. Imaging Setup
Our acquisition system consists of a rotation stage, LED
array, and two cameras as illustrated in Fig. 3. We assume
that the camera response functions are known. All imagesof the target object are captured automatically using a remotely controlled rotation stage with synchronized cameras
and LEDs. A typical acquisition captures 312 images (24viewpoints, 15 degrees apart illuminated by 13 differentLEDs) and takes about three minutes.

4.2. Camera Calibration and Multiview Stereo
We calibrate the camera intrinsics a priori and assume
that they remain constant during the acquisition. The
extrinsic parameters are estimated using a generic SfMMesh Perturbation
Nehab et al. [15]Hernandez et al. [6]Ours
Mesh Resolution
Nehab et al. [15]Hernandez et al. [6]
Ours

Bunny
Gargoyle
Happy-Buddha
Level 1
Level 2
Level 3
Level 1
Level 2
Level 3
Level 1
Level 2
Level 3
2.87, 99.7 2.93, 99.7 6.31, 93.6 4.72, 99.9 4.56, 99.9 4.58, 99.9 4.01, 99.8 3.95, 99.8 4.06, 99.8
1.66, 99.7 2.30, 99.7 5.51, 92.4 3.03, 100.0 3.40, 100.0 4.15, 100.0 2.92, 99.8 3.42, 99.9 3.04, 99.9
1.50, 100.0 1.94, 100.0 2.67, 100.0 3.43, 100.0 3.47, 100.0 3.70, 100.0 3.50, 99.5 3.65, 99.5 3.50, 99.5
70K
50K
25K
70K
50K
25K
70K
50K
25K
3.56, 99.9 5.44, 94.4 7.54, 67.8 6.02, 98.5 8.38, 83.6 11.56, 49.3 4.82, 99.2 6.51, 91.2 8.37, 64.8
1.11, 99.8 1.43, 96.7 1.67, 76.0 3.64, 96.7 4.14, 89.8 4.88, 64.5 2.76, 98.3 3.43, 93.4 4.29, 72.1
1.39, 100.0 1.40, 100.0 1.41, 100.0 3.33, 100.0 3.37, 100.0 3.45, 99.9 3.45, 99.6 3.48, 99.5 3.49, 99.5

Table 2. Comparison using synthetic dataset. In this experiment, each method reﬁnes degraded meshes, and the results are evaluated incomparison with the ground truth. Each cell of the table shows accuracy (×10−3 ) and completeness (%) for two experiments, meshperturbation and mesh resolution (see text for more details).

pipeline [20], which we found to be quite accurate. However, methods tailored to turn-tables [2] can also be used.
Stereo matching. Using the visibility of the SfM point
cloud, we estimate a depth range for each viewpoint andthen perform plane-sweep stereo matching for each viewpoint using two other images captured from adjacent viewpoints under identical lighting. Using normalized crosscorrelation as the matching cost and semi-global matchingbased cost aggregation [7], we ﬁrst estimate a dense depth
map with discrete depth estimates. Sub-pixel reﬁnement isthen performed on these depth maps using a standard local parabolic reﬁnement of the aggregated matching costs.
We compute per-pixel conﬁdence associated with the depth
map using the ratio of the minimum and the second smallest
costs to measure distinctiveness and prune depth estimatesat pixels with very low conﬁdence. See Fig. 1 for an example of a reﬁned depth map.
Mesh extraction. The ﬁltered depth maps are fused using an energy minimization framework based on volumetric graph-cuts [24]. The step computes an implicit 3D shapeof a closed object by labeling voxels on a uniform 3D gridwith binary labels – occupied, or empty. This optimization
is formulated using a discrete binary Markov Random Field
using unary and pairwise terms on a 6-connected voxel gridwith a typical resolution of 1003 . The unary potentials arecomputed using free space occupancy of the 3D points inthe depth map [5], where the contributions from depth maps
are weighted by their conﬁdences. The pairwise potentials
are derived from the sub-voxel positions of these 3D points.
As our acquisition setup allows simple foreground silhouette extraction, we also include a silhouette-based unaryterm in the energy – voxels that are projected outside thesilhouette are given a high penalty for taking the label occupied. The optimal binary labeling can be exactly computedin an efﬁcient manner using graph cuts [1]. Finally, fromthe labeled grid, we recover a triangulated mesh M using
marching cubes [13]. We prefer MVS in computing ourbase mesh over a visual-hull based approach [6], since MVS
yields more accurate mesh in our experience, especially forobjects with large concavities or complex topologies.

5. Results
We ﬁrst quantitatively evaluate our method using synthetic datasets and compare our method with existing stateof-the-art approaches [15, 6] focusing on the performanceof our mesh reﬁnement algorithm. In this evaluation, weused synthetically rendered images and the original mesh
as the preliminary mesh. In a second set of experiments, weshow 3D reconstruction results on various real-world objects where the level of detail in our reconstructed models
is of the order of a few millimeters.

5.1. Experiments on Synthetic Data
Using the Bunny, Gargoyle, and Happy-Buddha models,
we render 712 × 712 images under 8 different light directions and 16 distinct viewpoints using the Lambertian shading model. All the methods in the evaluation have access tothe true camera and light calibration parameters. The number of triangles for these models varies from 70K to 1M .
For consistency, these 3D models are scaled by setting theradius of their tightest bounding spheres to a unit. To simulate errors and irregularities of real data, these meshes arecorrupted by adding noise, and vertices are sub-sampled to
produce meshes with smaller triangle counts.
Mesh perturbation test. In this test, we perturb the original
mesh by adding random vertex displacements as noise andthen use the Taubin operator [22] to apply mesh smoothing.
The perturbation is performed at three levels, where level 3
has the highest error.
Mesh resolution test. In this test, we vary the number offaces of the base mesh to analyze the effect of mesh resolutions. Using a mesh simpliﬁcation technique [8], we generate meshes with 25K, 50K, and 70K vertices.
Evaluation metrics. Given the ground truth mesh G, wemeasure the accuracy of the reﬁned mesh R by computing the accuracy and completeness metrics that are used in
the Middlebury multiview stereo benchmark [18]. These
are based on asymmetric distances distR→G and distG→R ,where distA→B represents the minimum distance from vertices of A to vertices of B. Accuracy refers to the distanced ∈ distR→G such that x% of the points are within distanceGargoyle - perturbation level 3

HappyBuddha - perturbation level 3
100

80

80

80

60
40

Hernandez
Nehab
Ours

20
0

2

4

6

Accuracy (v10-3)

Ratio (%)

100

Ratio (%)

Ratio (%)

Bunny - perturbation level 3
100

60
40

Hernandez
Nehab
Ours

20
0

2

4

6

8

60
40

Hernandez
Nehab
Ours

20
0

10

Accuracy (v10-3)

2

4

6

Accuracy (v10-3)

Figure 4. Cumulative error distributions of Accuracy for three synthetic dataset; Bunny, Gargoyle, and Happy-Buddha. The graph corresponds to mesh perturbation experiment in Table 2 when perturbation level is 3. Except for Happy-Buddha, our curve locates above theother curves. This means larger proportion of per-vertex errors are smaller than others.

Perturbed Happy-Buddha

Zoom-in

Hernandez [6]Nehab [15]Ours

Ground truth

Figure 5. A perturbed Happy-Buddha model and reﬁnement results by three methods. The result corresponds to mesh perturbation experiment in Table 2 where perturbation level is 3.

d to G. Completeness refers to the proportion of vertices,where distG→R is less than threshold distth . In our experiments, we set x = 90 and distth = 0.01.
Results. Table 2 shows quantitative results on the synthetic
data. For the mesh perturbation experiment, our method
consistently performs better than Nehab et al. [15] because
our method naturally avoids mesh ﬂipping and overlapping
triangles. In this test, the accuracy and completeness of
our results are comparable to those of Hernandez et al. [6].
As our approach estimates a displacement map whose resolution is derived from the original image resolution, ourmethod recovers ﬁne geometric details regardless of the resolution of the base mesh (see mesh resolution in Table 2).
Figure 4 shows the cumulative error distributions for the
Accuracy metric. The percentage of vertices within an accuracy threshold is plotted for different thresholds. The plotshows that our method is consistently the most accurate,
except for the Happy-Buddha model where our method iscomparable to [6]. The reﬁned meshes for Happy-Buddha
are shown in Fig. 5. Our method faithfully reconstructs ﬁne
details such as the necklace and ﬂower on the model.
Computation time. We compare the computational cost of
our mesh reﬁnement method with that of Nehab et al. [15].
The result is shown in Fig. 6 where its clear that our method
is computationally more efﬁcient than theirs when bothmethods are conﬁgured to produce results with comparable
accuracy. On average, our proposed method (excluding ac-

Base
mesh

Nehab Nehab
Ours Ground
300K
1.8M
truth
4.6Ő10-3 3.5Ő10-3 3.3Ő10-3(1M)
Test model
2.9 sec. 18.3 sec. 4.2 sec.
Figure 6. Computational cost and accuracy of mesh reﬁnement.
Base mesh resolutions, reconstruction accuracies, and computation times are shown under sub-ﬁgures. Our method does not require tuning the mesh resolution because it is automatically determined by the input image resolution.

quisition and pre-processing) as described in Sec. 3 takes
less than a minute to run. The timings are measured on asystem equipped with an Intel i7 quad-core 3.06GHz CPU
and 8GB memory.

5.2. Experiments on Real Data
Figure 7 shows the result of ﬁve real scenes whose images are taken using the imaging setup described in Sec. 4.1.
The ﬁrst two objects, B UDDHA and AGRIPPA, have mostly
uniform albedos. However, B UDDHA statue is made of copper and has many specular reﬂections. Even though our normal estimation method assumes Lambertian reﬂectances,
the normal aggregation process of Eq. (3) effectively handles outliers arising from non-Lambertian reﬂectance.
The other three objects, D OLL -1, D OLL -2, andT EAPOT, show more interesting topologies and multiple
albedos. In D OLL -1, we can observe the detailed shape of
buttons on the jacket of the right doll as well as facial expression of the dolls, which cannot be seen in the original
base mesh. The English characters in the middle region of
D OLL -2 are clearly visible in the ﬁnal mesh. The geometricdetails on the T EAPOT model are faithfully reconstructed.
Note that these embossed patterns are only a few millimeters deep. On the other hand, an artifact can be seen below
the left doll’s skirt in D OLL -1 as indicated by red rectangles in Fig. 7. Since no valid normal could be estimatedfrom any of the viewpoints, our method is unable to reﬁnethe coarse mesh in this region.

6. Discussions
Our 3D reconstruction approach enables the acquisitionof high-ﬁdelity 3D models where a mesh parameterization
scheme is used to fuse photometric and geometric cues. Although our automatic pipeline demonstrates high accuracy,
there are currently a few limitations. First, we have useda linear photometric stereo approach for efﬁciency reasons,but the accuracy of our system can be potentially boosted
using recent advances in robust photometric stereo [10].
Dark and textureless surfaces are currently difﬁcult to handle in our method due to the lack of reliable photometric orgeometric cues. In the future, we plan to explore a joint optimization approach that simultaneously estimates surface
normals and scene depth for greater accuracy and robustness. Recovering surface reﬂectance as well as accurate 3D
shape is another interesting direction for future work.

References[1] Y. Boykov and V. Kolmogorov. An experimental comparison ofmin-cut/max-ﬂow algorithms for energy minimization in computer
vision. IEEE Trans. Pattern Anal. Mach. Intell., 26(9):1124–1137,
2004.
[2] A. W. Fitzgibbon, G. Cross, and A. Zisserman. Automatic 3D model
construction for turn-table sequences. In 3D Structure from Multiple
Images of Large-Scale Environments, pages 155–170. 1998.
[3] U. I. Gupta, D. T. Lee, and J. Y.-T. Leung. Efﬁcient algorithms forinterval graphs and circular-arc graphs. Networks, 12(4):459–467,
1982.
[4] H. Hayakawa. Photometric stereo under a light source with arbitrary
motion. Journal of the Optical Society of America, 11(11):3079–
3089, 1994.
[5] C. Hernández, G. Vogiatzis, and R. Cipolla. Probabilistic visibility for multi-view stereo. In Proc. of Computer Vision and Pattern
Recognition (CVPR), 2007.
[6] C. Hernández, G. Vogiatzis, and R. Cipolla. Multi-view photometric stereo. IEEE Trans. Pattern Anal. Mach. Intell., 30(3):548–554,
2008.
[7] H. Hirschmüller. Stereo processing by semiglobal matching and mutual information. IEEE Trans. Pattern Anal. Mach. Intell., 30(2):328–
341, 2008.
[8] H. Hoppe. New quadric metric for simplifying meshes with appearance attributes. In Proc. of IEEE Visualization Conference, 1999.

[9] B. K. P. Horn. Shape from shading: A method for obtaining the shapeof a smooth opaque object from one view. PhD thesis, MIT, 1970.
[10] S. Ikehata, D. Wipf, Y. Matsushita, and K. Aizawa. Robust photometric stereo using sparse regression. In Proc. of Computer Vision
and Pattern Recognition (CVPR), pages 318–325, 2012.
[11] H. P. A. Lensch, J. Kautz, M. Goesele, W. Heidrich, and H. peter Seidel. Image-based reconstruction of spatial appearance and geometricdetail. ACM Trans. on Graph., 22:234–257, 2003.
[12] H. C. Longuet-Higgins. A computer algorithm for reconstructing ascene from two projections. Nature, 193:133–135, 1981.
[13] W. E. Lorensen and H. E. Cline. Marching cubes: A high resolution
3d surface construction algorithm. In Proc. of SIGGRAPH, pages
163–169, 1987.
[14] D. M. Mount and S. Arya. ANN: A library for approximate nearest
neighbor searching. http://www.cs.umd.edu/˜mount/ANN/, 2010.
[15] D. Nehab, S. Rusinkiewicz, J. Davis, and R. Ramamoorthi. Efﬁciently combining positions and normals for precise 3D geometry.
ACM Trans. on Graph., 24(3), 2005.
[16] T. Okatani and K. Deguchi.
Optimal integration of photometric and geometric surface measurements using inaccurate reﬂectance/illumination knowledge. In Proc. of Computer Vision and
Pattern Recognition (CVPR), pages 254–261, 2012.
[17] H. Rushmeier and F. Bernardini. Computing consistent normals andcolors from photometric data. In 3.
[18] S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski.
A comparison and evaluation of multi-view stereo reconstruction
algorithms. In Proc. of Computer Vision and Pattern Recognition(CVPR), 2006.
[19] A. Sheffer, E. Praun, and K. Rose. Mesh parameterization methods and their applications. Found. Trends. Comput. Graph. Vis.,2(2):105–171, 2006.
[20] N. Snavely, S. M. Seitz, and R. Szeliski. Photo tourism: Exploring
photo collections in 3d. In Proc. of SIGGRAPH, pages 835–846.
ACM Press, 2006.
[21] L. Szirmay-Kalos and T. Umenhoffer. Displacement mapping on the
GPU - State of the Art. Computer Graphics Forum, 27(1), 2008.
[22] G. Taubin. A signal processing approach to fair surface design. In
Proc. of SIGGRAPH, pages 351–358, 1995.
[23] D. Vlasic, P. Peers, I. Baran, P. Debevec, J. Popović, S. Rusinkiewicz,
and W. Matusik. Dynamic shape capture using multi-view photometric stereo. ACM Trans. on Graph., 28(5), 2009.
[24] G. Vogiatzis, C. Hernández Esteban, P. H. S. Torr, and R. Cipolla.
Multiview stereo via volumetric graph-cuts and occlusion robust
photo-consistency. IEEE Trans. Pattern Anal. Mach. Intell., 29(12),Dec. 2007.
[25] R. J. Woodham. Photometric method for determining surface orientation from multiple images. Optical Engineering, 19(1), 1980.
[26] C. Wu, Y. Liu, Q. Dai, and B. Wilburn. Fusing multiview and photometric stereo for 3d reconstruction under uncalibrated illumination.
IEEE Trans. on Visualization and Computer Graphics, 17(8):1082–
1095, 2011.
[27] C. Wu, B. Wilburn, Y. Matsushita, and C. Theobalt. High-quality
shape from multi-view stereo and shading under general illumination. In Proc. of Computer Vision and Pattern Recognition (CVPR),
2011.
[28] L. Zhang, B. Curless, A. Hertzmann, and S. M. Seitz. Shape andmotion under varying illumination: Unifying structure from motion,
photometric stereo, and multi-view stereo. In Proc. of Int’l Conf. on
Computer Vision (ICCV), pages 618–625, 2003.
[29] Q. Zhang, M. Ye, R. Yang, Y. Matsushita, B. Wilburn, and H. Yu.
Edge-preserving photometric stereo via depth fusion. In Proc. ofComputer Vision and Pattern Recognition (CVPR), 2012.
[30] K. Zhou, J. Snyder, B. Guo, and H.-Y. Shum. Iso-charts: Stretchdriven mesh parameterization using spectral analysis. In Eurographics Symposium on Geometry Processing, pages 45–54, 2004.

Buddha-statue

Agrippa

Doll-1

Doll-2TeapotFigure 7. Reconstruction results by our method for B UDDHA - STATUE, AGRIPPA, D OLL -1, D OLL -2, and T EAPOT scenes. Each row shows
one of input images, the base mesh from MVS, and the reﬁned mesh. The corresponding surface normal and displacement maps are shownin the supplementary material. Clear failure cases are highlighted by red rectangles; these occur at textureless dark regions.