Edge-Preserving Photometric Stereo via Depth Fusion
Qing Zhang1

Mao Ye1

Ruigang Yang1
Yasuyuki Matsushita2
Bennett Wilburn2
Huimin Yu3
1
University of Kentucky
Microsoft Research Asia2 Zhejiang University3

Abstract
We present a sensor fusion scheme that combines active stereo with photometric stereo. Aiming at capturing
full-frame depth for dynamic scenes at a minimum of threelighting conditions, we formulate an iterative optimization
scheme that (1) adaptively adjusts the contribution fromphotometric stereo so that discontinuity can be preserved;(2) detects shadow areas by checking the visibility of theestimated point with respect to the light source, instead ofusing image-based heuristics; and (3) behaves well for illconditioned pixels that are under shadow, which are inevitable in almost any scene. Furthermore, we decompose
our non-linear cost function into subproblems that can be
optimized efficiently using linear techniques. Experiments
show significantly improved results over the previous stateof-the-art in sensor fusion.

(a) Input Image(b) Input Depth(c) Nehab et al. 2005(d) Our Method

1. Introduction
The recent availability of consumer low-cost depth cameras brings exciting opportunities for computer vision. By
providing the additional depth information, which is unavailable in traditional imaging techniques, they hold thepotential to make a number of hard problems in computer
vision more tractable in practice. It has already changed theway we interact with computers (in games) [25]. However,the depth quality from the current generation of depth cameras still has plenty of room for improvement. For example,
in the most popular Kinect sensor [21], the quantization effect is visible and surface details are lost, as shown in Figure 1(b).
We are motivated by both the opportunities and limitations in commodity depth sensors. Our goal is to significantly improve the depth sensor quality so that it canbe used for applications such as 3D modeling and view
synthesis. In this paper we present a novel sensor fusion
scheme that combines active stereo with photometric stereo.
The complimentary nature of stereovision and photometricstereo (PS) has long been recognized. Stereovision is simple to set up and can generate metric measurement, but its

Figure 1. A reconstructed scene after fusion. (a) one of the three
input images; (b) captured from a depth sensor; (c) and (d) thereconstructed mesh rendered from two view points, without andwith discontinuity and shadow handling.

accuracy is inversely proportional to the object distance. On
the other hand, photometric stereo is known for capturing
surface details, but the normal map produced by PS does
not provide metric depth measurement. In addition, the resulting 3D surface may suffer from global non-uniform distortions. Therefore several methods have been developed tofuse depth maps with normal maps (e.g., [9, 23, 2]). Our
solution is different from all these previous methods in that
we explicitly model the discontinuity of the captured scene.
In other words, we envision that our method can be usedas the foundation for the next generation depth sensors to1

Shadows in Photometric Stereo PS requires at least
three distinctive lighting conditions to resolve the surface
normal. Common approaches to address shadows are touse more lighting conditions and check the consistency
with the result where the minimum requirement can be
met [10, 4, 7]. In the minimum case of three lights, [14]modifies the cost in [7] to detect and deal with pixel shadows. The fundamental assumption made there is that theentire captured surface is continuous and smooth. That isexactly the type of assumption that we in this paper do notchoose to assume.

capture full-frame depth maps at high-quality, rather than amethod for an object scanner.
Another constraint from our design consideration is thenumber of lighting variations. With dynamic scene capturein mind, we limit the number of lighting conditions to three– the minimum to resolve the surface normal. In almost any
real world scene, shadows due to occlusions will unavoidably appear and photometric stereo will fail in there areas.
We therefore develop a framework to first identify areas inshadow and then allow insufficient number of lighting conditions.
In summary, this paper makes the following technicalcontributions. First, we develop an automatic fusion algorithm to use the intensity variations under different illumination conditions to refine the depth map produced by adepth sensor. Unlike previous fusion algorithms, an adaptive weighting scheme is devised to preserve surface discontinuities. Secondly, instead of requiring a given normal
map from a complete photometric stereo process, we extend the fusion scheme to use insufficient number of lightsources. We cast the full optimization problem into a set of
linear problems that can be solved efficiently. Our method
allows a principled way to deal with pixels in shadow. As
we will see later, we demonstrate quality results using only
two lights. Finally, our fusion scheme also enables us to
identify shadow regions in a way similar to shadow mapping in computer graphics.

Depth Sensor Fusion Given the tremendous interest incommodity depth cameras, there have been several papers
on improving depth sensor quality by using sensor fusion.
For example, stereo and time-of-flight sensors are fused together for improved quality(e.g., [28, 18]). There are also
numerous papers on the general problem of combining positions with normals (e.g., [9, 22, 8, 23, 26]). To deal withdynamic scenes, [17, 3, 2] utilize color information for nonrigid objects. Color lights will interfere the image quality
and therefore we do not choose. Moreover, none of thesemethods explicitly models the discontinuities in the scene.

3. Adaptive Depth-Normal Fusing
We first present notations and preliminaries and then describe our adaptive weighting method to combine the depth
and normal maps while preserving depth discontinuities.
For now, we assume that the corresponding normal map has
been calculated and given.

2. Related Work
Our proposed work is related to both photometric stereo(PS) and sensor fusion. The foundation for PS is describedin the pioneering work of Woodham in the eighties [27], inwhich the surface unit normal of a Lambertian object can be
estimated given three distant light sources and corresponding images. The focus of later PS research has two major
threads. One is on extending PS to non-Lambertian surfaces(e.g., [11, 24, 15]). While in this paper we make the basic
Lambertian surface assumptions, some of these techniques
can be incorporated into our framework.
The other thread of PS research is to integrate the normal
field from PS to a surface. Horn and Brooks [16] propose
the basic method to integrate from measured gradient fieldusing calculus of variations. Frankot and Chellappa [12]use the Fourier basis function to project the possibly nonintegrable gradient field onto the nearest integrable slopes.
Agrawal et al. [1] summarize a collection of integration
methods from the traditional approach to the affine transform approach. Harker and O’Leary [13] discretize the traditional cost function to determine a unique least square solution up to an integration constant. Basri et al. [5] use theproduct of normal and surface gradient to avoid the rim error of the gradients and the explicit normal integration.

3.1. Fusion of depths and normals
Suppose the 3D surface S(u, v) = [x, y, z]T is parameterized in a 2D field Ω = [u, v] and the initial measured
surface is S 0 . We use the superscript 0 to denote the initial
measure in this paper. Assume the field [u, v] coincides with
the image grid [i, j], and the normal and depth are denoted
as n(i, j) = Nij and z(i, j) = Zij , respectively. Considering the perspective projection of 3D positions and pixels,
the surface can be represented in terms of the depth map
Zij :
T

i − px j − py,
, 1 Zij ,(1)
S(i, j) =
fx
fy

T
i − px j − py
µij :=
,
,1 ,
(2)
fx
fy
where [fx , fy ] and [px , py ] are the camera focal length andprincipal point, respectively. Then the distance of the surface to the measurement is represented using depths:
X

0 2
Ep =
kµij k2 Zij − Zij
.
(3)
ij

2

3.2. Adaptive weighting algorithm

To use the normal information, we define the following costfunction (“·” refers to the dot product of 3D vectors in thispaper):
2 
2
X
0 ∂S
0 ∂S
+ Nij ·
,
En =
Nij ·
∂u
∂v
ij

Traditionally, the fusion weights λp and λn are prespecified constants, which are only suitable for a single continuous object. Adjusting them in a pixelwise manner turns
out ineffective to deal with discontinuities and may resultin unwanted artifacts. In order to handle generic scenes, wedevelop a new automatic weighting algorithm without requiring any manual hard segmentation. This is possible because the discontinuity at different objects’ boundaries canbe reliably detected from the measured depth map.

(4)

where the derivatives of the surface are:

T
∂S
∂Z
Zij
(i, j) = µij(i, j) +, 0, 0 ,
∂u
∂u
fx

T
∂S
∂Z
Zij
(i, j) = µij(i, j) + 0,
,0 ,
∂v
∂v
fy

(5)

According to the first order discrete form of eq. 4, we canconsider each pixel in the reconstructed surface as a quadrilateral (shown in Figure 2) that: each quadrilateral is a tiny
plane perpendicular to the measured normal and connected
to its four direct neighbors. The derivative is then approximated by the center difference (Zi+1 − Zi−1 )/2, which
can be further decomposed to an average of the forward andbackward difference (Zi+1 − Zi )/2 + (Zi − Zi−1 )/2 as thedotted line. The idea of our adaptive weighting algorithm is(6)∂Zand ∂Z∂u and ∂v are discrete derivatives. Due to the truncated error and the Gibbs phenomenon arising near the gradient discontinuities, we choose a 3-point derivative formula similar to [13].
Combining all terms above, the desired surface is obtained by minimizing the following function in terms of
depth values:

E

P

2

kµij k
2
0
P 0
Nxij
Z
+
+ λn
(Nij · µij ) ∂Z
ij
∂u ij
fx
i,j
2
0
P 0
Nyij
+
+ λn
Z
,
(Nij · µij ) ∂Z
ij
∂v ij
fy
= λp

i,j

N𝑖


0 2
Zij − Zij

N𝑖−1
Z𝑖−1

(7)

i,j

𝑖−1
𝑖
𝑖+1
Figure 2. The approximation of surface gradient in one dimension.



λp Iµ



λp Z 0 µNx∂0 λn (N · µ) ∂u + fx 0 [Z] = 
,
h
0i
N




y
∂
0
+ fx 
 λn (N 0 · µ) ∂v
0λs ∇ 2
h

Z𝑖+1

′
𝑍𝑖−1

where λp +λn = 1 and λp , λn ≥ 0 are blending weights foreach penalty. Since the differential operator over the depth
map will finally represent a matrix multiplication, the totalenergy is a quadratic form of depth values and therefore canbe solved by an over-constrained linear least square system:


N𝑖+1

Z𝑖

to design the differential operator as a bilateral filter: considering both neighbors and the depth range. By choosing aproper range filter, we can approximate the derivative acrossthe discontinuity by only using one side difference. If the0range difference is large (e.g., Zi−1
in Figure 2), the range
filter will assign a small weight to the “steep” side. If therange difference is small, the operator goes back to the center difference. In other words, in the continuous part allneighbors are weighted involved, and meanwhile on boundaries false neighbors are occluded adaptively.

i
0

(8)

where [Z] means stacking all the depth variables into a column vector, and the multiplications taken in the left handside are arranged in the order of each corresponding pixels.
The smoothness term is to suppress the quantization error
and Gibbs phenomenon, where ∇2 denotes a Laplacian operator on the 4-neighbor image grid and the weight λs ischosen small. λp and λn adjust how the depth and normal
affect the final reconstructed surface. In practice, we choosea large λn and small λp to ensure high quality details as longas the depth bias of the result is comparable to the originalone.

We choose a shift-invariant Gaussian filter for the depth
range at the location pi given a general pixel p:2|Zp − Zpi |s(p, pi ) = exp −
2σ 2
3

!
.

(9)

The weighted differential operators are:the ill-conditioned region, but also avoids the use of nonlinear optimization.
We decouple eq. 12 by iteratively optimizing two steps:
firstly, we compute an intermediate normal map Ñ from thefollowing subproblem:∂Zijsi−1,j (Zij − Zi−1,j ) + si+1,j (Zi+1,j − Zij )=, (10)∂usi−1,j + si+1,j∂Zijsi,j−1 (Zij − Zi,j−1 ) + si,j+1 (Zi,j+1 − Zij )=, (11)∂v
si,j−1 + si,j+1

min
t
Ñij

where si−1,j is short for s((i − 1, j), (i, j)) and so on.
The standard deviation σ should be small to make a good
distinction, and we choose σ = 0.1 in our experiment.
Besides, in the singular case where both sides are very
steep, we do not normalize the operator when the divisor
si−1,j + si+1,j or si,j−1 + si,j+1 is close to zero. In addition, the Laplacian operator in eq. 8 is also modified in asimilar form containing four neighbors.

subject tot−1td(Ñij
, Nij
)

(13)
t
k
Ñij
· Lk = Iij,

k = 1, 2, . . . , n,where d(·, ·) represents the geodesic distance between two
vectors, the superscript t indicates the iteration number andt
Nijis computed from the mesh after t times iterations. Thetintermediate normals Ñij
are then combined into the fusion
step by solving eq. 8 as our second step. If the number oflights is n ≥ 3, the solution is unique and hence the whole
optimization will converge after one iteration. When thenumber of lights is insufficient, experiments show that our
optimization usually converges in at most 10 iterations.

4. Dealing with Insufficient Lighting
In a general scene, shadows will be inevitable for photometric stereo. For dynamic scene capture, the smaller number of lights is preferred. Under our minimum 3-light setup,shadow pixels in any of the three images will not have sufficient information to fully resolve the normal. In this section
we extend our fusion algorithm to deal with an arbitrary
number of lights, including these ill-conditioned ones whena pixel is lit by less than three lights. We first formulate thefusion as a nonlinear optimization problem and later present
our efficient solution method. We then analyze the case of
optimizing the depth with n ≤ 2-light conditions. Note thatin this section, we do not assume a normal map is given.
Instead, we directly work with image intensities.

4.2. n ≤ 2 light sources
We illustrate the case of n = 2 in Figure 3. The constraints restrict the solution space to a 3D line l intersectedby two planes Π1 : Ñ · L1 = I 1 and Π2 : Ñ · L2 = I 2 .
The discussion in this section is for a single pixel and thus
we omit the subscript for conciseness.

𝑙Π2

4.1. Problem Statement
Suppose we have images I k illuminated by distant lightsources Lk from different directions k = 1, . . . , n. Insteadof computing the normal map a priori from the images, we
directly put the image appearance discrepancy to the objective function eq. 7. The goal is to optimize depths in thefollowing form:Z=

arg minZ λp

P
i,j

+ λn

n P
P
k=1 i,j

+ λn

P

+ λn

P

i,j

i,j

2

kµij kΠ1

Nxij
fx ZijNij · µij ∂Z∂v ij +Nyij
fy ZijBy projecting the line onto the unit sphere, we obtaina semi-circle representing the 1-DOF solution space of thenormal. The goal of eq. 13 is to find the closest point on
the arc to the given normal N t−1 . We observe that theline l cannot pass the origin since intensities cannot be zeros. Also the line cannot be parallel to the direction of thegiven normal N t−1 , otherwise N t−1 will be parallel to bothplanes and hence be perpendicular to both plane normals L1and L2 , i.e., L1 · N t−1 = 0 and L2 · N t−1 = 0, which is
impossible in our case.
Therefore, we can find the closest vector by projecting
the normal N t−1 onto the plane that contains both the line

(12)

2
2

𝛼

Figure 3. The illustration of optimizing the normal when only twolight conditions are given.

2

Nij · µij ∂Z∂u ij +

𝑁 𝑡−1

𝑂


0 2
Zij − Zijk
Lk · Nij − Iij

𝑁𝑡.

Differently from eq. 7, this optimization is nonlinear in thedepth map Zij due to unknown normals Nij . This optimization is experimentally found difficult to converge [26].
Our approach solves this problem by alternately optimizing
depths and normals. It not only allows convergence even in4

l and origin O:
Ñ t = N t−1 −(I2 L1 −I1 L2 )dark colored or specular material, where the image-based
approach will likely categorize such regions as the invisible part from all light sources. In our implementation, we
render the reconstructed scene in each iteration by Z-buffer
and determine that each pixel is visible to which light source
or totally invisible.

N t−1 · (I2 L1 − I1 L2 ), (14)
kI2 L1 − I1 L2 k2

and normalizing Ñ t . We note that since the solution space
is a semi-circle, there exists a failure case: Ñ t locates in
the other half of the sphere and does not intersect with thearc. This may happen when given severely incorrect initial
surface normal. In such a case, additional information or assumptions are required. In our implementation, we simply
skip the optimization if 1) the projected normal indicates anegative albedo according to lighting constraints; 2) the angle between Ñ t and N t−1 exceeds the threshold (60o in our
experiment). We do not require any regularization like thecurl-free integrability constraints [14] because we did not
find any improvement by using them.
It is worth noting that our approach can be extended todeal with the scene under ambient light. Given intensities
from three light sources, the ambient light A is constantly
added to the constraints, Ñ · Lk = I k + A, k = 1, 2, 3. By
subtracting from the one with the maximum intensity, we
obtain two virtual lights and their Lambertain constraints.
Consequently, the ambient light case becomes equivalent to
the two-light case.
For the n = 1 light source case, we have a hemisphereas the solution space and are unable to search for a reasonable solution. In n ≤ 1 region, we do not optimize thenormals but preserve the Laplacian coordinates [20]. Our
experiments show that although we do not perform any optimization directly on normals, the recovered mesh is robust
to the noise in n ≤ 1 region and maintains the local neighborhood well without large deformations.

Algorithm 1 Pseudo-code for the fusion framework.
OPTIMIZE(Z 0 , L, I)
Z 0 ← PREPROCESS(Z 0 )Z ← Z0while Z is not converged dovis ← VISIBILITY(Z, L, I)
N ← COMPUTE NORMAL(Z)Ñ ← OPTIMIZE NORMAL(Z, N, vis, L, I)
Z ← OPTIMIZE DEPTH(Z 0 , Ñ )end while

5. Evaluation and Results
Our approach is evaluated using both synthetic data andreal scenes. Using synthetic data, we assess the quantitative
accuracy with respect to the ground truth. Using real data,
we show the ability of our algorithm to automatically preserve surface discontinuities and also demonstrate the capability to deal with shadows, two light sources or threelights plus ambient light. Error analysis is conducted in a
simple case with the ground truth. We shall notice that in
our results we do not perform any hard segmentation on
the mesh but just not render the triangles that have longedges (greater than 15mm in our result). We set weights
λd = 0.01, λn = 0.99, λs = 0.1 described in eq. 8 forall of our experiment. For the sake of comparison, we also
implement the method in [23] by plugging in their gradientcalculation method in eq. 8. The global weights are set tobe the same as ours.
Our Matlab code run on an Intel i5 3.2GHz and 16GB
memory PC in less than 50 seconds per iteration at the resolution of 1024 × 768, in which the sparse linear equation
solver takes up about 30 seconds. We run 3 iterations for a
3-light scene and 10 iterations for a 2-light scene.

4.3. Fusion Framework Summary
The above analysis of the well-defined and the degenerated cases provides us with all the elements needed toformulate the normal optimization using arbitrary numberof light sources over the image domain. Although the total optimization is nonlinear, we include the pseudo-code of
our fusion scheme in Algorithm 1 to emphasize both OPTIMIZE NORMAL and OPTIMIZE DEPTH steps are linear in both space and time. The PREPROCESS smooths
the captured depth map using bilateral filtering. The COMPUTE NORMAL triangles the depth map to a 3D mesh bycamera intrinsic matrix and returns the facet normal for eachpixel. The VISIBILITY detects pixels in shadow.

5.1. Synthetic Data
We render a plane and sphere scene using three directional lights as shown in Figure 4. In the convex case, thehemisphere “pops up” from the plane and cast shadows canbe observed on the plane. In the concave case, the hemisphere “digs” into the plane and the shadow region contains
both one-light and two-light case, in particular the center
part is totally invisible. The sphere radius is 400mm and thecenter is 1200mm away from the camera center. Initially theground truth depth map is artificially corrupted with white

Shadow Pixel Detection Differently from the previous
shadow segmentation by checking the consistency with four
images [10, 4] or the graph-cut based scheme [7, 14], we detect shadow map using the reconstructed 3D mesh because:
1) it provides a principled way for shadow detection; 2) the
shadow map is robust to image noise and insensitive to the5noise of maximum 100mm as input. Results from both our
method and [23] are compared with the ground truth andshown in Table 1. Notice the significant improvement inaccuracy with our method.

Our method
Method from [23]ConvexMean Max
0.883 75.1
28.8 153.8

Concave
Mean Max
3.2
18.4
19.7 120.8

Table 1. Quantitative evaluation (depth error in mm) on synthetic
data.

(a) Input(b) Our Method

Figure 5. The dynamic scene capture setup.

(c) Method from [22]Figure 4. The synthetic data under three different illuminations.
Figure 6. Reconstruction of two balls. The first row contains one
input image and the depth map from Kinect. The second row is our
result, and the third row is the result of [23]. In the fourth column,
the error map of the reconstruction result is visualized.

5.2. Real Capture System
We build up the real scene capture system using a depth
camera - Kinect [21], and replace its low resolution andnonlinear response color camera with the Point Grey Research Flea2, which has a linear response and 1024 × 768
resolution at 30Hz. Figure 5 shows the actual hardware of
our system. We use three white Luxeon K2 LEDs that are
inexpensive, bright and switch instantly. We design a simple microcontroller circuit to trigger the camera at 30Hz andturn on each LED in a sequential manner. Only one of lights
is turned on throughout an exposure time. Kinect captures
independently and reports the latest buffered depth map.
The Flea2 and Kinect cameras are calibrated as well as light
directions using the color checker [17]. The depth maps are
finally registered and converted to the camera coordinate of
the Flea2, and upsampled to dense using the joint bilateral
upsampling [19].
Figure 6 shows the result of two painted balls where thebigger ball is occluded by the smaller one. We compute theerror map using the ground truth that is obtained by measuring the diameter of each ball in both 2D and 3D, and backperspective-projecting the ball center to a 3D location. Using the existing fusion method, the shape recovered in the
shadow region is totally smoothed out and stretches balls
closer to each other. Our method correctly optimizes thenormal map and the depth in the two-light region. Note that
artifacts arising around the lower part of the bigger ball arein the one-light region, we do not perform any optimization
but smoothing.
In Figure 7, we show an extreme case by setting one ofvisibility maps to false, the whole scene is therefore lit byonly two lights. Compared with the result of three lights,
our result misses some details but still contains much moredetails than the input. Meanwhile, compared with the existing approach, our final result using the adaptive differential
operator correctly maintains discontinuities between its two
legs.
Figure 8 shows the same static scene as Figure 1 with6

(a) Input Image and Depth(c) Our Method - 3 Lights(b) Our Method - 2 Lights(d) Method from [22] - 3 Lights

Figure 7. The frog model taken under two and three illuminations. The held leg is behind and mostly occluded by the frontal leg in thecapture view.

3 lights + ambient lightluminating conditions to significantly improve the scene reconstruction quality. The quality improvement is achievedby explicitly modeling discontinuities and shadow areas.
Our fusion framework is designed to work with the minimum number of lighting variations and even with ambient
light. With its modest computational overhead, we believe
our approach can extend the application of current depth
capture techniques by simply setting several LED lights
around and automatically generating much more detailed
shapes.

Our Method

Figure 8. The three-light sources plus ambient light result.

There are certainly rooms for improvement in our current approach. It will be stuck at local minima when the input depth resolution is too low or provides a severely wrong
normal in the ill-conditioned region (e.g., shadows). Thiscan only be remedied by performing strong smoothing. In
addition, our method cannot improve the results in areasthat receive only one or even no illumination. Fortunately
areas like these do not take up too much space in the scene.
Other limitations of our current method are like traditional
photometric stereos: error occurs in the presence of nonLambertian reflectance, interreflection, etc. The use of thedepth map, which is generated by active or passive stereo,
avoids catastrophic failure in such cases. A practical concern in our setup is the directional light assumption. We
have observed that within the effective distance of the depth
sensor, lights can become near and area, introducing soft
shadows. We hope to formulate the light source as a lightfield in next step to get more precise results.

ambient light on. Compared with the ambient-light-free result, artifacts arise in shadows where the constraints reduce
to 1-light case.
Figure 9 shows a dynamic sequence of crossing handsin front of the actor’s body. Because our camera is low
speed (at 30Hz), we have to align pixels across frames. We
compute both forward and backward optic flows using themethod of Black and Anandan [6] every other three frames.
For example, at frame 3, we warp frame 4 using one third of
the backward flow between frame 1 and frame 4, also, warp
frame 2 using one third of the forward flow between frame
2 and frame 5. Shadows can appear on both hands and thebody.

6. Summary and Future WorkIn this paper, we have presented an efficient approach for
combining captured depth map and images under various il7

(a) Input Image(b) Input Depth(c) Method from [22](d) Our Method

Figure 9. Two reconstructed frames from a dynamic sequence.

Acknowledgements[12] R. T. Frankot, R. Chellappa, and S. Member. A method for enforcing
integrability in shape from shading algorithms. PAMI, 10:439–451,
1988. 2[13] M. Harker and P. O’Leary. Least squares surface reconstruction frommeasured gradient fields. In CVPR, 2008. 2, 3
[14] C. Hernández, G. Vogiatzis, and R. Cipolla. Shadows in three-source
photometric stereo. In ECCV, pages 290–303, 2008. 2, 5
[15] T. Higo, Y. Matsushita, and K. Ikeuchi. Consensus photometricstereo. In CVPR, pages 1157–1164, 2010. 2[16] B. K. P. Horn and M. J. Brooks. The variational approach to shapefrom shading. Computer Vision, Graphics, and Image Processing,33(2):174–208, 1986. 2[17] H. Kim, B. Wilburn, and M. Ben-Ezra. Photometric stereo for dynamic surface orientations. In ECCV, pages 59–72, 2010. 2, 6
[18] Y. M. Kim, C. Theobalt, J. Diebel, J. Kosecka, B. Micusik, and
S. Thrun. Multi-view image and tof sensor fusion for dense 3d reconstruction. In Proc. of 3DIM 2009, 2009. 2[19] J. Kopf, M. F. Cohen, D. Lischinski, and M. Uyttendaele. Joint bilateral upsampling. SIGGRAPH, 26(3):96, 2007. 6[20] M. Liao, Q. Zhang, H. Wang, R. Yang, and M. Gong. Modelingdeformable objects from a single depth camera. In ICCV, pages 167–
174, 2009. 5[21] Microsoft.
Kinect camera.
http://www.xbox.com/enUS/kinect/default.htm, 2010. 1, 6
[22] M. G.-H. Mostafa, S. M. Yamany, and A. A. Farag. Integrating shapefrom shading and range data using neural networks. In CVPR, pages
2015–2020, 1999. 2[23] D. Nehab, S. Rusinkiewicz, J. Davis, and R. Ramamoorthi. Efficiently combining positions and normals for precise 3D geometry.
SIGGRAPH, 24(3), aug 2005. 1, 2, 5, 6
[24] M. Oren and S. K. Nayar. Generalization of the lambertian modeland implications for machine vision. IJCV, 14(3):227–251, 1995. 2[25] J. Shotton, A. FItzgibbon, M. Cook, T. Sharp, M. Finocchio,
R. Moore, A. Kipman, and A. Blake. Real-time human pose recognition in parts from single depth images. In CVPR, June 2011. 1[26] G. Vogiatzis, C. Hernández, and R. Cipolla. Reconstruction in theround using photometric normals and silhouettes. In CVPR, pages
1847–1854, 2006. 2, 4[27] R. J. Woodham. Photometric method for determining surface orientation from multiple images. In Optical Engineering, volume 19(1),pages 139–144, 1980. 2[28] J. Zhu, L. Wang, R. Yang, and J. Davis. Fusion of time-of-flight
depth and stereo for high accuracy depth maps. In CVPR, 2008. 2

This work is supported in part by University of Kentucky Research Foundation, US National Science Foundation award IIS-0448185, CPA-0811647, MRI-0923131, National Science Foundation of China grant No. 60872069,
and Zhejiang Provincial Natural Science Foundation ofChina grant 2011C11053.

References[1] A. K. Agrawal, R. Raskar, and R. Chellappa. What is the range of
surface reconstructions from a gradient field? In ECCV, pages 578–
591, 2006. 2[2] R. Anderson, B. Stenger, and R. Cipolla. Color photometric stereo
for multicolored surfaces. In ICCV, 2011. 1, 2
[3] R. Anderson, B. Stenger, and R. Cipolla. Augmenting depth camera
output using photometric stereo. In Machine Vision Applications,
June 2011. 2[4] S. Barsky and M. Petrou. The 4-source photometric stereo technique
for three-dimensional surfaces in the presence of highlights and shadows. PAMI, 25(10):1239–1252, 2003. 2, 5
[5] R. Basri, D. W. Jacobs, and I. Kemelmacher. Photometric stereo withgeneral, unknown lighting. IJCV, 72(3):239–257, 2007. 2[6] M. J. Black and P. Anandan. A framework for the robust estimationof optical flow. In ICCV, pages 231–236, 1993. 7[7] M. K. Chandraker, S. Agarwal, and D. J. Kriegman. Shadowcuts:
Photometric stereo with shadows. In CVPR, pages 1–8, 2007. 2, 5
[8] C.-Y. Chen, R. Klette, and C.-F. Chen. Shape from photometricstereo and contours. In CAIP, pages 377–384, 2003. 2[9] J. E. Cryer, P.-S. Tsai, and M. Shah. Integration of shape from shading and stereo. Pattern Recognition Society, 28(7):1033–1043, 1995.
1, 2[10] M. S. Drew. Reduction of rank–reduced orientation–from–color
problem with many unknown lights to two-image known-illuminantphotometric stereo. In In: IEEE International Symposium on Computer Vision, pages 419–424, 1995. 2, 5
[11] M. A. Fischler and R. C. Bolles. Random sample consensus: A
paradigm for model fitting with applications to image analysis andautomated cartography. Commun. ACM, 24(6):381–395, 1981. 2

8