Bayesian Depth-from-Defocus with Shading Constraints
Chen Li1
1

Shuochen Su3

Yasuyuki Matsushita22State Key Lab of CAD&CG, Zhejiang University

Abstract

Kun Zhou1

Microsoft Research Asia

3

Stephen Lin2

Tsinghua University

but also reveals high-frequency shape variations that allow
shape-from-shading (SFS) methods to match or exceed thelevel of detail obtainable by active sensing [10, 32]. We
therefore seek to capitalize on shading data to refine andcorrect the coarse depth maps obtained from DFD. The
utilization of shading in conjunction with DFD, however,poses a significant challenge in that the scene texture generally needed for DFD interferes with the operation of shapefrom-shading, which requires surfaces to be free of albedo
variations. Moreover, DFD and SFS may produce incongruous depth estimates that need to be reconciled.
To address these problems, we first propose a Bayesian
formulation of DFD that incorporates shading constraints
in a manner that locally emphasizes shading cues in areas
where there are ambiguities in DFD. To enable the use of
shading constraints in textured scenes, this Bayesian DFD
is combined in an iterative framework with a depth-guided
intrinsic image decomposition that aims to separate shading from texture. These two components mutually benefit each other in the iterative framework, as better depth
estimates lead to improvements in depth-guided decomposition, while more accurate shading/texture decomposition
amends the shading constraints and thus results in better
depth estimates.
In this work, the object surface is assumed to be Lambertian, and the illumination environment is captured by imaging a sphere with a known reflectance. Our experiments
demonstrate that the performance of Bayesian DFD withshading constraints surpasses that of existing DFD techniques over both coarse and fine scales. In addition, the useof shading information allows our Bayesian DFD to work
effectively even for the case of untextured surfaces.

We present a method that enhances the performanceof depth-from-defocus (DFD) through the use of shading
information. DFD suffers from important limitations –namely coarse shape reconstruction and poor accuracy on
textureless surfaces – that can be overcome with the help of
shading. We integrate both forms of data within a Bayesianframework that capitalizes on their relative strengths. Shading data, however, is challenging to recover accurately fromsurfaces that contain texture. To address this issue, we propose an iterative technique that utilizes depth information
to improve shading estimation, which in turn is used to elevate depth estimation in the presence of textures. With thisapproach, we demonstrate improvements over existing DFD
techniques, as well as effective shape reconstruction of textureless surfaces.

1. Introduction
Depth-from-defocus (DFD) is a widely-used techniquethat utilizes the relationship between depth, focus setting,and image blur to passively estimate a range map. A pairof images is typically acquired with different focus settings,
and the differences between their local blur levels are thenused to infer the depth of each scene point. In contrast to
active sensing techniques such as 3D scanning, DFD does
not require direct interaction with the scene. Additionally,
it offers the convenience of employing a single stationary
camera, unlike methods based on stereo vision.
With the rising popularity of large format lenses for highresolution imaging, DFD may increase in application due
to the shallow depth of field of such lenses. However, there
exist imaging and scene factors that limit the estimation accuracy of DFD. Among these is the limited size of lens apertures, which leads to coarse depth resolution. In addition tothis, depth estimates can be severely degraded in areas withinsufficient scene texture for measuring local blur levels.
We present in this paper a technique that aims to mitigate the aforementioned drawbacks of DFD through theuse of shading information. In contrast to defocus blur,shading not only indicates the general shape of a surface,

2. Related Work
Depth-from-defocus There exists a substantial amount ofliterature on DFD, beginning with works that handle objects
whose brightness consists of step edges [18, 25, 9]. Since
the in-focus intensity profile of these edges is known, theirdepth can be determined from the edge blur. Later methods
have instead assumed that object surfaces can be locally approximated by a plane parallel to the sensor [33, 26, 30],
such that local depth variations can be disregarded in the1estimation. Some techniques utilize structured illumination
to deal with textureless surfaces and improve blur estimation [15, 14, 29]. DFD has been formulated as a Markov
random field (MRF) problem, which allows inclusion of
constraints among neighboring points [21, 22, 20]. Defocus has also been modeled as a diffusion process that does
not require recovery of the in-focus image when estimating
depth [6].
Shape-from-shading Considerable work has also been
done on shape-from-shading. We refer the reader to the
SFS surveys in [34, 5], and review only the most relevant methods here. SFS has traditionally been applied under restrictive settings (e.g., Lambertian surfaces, uniform
albedo, directional lighting, orthographic projection), andseveral works have aimed to broaden its applicability, suchas to address perspective projection [19], non-Lambertianreflectance [16], and natural illumination [10, 8]. Nonuniform albedo has been particularly challenging to overcome, and has been approached using smoothness and entropy priors on reflectance [3]. Our work instead takes advantage of defocus information to improve estimation for
textured surfaces. Shape-from-shading has also been used
to refine the depth data of uniform-albedo objects obtainedby multi-view stereo [32]. In our method, SFS is used in thecontext of DFD with scenes containing albedo variations.

Figure 1. Imaging model used in depth-from-defocus.

handling surface textures, and finally the iterative algorithm
that integrates all of these components.

3.1. Basic principles of DFD
DFD utilizes a pair of images taken with different focus
settings. The effects of these focus settings on defocus blurwill be described in terms of the quantities shown in Fig. 1.
Let us consider a scene point P located at a distance d fromthe camera lens. The light rays radiated from P to the camera are focused by the lens to a point Q according to the thinlens equation:
1
1
1
+
= ,
(1)
d vd
F
where vd is the distance of Q from the lens, and F is the
focal length. When the focus setting v, which represents
the distance between the lens and sensor plane, is equal tovd , the rays of P converge onto a single point on the sensor,
and P is thus in focus in the image. However, if v ̸= vd , thefocused point Q does not lie on the sensor plane, and P thenappears blurred because its light is distributed to different
points on the sensor. Because of the rotational symmetry oflenses, this blur is in the form of a circle. The radius b ofthis blur circle can be geometrically derived as

Intrinsic images Intrinsic image decomposition aims toseparate an image into its reflectance and shading components. This is an ill-posed problem, since there are twice
as many unknowns (reflectance, shading) as observations(image intensities) per pixel. The various approaches thathave been employed make this problem tractable throughthe inclusion of additional constraints, such as those derived
from Retinex theory [11], trained classifiers [28], and multiple images under different lighting conditions [31]. Despite the existence of these different decomposition cues,
the performance of intrinsic image algorithms has in general been rather limited [7]. Recently, range data has been
exploited to provide strong constraints for decomposition,and this has led to state-of-the-art results [12]. Inspired bythis work, we also utilize depth information to aid intrinsic
image decomposition. However, our setting is considerably
more challenging, since the depth information we start withis very rough, due to the coarse depth estimates of DFD andthe problems of SFS when textures are present.

b=

Rv 1
1 1
− − ,
2 F
v d

(2)

where R is the radius of lens. As seen from this equation,
there is a direct relationship between depth d and blur radiusb for a given set of camera parameters.
The light intensity of P within the blur circle can be expressed as a distribution function known as the point spreadfunction (PSF), which we denote by h. In this paper, we
model the PSF h using a 2D Gaussian function [18]:3. Approachh(x, y; σ) =In this section, we present our method for Bayesian DFD
with shading constraints. We begin with a review of basic
DFD principles, followed by a description of our Bayesian
DFD model, our shading-based prior term, the method for1 − x2 +y2 2
e 2σ
2πσ 2

(3)

with standard deviation σ = γb where the constant γ canbe determined by calibration [9]. Using the PSF, we can2express the irradiance I measured on the image plane as thefollowing convolution:
I(x, y) = If ∗ h(x, y, b),

where P (d) is the prior distribution of depth map d,P (I(1) , I(2) |d) is the likelihood of observations I(1) , I(2) ,and L is the log likelihood of P , i.e. L = − log P .
The likelihood term can be modeled as the basic DFD
energy from Eq. (8), and the prior term as depth smoothness
along the links [22]:(4)

where If is the all-focused image of the scene, such as that
captured by a pinhole camera.
In DFD, we have two images, I1 and I2 , which are captured at the same camera position but with different focus
settings, v1 and v2 :I1 (x, y) = If ∗ h(x, y; σ1 ),I2 (x, y) = If ∗ h(x, y; σ2 ),L(I(1) , I(2) |d) =I2 (x, y) = I1 (x, y) ∗ h(x, y; ∆σ),(5)

L(d) = λ(6)The smoothness prior of Eq. (11) can reduce noise in
the reconstructed depth, but does not provide any additional
knowledge about the scene. We propose to use a more informative prior based on the shading observed in the DFD
image pair, which is helpful both for reconstructing surfaces
with little texture content and for incorporating the finescale shape details that shading exhibits. In this section,
we consider the case of uniform-albedo surfaces, for whichshading can be easily measured. The more complicated caseof textured surfaces will be addressed in Sections 3.4-3.5.
Lambertian shading can be modeled as a quadratic function of the surface normal [23, 10]:(8)d3.2. Bayesian depth-from-defocus model
We now formulate the DFD problem within a Bayesianframework and obtain a solution using a Markov random field (MRF). A basic review of Bayesian models andMarkov random fields can be found in [4, 17]. MRF-based
solutions of DFD have also been used in [22, 20], and a
Bayesian analysis of the larger light-field problem was addressed in [13].
Let i = 1, . . . , N index a 2D lattice G(ν, ε) of image pixels, where ν is the set of pixels and ε is the setof links between pixels in a 4-connected graph. In correspondence with G, let d = (d1 , d2 , .., dN ) denote values of(1) (1)(1)
the depth map D, and let I(1) = (I1 , I2 , . . . , IN ) and
(2)
(2)
(2)
I(2) = (I1 , I2 , . . . , IN ) be the observations at the pixels. Depth estimation can then be formulated as a maximum
a posteriori estimation problem, expressed using Bayes’
theorem as follows:d̂ = argmax P (d|I(1) , I(2) )(11)

3.3. Shading-based prior term

(7)

where d is the depth of pixel Px,y .
Based on Eq. (7), most DFD algorithms solve for depth
by minimizing the following energy function or some variant of it:
argmin(I1 (x, y) ∗ h(x, y, d) − I2 (x, y))2 .

(di − dj )2 .

Hereafter, this particular formulation will be referred to as
standard DFD.
To optimize the MRF model of Eqs. (10)-(11), we use
the max-product variant of the belief propagation algorithm [27], with a message update schedule that propagates
messages in one direction and updates each node immediately.

2I2 (x, y) = I1 (x, y) ∗ h(x, y, d),∑(i,j)∈ε

where ∆σ = σ2 − σ1 . In the preceding equations, it canbe seen that the defocus difference, ∆σ, is determined by
the depth d and the two known focal settings v1 and v2 , so
Eq. (6) can be represented as
2

(10)

i∈ν

where σ1 = γ ∗ b1 and σ2 = γ ∗ b2 . Without loss of generality, assume that σ1 < σ2 . I2 can then be expressed as thefollowing convolution on I1
2

∑ (1)
(2)
(Ii ∗ h(i, d) − Ii )2 ,s(n) = nT M n,

(12)

where nT = (nx , ny , nz , 1) for surface normal n, and M isa symmetric 4 × 4 matrix that depends on the lighting environment. With this shading model, we solve for the surface
normal of each pixel using the method in [10]. We also
obtain the 3D coordinates for each point by re-projecting
each pixel into the scene according to its image coordinates(x, y), depth value d from DFD, and the perspective projection model:
(
)
)
((
h
w)
ud, y −
ud, d ,
x−
2
2

(9)

where w×h is the resolution of the image, and u is the pixelsize.
For each pair of linked pixels i, j in the MRF, we now
have their depths di , dj , 3D positions pi , pj , and normals→
ni , nj . Since the vector −p−i pj should be perpendicular tod= argmax P (I(1) , I(2) |d)P (d)d[]= argmin L(I(1) , I(2) |d) + L(d)d3ture. This problem arises because the brightness variations
from shading and texture are intertwined in the image intensities. To separate shading from texture, methods for intrinsic image decomposition solve the following equation foreach pixel p:
ip = sp + rp ,(14)
where i, s and r are respectively the logarithms of the image
intensity, shading value, and reflectance value.
In this paper, we decompose an image into its shading
and reflectance components with the help of shape information provided by DFD. The method we employ is basedon the work in [12], which uses streams of video and depth
maps captured by a moving Kinect camera. In contrast to
their work, we do not utilize temporal constraints on thedecomposition, since video streams are unavailable in our
setting. Also, we are working with depth data that is often
of much lower quality.
The decomposition utilizes the conventional Retinex
model with additional constraints on non-local reflectance [24] and on similar shading among points thathave the same surface normal direction. Let Ω be the setof all pixels, ℵ be the set of 8-connected pixel pairs, Gr (p)be the set of pixels having a similar local texture pattern asp (computed as in [24]), and Gs (p) be the set of pixels withthe same surface normal as p. Then the shading componentof the image is computed through the following minimization:∑ []argminωp,q s (sp − sq )2 + ωp,q r ((ip − sp ) − (iq − sq ))2

(a)
(b)
(c)
Figure 2. Effect of different prior terms on DFD. (a) Original image (synthesized so that ground truth depth is available). (b/c)
Close-up of depth estimates for the red/green box in (a). From
top to bottom: DFD with no prior, the smoothness prior, and theshading-based prior, followed by the ground truth depth.

the normal direction ni + nj , we formulate the shadingbased prior term as)2∑ (ni + nj(pj − pi )TL(d) = λ.
(13)∥ni + nj ∥(i,j)∈ε

where ε denotes the set of 4-connected neighbors over the
MRF. DFD with this shading-based prior in place of thesmoothness prior will be referred to as DFD with shading
constraints.
In the practical application of this shading constraint, wehave a pair of differently focused images from which to obtain the shading data. The most in-focus parts of the two
images are combined into a single image by focal stacking using Microsoft Research’s Image Composite Editor(ICE) [1], which also automatically aligns different magnifications among the defocused images. This image is thenused for surface normal estimation, with the lighting environment measured using a sphere placed in the scene asdone in [10].
As shown in Fig. 2, the incorporation of shading constraints leads to improvements in DFD, especially in areas
with little intensity variation. Such areas have significant
depth ambiguity in DFD, because the likelihood energy in
Eq. (10) varies little with respect to estimated depth. In suchcases, DFD needs to rely on a prior term to obtain a distinct
solution. The simple smoothness prior of Eq. (11) helps byusing the depths of neighbors as a constraint, but this may
blur high frequency details. By contrast, the shape-basedprior term of Eq. (13) provides fine-scale shape informationthat more effectively resolves uncertainties in DFD.

s(p,q)∈ℵ+∑ ∑ []ωnlr ((ip − sp ) − (iq − sq ))2p∈Ω q∈Gr (p)

+

∑ ∑ [
]
ωnls (sp − sq )2 , (15)p∈Ω q∈Gs (p){ωp,q r =

ωp,q

s

ωr
0

{
ωs
=
0.1ωs

if (1 − ĉTp ĉq ) < τr ,otherwise(16)

if (1 − n̂Tp n̂q ) < τr ,otherwise(17)

where ĉ denotes chromaticity, n̂ denotes surface normal,and ωr , ωnlr , ωs and ωnls are coefficients that balance thelocal and non-local reflectance constraints, and local andnon-local shading constraints, respectively.
We note that Eq. (15) is a quadratic function which canbe simplified to a standard quadratic form:
1
argmin sT As − bT s + c.
2s

3.4. Texture Handling
Shading information becomes considerably more difficult to extract from an image when its surfaces contain tex-

(18)

It is optimized in our implementation using the preconditioned conjugate gradient algorithm.
4

(a)
(b)
(c)
(d)
(e)
Figure 3. Depth and shading refinement in our iterative approach.
(a) Defocus pair. (b-d) Estimated shading (top) and depth (bottom)
for (b) first iteration, (c) middle iteration, (d) final iteration. (e)Ground truth.

3.5. Iterative algorithm
The performance of depth-guided intrinsic image decomposition depends on the accuracy of the input depth.
Likewise, the utility of shading constraints in DFD rests on
how well shading is extracted from the image. Since DFD
and intrinsic image decomposition facilitate each other, we
use them in alternation within an iterative framework. Each
iteration begins with the DFD step, followed by decomposition. This cycle is repeated until the average change in
depth within each local region (which is partitioned in our
implementation by a 10x10 grid on the image) lies below a
threshold. We solved the MRF using a multi-scale refinement with 200 depth labels per depth range and reducing
the range by 15% with each iteration. We used 15 iterations
which equivalently gives about 2000 depth labels in total.
Since the estimated shading and depth are less accurate
in earlier iterations, the parameters in DFD and decomposition are set in each iteration to account for this. Initially,
the shading constraint weight λ in Eq. (13) for DFD andthe non-local shading coefficient ωnls in Eq. (15) for decomposition are set to relatively low values (0.5 and 0.05,
respectively, in our implementation). At each subsequent iteration, both of these values are increased by a factor of 1.1
until reaching a maximum of twice the initial value, after
which these coefficients are no longer made larger.
As illustrated in Fig. 3, the iterations bring improvements
to both the estimated depth and shading. This iterative algorithm has converged to a significantly better result for all
the examples we tested.

(a)
(b)
(c)
(d)
Figure 4. Comparison of estimated normal maps. (a) From standard DFD. (b) From SFS with natural illumination [10]. (c) Computed from our estimated depth maps. (d) Ground truth.
0.4

0.3

0.4

SIRFS
DFD via Diffusion
Standard DFD
DFD with Shading

0.3

0.2

0.2

0.1

0.1

0

BuddhaFeline

Zebra

0

SIRFS
DFD via Diffusion
Standard DFD
Our Iterative Approach

BuddhaFeline

Zebra

(a)
(b)
Figure 5. Depth error analysis. (a) For synthetic non-textured objects. (b) For synthetic textured objects.

niques: standard MRF-based DFD with smoothness constraints, DFD via diffusion [6], and the single-image SIRFS
method [3]1 . In these experiments, a foreground mask isused to discard the background, and depth maps are scaled
to the range of [0,1] for visualization.

4.1. Synthetic images
The first set of experiments uses synthetic data to provide
comparisons to ground truth. Three object models – Buddha, feline and zebra [35] – are handled with and without
texture, under illumination from the Eucalyptus Grove environment map [2, 23]. The defocus pair is rendered with blur

4. Results

1 The

results for DFD via diffusion and SIRFS weregeneratedusing
theauthors’
downloadablecodeat
http://home.eps.hw.ac.uk/˜pf21/pages/page4/page4.html
andhttp://www.cs.berkeley.edu/˜barron, respectively.

We evaluated our method on synthetic and real images,
both with and without texture. The depth estimates ofour method are compared to those of three previous tech5

(a)

(b)

(c)

(d)
(e)
(f)
(g)
(h)
(i)
(j)
(k)
Non-texture results
Texture results
Figure 6. Synthetic data results. (a) Ground truth depth maps. (b/g) Non-textured/textured input defocus pairs. Depth estimate results for(c/h) SIRFS [3], (d/i) DFD via diffusion [6], (e/j) standard DFD, (f/k) our method.

according to Eq. (2) and with the virtual camera parameters
set to F = 0.01, F num = 2.0, and γ = 1000000. The
two focal settings are chosen such that their focal planes
bound the ground truth depth map, and random Gaussian
noise with a standard deviation of 1.0 is added to simulate
real images.
The benefits of utilizing shading information with DFD
are illustrated in Fig. 4 for normal map estimation on textureless objects. Here, the normal maps are constructedfrom gradients in the estimated depth maps. The uncertaintyof DFD in areas with little brightness variation is shown
to be resolved by the shading constraints. As we use themethod of SFS with natural illumination [10] to obtain surface normals, our technique is able to recover a similar levelof shape detail.
Our depth estimation results are exhibited together withthose of the comparison techniques in Fig. 6. The average errors for each method within the foreground masks areshown in Fig. 52 . With the information in a defocus pair,
our method can obtain results more reliable than that of thesingle-image SIRFS technique. In comparison to the two
DFD methods, ours is able to recover greater shape detailthrough the use of shading.

4.2. Real images
We also compared our method to related techniques using real images. As with the synthetic data, the comparison
methods are SIRFS [3], DFD via diffusion [6], and standard
DFD. The images were captured using a Canon 5D Mark II
camera with a 100mm lens. We mounted the camera on a
tripod and shot the images in RAW mode with the objects
about 50cm away.
In order to use our shading constraints, we first calibrate
the natural illumination using a white Lambertian sphere,
and then use the known surface normals of the sphere tosolve the shading matrix in Eq. (12) by least-squares optimization. Because the albedo of the sphere may differ fromthose of our target objects, we estimate the relative albedo
between target objects and the sphere simply by comparing
the brightness of manually identified local areas that have asimilar normal orientation. For objects with surface texture,
the albedo of the local area used in this comparison is usedas the reference albedo for the object.

2 DFD by diffusion does not work as well as standard DFD on our objects because its preconditioning is less effective when the intensity variations are not large.

6

(a)
(b)
(c)
(d)
(e)
Figure 7. Real image results. (a) Input defocus pairs. Depth estimate results for (b) SIRFS [3], (c) DFD via diffusion [6], (d) standard
DFD, (e) our method.

The results for real images are shown in Fig. 7. The
first example is a plaster bust of uniform color. With the
SIRFS method, the depth variations on the body correctly
follow the object shape, but the head is shown to be closer
than it actually is. The depth estimates of DFD via diffusion and standard DFD are both generally accurate for thehead and body. They however exhibit some false fine-scalevariations, perhaps due to DFD ambiguity in non-textured
regions. Our result conforms most closely to the actual object, with shading information to provide shape details andhelp resolve DFD uncertainties.
The last three examples contain albedo variations. For
the dress in the second row, our results exhibit a finer levelof detail than the others. The general depth trends shownwith SIRFS are accurate, but the albedo change and shapedetails are missed. DFD via diffusion performs relatively
well on this example. Some detail is visible, but not as much
as what our method obtains through shading. Standard DFD
shows some shape detail as well, but also displays someobvious errors, such as near the top-right corner.
For the turtle in the third row, the depth estimates of our
method show greater accuracy. The SIRFS method does afairly good job, but does not indicate the nearness of theright leg. It also shows the shell and neck at the same depth,
and a smooth depth transition from the head to the shell.
DFD via diffusion does not exhibit the gradual changes ofdepth over the object, while standard DFD displays incorrect depth variations in areas with little texture.
The final example, in the fourth row, is of a bunny figurine. With SIRFS, the head and far arm are well reconstructed. The depth of the closer arm, however, is off, andthe left foot is not shown to be closer. Both this result andthe one of DFD via diffusion exhibit less shape detail thanour depth estimate. Standard DFD displays some shape detail, but has problems on the mostly textureless head.

5. Conclusion
In this paper, we presented a method to enhance depthfrom-defocus by incorporating shading constraints. To effectively utilize the shading information on objects withvarying albedo, we proposed an iterative technique that uses
DFD and shading estimation in manner in which they facilitate each other. Our experiments demonstrate that the useof shading constraints brings greater accuracy and detail to
DFD, especially in areas without clear DFD solutions.
7

In future work, we plan to investigate ways to increase
the accuracy of our depth estimates. Our current implementation assumes the incident illumination to be the same at all
surface points. However, this will not be the case due to different self-occlusions of an object towards different lighting
directions. This issue could be addressed by computing thelight visibility of each point from the estimated depth.

[17] B. Peacock, N. Hastings, and M. Evans. Statistical Distributions. Wiley-Interscience, June 2000.
[18] A. P. Pentland. A new sense for depth of field. IEEE Trans.
Patt. Anal. and Mach. Intel., 9(4):523–531, Apr. 1987.
[19] E. Prados and O. Faugeras. A generic and provably convergent shape-from-shading method for orthographic and pinhole cameras. Int. Journal of Computer Vision, 65(1):97–
125, 2005.
[20] A. Rajagopalan, S. Chaudhuri, and U. Mudenagudi. Depth
estimation and image restoration using defocused stereo
pairs. IEEE Trans. Patt. Anal. and Mach. Intel., 26(11):1521–1525, nov. 2004.
[21] A. N. Rajagopalan and S. Chaudhuri. Optimal selection ofcamera parameters for recovery of depth from defocused images. In CVPR, 1997.
[22] A. N. Rajagopalan and S. Chaudhuri. An mrf model-based
approach to simultaneous recovery of depth and restorationfrom defocused images. IEEE Trans. Patt. Anal. and Mach.
Intel., 21(7):577–589, July 1999.
[23] R. Ramamoorthi and P. Hanrahan. An efficient representation for irradiance environment maps. In ACM SIGGRAPH,
pages 497–500, 2001.
[24] L. Shen, P. Tan, and S. Lin. Intrinsic image decomposition
with non-local texture cues. In CVPR, pages 1 –7, june 2008.
[25] M. Subbarao and N. Gurumoorthy. Depth recovery fromblurred edges. In CVPR, pages 498 –503, jun 1988.
[26] M. Subbarao and G. Surya. Depth from defocus: A spatial domain approach. Int. Journal of Computer Vision,
13(3):271–294, 1994.
[27] R. Szeliski, R. Zabih, D. Scharstein, O. Veksler, V. Kolmogorov, A. Agarwala, M. Tappen, and C. Rother. A comparative study of energy minimization methods for markov
random fields with smoothness-based priors. IEEE Trans.
Patt. Anal. and Mach. Intel., 30(6):1068 –1080, june 2008.
[28] M. F. Tappen, E. H. Adelson, and W. T. Freeman. Estimating
intrinsic component images using non-linear regression. In
CVPR, pages 1992–1999, 2006.
[29] M. Watanabe, S. Nayar, and M. Noguchi. Real-Time Computation of Depth from Defocus. In Proc. SPIE, volume
2599, pages 14–25, Jan 1996.
[30] M. Watanabe and S. K. Nayar. Rational filters for passive depth from defocus. Int. Journal of Computer Vision,27(3):203–225, May 1998.
[31] Y. Weiss. Deriving intrinsic images from image sequences.
In ICCV, pages 68–75, 2001.
[32] C. Wu, B. Wilburn, Y. Matsushita, and C. Theobalt. Highquality shape from multi-view stereo and shading under general illumination. In CVPR, pages 969–976, 2011.
[33] Y. Xiong and S. A. Shafer. Depth from focusing and defocusing. In CVPR, pages 68–73, 1993.
[34] R. Zhang, P.-S. Tsai, J. Cryer, and M. Shah. Shape-fromshading: a survey. IEEE Trans. Patt. Anal. and Mach. Intel.,21(8):690 –706, aug 1999.
[35] K. Zhou, X. Wang, Y. Tong, M. Desbrun, B. Guo, and H.-Y.
Shum. Texturemontage: Seamless texturing of arbitrary surfaces from multiple images. ACM Transactions on Graphics,
24(3):1148–1155, 2005.

Acknowledgements
This work was partially supported by NSFC (No.
61272305). It was done while Chen Li and Shuochen Su
were visiting students at Microsoft Research Asia.

References[1] Image composite editor. http://research.microsoft.com/
en-us/um/redmond/groups/ivm/ice.
[2] Light probe image gallery.
http://www.pauldebevec.com/Probes/.
[3] J. T. Barron and J. Malik. Color constancy, intrinsic images,
and shape estimation. In ECCV, 2012.
[4] A. Blake, P. Kohli, and C. Rother. Markov Random Fields
for Vision and Image Processing. The MIT Press, 2011.
[5] J.-D. Durou, M. Falcone, and M. Sagona. Numerical methods for shape-from-shading: A new survey with benchmarks.
Compt. Vision and Image Underst., 109(1):22–43, 2008.
[6] P. Favaro, S. Soatto, M. Burger, and S. Osher. Shape fromdefocus via diffusion. IEEE Trans. Patt. Anal. and Mach.
Intel., 30(3):518–531, 2008.
[7] R. Grosse, M. K. Johnson, E. H. Adelson, and W. T. Freeman. Ground truth dataset and baseline evaluations for intrinsic image algorithms. In ICCV, 2009.
[8] R. Huang and W. Smith. Shape-from-shading under complex
natural illumination. In ICIP, pages 13–16, 2011.
[9] T.-l. Hwang, J. Clark, and A. Yuille. A depth recovery algorithm using defocus information. In CVPR, pages 476 –482,
jun 1989.
[10] M. K. Johnson and E. H. Adelson. Shape estimation in natural illumination. In CVPR, pages 2553–2560, 2011.
[11] R. Kimmel, M. Elad, D. Shaked, R. Keshet, and I. Sobel. A
variational framework for retinex. Int. Journal of Computer
Vision, 52:7–23, 2003.
[12] K. J. Lee, Q. Zhao, X. Tong, M. Gong, S. Izadi, S. U. Lee,
P. Tan, and S. Lin. Estimation of intrinsic image sequencesfrom image+depth video. In ECCV, pages 327–340, 2012.
[13] A. Levin, W. T. Freeman, and F. Durand. Understanding
camera trade-offs through a bayesian analysis of light fieldprojections. In ECCV, pages 88–101, 2008.
[14] S. Nayar, M. Watanabe, and M. Noguchi. Real-time focus
range sensor. In ICCV, pages 995 –1001, jun 1995.
[15] M. Noguchi and S. Nayar. Microscopic shape from focus
using active illumination. In ICPR, volume 1, pages 147–
152, oct 1994.
[16] G. Oxholm and K. Nishino. Shape and reflectance from natural illumination. In ECCV, pages I:528–541, 2012.

8