IEEE TRANSACTIONS ON PATTERN RECOGNITION AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MONTH YEAR

1

Semi-Calibrated Photometric Stereo
Donghyeon Cho, Student Member, IEEE, Yasuyuki Matsushita, Member, IEEE,
Yu-Wing Tai, Member, IEEE, and In So Kweon, Member, IEEE
Abstract—While conventional calibrated photometric stereo methods typically assume that light intensities are identical and sensor
exposure is constant across observed images taken under varying lightings, these assumptions easily break down in practical settings
due to individual light bulb’s characteristics and limited control over sensors. In this paper, we study the effects of non-uniform lightintensities and sensor exposures across observed images. In order to explicitly model these non-uniformities, we define ”semi-calibrated”
photometric stereo framework that light directions are assumed to be known or calibrated, but light intensities and exposures arevariables to be estimated. Then, we propose methods for accurately determining surface normal in ”semi-calibrated” photometric stereo
framework. We consider how the new proposed concept, ”semi-calibrated” photometric stereo, differs from conventional ”uncalibrated”
and ”calibrated” photometric stereo and looks at its practical usefulness. In addition, we show that our methods are advantageous forgeneral photometric stereo settings, where auto-exposure control is desirable. We compare our method with conventional Lambertiancalibrated photometric stereo and robust photometric stereo methods as well as uncalibrated photometric stereo, and the experimental
result shows superior accuracy of our method in this practical circumstance.
Index Terms—Photometric stereo, Light intensity, Exposure, Surface normal

F

1

I NTRODUCTION

N

ON - UNIFORM light intensities and exposures across observed images are a practical and common circumstancein data acquisition for photometric stereo that uses multiple
images under distinct light directions. For example, different
light bulbs with different intensity characteristics may be used
for illuminating a scene. Even with identical light bulbs, due tothat scene radiance is determined by surface normal and light
directions, auto-adjusted sensor exposure is desirable dependingon the light directions to avoid over-/under-exposures. Therefore,
the capability of properly handling varying light intensities andexposures across observed images is an important feature formaking photometric stereo more practical.
In the Lambertian image formation model, a measured intensity m is written as

mi,j = Ei ρj n>
j li ,(1)

where i and j are indices of light direction and pixel location,
li , nj ∈ R3×1 are unit vectors of light direction and surface
normal, ρj ∈ R is a Lambertian diffuse albedo, and Ei ∈ R
is a light source intensity. In a matrix form for representing all
pixels and light directions at a time, it can be written as

M = ELN> P,(2)

where M ∈ Rf ×p is an observation matrix, E is an f ×f diagonallight intensity matrix, L ∈ Rf ×3 is a light direction matrix,
N ∈ Rp×3 is a surface normal matrix, P is a p × p diagonal
diffuse albedo matrix, and f and p are the number of images andpixels, respectively. Conventional photometric stereo [1] assumes•••

D. Cho and I.S. Kweon are KAIST, Daejeon, Korea.
E-mail: cdh12242@gmail.com, iskweon@kaist.ac.kr
Y. Matsushita is with Osaka University, Japan.
E-mail: yasumat@ist.osaka-u.ac.jp
Y-W. Tai is with Tencent Youtu, HongKong.
E-mail: yuwing@gmail.com

Manuscript received XXXX XX, XXXX; revised XXXX XX, XXXX.

that light source intensities are identical across images, where
the matrix E becomes a scaled identity matrix (E = eI), andcomputes albedo-scaled surface normal B(= P> N) by∗

eB> = L† M,(3)

up to a scale ambiguity e, where the superscript † indicates a
generalized inverse when f ≥ 3.
Clearly, when the light source intensities are non-uniform orcamera exposures vary across images, the assumption E = eI
does not hold, but instead its diagonal elements have individual
scales. When this non-uniformity is present, the surface normal
estimates by Eq. (2) naturally becomes biased by greater scales
as illustrated in Fig. 1. While there are recently various robust
estimation techniques used for photometric stereo [2], [3], [4],[5], [6], because the effect of non-uniform E neither increases
the rank of the observation matrix nor sparsifies outliers, robust
techniques such as rank minimization or `0 -norm minimization
techniques cannot resolve this issue. In the rest of the paper, wecollectively call this problem setting, non-uniform light intensities
and exposures across images, a varying light intensity condition,
because they are both considered intensity scaling on individual
images.
Under varying light intensity condition, as well as light direction, light intensity is the factor which is pre-calibrated or
estimated with surface normal. Depending on whether the directions and intensities of light sources are calibrated, photometricstereo problem can be categorized into four cases: knowing
both, not knowing both, and knowing only one of them. In thispaper, we focus on a case that light directions are assumed tobe known or calibrated, but light intensities and exposures arevariables to be estimated. We call this case as ”semi-calibrated”
photometric stereo. We considers a method to effectively solve”semi-calibrated” photometric stereo under varying light intensity
condition. The problem that we deal with in this paper is a bilinear
problem written as following.

IEEE TRANSACTIONS ON PATTERN RECOGNITION AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MONTH YEAR

Exposuretime

Exposuretime

2

Exposure
time

Same light brightness and exposure time

Different light brightness

Different exposure time

(a)

(b)

(c)

Fig. 1: (a) Conventional photometric stereo setting where constant light intensities and exposures are used, (b)(c): Varying lighting
intensity/exposure conditions. Estimated surface normal are biased toward brighter light source or images captured with longer
exposures with a conventional solution method.
Problem (Semi-calibrated photometric stereo under varying
light intensity condition). Given observations M and light
directions L, estimate a diagonal light intensity matrix E andan albedo-scaled surface normal matrix B from the following
relationship:
M = ELB> .
(4)

2)
3)

2
We first show that there exists a linear closed-form solution
method, which simultaneously estimates scales of light intensities(or exposures) E and albedo-scaled surface normal B. We callthis method a linear joint estimation method. This method is
straightforward to implement; however, inefficient in terms ofcomputation time and memory consumption. We then introduce afactorization based method for determining only surface normal B
in Eq. (4) without being affected by E. It bypasses the estimationof E using algebraic distance minimization (or, cross productminimization) by making the problem independent of vector
magnitudes. Next, we show that this bilinear problem can be
efficiently solved by an alternating minimization technique that
determines E and B in each step. Finally, in order to handle
non-Lambertian components, we introduce robust alternating minimization method. We discuss details and characteristics of eachmethod later in this paper.
We further show that our method is advantageous in improving
signal-to-quantization-noise ratio (SQNR) in comparison to a standard photometric stereo method when an auto-exposure control isused, and as a result more accurate surface normal estimates canbe obtained. Experimental results show the effectiveness of the
proposed method in practical settings. In this paper, we assumes
a directional light setting where radiance from a light source toa scene is constant except for shadowing, i.e., spatially varying
incident radiance within a scene is not assumed.
A shorter version of this work appeared in [7]. This paper
extends our previous work with defining ”semi-calibrated” photometric stereo, and additional experimentations. Also, the L1-norm
based alternating minimization algorithm is newly introduced. Tosummarize, our contributions are as follows:1)

We introduce a concept of ”semi-calibrated” photometricstereo under varying light intensity condition, and analyze
differences with uncalibrated and calibrated photometricstereo.

In order to handle non-Lambertian components, we introduce robust alternating minimization method.
We evaluate our methods on new photometric stereo
dataset, and demonstrate effectiveness of proposed methods.

R ELATED W ORKS

Photometric stereo was first introduced by Woodham [1] in 1980’s
for determining surface normal from images taken under known
and varying light directions with a Lambertian reflectance assumption. After Woodham’s work, there have been various techniques
proposed for making photometric stereo more practical. Theirmain focuses are to relax the assumptions of (1) calibrated lightsources and (2) Lambertian image formation model.
The first class of the methods, called uncalibrated photometricstereo, tries to eliminate the need for calibrating light directions.
When the light directions are unknown, it is understood that thesolution can be obtained up to a 3 × 3 linear ambiguity [8]. If
the integrability [9] of the surface is assumed, it has been shownthat the linear ambiguity can be reduced to a generalized basrelief (GBR) ambiguity [10], which only has three parameters. To
fully resolve these ambiguities, various types of external clues
have been used. For example, there are methods that use theentropy of albedo distributions [11], specular observations [12],shadows [13], and groups of color and intensity profiles [14]. Our
problem setting has a similarity to the uncalibrated photometricstereo scenario in that we relax the assumptions of known lightintensities and constant light intensity across varying light directions. And none of the previous uncalibrated photometric stereo
works can effectively solve the problem.
The second class of the methods tries to make photometricstereo applicable to non-Lambertian scenes. There are methods
that use more sophisticated reflectance models than Lambertian reflectance model, such as the works that use Torrance-Sparrow [15],[16], Cook-Torrance [17], Phong [18], Blinn-Phong [19]. Morerecently, Shi et al. [20] propose a bi-polynomial reflectance modelthat produces successful results for non-Lambertian diffusive
scenes. There are approaches that use robust estimation techniquesby treating non-Lambertian reflectances and shadows as outliers.
In [2], the robustness against outliers is achieved by capturing
hundreds of input images coupled with Markov Random Field

IEEE TRANSACTIONS ON PATTERN RECOGNITION AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MONTH YEAR

(MRF) to maintain neighborhood smoothness. Verbiest and Van
Gool [3] use a confidence approach to reject outliers in input images of photometric stereo. Wu et al. [4] proposed a robust method
based on low-rank matrix factorization. Oh et al. [5] introduced apartial sum of singular values for rank minimization, and showed
good performance in photometric stereo. Ikehata et al. [21] useda sparse Bayesian regression for effectively neglecting sparse
outliers (specularities and shadows). While these techniques are
effective, they are built upon the assumption of constant lightintensity, and cannot directly address the issue of varying lightintensities and exposures.

3

Calibrated Case

Usually, in conventional calibrated photometric stereo, light directions L are pre-calibrated while constant intensities acrossimages are assumed. As a result, estimated surface normal canbe globally biased toward brighter light source under non-uniform
light intensities and exposures across images. In order to remove
distortion from varying light intensity condition, light intensities
E can be also pre-calibrated. However, sophisticated operations
using Lambertian sphere are required to calibrate intensities oflight sources. Taking into consideration that calibration using thechrome spheres to obtain directions of light sources, calibrating
intensities of light sources using another Lambertian sphere inadvance means that two different pre-calibration with two different
types of spheres are required. It is very inefficient and makes
photometric stereo impractical.
3.2

Hayakawa Case

As mentioned above, we touch cases that one of light directions
and intensities is known. Firstly, we think of light intensity
calibration without knowing light directions. As reported in [8] by
Hayakawa et al., if relative intensities across more than 6 frames
are known, 3 × 3 non-singular matrix H can be computed bysolving following relationship, ŜH = Ê where Ê is known
relative intensities. However, it depends on a few known intensities
of some frames, and is vulnerable to errors and outliers. Also, as
discussed later section Sec. 6, light intensity calibration is noteasy as light direction calibration. To prevent confusion, we callthis case as Hayakawa case.

C ATEGORIZATION OF P HOTOMETRIC S TEREOIn this section, we categorize Lambertian photometric stereo into
four cases depending on whether the directions and intensities
of light sources are known. Also, we study effectiveness of eachapproach for non-uniform light intensities and exposures acrossimages.
3.1

3.3

3

3.4

Semi-calibrated Case with Known Light Direction

In this paper, we focus on a case that light directions are known
or calibrated, but light intensities and exposures are variable tobe estimated with albedo-scaled surface normal. We call it as”semi-calibrated” photometric stereo that can effectively handle
non-uniform light intensities and sensor exposures across images. It is more practical than calibrated case and more generalthan uncalibrated photometric stereo. Also, unlike calibrated andHayakawa cases, ”semi-calibrated” photometric stereo is suitable
to auto-adjusted sensor exposure function for avoiding over/under-exposures because intensities of each frame are not fixedin advance. So far, since there is no studies that explictly focus on”semi-calibrated” photometric stereo under varying light intensity
condition, we propose four solution methods in Sec. 4.

4

S EMI -C ALIBRATED P HOTOMETRIC STEREO UN -DER VARYING LIGHT INTENSITY CONDITIONSAs discussed in Eq. (4) and Sec. 3.4, we are interested indetermining albedo-scaled surface normal B with unknown nonuniform scalings of light intensities or exposures E. In a leastsquares framework, the problem can be written as{E∗ , B∗ } = argminkM − ELB> k2F

Uncalibrated Case

(7)

E,B

To make photometric stereo practical, in uncalibrated photometricstereo, surface normal, albedo, light direction and intensity are
directly estimated from captured images without calibrating light
directions and intensities in advance. Solutions of uncalibrated
photometric stereo can be obtained via matrix factorization:

M = ŜB̂> ,(5)where Ŝ and B̂ are biased intensity-scaled light direction andalbedo-scaled surface normal, respectively. With an arbitrary 3×3
non-singular matrix H, Eq. (5) can be re-written as−1

M = (ŜH)(H

>

B̂ ).

(6)

Since any non-singular 3 × 3 matrix H satisfy Eq. (6), there
is no unique solution for surface normal. Therefore, previous
approaches [8], [9], [10], [11], [12], [13], [14] utilize prior knowledge about albedo or surface normal. In examples that satisfy
prior assumptions, light direction and intensity are obtained aswell as surface normal and albedo. However, when assumptions
are violated, all of estimates become incorrect. In other words,
uncalibrated photometric stereo lacks of generality even though it
has benefits of practicality.

given the observations M and light directions L.
We first present a linear estimation method that simultaneously
estimates B and E in Sec. 4.1. We then describe a factorization
based method in Sec. 4.2, which bypasses the estimation of
unknown scalings E. Then, we describe an efficient alternating
minimization method in Sec. 4.3. Finally, we introduce robust
alternating minimization method to handle non-Lambertian components.
4.1

Linear joint estimation method

The original form M = ELB> can be re-written as E−1 M =
LB> , because E is always invertible as it is a positive diagonal
matrix. Given known M and L, it can be viewed as a variant of a
Sylvester equation [22]:E−1 M − LB> = 0.

(8)

By vectorizing unknown variables E−1 and B> , Eq. (8) can be
written as>
diag(m1 )| · · · |diag(mp ) E−1 1(9)− (Ip ⊗ L)vec(B> ) = 0,

IEEE TRANSACTIONS ON PATTERN RECOGNITION AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MONTH YEAR

where diag(·), vec(·) and ⊗ are diagonalization, vectorization,
and Kronecker product operators, respectively. Ip is a p × p
identity matrix, and 1 indicates a vector whose elements are
all one. By concatenating matrices and vectors in Eq. (9), a
homogeneous equation can be obtained:h
> i vec(B> )= 0, (10)−Ip ⊗ L| diag(m1 )| · · · |diag(mp )−1|{z} | E{z 1 }D

Factorization based method

Although the linear joint estimation method is simple to implement, it has practical limitations in terms of its computationaltime and memory requirement when the sparse matrix D is large;
not only constructing D but also computing SVD of D. This
limitation can be relaxed by dividing the observation matrix into
small groups and deriving solutions for each group. However, thisgrouping should be performed carefully to avoid the condition
numbers of divided sub-matrices to be high. The condition number
increases when observations within each divided group are similar
to each other, and as a result, the numerical error becomes greater.
To avoid these issues, we develop a factorization based method
described in this section.
Like solution methods of uncalibrated photometric stereo, light
directions and surface normal can be solved directly via matrix
factorization as Eq. (5) and Eq. (6). In our setting, since we know
the light directions L, we can find an appropriate non-singularmatrix H for resolving the biases. Regardless of the effect of lightintensities, direction of ŜH should be the same with L. Thus, we
can use this constraint, (ŜH) × L = 0 where × indicates a crossproduct for determining H as
 h1 >
0−li,3 ŝi li,2 ŝi  > 
= 0,
(11)
h
li,3 ŝi0−li,1 ŝi  2 > h3where H = [h1 |h2 |h3 ], li,∗ and ŝi are i-th row of L and Ŝ,respectively. The solution of Eq. (11) is unique up to scale when
there are more than 4 distinct light directions. Using estimated Ĥ,we can compute unbiased albedo-scaled surface normal H−1 B̂> .
Interestingly, this factorization based method can naturally bypass
the light intensity estimation; thus, it is suitable for our setting.
Compared to the linear joint estimation method, the computational
cost of the factorization based method is lower, even without
dividing observations M into small groups.
4.3

such as shadows and pixel saturations. To avoid this problem, wedevelop a robust method that is based on alternating minimization
for solving Eq. (7).
Our method computes albedo-scaled surface normal B(t) andnon-uniform scalings E(t) in an alternating manner using theirintermediate estimates from the previous iteration. Using E(t)from the previous iteration and by fixing it, albedo-scaled surface
normal B(t+1) is updated byywhere D ∈ Rpf ×(3p+f ) is a sparse design matrix and y ∈R(3p+f )×1 is an unknown vector. The homogeneous system
always has a trivial solution y = 0. To have a unique (up to scale)
non-trivial solution, the matrix D should have a one dimensionalnull space, i.e., when rank of D is (3p + f − 1), a unique solution
can be obtained via singular value decomposition (SVD). The
minimum condition to have a unique solution up to scale is f ≥ 5
and p ≥ 3, or f = 4 and p ≥ 2. Unlike conventional photometricstereo settings, increasing the number of light directions does notnecessarily make the problem easier in this setting, because it also
increases the unknowns about light intensities.
4.2

4

Alternating minimization method

While the previous two methods are effective in ideal settings,
they are prone to large errors due to un-modelled observations,

B(t+1) = argmin M − E(t) LB>
B

2
F

.

(12)

The above problem is a linear problem with respect to B and canbe solved efficiently. Once matrix B(t+1) is determined, E(t+1) isthen updated by solving

E(t+1) = argmin M − ELB(t+1)

> 2
F

E

.

(t+1)

Since matrix E is diagonal, each element Eidetermined byP(t+1) >>)
j mi,j (li bj
(t+1)
Ei
=P
.
(t+1)
(t+1) >>>)(li bj)j (li bj

(13)
is simply(14)

The initial scaling matrix E(0) is set to an identity matrix, and theconvergence criteria is defined by the magnitude of variation of
matrix B, i.e., kB(t+1) − B(t) kF < , where  is set to a smallvalue (in our implementation,  = 1.0e-8).
If we consider E as weights, this alternating minimization issimilar to iteratively re-weighted least squares (IRLS) [23] exceptthat weights are defined row-wise (each image has same weight).
We show how the alternating method operates in the following.
Let us consider updating B(t+1) with fixing E(t) , then Eq. (12)becomes

B(t+1)=

argmin M − E(t) LB>
B=

2

argmin M − E∗ LB> − Er LB >
B

(15)

F
2
F

,

where E(t) = E∗ + Er , E∗ is the ground truth (that we do notknow), and Er is the error from t-th iteration. It shows that thesmaller the scaling error Er is, the smaller objective cost becomes.
The elements of E(t) can also be written as
P
(t) >
>
j mi,j (li bj )(t)Ei = P
(t)
(t) >
>
>
j (li bj )(li bj )
(16)
P
P
∗>
r>
>
+ j mi,j l>i bj
j mi,j li bj=P > ∗
r >
∗
r > ,
>
j (li (bj + bj ) )(li (bj + bj ) )where b(t+1) = b∗ + br , b∗ is the ground truth, and br is theerror from t-th iteration. Since the denominator is fixed for all
images, and the left-hand side of the numerator is proportional to
the ground truth scaling E∗ , the smaller the error br becomes, thebetter scaling elements E becomes. To summarize, if the current
estimate of albedo-scaled surface normal B(t) is better than theprevious one, E(t) is better updated. In our experiments, above
improvements are always observed since updated E(1) becomescloser to the ground truth than E(0) . Then, B(t) and E(t) arealternatingly updated. The minimum condition for obtaining astable solution is experimentally found to be f ≥ 5 and p ≥ 3.

IEEE TRANSACTIONS ON PATTERN RECOGNITION AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MONTH YEAR

4.4

Robust Alternating minimization method

solving Eq. (7) by the linear joint estimation, factorization based,alternating minimization (AM) can be biased by outliers. One
possible solution to resolve outliers, is to reformulate Eq. (7)
as
(17)

E,B

In order to solve L1-norm minimization in Eq. (17), we adopt
iteratively re-weighted least squares (IRLS) [23]. At each iteration,
we multiply a weight value for each pixel. Pixels that are considered outliers are multiplied by small weights, and pixels considered inliers are multiplied by high weights. Therefore, Eq. (12),and Eq. (13) are modified as(M − E(t) LB> )B

E(t+1) = argmin W(t)>(M − ELB(t+1) )E

where W(t) is weight matrix and
Weight matrix is defined as

2
F

,

(18)

2
F

,

(19)

element-wise multiplication.
>

W(t) = γ · max(|(M − E(t−1) LB(t−1) )|, β)−1 ,(20)

where γ , and β are a scaling factor and a thresholding value,
respectively. Compared to proposed three methods in Sec. 4.1,
Sec. 4.2 and Sec. 4.3, L1-based robust method can take care about
sparse outliers effectively.

5

S IGNAL - TO -Q UANTIZATION -N OISE RATIO ANAL -YSISOne of the important benefits of our method is its compatibility
to the sensor’s auto-exposure function that makes non-uniform
scaling of observations. With auto-exposure, SQNR of observations is effectively increased by avoiding over-/under-exposures.
As a result, the surface normal estimates are less suffered fromquantization noise, and thus, a greater accuracy can be obtained.
Based on the previous study of quantization noise [24], SQNR iswritten as
CµQµQµsignal∝ CR ==,(21)
SQN R =noiseR
(Vh − Vl )Q
where µ is the expectation of the signal, Q is the number ofquantization levels, and C is a scaling factor representing theamount of exposure. Also, R = Vh − Vl , where Vl and Vh
are the minimum and maximum scene irradiance. Thus, R andµ are both the functions of exposure time. From Eq. (21), we canobserve that SQNR without saturation is dependent of the numberof quantization levels Q; thus, better exposed signals produce ahigher SQNR.
When the signals are over-exposed, the SQNR expression
becomes more complicated due to saturation, as

SQN R =

signalCo µ − α∝ (λ−C V )
,
o l
noise
+α

Let us assume that not all signals are saturated. Then, thecondition that the well-exposed case has a greater SQNR than theover-exposed case is following:

Co µ − α
Co Qµ − QαQµ=≥ (λ−C V ).
ol(Vh − Vl )λ − Co Vl + Qα
+α

(23)

Q

{E∗ , B∗ } = argminkM − ELB> k1 .

B(t+1) = argmin W(t)

5

(22)

Above can be simplified by some algebraic operations into:

Qµ
Qα
≥
.
(Vh − Vl )(Co Vh − λ) − Qα(24)

The condition to satisfy Eq. (23) with respect to Q isQ≤

(Co Vh − λ)(Co Vh − λ) (Vh − Vl )
−
,Q >
,
α
µ
α

(25)

where (Co Vh − λ) is the maximum error. Mathematically, overexposed case can produce higher SQNR than well-exposed case.
However, in general situations, SQNR of well-exposed case isbetter than over-exposed case because the number of quantizationlevels Q is usually larger enough than maximum error (Co Vh −λ)over expectation error α. Therefore, well-exposed signals havehigher SQNR than over- or under-exposure cases in terms ofquantization if the number of quantization levels is sufficient.
Then, our methods are beneficial with auto-exposure to increase
SQNR since it can handle non-uniformity caused by auto-exposure
effectively.
If there are quantization noise in the images, the observation
matrix M becomes

M = M∗ + ζ = ELB> + ζ,(26)

where M∗ and ζ are the ideal observation and quantization noise,respectively. Using the noisy input in Eq. (26) , the objectivefunction in Eq. (7) becomes{E∗ , B∗ } = argminkζk2F , s.t. ζ = M − ELB> .

(27)

E,B

Therefore, in the cases of high SQNR data, we can computesurface normal and intensities by optimizing Eq. (27) withoutbiases since ζ is close to zero (M ≈ M∗ ). However, in low
SQNR inputs, minimizing Eq. (27) can produce biased results
because ζ is not small anymore (M 6= M∗ ). As a result, autoexposure can help to estimate surface normal by increasing SQNR
of images, and our method is suitable for dealing with the exposure
variations.

6

L IGHT INTENSITY CALIBRATION ANALYSIS

One may think that light intensity calibration is an easy task, but
it actually requires both careful control over the environment andexplicit knowledge about the reflectance of a calibration target. To
show this, we perform light intensity calibration using a diffuse
sphere1 . Assuming a Lambertian reflectance model and known
surface normal N, the scaled light matrix S can be estimatedfrom a set of measurements M asS∗ = argmin πΩc (M) − πΩc (SN> )
S

2
F

,

(28)

Q

where λ, α, and Co are saturation threshold, expectation of error
within saturation sub-interval, and scaling factor of the overexposure case, respectively. Here, Co Vh is replaced by λ due tothe saturation.

where Ω denotes the locations of shadowed entries in the observation M, and πΩc represents an operator that extracts entries
1. Due to the presence of saturations, a chrome sphere with specular
highlights is not a proper calibration object for this task.

IEEE TRANSACTIONS ON PATTERN RECOGNITION AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MONTH YEAR

6

Direction-1

Direction-2

Direction-3

Direction-4(a) 1.0(b) 0.948(c) 0.939(d) 0.988

Fig. 2: Light intensity calibration. A diffuse sphere is illuminated under different light directions by moving an identical light source.
The red point indicates the lighting direction, and a blue circle is a circle fitting to the image of a sphere. The numbers under photographs
are the estimated light source intensities, that are relative to that of Direction 1.
that are not shadowed (Ωc ). Since S = EL, with known light
directions L, we can determine the light intensities by

E = S∗ L† .

(29)

We recorded images of a diffuse sphere by changing the light
directions of an identical light source with retaining its distance tothe target object approximately the same. The camera response
function is linear and uncompressed RAW images are used.
Exposure times are set constant with making sure that there is nounder- or over-exposures. In addition, to neglect the perspective
effect, a camera is placed far enough from the target object so that
we can assume an orthographic projection model. Fig 2 showssome of the recorded images, and the light intensity matrix E isobtained by Eqs. (28) and (29).
As summarized in the numbers in Fig. 2, the estimated lightintensities have variations while they are supposed to be uniform
under this setting. The variations may be caused due to that (1)although the sphere is carefully selected, it still deviates fromthe Lambertian assumption, and (2) the assumed surface normal
directions may be different from the truth due to errors of circle
fitting. As such, even with a careful procedure, the light source
intensity calibration is not a straightforward task. And in our
setting, it had a non-negligible spread of estimated intensities(maximum 0.052 when the intensities are normalized to one,corresponding to 5% error). Therefore, it is needed to directlymodel the variations of light intensities in the photometric stereo
formulation.

7

E XPERIMENTS

We evaluate the proposed methods, linear joint estimation,
factorization based, alternating minimization (AM), and robust alternating minimization (Robust-AM) methods, using synthetic (Sec. 7.1), benchmark (Sec. 7.2), and real-world (Sec. 7.3)
scenes in the setting of non-uniform intensities and exposures.
Although none of the previous techniques are designed for the
non-uniform intensity setting, as previous methods to compare,
we use standard Frobenius-norm minimization [1], robust L1norm minimization used as a baseline method in [25], and thestate-of-the-art photometric stereo algorithm based on constrained
bivariate regression (CBR) [25].
7.1

Synthetic data

We first test our methods using synthetic examples that are
textured and rendered with a Lambertian reflectance model withshadows. For qualitative and quantitative comparisons, we analyze
the effects of non-uniform light intensities and auto-exposure.
In addition, we demonstrate how well previous and proposed
methods handle non-Lambertian components.
Non-uniform light intensities: We first test the setting of nonuniform light intensities. The scenes are rendered under 20 varying
light directions with their intensity variance 0.05. The qualitativevisualization of surface normal estimates and error maps are
summarized in Fig. 3 with comparison to other previous methods,i.e., Frobenius-norm, L1-norm, CBR. Our methods, namely, linear
joint, Factorization, and AM methods correspond to the ones
described in Secs. 4.1, 4.2, and 4.3, respectively. Our proposed
methods produce results close to the ground truth compared toother techniques that do not explicitly consider the non-uniform
light intensities. The quantitative results are reported under eacherror map. The superior performance is consistently observed
under varying numbers of images and light intensity variances
as shown in Fig. 4.
Auto-exposure case: Auto-exposure allows us to obtain measurements with a higher Signal-to-Quantization-Noise ratio (SQNR).
To assess the benefit of auto-exposure in photometric stereo
and effectiveness of our methods in this setting, we render
two datasets; one with auto-exposure and the other with fixedexposure. In the auto-exposure dataset, the sensor irradiances are
stretched to properly include the most of dynamic range before
quantization. For the fixed-exposure dataset, sensor irradiances arequantized without stretching. From the two types of dataset, weapply the same set of photometric stereo methods for performance
evaluation. The results are summarized in Table. 1. While thefixed-exposure setting suffers from a low SQNR (which leads tolower accuracy of surface normal estimates), the auto-exposure
retains a higher SQNR. And with our methods, this setting
is properly handled and accurate surface normal estimates are
obtained.
Non-Lambertian cases: Since our proposed methods, especially,
linear joint estimation and factorization based method are generalsolutions for ideal Lambertian objects, it is difficult to deal
with non-Lambertian components. In order to see how well ourfour methods cope with non-Lambertian components, we perform
experiments on non-Lambertian examples under non-uniform lightintensities. To render non-Lambertian examples, we adopt CookTorrance model [17]. Similar to Lambertian cases, we produce 20
images which are rendered varying light directions and intensities
as shown Fig. 5. Input images have a lot of non-Lambertian
regions and saturated pixels. Fig. 5 shows qualitative and quan-

74.237712.816

4.1685e-06 0.0016186 1.4663e-06 1.1618e-06

6.07924.221612.7984.4372e-06 0.0016186 1.6798e-06 1.1742e-06

5.768
4.2379Image Frobenius-norm L1-norm

11.153CBR

0.02036 0.00026724 1.7061e-06 9.8699e-07
Linear joint Factorization
AM
Robust-AM

Error map

6.0792Error map

CaesarTextured sphere

Error map

SphereIEEE TRANSACTIONS ON PATTERN RECOGNITION AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MONTH YEAR

Fig. 3: (Lambertian cases) Photometric stereo experiment under non-uniform light intensities. The scenes are rendered under 20 distinct
light directions with their intensity variance 0.05. Our methods (linear joint, factorization, AM and Robust-AM) effectively handle thecondition of non-uniform light intensities. Error maps are scaled by 4 times. The numbers indicate the mean angular errors in degree.

titative comparisons with other previous methods. Linear joint
and factorization based methods are severely broken down while
previous method cannot deal with non-uniform light intensities.
Compared to other methods, AM and Robust-AM methods effectively handle non-uniform light intensities and non-Lambertiancomponents.
7.2

Benchmark data

Shi et al. [26] provide photometric stereo dataset which contains
10 sets of input images, calibrated light directions, intensities
for each image as well as ground truth surface normals. Unlike
dataset in Sec. 7.1, input images are not synthetic, and also containnoise and outliers. Dataset covers various types of surface suchas complicated BRDF and spatially-varying materials. Since theirimages are captured under varying light intensity condition, it isappropriate to experiment our proposed methods on their dataset.
Table. 2 shows angular errors of each method for each example.
In the most of examples, AM and robust-AM methods produce
smaller errors than previous methods which do not consider nonuniform light intensities and sensor exposures. However linear
joint method is vulnerable to outliers and easily broken down.

Interestingly, for complicated cases (POT2, COW, HARVEST),
our methods are not much different from conventional methods
or rather are not good. Surface materials of those examples
are spatially-varying or complicated, thus consideration for nonLambertian and outliers is more critical factor than non-uniform
intensities and sensor exposures. Because our methods are solutions to the basic Lambertian model, considering non-uniform
intensity to an example such as the Harvest, which includes aconsiderable number of outliers, can have a negative impact on the
results. In most cases, however, handling non-uniform intensities
can be made for non-Lambertian objects with an simple method
such as L1-norm minimization Sec. 4.4.
7.3

Real data

We design three different settings for the real-world experiment;(A) non-uniform light source intensities across images, (B) recording with auto-exposures under identical light intensities (by moving the same light source), and (C) use of an uncontrolled mobile
phone camera for imaging where auto-exposure is turned on undervarying light source intensities. For all real-world examples, weuse a shiny sphere to calibrate the light directions. To suppress

IEEE TRANSACTIONS ON PATTERN RECOGNITION AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MONTH YEAR(a) Sphere
40

25
20
15

Angular error

10

30
25
20
15
10

5

5

0
0.05

0
0.05

0.055

0.06

0.065

0.07

0.075

0.08

0.055

0.06

0.065

0.07

0.075

20
15
10
5
0
15

20

25

30

10

35

0.055

0.06

0.065

0.07

0.075

0.08

40

30
25
20
15

Frobenius
L1
CBR
Linear joint
Factorization
AM
Robust-AM

35

10

30
25
20
15
10
5
0

0
10

15

Variance of light intensities(f) Caesar

5

5

20

0
0.05

0.08

Frobenius
L1
CBR
Linear joint
Factorization
AM
Robust-AM

35

Angular error

25

25

5

40

Frobenius
L1
CBR
Linear joint
Factorization
AM
Robust-AM

30

30

Variance of light intensities(e) Textured Sphere

40
35

Frobenius
L1
CBR
Linear joint
Factorization
AM
Robust-AM

35

Angular error

Angular error

30

Frobenius
L1
CBR
Linear joint
Factorization
AM
Robust-AM

35

Angular error

Frobenius
L1
CBR
Linear joint
Factorization
AM
Robust-AM

40

Variance of light intensities(d) Sphere
Angular error(c) Caesar(b) Textured Sphere

40
35

8

5

10

Number of images

15

20

25

30

35

5

10

15

20

25

30

35

Number of images

Number of images

Fig. 4: Variations of mean angular errors of surface normal estimates over variance of light intensities (top row) and the number ofimages (bottom row) for the three datasets. (a,d) Sphere, (b,e) Textured Sphere, (c,f) Caesar. Our methods consistently yield favorable
results across these variations.

SphereSQNR
Frobenius
L1
CBR [25]
Linear joint
Factorization
AM
Robust-AM

Fixed
48.3951.347
1.505217.2929.2883
2.699
2.19282.2852

TexturedAuto
126.4724.86524.355
14.4640.80469
0.71346
0.15501
0.15501

Fixed
42.992
1.92962.135
19.59913.3514.3541
3.04713.1601

Auto
126.6524.98924.48114.7520.804130.73457
0.16254
0.16254

CaesarFixed
41.6762.62842.842720.878
11.6675.8064
3.4081
3.5626Auto
128.7
24.43421.85321.5142.48142.36
0.416890.41689

TABLE 1: Comparison under auto-exposure (Auto) and fixed-exposure (Fixed) settings. SQNR and the mean angular errors of surface
normal estimates in degree are shown.
other disturbing factors, our experiments are carried out in a darkroom.
Non-uniform light source intensities: To record images under
different light intensities and directions, we use controllable lightsources whose brightnesses can be manually controlled by thegain of power supply. The camera setting, such as shutter speed
and aperture, are all fixed in this experiment, and a linear sensor
response is used. In this experimental setting, we record 20 images
for each static scene. The results are summarized in Fig. 6, in
which the estimated surface normal and their 3D reconstruction
using [27] are presented. As shown in the figures, our methods
properly handle the varying light source intensities compared to
Frobenius-norm, L1-norm and CBR methods, with which severe
distortions are observed in their reconstructed surfaces.
Auto-exposure: When auto-exposure is used, the shutter speed
and/or aperture size of a camera is automatically adjusted to record

well-exposed images according to the amount of incoming light.
While it increases SQNR, it results in the non-uniform intensity
setting.
For this experiment, we record 20 images of each static scene
with auto-exposure. Fig 7 shows the comparative result. As shownin the figure, our methods consistently yield higher quality outputs
than the other methods because our method explicitly accounts forthe non-uniform exposures.
Mobile phone cameras: Our method is suitable for uncontrollable
cameras like many of mobile phone cameras, where we cannot
turn off the auto-exposure setting. With such cameras, recorded
images are in the condition of non-uniform exposures acrossimages. From recorded images from a mobile phone camera,
we linearize the intensity observations using the method of [28]as preprocessing. Fig 8 shows the surface normal estimates andtheir 3D reconstructions. While the 3D reconstructions of conventional methods are severely deformed, our methods show better94.324412.71

17.16332. 083

1.56150.48403

6.533

4.381612.71717.604

38.63

2.155

0.74433

6.438
4.4077
Image Frobenius-norm L1-norm

11.009CBR

5.0061AM

0.5331
Robust-AM

Error map

6.4467

Error map

CaesarTextured sphere

Error map

SphereIEEE TRANSACTIONS ON PATTERN RECOGNITION AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MONTH YEAR

16.51832.677
Linear joint Factorization

Fig. 5: (Non-Lambertian cases) Photometric stereo experiment under non-uniform light intensities. The scenes are rendered under 20
distinct light directions with their intensity variance 0.05. Our methods (linear joint, factorization, AM and Robust-AM) effectively
handle the condition of non-uniform light intensities. Error maps are scaled by 4 times. The numbers indicate the mean angular errors
in degree.
reconstructions in general. The linear joint estimation method
suffered from the outliers in this case, but that is not observedin factorization based and AM methods.

7.4

Difference from spatially-varying light case

Spatially-varying light intensities can be caused by near-light or
non-isotropic light source cases, and there are a few approaches
to dealing with this [29], [30], [31]. As stated, our setting isdifferent from the near-light setting, and we primarily consider
the situation where the directional light intensities are different
across images. The setting is not a subset of the spatially-varying
light case, and to verify the difference, we show the results ofapplying our method and the NLPS method [29] to both spatiallyvarying (near-light) and non-uniform light (our setting) cases. The
results are summarized in Table. 3. This result is rather natural inthat the near-light setting is better handled by NLPS [29], while
the non-uniform light case (our setting) cannot be dealt well withthe method based on the near-light assumption, because these are
two distinct settings.

8

C ONCLUSION

This paper described photometric stereo methods that can handle
the non-uniform light source intensities and exposures acrossimages. We showed the effect of varying light intensity conditions
in photometric stereo that is relevant in practical settings. We
then developed solution methods that explicitly account for the
non-uniform light intensities and exposures; namely, linear joint
estimation, factorization based, alternating minimization, robust
alternating minimization methods. The linear joint estimationand factorization based methods are simple and easy to implement, they occasionally suffer from numerical instability due toun-modelled observations. The alternating minimization method
showed a greater robustness over these techniques, while retaining
the efficiency in computation. The robust-AM methods effectively
handle sparse outliers which cannot be detected by linear joint
estimation, factorization based, and AM methods. They are all
effective in the non-uniform intensities setting compared to methods that neglect the effect of the setting. We further illustrate thatour proposed methods can benefit from auto-exposure, with whichmeasurements with a greater SQNR can be obtained. Our experiments on synthetic and real-world examples show the importanceIEEE TRANSACTIONS ON PATTERN RECOGNITION AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MONTH YEAR

BALL
CAT
POT1
BEAR
POT2
BUDDA
GOBLET
READINGCOW
HARVEST

Frobenius
16.7
17.34
18.34420.86321.562
20.94623.29824.87726.97633.089L1
19.10719.42719.30822.04521.48122.85223.64324.43427.34733.424

CBR [25]
18.61818.85618.96823.01
20.83721.48523.29924.90625.94231.831

Linear joint
30.05718.56614.75226.02638.15834.40847.50955.75947.61
48.697

Factorization
4.6221
8.94889.63669.5641
20.16915.22923.42820.20728.5
45.017

10

AM
3.746
8.8481
9.622
9.263819.652
15.11322.56318.63928.30741.224

Robust-AM
2.77668.04829.46548.071719.58413.37818.18414.18526.89633.347

Budda

Doll

Budda

Doll

TABLE 2: Comparison under auto-exposure (Auto) and fixed-exposure (Fixed) settings. SQNR and the mean angular errors of surface
normal estimates in degree are shown.

Frobenius-norm

L1-norm

CBR

Linear joint

Factorization

AM

Robust-AM

Fig. 6: Result of varying light source intensities case. From left to right, one of input images, results from Frobenius-norm, L1-norm,
CBR [25], linear joint estimation, factorization, alternating minimization (AM) and Robust-AM methods are shown.

of properly handling varying light intensities and exposures.

R EFERENCES
[1]

[2]

[3]

[4]

R. J. Woodham, “Photometric method for determining surface orientationfrom multiple images,” Optical Engineering, vol. 19, no. 1, pp. 139–144,
January 1980.
T. Wu, K. Tang, C. Tang, and T. Wong, “Dense photometric stereo: A
markov random field approach,” IEEE Trans. Pattern Anal. Mach. Intell.
(TPAMI), vol. 28, no. 11, pp. 1830–1846, 2006.
F. Verbiest and L. Van Gool, “Photometric stereo with coherent outlier
handling and confidence estimation,” in Proc. of Computer Vision and
Pattern Recognition (CVPR), 2008.
L. Wu, A. Ganesh, B. Shi, Y. Matsushita, Y. Wang, and Y. Ma, “Robust
photometric stereo via low-rank matrix completion and recovery,” inProc. of Asian Conf. on Computer Vision (ACCV), 2010.

[5]

T.-H. Oh, H. Kim, Y.-W. Tai, J.-C. Bazin, and I. S. Kweon, “Partial sum
minimization of singular values in rpca for low-level vision,” in Proc. ofInt’l Conf. on Computer Vision (ICCV), 2013.
[6] Y. Zhang, C. Mu, H.-W. Kuo, and J. Wright, “Toward guaranteedillumination models for non-covex objects,” in Proc. of Int’l Conf. on
Computer Vision (ICCV), 2013.
[7] D. Cho, Y. Matsushita, Y.-W. Tai, and I. S. Kweon, “Photometric stereo
under non-uniform light intensitiesand exposures,” in ECCV, 2016.
[8] H. Hayakawa, “Photometric stereo under a light-source with arbitrary
motion,” Journal of the Optical Society of America (JOSA), vol. 11,
no. 11, pp. 3078–3089, 1994.
[9] A. L. Yuille and D. Snow, “Shape and albedo from multiple images
using integrability,” in Proc. of Computer Vision and Pattern Recognition(CVPR), 1997, pp. 158–164.
[10] P. N. Belhumeur, D. J. Kriegman, and A. L. Yuille, “The bas-relief
ambiguity,” Int’l Journal of Computer Vision (IJCV), vol. 35, no. 1, pp.
33–44, 1999.

11

Budda Head

Pottery

Budda Head

PotteryIEEE TRANSACTIONS ON PATTERN RECOGNITION AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MONTH YEAR

Frobenius-norm
L1-norm
CBR
Linear joint
Factorization
AM
Robust-AM
Fig. 7: Result of auto-exposure case. From left to right, one of input images, results from Frobenius-norm, L1-norm, CBR [25], linear
joint estimation, factorization, alternating minimization (AM) and Robust-AM methods are shown.

Frobeniusnorm

L1normCBR

Linearjoint

Factorization

AM

Robust-AM

Fig. 8: Result using a mobile phone camera. Top: Estimated surface normal, Bottom: 3D reconstruction. Our methods (linear joint
estimation, factorization, alternating minimization (AM) and Robust-AM methods) produce more faithful results than the conventional
methods.

[11] N. G. Alldrin, S. P. Mallick, and D. J. Kriegman, “Resolving thegeneralized bas-relief ambiguity by entropy minimization,” in Proc. ofComputer Vision and Pattern Recognition (CVPR), 2007.
[12] O. Drbohlav and M. Chantler, “Can two specular pixels calibrate photometric stereo?” in Proc. of Int’l Conf. on Computer Vision (ICCV), 2005,
pp. 1850–1857.
[13] K. Sunkavalli, T. Zickler, and H. Pfister, “Visibility subspaces: Uncalibrated photometric stereo with shadows,” in Proc. of European Conf. on
Computer Vision (ECCV), 2010, pp. 251–264.
[14] B. Shi, Y. Matsushita, Y. Wei, C. Xu, and P. Tan, “Self-calibrating
photometric stereo,” in Proc. of Computer Vision and Pattern Recognition(CVPR), 2010, pp. 1118–1125.
[15] F. Solomon and K. Ikeuchi, “Extracting the shape and roughness ofspecular lobe objects using four light photometric stereo,” IEEE Trans.
Pattern Anal. Mach. Intell. (TPAMI), vol. 18, no. 4, pp. 449–454, 1996.
[16] A. Georghiades, “Incorporating the torrance and sparrow model ofreflectance in uncalibrated photometric stereo,” in Proc. of Int’l Conf.
on Computer Vision (ICCV), 2003, pp. 816–823.
[17] S. Lin and S. W. Lee, “Estimation of diffuse and specular appearance,”in Proc. of Int’l Conf. on Computer Vision (ICCV), 1999, pp. 855–860.
[18] H. T. Y. Iwahori, R. J. Woodham and N. Ishii, “Neural network toreconstruct specular surface shape from its three shading images,” inProc. International Joint Conference on Neural Networks, 1993, pp.
1181–1184.
[19] T. Malzbender, B. Wilburn, D. Gelb, and B. Ambrisco, “Surface enhancement using real-time photometric stereo and reflectance transformation,” in Proceedings of the Eurographics Symposium on Rendering
Techniques, 2006, pp. 245–250.
[20] B. Shi, P. Tan, Y. Matsushita, and K. Ikeuchi, “Bi-polynomial modelingof low-frequency reflectances,” IEEE Trans. Pattern Anal. Mach. Intell.
(TPAMI), vol. 36, no. 6, pp. 1078–1091, 2014.
[21] S. Ikehata, D. P. Wipf, Y. Matsushita, and K. Aizawa, “PhotometricIEEE TRANSACTIONS ON PATTERN RECOGNITION AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MONTH YEAR

Condition
NLPS [29]AM

SphereSpatial varying
Non-uniform
0.002
5.212
3.469
1.7555e-06

Textured
Spatial varying
Non-uniform
0.002
5.212
3.443
2.3556e-06

12

CaesarSpatial varying
Non-uniform
0.184
5.342
4.066
2.0298e-06

TABLE 3: Comparison of methods under spatially-varying and non-uniform (ours) light settings. The NLPS method shows lower
accuracy for the non-uniform intensity case, while it is effective in the spatially-varying light setting. Our method works in acomplementary manner; it does not handle the spatially-varying light case, but is effective for non-uniform light and exposure case.

stereo using sparse bayesian regression for general diffuse surfaces,”
IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI), vol. 36, no. 9, pp.
1816–1831, 2014.
[22] J. Sylvester, “Sur lequations en matrices px = xq,” C. R. Acad. Sc. Paris,
vol. 99, no. 2, pp. 67–71,115–116, 1884.
[23] P. W. Holland and R. E. Welsch, “Robust regression using iteratively
reweighted least-squares,” Communications in Statistics: Theory andMethods, vol. 6, no. 9, pp. 813–827, 1977.
[24] B. Kamgar-Parsi and B. Kamgar-Parsi, “Evaluation of quantization error
in computer vision,” C. R. Acad. Sc. Paris, vol. 11, no. 9, pp. 929–940,
1989.
[25] S. Ikehata and K. Aizawa, “Photometric stereo using constrained bivariate regression for general isotropic surfaces,” in Proc. of Computer Vision
and Pattern Recognition (CVPR), 2014.
[26] B. Shi, Z. Wu, Z. Mo, D. Duan, S.-K. Yeung, and P. Tan, “A benchmark
dataset and evaluation for non-lambertian and uncalibrated photometricstereo,” in Proc. of Computer Vision and Pattern Recognition (CVPR),
2016.
[27] W. Xie, Y. Zhang, C. C. L. Wang, and R. C.-K. Chung, “Surface-fromgradients: An approach based on discrete geometry processing,” in Proc.
of Computer Vision and Pattern Recognition (CVPR), 2014.
[28] J.-Y. Lee, Y. Matsushita, B. Shi, I. S. Kweon, and K. Ikeuchi, “Radiometric calibration by rank minimization,” IEEE Trans. Pattern Anal. Mach.
Intell. (TPAMI), vol. 35, no. 1, pp. 144–156, Jan. 2013.
[29] T. Papadhimitri and P. Favaro, “Uncalibrated near-light photometricstereo,” in Proceedings of the British Machine Vision Conference.
BMVA Press, 2014.
[30] F. Sakaue and J. Sato, “A new approach of photometric stereo fromlinear image representation under close lighting,” in Proc. of Int’l Conf.
on Computer Vision Workshops (ICCV Workshops), 2011, pp. 759 – 766.
[31] W. Xie, C. Dai, and C. C. L. Wang, “Photometric stereo with near point
lighting: A solution by mesh deformation,” in Proc. of Computer Vision
and Pattern Recognition (CVPR), 2015.

Donghyeon Cho is a Ph.D. student in EE department of KAIST, South Korea. From Sept2014 to Feb 2015, he worked as a full-time student internship in the Microsoft Research Asia
(MSRA). He received the MS degree in Computer Science from Korea Advanced Institute of
Science and Technology (KAIST), Daejeon, Korea in 2014. He received the BS degree in information and communication engineering from
Inha University, Incheon, Korea, in 2012. His
research interests include image/video processing, computer vision and computational photography. He is a student
member of the IEEE.

Yasuyuki Matsushita received his B.S., M.S.
and Ph.D. degrees in EECS from the Universityof Tokyo in 1998, 2000, and 2003, respectively.
From April 2003 to March 2015, he was with
Visual Computing group at Microsoft Research
Asia. In April 2015, he joined Osaka University
as a professor. His research area includes computer vision, machine learning and optimization.
He is on the editorial board of IEEE Transactions
on Pattern Analysis and Machine Intel- ligence(TPAMI), International Journal of Computer Vision (IJCV), IPSJ Journal of Computer Vision and Applications (CVA),The Visual Computer Journal, and Encyclopedia of Computer Vision.
He served/is serving as a Program Co-Chair of PSIVT 2010, 3DIMPVT
2011, ACCV 2012, ICCV 2017, and a General Co-Chair for ACCV 2014.
He is a senior member of IEEE.

Yu-Wing Tai is a research director of youtu research lab of Tencent SNG. He was a principle
research scientist of SenseTime Group Limited
from September 2015 to December 2016. He
was an associate professor working at the Korea Advanced Institute of Science and Technology (KAIST) from July 2009 to August 2015.
He received his Ph.D. degree in Computer Science from the National University of Singapore(NUS) in 2009. From Sept 2007 to June 2008,
he worked as a full-time student internship in
the Microsoft Research Asia (MSRA). He was awarded the Microsoft
Research Asia Fellowship in 2007, and the KAIST 40th Anniversary
Academic Award for Excellent Professor in 2011 respectively. He received a M.Phil and B.Eng (First Class Honors) degree in Computer
Science from the Hong Kong University of Science and Technology(HKUST) in 2005 and 2003 respectively. He is a senior member of theIEEE.

Inso Kweon is an professor in EE departmentof KAIST, South Korea. He received the BS
and MS degrees in mechanical design and production engineering from Seoul National University, Seoul, Korea, in 1981 and 1983, respectively, and the PhD degree in robotics from the
Robotics Institute, Carnegie Mellon University,
Pittsburgh, Pennsylvania, in 1990. He worked
for the Toshiba R&D Center, Japan, and joined
the Department of Automation and Design Engineering, KAIST, Seoul, Korea, in 1992, where
he is now a professor with the Department of Electrical Engineering.
He is a recipient of the best student paper runner-up award at theIEEE Conference on Computer Vision and Pattern Recognition (CVPR
09). His research interests are in camera and 3D sensor fusion, color
modeling and analysis, visual tracking, and visual SLAM. He was the
program co-chair for the Asian Conference on Computer Vision (ACCV07) and was the general chair for the ACCV 12. He is also on the editorial
board of the International Journal of Computer Vision. He is a memberof the IEEE and the KROS.