Photometric Stereo under Non-uniform Light Intensities
and Exposures
Donghyeon Cho1?
cdh12242@gmail.com
1

Yasuyuki Matsushita2

Yu-Wing Tai3

Inso Kweon1

yasumat@ist.osaka-u.ac.jp yuwing@gmail.com iskweon@kaist.ac.kr

KAIST

2

Osaka University

3

SenseTime Group Limited

Abstract. This paper studies the effects of non-uniform light intensities and sensor exposures across observed images in photometric stereo. While conventional
photometric stereo methods typically assume that light intensities are identicaland sensor exposure is constant across observed images taken under varying lightings, these assumptions easily break down in practical settings due to individual
light bulb’s characteristics and limited control over sensors. Our method explicitly models these non-uniformities and develops a method for accurately determining surface normal without affected by these factors. In addition, we showthat our method is advantageous for general photometric stereo settings, whereauto-exposure control is desirable. We compare our method with conventional
least-squares and robust photometric stereo methods, and the experimental resultshows superior accuracy of our method in this practical circumstance.
Keywords: Photometric stereo, shape estimation, unknown light intensity andexposure, surface normal

1

Introduction

Non-uniform light intensities and exposures across observed images are a practical andcommon circumstance in data acquisition for photometric stereo that uses multiple
images under distinct light directions. For example, different light bulbs with different intensity characteristics may be used for illuminating a scene. Even with identicallight bulbs, due to that scene radiance is determined by surface normal and light directions, auto-adjusted sensor exposure is desirable depending on the light directions
to avoid over-/under-exposures, which results in non-uniform exposures (equivalently,
non-uniform light intensities). Therefore, the capability of properly handling varying
and unknown light intensities and exposures across observed images is an importantfeature for making photometric stereo practical.
The setting can be regarded as a “semi-calibrated” photometric stereo, where thelight directions are known but their intensities are unknown. We argue that accurate
light intensity calibration is practically a hard task to perform due to that the light bulb’s
luminous efficiency varies over time and quantization error in the measurement evenwith high-dynamic range imaging. This paper provides a way to bypass the difficult
intensity calibration in photometric stereo.
?

Part of this work was done while the first author was an intern at Microsoft Research Asia.

2

Donghyeon Cho, Yasuyuki Matsushita, Yu-Wing Tai, Inso Kweon

Exposuretime

Exposuretime

Exposure
time

Same light brightness and exposure time

Different light brightness

Different exposure time

(a)

(b)

(c)

Fig. 1. (a) Conventional photometric stereo setting where constant light intensities and exposures
are used, (b)(c): Varying lighting intensity/exposure conditions. Estimated surface normal arebiased toward brighter light source or images captured with longer exposures with a conventional
solution method.

In the Lambertian image formation model, a measured intensity m is written as
mi,j = Ei ρj n>
j li ,(1)

where i and j are indices of light direction and pixel location, li , nj ∈ R3×1 are unit
vectors of light direction and surface normal, ρj ∈ R is a Lambertian diffuse albedo,
and Ei ∈ R is a light source intensity. In a matrix form for representing all pixels andlight directions at a time, it can be written as
M = ELN> P,(2)

where M ∈ Rf ×p is an observation matrix, E is an f × f diagonal light intensity
matrix, L ∈ Rf ×3 is a light direction matrix, N ∈ Rp×3 is a surface normal matrix,
P is a p × p diagonal diffuse albedo matrix, and f and p are the number of images
and pixels, respectively. Conventional photometric stereo [1] assumes that light source
intensities are identical across images, where the matrix E becomes a scaled identity
matrix (E = eI), and computes albedo-scaled surface normal B(= P> N) by
eB∗ > = L† M,(3)

up to a scale ambiguity e, where the superscript † indicates a generalized inverse whenf ≥ 3.
Clearly, when the light source intensities are non-uniform or camera exposures varyacross images, the assumption E = eI does not hold, but instead its diagonal elements
have individual scales. When this non-uniformity is present, the surface normal estimates by Eq. (2) naturally becomes biased by greater scales as illustrated in Fig. 1.
While there are recently various robust estimation techniques used for photometricstereo [2–6], because the effect of non-uniform E neither increases the rank of the observation matrix nor sparsifies outliers, robust techniques such as rank minimization or`0 -norm minimization techniques cannot resolve this issue. In the rest of the paper, wecollectively call this problem setting, non-uniform light intensities and exposures acrossPhotometric Stereo under Non-uniform Light Intensities and Exposures

3

images, a varying light intensity condition, because they are both considered intensity
scaling on individual images.
This paper considers a method to effectively deal with the non-uniform light intensities and exposures. The problem that we deal with in this paper is a bilinear problem
written as following.
Problem (Photometric stereo under varying light intensity condition). Given observations M and light directions L, estimate a diagonal light intensity matrix E andan albedo-scaled surface normal matrix B from the following relationship:
M = ELB> .

(4)

We first show that there exists a linear closed-form solution method, which simultaneously estimates scales of light intensities (or exposures) E and albedo-scaled surface
normal B. We call this method a linear joint estimation method. This method is straightforward to implement; however, inefficient in terms of computation time and memory
consumption. We then introduce a factorization based method for determining only surface normal B in Eq. (4) without being affected by E. It bypasses the estimation of E
using algebraic distance minimization (or, cross product minimization) by making the
problem independent of vector magnitudes. Finally, we show that this bilinear problem
can be efficiently solved by an alternating minimization technique that determines E
and B in each step. We discuss details and characteristics of each method later in thispaper.
We further show that our method is advantageous in improving signal-to-quantizationnoise ratio (SQNR) in comparison to a standard photometric stereo method when an
auto-exposure control is used, and as a result more accurate surface normal estimates
can be obtained. Experimental results show the effectiveness of the proposed method inpractical settings. In this paper, we assumes a directional light setting where radiancefrom a light source to a scene is constant except for shadowing, i.e., spatially varying
incident radiance within a scene is not assumed.

2

Related Works

Photometric stereo was first introduced by Woodham [1] in 1980’s for determining
surface normal from images taken under known and varying light directions with aLambertian reflectance assumption. After Woodham’s work, there have been various
techniques proposed for making photometric stereo more practical. Their main focuses
are to relax the assumptions of (1) calibrated light sources and (2) Lambertian image
formation model.
The first class of the methods, called uncalibrated photometric stereo, tries to eliminate the need for calibrating light directions. When the light directions are unknown,
it is understood that the solution can be obtained up to a 3 × 3 linear ambiguity [7].
If the integrability [8] of the surface is assumed, it has been shown that the linear ambiguity can be reduced to a generalized bas-relief (GBR) ambiguity [9], which only
has three parameters. To fully resolve these ambiguities, various types of external clues
have been used. For example, there are methods that use the entropy of albedo distributions [10], specular observations [11], shadows [12], and groups of color and intensity profiles [13]. Our problem setting has a similarity to the uncalibrated photometric4

Donghyeon Cho, Yasuyuki Matsushita, Yu-Wing Tai, Inso Kweon

stereo scenario in that we relax the assumptions of known light intensities and constantlight intensity across varying light directions. And there has not been uncalibrated PS
works that derive disambiguated solution without external assumptions such as albedo
entropy [10] and pixel profiles [13].
The second class of the methods tries to make photometric stereo applicable to nonLambertian scenes. There are methods that use more sophisticated reflectance models
than Lambertian reflectance model, such as the works that use Torrance-Sparrow [14,15], Cook-Torrance [16], Phong [17], Blinn-Phong [18]. More recently, Shi et al. [19]propose a bi-polynomial reflectance model that produces successful results for nonLambertian diffusive scenes.
There are approaches that use robust estimation techniques by treating non-Lambertianreflectances and shadows as outliers. In [2], the robustness against outliers is achievedby capturing hundreds of input images coupled with Markov Random Field (MRF) to
maintain neighborhood smoothness. Verbiest and Van Gool [3] use a confidence approach to reject outliers in input images of photometric stereo. Wu et al. [4] proposeda robust method based on low-rank matrix factorization. Oh et al. [5] introduced a partial sum of singular values for rank minimization, and showed good performance inphotometric stereo. Ikehata et al. [20] used a sparse Bayesian regression for effectively
neglecting sparse outliers (specularities and shadows). While these techniques are effective, they are built upon the assumption of constant light intensity, and cannot directly
address the issue of varying light intensities and exposures.

3

Photometric stereo under varying light intensity conditions

As discussed in Eq. (4), we are interested in determining albedo-scaled surface normal
B with unknown non-uniform scalings of light intensities or exposures E. In a leastsquares framework, the problem can be written as{E∗ , B∗ } = argminkM − ELB> k2F

(5)

E,B

given the observations M and light directions L.
We first present a linear estimation method that simultaneously estimates B and E
in Sec. 3.1. We then describe a factorization based method in Sec. 3.2, which bypasses
the estimation of unknown scalings E. Finally, we describe an efficient alternating minimization method in Sec. 3.3.
3.1

Linear joint estimation method

The original form M = ELB> can be re-written as E−1 M = LB> , because E isalways invertible as it is a positive diagonal matrix. Given known M and L, it can beviewed as a variant of a Sylvester equation [21]:E−1 M − LB> = 0.
By vectorizing unknown variables E−1 and B> , Eq. (6) can be written as>
diag(m1 )| · · · |diag(mp ) E−1 1 − (Ip ⊗ L)vec(B> ) = 0,

(6)

(7)

Photometric Stereo under Non-uniform Light Intensities and Exposures

5

where diag(·), vec(·) and ⊗ are diagonalization, vectorization, and Kronecker productoperators, respectively. Ip is a p × p identity matrix, and 1 indicates a vector whose
elements are all one. By concatenating matrices and vectors in Eq. (7), a homogeneous
equation can be obtained:h
> i vec(B> )= 0,(8)−Ip ⊗ L| diag(m1 )| · · · |diag(mp )−1{z} | E{z 1 }|Dywhere D ∈ Rpf ×(3p+f ) is a sparse design matrix and y ∈ R(3p+f )×1 is an unknown
vector. The homogeneous system always has a trivial solution y = 0. To have a unique(up to scale) non-trivial solution, the matrix D should have a one dimensional nullspace, i.e., when rank of D is (3p+f −1), a unique solution can be obtained via singularvalue decomposition (SVD). The minimum condition to have a unique solution up toscale is f ≥ 5 and p ≥ 3, or f = 4 and p ≥ 2. Unlike conventional photometric stereo,
increasing the number of light directions does not necessarily make the problem easierin this setting, because it also increases the unknowns about light intensities.
3.2

Factorization based method

Although the linear joint estimation method is simple to implement, it has practicallimitations in terms of its computational time and memory requirement when the sparse
matrix D is large; not only constructing D but also computing SVD of D. This limitation can be relaxed by dividing the observation matrix into small groups and deriving solutions for each group. However, this grouping should be performed carefully to
avoid the condition numbers of divided sub-matrices to be high. The condition number
increases when observations within each divided group are similar to each other, andas a result, the numerical error becomes greater. To avoid these issues, we develop afactorization based method described in this section.
Like solution methods of uncalibrated photometric stereo, light directions and surface normal can be solved directly via matrix factorization:
M = ŜB̂> ,(9)where Ŝ and B̂ are biased intensity-scaled light direction and albedo-scaled surface
normal, respectively. With an arbitrary 3 × 3 non-singular matrix H, Eq. (9) can be
re-written as
M = (ŜH)(H−1 B̂> ).

(10)

In our setting, since we know the light directions L, we can find an appropriate nonsingular matrix H for resolving the biases. Regardless of the effect of light intensities,
direction of ŜH should be the same with L. Thus, we can use this constraint, (ŜH) ×
L = 0 where × indicates a cross product, for determining H as



 h >
0 −li,3 ŝi li,2 ŝi  1 > 
= 0,
(11)
h2
li,3 ŝi 0 −li,1 ŝi
h3 >

6

Donghyeon Cho, Yasuyuki Matsushita, Yu-Wing Tai, Inso Kweon

where H = [h1 |h2 |h3 ], li,∗ and ŝi are the i-th row of L and Ŝ, respectively. The
solution of Eq. (11) is unique up to scale when there are more than 4 distinct light
directions. Using estimated Ĥ, we can compute unbiased albedo-scaled surface normal
H−1 B̂> . Interestingly, this factorization based method can naturally bypass the lightintensity estimation; thus, it is suitable for our setting. Compared to the linear joint
estimation method, the computational cost of the factorization based method is lower,
even without dividing observations M into small groups.
3.3

Alternating minimization method

While the previous two methods are effective in ideal settings, they are prone to large
errors due to un-modelled observations, such as shadows and pixel saturations. To avoid
this problem, we develop a robust method that is based on alternating minimization forsolving Eq. (5).
Our method computes albedo-scaled surface normal B(t) and non-uniform scalings(t)
E in an alternating manner using their intermediate estimates from the previous iteration. Using E(t) from the previous iteration and by fixing it, albedo-scaled surface
normal B(t+1) is updated by
B(t+1) = argmin M − E(t) LB>

2

.

(12)

F

B

The above problem is a linear problem with respect to B and can be solved efficiently.
Once matrix B(t+1) is determined, E(t+1) is then updated by solving
E(t+1) = argmin M − ELB(t+1)(t+1)

Since matrix E is diagonal, each element Ei
(t+1)

.

(13)

F

E

Ei

> 2

is simply determined by(t+1) >>)
j mi,j (li bj.
(t+1)(t+1) >
>
)(l>
)
i bj
j (li bj

P

=P

(14)

The initial scaling matrix E(0) is set to an identity matrix, and the convergence criteria
is defined by the magnitude of variation of matrix B, i.e., kB(t+1) − B(t) kF < , where is set to a small value (in our implementation,  = 1.0e-8).
If we consider E as weights, this alternating minimization is similar to iteratively
re-weighted least squares (IRLS) [22] except that weights are defined row-wise (eachimage has same weight). We show how the alternating method operates in the following.
Let us consider updating B(t+1) with fixing E(t) , then Eq. (12) becomesB(t+1) = argmin M − E(t) LB>
B

2= argmin M − E∗ LB> − Er LB >
B

(15)

F
2

,
F

Photometric Stereo under Non-uniform Light Intensities and Exposures

7

where E(t) = E∗ + Er , E∗ is the ground truth (that we do not know), and Er is theerror from t-th iteration. It shows that the smaller the scaling error Er is, the smaller
objective cost becomes. The elements of E(t) can also be written as
P
(t) >
>
j mi,j (li bj )(t)Ei = P
(t)
(t) >
>
>
j (li bj )(li bj )
(16)
P
P
∗>
r>
>
+ j mi,j l>i bj
j mi,j li bj=P > ∗
r >
∗
r > ,
>
j (li (bj + bj ) )(li (bj + bj ) )where b(t+1) = b∗ + br , b∗ is the ground truth, and br is the error from t-th iteration.
Since the denominator is fixed for all images, and the left-hand side of the numerator
is proportional to the ground truth scaling E∗ , the smaller the error br becomes, thebetter scaling elements E becomes. To summarize, if the current estimate of albedoscaled surface normal B(t) is better than the previous one, E(t) is better updated. In our
experiments, above improvements are always observed since updated E(1) becomescloser to the ground truth than E(0) . Then, B(t) and E(t) are alternately updated. The
minimum condition for obtaining a stable solution is experimentally found to be f ≥ 5
and p ≥ 3.

4

Signal-to-Quantization-Noise ratio analysis

One of the important benefits of our method is its compatibility to the sensor’s autoexposure function that makes non-uniform scaling of observations. With auto-exposure,
SQNR of observations is effectively increased by avoiding over-/under-exposures. As
a result, the surface normal estimates are less suffered from quantization noise, andthus, a greater accuracy can be obtained. Based on the previous study of quantizationnoise [23], SQNR is written as
SQNR =CµQµsignalQµ∝ CR =
=
,
noise
R
(V
h − Vl )Q(17)where µ is the expectation of the signal, Q is the number of quantization levels, andC is a scaling factor representing the amount of exposure. Also, R = Vh − Vl , whereVl and Vh are the minimum and maximum scene irradiance. Thus, R and µ are boththe functions of exposure time. From Eq. (17), we can observe that SQNR without
saturation is dependent of the number of quantization levels Q; thus, better exposed
signals produce higher SQNR.
When the signals are over-exposed, the SQNR expression becomes more complicated due to saturation, as
SQNR =

signalCo µ − α∝ (λ−C V )
,
o l
noise
+α

(18)

Q

where λ, α, and Co are saturation threshold, expectation of error within saturation subinterval, and scaling factor of the over-exposure case, respectively. Here, Co Vh is replaced by λ due to saturation.

8

Donghyeon Cho, Yasuyuki Matsushita, Yu-Wing Tai, Inso Kweon

Let us assume that not all signals are saturated. Then, the condition that the wellexposed case has a greater SQNR than the over-exposed case is following:
Co Qµ − Qα
Qµ
Co µ − α=≥ (λ−C V ).
o l(Vh − Vl )λ− Co Vl + Qα
+α

(19)

Q

Above can be simplified by some algebraic operations into:
Qα
Qµ
≥
.
(Vh − Vl )(Co Vh − λ) − Qα(20)

The condition to satisfy Eq. (19) with respect to Q isQ≤

(Co Vh − λ) (Vh − Vl )(Co Vh − λ)
−
,Q >
,
α
µ
α

(21)

where (Co Vh − λ) is the maximum error. Mathematically, over-exposed case can produce a higher SQNR than the well-exposed case. However, in general situations, SQNR
of well-exposed case is better than over-exposed case because the number of quantization levels Q is usually larger enough than maximum error (Co Vh − λ) over expectation error α. Therefore, well-exposed signals have higher SQNR than over- or
under-exposure cases in terms of quantization if the number of quantization levels is
sufficient. Our method is beneficial with auto-exposure to increase SQNR since it caneffectively handle non-uniformity caused by auto-exposure.
If there are quantization noise in the images, the observation matrix M becomes
M = M∗ + ζ = ELB> + ζ,

(22)

where M∗ and ζ are the ideal observation and quantization noise, respectively. Using
the noisy input in Eq. (22) , the objective function in Eq. (5) becomes{E∗ , B∗ } = argminkζk2F , s.t. ζ = M − ELB> .

(23)

E,B

Therefore, in the cases of high SQNR data, we can compute surface normal and intensities by optimizing Eq. (23) without biases since ζ is close to zero (M ≈ M∗ ). However,in low SQNR inputs, minimizing Eq. (23) can produce biased results because ζ is not
small anymore (M 6= M∗ ). As a result, auto-exposure can help to estimate surface
normal by increasing SQNR of images, and our method is suitable for dealing with theexposure variations.

5Light intensity calibration analysis

One may consider that light intensity calibration is an easy task, but it actually requires
both careful control over the environment and explicit knowledge about the reflectanceof a calibration target. To show this, we perform light intensity calibration using a diffuse sphere1 . Assuming a Lambertian reflectance model and known surface normal N,
1

Due to the presence of saturations, a chrome sphere with specular highlights is not a proper
calibration object for this task.

Photometric Stereo under Non-uniform Light Intensities and Exposures
Direction-1

Direction-2

Direction-3

Direction-4(a) 1.0(b) 0.948(c) 0.939(d) 0.988

9Fig. 2. Light intensity calibration. A diffuse sphere is illuminated under different light directions
by moving an identical light source. The red point indicates the lighting direction, and a blue
circle is a circle fitting to the image of a sphere. The numbers under photographs are the estimatedlight source intensities, that are relative to that of Direction 1.

the scaled light matrix S can be estimated from a set of measurements M as
2

S∗ = argmin πΩc (M) − πΩc (SN> ) F ,S(24)

where Ω denotes the locations of shadowed entries in the observation M, and πΩ crepresents an operator that extracts entries that are not shadowed (Ω c ). Since S = EL,with known light directions L, we can determine the light intensities by
E = S∗ L† .

(25)

We recorded images of a diffuse sphere by changing the light directions of an identical light source with retaining its distance to the target object approximately the same.
The camera response function is linear and uncompressed RAW images are used. Exposure times are kept constant with making sure that there is no under- or over-exposures.
In addition, to neglect the perspective effect, a camera is placed far enough from thetarget object so that we can assume an orthographic projection model. Figure 2 showssome of the recorded images, and the light intensity matrix E is obtained by Eqs. (24)and (25).
As summarized in the numbers in Fig. 2, the estimated light intensities have variations while they are supposed to be uniform under this setting. The variations may becaused due to that (1) although the sphere is carefully selected, it still deviates from theLambertian assumption, and (2) the assumed surface normal directions may be different
from the truth due to errors of circle fitting. As such, even with a careful procedure, thelight source intensity calibration is not a straightforward task. And in our setting, it hada non-negligible spread of estimated intensities (maximum 0.052 when the intensities
are normalized to one, corresponding to 5% error). Therefore, it is needed to directlymodel the variations of light intensities in the photometric stereo formulation.

6

Experiments

We evaluate the proposed methods, linear joint estimation, factorization based, and alternating minimization (AM) methods, using synthetic (Sec. 6.1) and real-world (Sec. 6.2)

Donghyeon Cho, Yasuyuki Matsushita, Yu-Wing Tai, Inso Kweon

4.237712.816

4.1685e-06 0.0016186 1.4663e-06

6.07924.221612.7984.4372e-06 0.0016186 1.6798e-06

5.768
4.2379Image Frobenius-norm L1-norm

11.153CBR

0.02036 0.00026724 1.7061e-06
Linear joint Factorization AM

Error map

6.0792Error map

CaesarTextured sphere

Error map

Sphere10Fig. 3. Photometric stereo experiment under non-uniform light intensities. The scenes are rendered under 20 distinct light directions with their intensity variance 0.05. Our methods (linear
joint, factorization and AM) effectively handle the condition of non-uniform light intensities.
Error maps are scaled by 4. The numbers indicate the mean angular errors in degree.

scenes in the setting of non-uniform intensities and exposures. Although none of theprevious techniques are designed for the non-uniform intensity setting, as previous
methods to compare, we use standard Frobenius-norm minimization [1], robust L1norm minimization used as a baseline method in [24], and the state-of-the-art photometric stereo method based on constrained bivariate regression (CBR) [24].
6.1

Synthetic data

We first test our methods using synthetic examples that are textured and rendered witha Lambertian reflectance model with shadows. For qualitative and quantitative comparisons, we analyze the effects of non-uniform light intensities and auto-exposure.
Non-uniform light intensities: We first test the setting of non-uniform light intensities.
The scenes are rendered under 20 varying light directions with their intensity variance0.05. The qualitative visualization of surface normal estimates and error maps are summarized in Fig. 3 with comparison to other previous methods, i.e., Frobenius-norm,Photometric Stereo under Non-uniform Light Intensities and Exposures(b) Textured Sphere(a) Sphere2030

Angular error

15

10

25

20

15

10

5

5

0
0.05

0.055

0.06

0.065

0.07

0.075

Variance of light intensities(d) Sphere

0.06

0.065

0.07

0.075

30

20

10

10

15

20

25

30

Number of images

10

35

0.06

0.065

0.07

0.075

0.08

70

50

40

30

20

0

0.055

Variance of light intensities(f) Caesar
Frobenius
L1
CBR
Linear joint
Factorization
AM

60

50

40

30

20

10

10

5

15

0
0.05

0.08

Frobenius
L1
CBR
Linear joint
Factorization
AM

60

Angular error

Angular error

40

0

0.055

70

Frobenius
L1
CBR
Linear joint
Factorization
AM

50

20

Variance of light intensities(e) Textured Sphere

70

60

25

5

0
0.05

0.08

Frobenius
L1
CBR
Linear joint
Factorization
AM

30

Angular error

Angular error

25

35

Frobenius
L1
CBR
Linear joint
Factorization
AM

Angular error

Frobenius
L1
CBR
Linear joint
FactorizationAM30(c) Caesar

35

35

11

5

10

15

20

25

30

35

Number of images

0

5

10

15

20

25

30

35

Number of images

Fig. 4. Variations of mean angular errors of surface normal estimates over variance of light intensities (top row) and the number of images (bottom row) for the three datasets. (a,d) Sphere,(b,e) Textured Sphere, (c,f) Caesar. Our methods consistently yield favorable results across thesevariations.

SQNR
Frobenius
L1
CBR [24]
Linear joint
Factorization
AM

Sphere
Fixed
Auto
42.52
129.8
2.383
24.35
2.507
23.85
21.47
14.54
20.49
0.886
6.569
0.715
3.970
0.256

Textured
Fixed
Auto
33.68
135.6
27.93
23.61
28.05
23.41
47.15
16.53
59.63
1.561
42.12
1.381
29.74
0.565

Caesar
Fixed
Auto
33.56
133.3
8.415
23.86
8.320
21.03
33.02
21.49
43.72
1.725
26.16
1.504
9.684
0.564

Table 1. Comparison under auto-exposure (Auto) and fixed-exposure (Fixed) settings. SQNR andthe mean angular errors of surface normal estimates in degree are shown.

L1-norm, CBR. Our methods, namely, linear joint, Factorization, and AM methods correspond to the ones described in Secs. 3.1, 3.2, and 3.3, respectively. Our proposed
methods produce results close to the ground truth compared to other techniques that do
not explicitly consider the non-uniform light intensities. The quantitative results are reported under each error map. The superior performance is consistently observed undervarying numbers of images and light intensity variances as shown in Fig. 4.
Auto-exposure case: Auto-exposure allows us to obtain measurements with a higher
Signal-to-Quantization-Noise ratio (SQNR). To assess the benefit of auto-exposure in
photometric stereo and effectiveness of our methods in this setting, we render two
datasets; one with auto-exposure and the other with fixed-exposure. In the auto-exposure
dataset, the sensor irradiances are stretched to properly include the most of dynamic
range before quantization. For the fixed-exposure dataset, sensor irradiances are quan-Donghyeon Cho, Yasuyuki Matsushita, Yu-Wing Tai, Inso Kweon

Budda

Doll

Budda

Doll

12

Frobenius-norm L1-norm

CBR

Linear joint Factorization

AM

Fig. 5. Result of varying light source intensities case. From left to right, one of input images,
results from Frobenius-norm, L1-norm, CBR [24], linear joint estimation, factorization and alternating minimization (AM) methods are shown.

tized without stretching. From the two types of dataset, we apply the same set of photometric stereo methods for performance evaluation. The results are summarized in Table 1. While the fixed-exposure setting suffers from a low SQNR (which leads to lower
accuracy of surface normal estimates), the auto-exposure retains a higher SQNR. And
with our methods, this setting is properly handled and accurate surface normal estimates
are obtained.
6.2

Real data

We design three different settings for the real-world experiment; (A) non-uniform lightsource intensities across images, (B) with auto-exposures under identical light intensities (by moving the same light source), and (C) use of an uncontrolled mobile phone
camera for imaging where auto-exposure is turned on under varying light source intensities. For all real-world examples, we use a shiny sphere to calibrate the light directions.
To suppress other un-modelled factors, our experiments are carried out in a dark room.
Non-uniform light source intensities: To record images under different light intensities and directions, we use controllable light sources whose brightnesses can be manually controlled by the gain of power supply. The camera setting, such as shutter speed
and aperture, are all fixed in this experiment, and a linear sensor response is used. In

13

Budda Head Pottery

Budda Head

Pottery

Photometric Stereo under Non-uniform Light Intensities and Exposures

Frobenius-norm L1-norm

CBR

Linear joint

Factorization

AM

Fig. 6. Result of auto-exposure case. From left to right, one of input images, results from
Frobenius-norm, L1-norm, CBR [24], linear joint estimation, factorization and alternating minimization (AM) methods are shown.

this experimental setting, we recorded 20 images for each static scene. The results are
summarized in Fig. 5, in which the estimated surface normal and their 3D reconstruction using [25] are presented. As shown in the figures, our methods properly handle
the varying light source intensities compared to Frobenius-norm, L1-norm and CBR
methods, with which severe distortions are observed in their reconstructed surfaces.
Auto-exposure: When auto-exposure is used, the shutter speed and/or aperture sizeof a camera is automatically adjusted to record well-exposed images according to theamount of incoming light. While it increases SQNR, it results in the non-uniform intensity setting.
For this experiment, we recorded 20 images of each static scene with auto-exposure.
Figure 6 shows the comparative result. As shown in the figure, our methods consistently
yield higher quality outputs than the other methods because our method explicitly accounts for the non-uniform exposures.
Mobile phone cameras: Our method is suitable for uncontrollable cameras like manyof mobile phone cameras, where we cannot turn off the auto-exposure setting. With
such cameras, recorded images are in the condition of non-uniform exposures acrossimages. From recorded images from a mobile phone camera, we linearize the intensity
observations using the method of [26] as preprocessing. Figure 7 shows the surface
normal estimates and their 3D reconstructions. While the 3D reconstructions of conventional methods are severely deformed, our methods show better reconstructions ingeneral. The linear joint estimation method suffered from the outliers in this case, butthat is not observed in factorization based and AM methods.

14

Donghyeon Cho, Yasuyuki Matsushita, Yu-Wing Tai, Inso Kweon

Frobenius-norm

L1-norm

CBR

Linear joint

Factorization

AM

Fig. 7. Result using a mobile phone camera. Top: Estimated surface normal, Bottom: 3D reconstruction. Our methods (linear joint estimation, factorization, and alternating minimization (AM)methods) produce more faithful results than the conventional methods.

7

Conclusion

This paper described photometric stereo methods that can handle the non-uniform lightsource intensities and exposures across images. We showed the effect of varying lightintensity conditions in photometric stereo that is relevant in practical settings. We thendeveloped solution methods that explicitly account for the non-uniform light intensities and exposures; namely, linear joint estimation, factorization based, and alternating
minimization methods. The linear joint estimation and factorization based methods are
simple and easy to implement, they occasionally suffer from numerical instability due
to un-modelled observations. The alternating minimization method showed a greaterrobustness over these techniques, while retaining the efficiency in computation. They
are all effective in the non-uniform intensities setting compared to methods that neglect
the effect of the setting. We further illustrate that our proposed methods can benefit
from auto-exposure, with which measurements with a greater SQNR can be obtained.
Our experiments on synthetic and real-world examples show the importance of properly
handling varying light intensities and exposures.

Acknowledgement
This work is partly supported by JSPS KAKENHI Grant Number JP16H01732 and theMinistry of Trade, Industry & Energy and the Korea Evaluation Institute of Industrial
Technology (KEIT) with the program number of 10060110.

Photometric Stereo under Non-uniform Light Intensities and Exposures

15

References
1. Woodham, R.J.: Photometric method for determining surface orientation from multiple images. Optical Engineering 19(1) (January 1980) 139–144
2. Wu, T., Tang, K., Tang, C., Wong, T.: Dense photometric stereo: A markov random fieldapproach. IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI) 28(11)(November 2006) 1830–1846
3. Verbiest, F., Van Gool, L.: Photometric stereo with coherent outlier handling and confidence
estimation. In: Proceedings of Computer Vision and Pattern Recognition (CVPR). (2008)
2886–2893
4. Wu, L., Ganesh, A., Shi, B., Matsushita, Y., Wang, Y., Ma, Y.: Robust photometric stereo via
low-rank matrix completion and recovery. In: Proceedings of Asian Conference on Computer
Vision (ACCV). (2010) 703–717
5. Oh, T.H., Kim, H., Tai, Y.W., Bazin, J.C., Kweon, I.S.: Partial sum minimization of singularvalues in rpca for low-level vision. In: Proceedings of International Conference on Computer
Vision (ICCV). (2013) 145–152
6. Zhang, Y., Mu, C., Kuo, H.W., Wright, J.: Toward guaranteed illumination models for noncovex objects. In: Proceedings of International Conference on Computer Vision (ICCV).
(2013) 937–944
7. Hayakawa, H.: Photometric stereo under a light-source with arbitrary motion. Journal of the
Optical Society of America (JOSA) 11(11) (November 1994) 3078–3089
8. Yuille, A.L., Snow, D.: Shape and albedo from multiple images using integrability. In:
Proceedings of Computer Vision and Pattern Recognition (CVPR). (1997) 158–164
9. Belhumeur, P.N., Kriegman, D.J., Yuille, A.L.: The bas-relief ambiguity. International Journal of Computer Vision (IJCV) 35(1) (November 1999) 33–44
10. Alldrin, N.G., Mallick, S.P., Kriegman, D.J.: Resolving the generalized bas-relief ambiguity by entropy minimization. In: Proceedings of Computer Vision and Pattern Recognition(CVPR). (2007) 1–7
11. Drbohlav, O., Chantler, M.: Can two specular pixels calibrate photometric stereo? In: Proceedings of International Conference on Computer Vision (ICCV). (2005) 1850–1857
12. Sunkavalli, K., Zickler, T., Pfister, H.: Visibility subspaces: Uncalibrated photometric stereo
with shadows. In: Proceedings of European Conference on Computer Vision (ECCV). (2010)
251–264
13. Shi, B., Matsushita, Y., Wei, Y., Xu, C., Tan, P.: Self-calibrating photometric stereo. In:
Proceedings of Computer Vision and Pattern Recognition (CVPR). (2010) 1118–1125
14. Solomon, F., Ikeuchi, K.: Extracting the shape and roughness of specular lobe objects using
four light photometric stereo. IEEE Transactions Pattern Analysis and Machine Intelligence(TPAMI) 18(4) (April 1996) 449–454
15. Georghiades, A.: Incorporating the torrance and sparrow model of reflectance in uncalibrated photometric stereo. In: Proceedings of International Conference on Computer Vision(ICCV). (2003) 816–823
16. Lin, S., Lee, S.W.: Estimation of diffuse and specular appearance. In: Proceedings of International Conference on Computer Vision (ICCV). (1999) 855–860
17. Y. Iwahori, R. J. Woodham, H.T., Ishii, N.: Neural network to reconstruct specular surface
shape from its three shading images. In: Proceedings of International Joint Conference on
Neural Networks. (1993) 1181–1184
18. Malzbender, T., Wilburn, B., Gelb, D., Ambrisco, B.: Surface enhancement using real-time
photometric stereo and reflectance transformation. In: Proceedings of the Eurographics Symposium on Rendering Techniques. (2006) 245–250

16

Donghyeon Cho, Yasuyuki Matsushita, Yu-Wing Tai, Inso Kweon

19. Shi, B., Tan, P., Matsushita, Y., Ikeuchi, K.: Bi-polynomial modeling of low-frequency reflectances. IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI) 36(6)(October 2014) 1078–1091
20. Ikehata, S., Wipf, D.P., Matsushita, Y., Aizawa, K.: Photometric stereo using sparse bayesian
regression for general diffuse surfaces. IEEE Transactions Pattern Analysis and MachineIntelligence (TPAMI) 36(9) (September 2014) 1816–1831
21. Sylvester, J.: Sur lequations en matrices px = xq. Comptes Rendus Acad. Sci. Paris 99(2)(1884) 67–71,115–116
22. Holland, P.W., Welsch, R.E.: Robust regression using iteratively reweighted least-squares.
Communications in Statistics: Theory and Methods 6(9) (1977) 813–827
23. Kamgar-Parsi, B., Kamgar-Parsi, B.: Evaluation of quantization error in computer vision.
IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI) 11(9) (September
1989) 929–940
24. Ikehata, S., Aizawa, K.: Photometric stereo using constrained bivariate regression for generalisotropic surfaces. In: Proceedings of Computer Vision and Pattern Recognition (CVPR).
(2014) 2187–2194
25. Xie, W., Zhang, Y., Wang, C.C.L., Chung, R.C.K.: Surface-from-gradients: An approachbased on discrete geometry processing. In: Proceedings of Computer Vision and Pattern
Recognition (CVPR). (2014) 2203–2210
26. Lee, J.Y., Matsushita, Y., Shi, B., Kweon, I.S., Ikeuchi, K.: Radiometric calibration by rank
minimization. IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI) 35(1)(January 2013) 144–156