Simultaneous Estimation of Spectral Reflectance and Normal
from a Small Number of Images
Masahiro Kitahara1 , Takahiro Okabe1 , Christian Fuchs2 and Hendrik P. A. Lensch2
1 Department of Artificial Intelligence, Kyushu Institute of Technology, Iizuka, Japan
2 Computer Graphics Group, Tübingen University, Tübingen, Germany

m kitahara@pluto.ai.kyutech.ac.jp, okabe@ai.kyutech.ac.jp,
christian.fuchs@graphics.uni-tuebingen.de, hendrik.lensch@uni-tuebingen.de

Keywords:

Multispectral Imaging, Photometric Stereo, Spectral Reflectance, Surface Normal.

Abstract:

Spectral reflectance is inherent characteristics of an object surface and therefore useful not only for computer
vision tasks such as material classification but also compute graphics applications such as relighting. In thisstudy, by integrating multispectral imaging and photometric stereo, we propose a method for simultaneously
estimating the spectral reflectance and normal per pixel from a small number of images taken under multispectral and multidirectional light sources. In addition, taking attached shadows observed on curved surfaces into
consideration, we derive the minimum number of images required for the simultaneous estimation and propose
a method for selecting the optimal set of light sources. Through a number of experiments using real images,
we show that our proposed method can estimate spectral reflectances without the ambiguity of per-pixel scales
due to unknown normals, and that, when the optimal set of light sources is used, our method performs as wellas the straightforward method using a large number of images. Moreover, we demonstrated that estimating
both the spectral reflectances and normals is useful for relighting under novel illumination conditions.

1

INTRODUCTION

The appearance of an object depends not only on theobject itself but also on the light source illuminating the object and on the camera capturing its image. Therefore, the same object appears differently
under different light sources and with different cameras. This appearance variation often causes the performance degradation of various computer vision algorithms.
The fraction of incident light power at each wavelength that is reflected on an object surface is called
spectral reflectance. Since the spectral reflectance isinherent characteristics of an object surface and independent of light sources and cameras, it is useful forcomputer vision tasks such as material classificationand scene segmentation as well as computer graphics applications such as relighting. In particular, theuse of spectral reflectance can prevent the occurrenceof so-called metamerism, i.e. a coincidental match ofapparent RGB colors of object surfaces with different
spectral reflectances.
When the spectral distribution of the incident lightto an object surface is known in advance, the spectral reflectance is computed by division, i.e. the ratio

between the reflected light observed on the surface tothe incident light at each wavelength. We can measure
the spectral distribution of the reflected light by using
point sensors such as spectrometers (Wellman, 1981)
and area sensors such as multispectral cameras (Yamaguchi et al., 2006) and hyperspectral cameras (Gat,
2000; Schechner and Nayar, 2002). Instead of using
those special sensors, RGB cameras can be used together with multispectral light sources (Park et al.,2007; Han et al., 2013).
Unfortunately, however, the reflected light depends not only on the spectral reflectance but also
on the normal of an object surface, and the above
techniques cannot estimate surface normals because
they assume that the direction or location of the lightsource is fixed. Therefore, the estimated spectral reflectance has the ambiguity of a per-pixel unknown
scale, i.e. the inner product between the light source
direction and the surface normal at each pixel. In
other words, we cannot tell whether the reflectance islarge (small) or the normal faces in a similar (dissimilar) direction to the light source. Such a per-pixel unknown scale could degrade the performance of material classification and scene segmentation, and moreover relighting under novel light source directions is303

VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications

tral reflectances without the ambiguity of per-pixel
unknown scales, and demonstrated that the estimatedspectral reflectances and normals enable relighting
under novel light source spectral distributions as wellas under novel light source directions.

2
400

500

600

700

800

900

1000

Figure 1: The images of a plaster relief (left) captured byusing the multispectral light stage (right). The pixel values
are scaled for display purpose.

impossible without using normals.
In this paper, we address the estimation of thespectral reflectance and normal of an object surface
by integrating multispectral imaging and photometricstereo. Specifically, we assume the Lambertian modeland the low-dimensional linear model of spectral reflectance (Parkkinen et al., 1989), and estimate boththe coefficients of the spectral reflectance and normal
per pixel from the images taken under multispectral
and multidirectional light sources as shown in Figure 1. It is obvious that the straightforward method,i.e. photometric stereo (Woodham, 1980) followed bymultispectral imaging (Park et al., 2007; Han et al.,2013) can estimate both the normals and spectral reflectances of matte surfaces from a large number ofimages. However, there is a room for significantly reducing the number of images.
Accordingly, we propose a method for simultaneously estimating spectral reflectances and normals
from a small number of images taken under multispectral and multidirectional light sources on the basis of the alternating least square (ALS) algorithm. In
addition, taking attached shadows observed on curved
surfaces under varying light source directions into
consideration, we derive the minimum number of images required for estimating the spectral reflectanceand normal per pixel and propose a method for selecting the optimal set of light sources in terms of noisefrom given light sources.
The main contribution of this study is threefold; (i) the simultaneous estimation of spectral reflectances and normals from a small number of images, (ii) the derivation of the minimum number ofimages required for the simultaneous estimation, and(iii) the light source optimization for robust estimation from a small number of images. Through anumber of experiments using real images, we confirmed that, even from a small number of images,
the proposed method can accurately estimate spec-

304

REFLECTION MODEL

Assuming the Lambertian model, the pixel value i at asurface point illuminated by a directional light source
is described asi=Zl(λ)ρ(λ)c(λ)dλ s ⊤ n ,(1)

where λ is the wavelength of incident and reflected
light, and l(λ), ρ(λ), and c(λ) are the spectral distribution of the light source, the spectral reflectance atthe point, and the spectral sensitivity of a camera respectively. The direction of the light source and thenormal at the point are denoted by s and n . Our proposed method assumes that the directions and spectral distributions of the light sources and the spectral sensitivities of the camera are known, i.e. they
are calibrated in advance, and estimates the spectral reflectance and normal from the pixel values observed under multispectral and multidirectional lightsources.
The spectral reflectance is a continuous function
with respect to wavelength, and describes how thereflectance changes depending on the wavelength ofincident and reflected light. Since the number of
unknowns is large, e.g. about 80 unknowns whenestimating spectral reflectances in the visible range
every 5 nm, the estimation of spectral reflectances
is prone to an ill-posed and/or ill-conditioned problem. Accordingly, our proposed method stably estimates spectral reflectances by constraining the spaceof spectral reflectances on the basis of their statistical characteristics. Specifically, our method makes
use of the low-dimensional model of spectral reflectance (Parkkinen et al., 1989). They apply PCA
to the dataset of spectral reflectances, and show thatany spectral reflectance is approximately represented
by a linear combination of basis functions asK

ρ(λ) = ∑ αk bk (λ),

(2)

k=1

where K, αk , and bk (λ) are the number of basis functions, the coefficients of the linear combination, andthe basis functions respectively. In this study, we use
the same basis functions and set K = 8 according to
Parkkinen et al.

SimultaneousEstimationofSpectralReflectanceandNormalfromaSmallNumberofImages

Substituting eq.(2) into eq.(1), we obtain
8i = ∑ αk
k=1

Zl(λ)bk (λ)c(λ)dλ s ⊤ n .

(3)

Therefore, the estimation of the spectral reflectanceand normal at a surface point results in estimating thenormal n and the coefficients of the linear combination αk (k = 1, 2, ..., 8). The number of unknowns at
each pixel is 10 in total, i.e. 2 for the normal and 8 forthe spectral reflectance. Note that a normal is a 3D
vector with unit length.

3

STRAIGHTFORWARD
METHODIn this section, we explain the straightforward
method, i.e. photometric stereo for estimating normals followed by multispectral imaging for estimating spectral reflectances.

3.1 Estimating Surface Normal
We assume that images under multispectral andmultidirectional light sources are captured by using a multispectral light stage similar to the existingones (Ajdin et al., 2012; Gu and Liu, 2012). Specifically, the light stage has P clusters of light sources
at different directions, and each cluster has Q lightsources with different spectral distributions. We denote the pixel value observed under the p-th lightsource direction (p = 1, 2, ..., P) and the q-th lightsource spectral distribution (q = 1, 2, ..., Q) and by ther-th channel of an RGB camera (r = 1, 2, 3) by
8i pqr = ∑ αk
k=1Zlq (λ)bk (λ)cr (λ)dλ s ⊤
p n.

(4)

Taking summation with respect to the light source
spectral distribution q and the camera channel r, we
obtain the gray scale
i′p

Q

=

3

∑ ∑ i pqr=

38

∑ ∑ ∑ αk

q=1 r=1 k=1= ρ′ s ⊤
p n,


..
. 
s pz ..
.

lq (λ)bk (λ)cr (λ)dλ s ⊤
pn

, (6)
(7)

Once the normal is estimated, by using P × Q color
images, a set of liner equations with respect to thecoefficients of the spectral reflectance αk is derived
from eq.(4) as


 

..
..
..
.
 .   . 
R⊤
 i pqr  =  · · · αk  .
s
l
b
c
dλs
n
·
·
·
q
k
r
p


 

..
..
..
.
.
.
(9)

In theory, we can estimate the coefficients of the spectral reflectance in a similar manner to the above byusing the pseudo inverse matrix.
Unfortunately, however, it is reported that sucha naive estimation tends to be unstable, when thenumber of light source spectral distributions is small
and/or the spectral distributions are not optimally chosen (Park et al., 2007; Han et al., 2013). Therefore, similar to those existing techniques, we incorporate the smoothness and non-negativity constraints
into the estimation;("

8

i pqr − ∑ αk(5)where ρ′ is an unknown scalar independent of the index of the light source direction p.
By using the P gray images, we rewrite eq.(5) in amatrix form as!

3.2 Estimating Spectral ReflectanceP{α1 ,...,α8 }

Z

ρ′ nx
ρ′ ny
ρ′ nz

This means that we can estimate normals in a similarmanner to the classic photometric stereo (Woodham,1980). In general, if the number of light sources islarger than three, we can estimate the normal up to an
unknown scale by using the pseudo inverse matrix ofS as
−1 ⊤ ′
S i = S+ i ′ .
(8)
ρ′ n̂n = S⊤ S
Since a normal has unit length, the normal is givenby n̂n = ρ′ n̂n/|ρ′ n̂n|. Note that we remove the p-th lightsource direction from the equations above if a surface
point is shadowed under that light source direction.
In our experiments, we detect shadows by using athreshold on pixel values.

{α̂1 , ..., α̂8 } = arg min

q=1 r=1
Q



..
..
..
.
.
 ′ 
 .
 i p  =  s px s py



..
..
..
.
.
.
′
′
i = S (ρ n ) .


+w

k=1

Z

"

Z

8

8

3

p=1 q=1 r=1#2lq (λ)bk (λ)cr (λ)dλss ⊤p n

d 2 bk (λ)∑ αk dλ2
k=1

subject to

Q

∑∑∑

#2



dλ


∑ αk bk (λ) ≥ 0,

(10)

k=1

305

VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications

where w is an empirical parameter that balances thelikelihood term and the smoothness term that tries
to minimize the second order derivatives. We set w
in eq.(10) and eq.(11) to 300 throughout our experiments. We used the MATLAB implementation of
the active-set algorithm for solving the above linear
least-square problem with linear constraints. Once thecoefficients of the linear combination are estimated,
we can obtain the spectral reflectance by substituting
them into eq.(2).

stage on the basis of a variant of the noise propagation
analysis (Drbohlav and Chantler, 2005).

4.2 Simultaneous Estimation
We propose a method for simultaneously estimating
spectral reflectances and normals from a small number of images. By integrating spectral imaging andphotometric stereo, our proposed method is formulated as(α} = arg min{n̂n, α̂4PROPOSED METHOD

4.1 Overview
The straightforward method described in Section 3
uses photometric stereo and multispectral imaging
separately. Therefore, it requires the images taken under multidirectional light sources with the same spectral distribution1 for estimating normals and the images taken under multispectral light sources for estimating spectral reflectances. Specifically, ignoring
attached shadows observed on curved surfaces, thestraightforward method requires 3 images taken under
the light sources at different non-coplanar directions
but with the same spectral distribution, and 3 color
images taken under the light sources with different
spectral distributions2 since eq.(3) has 8 unknowns
with respect to the spectral reflectance and each image yields 3 constraints (8 < 3 × 3).
On the other hand, ignoring attached shadows, wemust be able to estimate the spectral reflectance andnormal at a surface point from at least 4 color images
in theory, since the number of unknowns is 10 in total as described after eq.(3) and each image yields 3
constraints (10 < 4 × 3). This motivates us to propose a method for simultaneously estimating spectralreflectances and normals from a small number of images by integrating multispectral imaging and photometric stereo.
In the rest of this section, we formulate the simultaneous estimation of the spectral reflectance and normal per pixel from a small number of images. Then,
taking attached shadows observed on curved surfaces
into consideration, we derive the minimum numberof images required for the simultaneous estimation.
Finally, we propose a method for selecting the optimal light sources from those of the multispectral light
1 The gray scale images defined by eq.(5) are used insubsection 3.2.
2 Those spectral distributions should be chosen carefully
so that eq.(9) or eq.(10) can be solved.

306

α}
{nn,α

"

8

i pqr − ∑ αk

+w

k=1

"
Z

Z

8

3

∑ ∑

(p,q)∈I r=1

lq (λ)bk (λ)cr (λ)dλss⊤pn

d 2 bk (λ)
α
k
∑
dλ2
k=1
8

subject to#2dλ∑ αk bk (λ) ≥ 0,

#2





(11)

k=1

where α = (α1 , α2 , α3 , · · · , α8 )⊤ is the coefficient
vector of the spectral reflectance and I specifies theset of images from which the spectral reflectance andnormal are estimated.
We can see that the cost function in eq.(11) has abilinear form with respect to two variables; it is linear with respect to the normal n when the coefficient
vector α is fixed, and vice versa. Accordingly, we use
the ALS algorithm, which sets an initial value for one
variable and then iteratively updates one of the twovariables while the other is fixed in turn, for optimizing eq.(11).
More specifically, the normal n is updated in asimilar manner to eq.(8) when the coefficient vector
α is fixed, and the coefficient vector α is updated in asimilar manner to eq.(10) when the normal n is fixed.
In our experiments, we tested two initializations. One
is n = (0, 0, 1)⊤ , i.e. the normal faces toward a camera. Another is α = (1, 0, 0, · · · , 0)⊤ , i.e. the spectralreflectance is the same as the first principal component. We experimentally confirmed that both of theinitializations achieve similar performance, and thatthe optimization converges within a few iterations.
It takes about 140 msec to estimate the spectral reflectance and normal at each pixel by using MATLAB
on a PC with Core i7.

4.3 Number of Required Images
The point on an object surface is in attached shadow
under a light source, when the angle between the lightSimultaneousEstimationofSpectralReflectanceandNormalfromaSmallNumberofImages

source direction s and the normal n at the point islarger than π/2, i.e. s⊤ n < 0. In general, attachedshadows are inevitably observed on curved surfaces
such as a sphere under varying light source directions. Since the pixel intensity in attached shadow iszero, i.e. the left hand side of eq.(3) is 0, we cannot
obtain any constraint about the spectral reflectanceand normal from the shadowed pixel except that thesurface normal faces in the opposite direction to thelight source (ss⊤ n < 0)3 . Therefore, in order to estimate the spectral reflectance and normal per pixel on a
curved surface, we need to take attached shadows into
consideration, and use a sufficient number of images
taken under different light sources so that each pointon the surface is illuminated by the required numberof light sources.
In this study, we derive the number of required images under the following two assumptions. First, we
assume that the shape of an object of interest is arbitrary but convex; denoting the viewing direction by v ,we assume arbitrary normals n such that v ⊤ n > 0 but
do not take cast shadows4 into consideration. Notethat the number of required images could be arbitrarily large, if we assume arbitrary complex shapes suchas a tree with a large number of branches and leaves.
Second, in the numerical analysis below, we assume
that a point on an object surface is illuminated by alight source, if the inner product between the lightsource direction and the normal is larger than a small
threshold ε;s ⊤ n > ε.
(12)This is because we detect shadows by using a threshold on pixel values and dark pixels are more likely tobe affected by noise.
Thus, in order to estimate the spectral reflectanceand normal at every point on an arbitrary convex surface, the set of color images taken under multispectral and multidirectional light sources has to satisfy
the following conditions.
(A) Each point is illuminated in at least 4 images because eq.(11) has 10 unknowns in total and each
image yields 3 constraints (10 < 4 × 3).
(B) Each point is illuminated in at least 3 images
taken under different light source spectral distributions for updating α in the ALS algorithm(8 < 3 × 3).
3 It is reported that normals can be recovered from at-tached shadows by using a large number of images taken
under varying light source directions (Okabe et al., 2009).
4 The cast shadows are observed on concave surfaces,
when s ⊤ n > 0 but the light source is occluded by the othersurface or the other part of the same surface.

(C) Each point is illuminated in at least 3 images
taken under different light source directions for
updating n in the ALS algorithm.
Based on the assumption about illuminated surface points by using a threshold in eq.(12), it is trivial that at least 3 images taken under different lightsource directions are required for illuminating every
point on an arbitrary convex surface at least once (seeAppendix A). Therefore, for satisfying the conditions(B), 3 images (a triplet) taken under different lightsource directions are required for each spectral distribution, i.e. 9 images (3 triplets) are required in total. In our experiments, we capture each image by simultaneously turning on two light sources at the same
direction but with different spectral distributions sothat the combination of the two spectral distributions
has overlap with the spectral sensitivities of the RGB
channels of a camera.
By using the above 9 images, every point on
an arbitrary convex surface is illuminated by 3 lightsources with different spectral distributions at least
once. Therefore, the condition (C) is satisfied, when
the light source directions for the triplets are different
from each other, i.e. when a set of 9 images (a nonuplet) is taken under different light source directions.
Moreover, we can numerically show that some of thenonuplets satisfying the conditions (B) and (C) alsosatisfy the condition (A) (see Appendix B). Hence,
we can estimate the spectral reflectance and normal at
every point on an arbitrary convex surface from 9 images. In our experiments, we confirmed that our lightstage has a number of nonuplet candidates which satisfy the conditions (A), (B), and (C).

4.4 Optimizing Light Sources
In the previous subsection, we show that a set of 9 images (a nonuplet) is required for estimating the spectral reflectance and normal at every point on an arbitrary convex surface, and that our light stage has anumber of nonuplet candidates. Since the accuracy ofthe estimated spectral reflectances and normals could
depend on the nonuplet used for the estimation, we
propose a method for selecting the optimal nonuplet,in other words, selecting the optimal light sources under which the optimal nonuplet is taken. In particular, we focus on the optimization of light source directions, since our light sources have only 6 different
spectral distributions in visible range and we have already used all of them.
The optimization of light source directions is
discussed in the context of the classic photometricstereo (Drbohlav and Chantler, 2005). They study
how the zero-mean Gaussian noises in pixel intensi-

307

VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications

ties propagate to the normals estimated by using thepseudo inverse matrix, and show that the noises areamplified by
−1σ2 Tr S⊤ S
(13)through the propagation, where σ is the standard deviation of the Gaussian noises and S is a light source
matrix defined in eq.(6) and eq.(7). By minimizing
eq.(13), they find the optimal light source configurations in terms of noise, e.g. 3 orthogonal directions
when the number of light sources is 3. Unfortunately,
however, they ignore attached shadows which are inevitably observed on curved surfaces under varying
light source directions. Actually, on curved surfaces,
surface points with different normals could be illuminated by different sets of light sources.
Accordingly, we extend Drbohlav and Chantler bytaking attached shadows into account, and select the
optimal set of light sources from its candidates L in
the minimax manner. Specifically, we evaluate themaximum error of the estimated surface normal foreach candidate S = {ss1 , s 2 , s 3 , · · · , s 9 } with respect toarbitrary normals n such that v ⊤ n > 0, and then selectthe candidate that minimizes the maximum error fromthe candidates L;hi−1
Ŝ = arg min max Tr S⊤ (nn)S(nn)
.
(14)
S∈L

n

Here, S(nn) consists of light sources included in a candidate and illuminating a surface point with normaln;S⊤ (nn) = (· · · , s p , · · · ),(15)

where {ss p ∈ S|ss⊤
p n > ε}.

5

EXPERIMENTS

5.1 Multispectral Light StageWe implemented a multispectral light stage similar
to the existing ones (Ajdin et al., 2012; Gu and Liu,2012), and captured images under multispectral andmultidirectional light sources on the basis of multiplexed sensing (Schechner et al., 2003), which is awell-known technique for increasing signal-to-noise
ratio without increasing measurement time. In thisstudy, we used 120 images in total, i.e. the number oflight source directions is 20 (P = 20) and the numberof light source spectral distributions is 6 (Q = 6)5 .
5 We removed a few images from the 120 images andused the remaining images for the following analysis andestimation, because the corresponding light sources did notwork well when those images were captured.

308

1

1

0.5

0.5

+ (i)
+ (ii)++ (iii)

(iv)

(i)
0
400
1

(ii)
500

600

700

500

600

700

0.5

0
400
1

600

700

500

600

700

0.5

(iii)
0
400

(iv)

(a)

(c)

500

0
400

(b)

(d)

(e)

Figure 2: The estimated spectral reflectances (b) at four
points on the plaster relief (a). The red-lines stand for thestraightforward method using 120 images and the greendotted lines stand for our proposed method using 9 images.
(c), (d), and (e) show the normals estimated by using thestraightforward method and our method and their difference
respectively. Those results demonstrates that our method
performs as well as the straightforward method.

Figure 1 shows the 120 images of a plaster relief. Here, all the LEDs are calibrated so that they
have the same intensity. The first two columns show
the images under purple light, followed by those under blue, green, yellow-green, orange, and red lights.
We can see that the color observed on the surfacechanges according to the light source spectral distribution. Moreover, when we focus on the 20 images under the same light source spectral distribution,
we can see that the intensity observed on the surface changes according to the light source direction.
Our proposed method estimates both the spectral reflectances and normals from the color and shading observed under multispectral and multidirectional lightsources.
In the rest of this section, we first confirmed thatour proposed method using a small number of images
performs as well as the straightforward method usinga large number of images. Second, we confirmed thatthe use of the optimal set of light sources is effective
for robust estimation. Finally, we evaluated the accuracy of our method by comparing the estimated spectral reflectances and normals with their ground truth
values.

5.2 Number of Images
Figure 2 (b) shows the estimated spectral reflectances
at four points on the plaster relief (a). We can seethat the spectral reflectances estimated from 9 imagesby using our proposed method (green-dotted lines)
are consistent with those estimated from 120 images

SimultaneousEstimationofSpectralReflectanceandNormalfromaSmallNumberofImagesby using the straightforward method (red-solid lines).
In addition, this result qualitatively demonstrates that
our method can estimate spectral reflectances accurately because the plaster relief is made of a uniform
material, and the estimated spectral reflectances lookalmost the same.
Figure 2 (c), (d), and (e) show the normals estimated from 120 images by using the straightforward
method, those estimated from 9 images by using our
proposed method, and their difference in which theangle from 0 to π/2 is linearly mapped to 8 bit grayscale. Here, normals are represented by using a color
map. Specifically, the x, y, and z elements of a normal
is linearly mapped to R, G, and B channels (see a reference sphere attached to the normal map). This result
qualitatively demonstrates that our method can estimate normals accurately; e.g. a surface point towarda camera is bluish and a surface point toward rightis greenish, and that (d) is consistent with (c) except
for concave areas. Note that both the straightforward
method and our method do not necessarily work wellin those areas because they do not take interreflection
into consideration.
Table 1 shows the average difference between
the normals estimated by using the straightforward
method and those estimated by using our proposed
method for four different objects; relief, bread,checker, and ball (see “best” row). This result quantitatively demonstrates that our method using a small
number of images performs as well as the straightforward method using a large number of images because
the differences are small enough.

5.3 Light Source Optimization
As described in subsection 4.4, in general, there are anumber of sets of images (and corresponding sets oflight sources) from which we can estimate spectral reflectances and normals of an arbitrary convex surface,
but the accuracy of the estimated spectral reflectancesand normals could depend on the set of light sources
used for the estimation. In Figure 3, (b) and (d) show
the images of a wooden bread taken under the optimal, i.e. the best (a) and the worst (c) sets of lightsources derived from eq.(14). Here, we show the lightsource directions in the 3D (xyz) space by projecting them on the 2D (xy) plane. The inner and outercircles correspond to the zenith angle θ = π/4 andθ = π/2 respectively. The selected light sources are
represented by symbols in cyan, and the light sources
represented by the same symbol have the same spectral distribution. We can see that the best set of lightsources distributes at wider angles than the worst one.
Figure 4 (b) shows the estimated spectral re-

(a)
(b)

(c)
(d)
Figure 3: (b) and (d) show the images of the wooden breadtaken under the best (a) and the worst (c) sets of lightsources represented in cyan respectively. The best set oflight sources distributes at wider angles than the worst one.

flectances at four points on the wooden bread (a). (c),(d), and (f) show the normals estimated from 120 images, those estimated from the 9 images taken under the best and the worst sets of light sources. (e)and (g) show the difference between (c) and (d) andthe difference between (c) and (f) respectively. Notethat some artifacts due to specular reflection components are visible since we assume the Lambertianmodel. We can see that spectral reflectances and normals can be estimated from both of the best and theworst sets of light sources, but that the estimated spectral reflectances and normals depend on the set of lightsources used for the estimation. In particular, (e) and(g) clearly demonstrate that our proposed method performs as well as the straightforward method when thebest set of light sources is used, but does not perform
well when the worst set of light sources is used. Thismeans that the optimization of light sources is critically important for robust estimation when the number of images is small.
Table 1 also shows the difference between the normals estimated by using the straightforward method
and those estimated by using our proposed methodwith the best or the worst set of light sources for thefour objects. This result quantitatively demonstrates

309

VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications

1+ (ii)

1

(i)

0.5

1

(ii)

2

3

4

1

0.5

+ (i)

2

+ (iv)

0
400
1

500

600

(iii)

0
400

0
400
1

500

600

700

3

(iv)

0.5

+ (iii)

700

4

0.5

500

600

(a)

700

0
400

500

600

700

(b)

5
6

(a)

(e)

(c)

1

0.5

0.5

0.5

0.5

Table 1: The difference between the normals estimated byusing the straightforward method and our proposed methodwith the best or the worst set of light sources.

relief
1.68◦
3.07◦

bread
3.17◦
5.42◦

checker
1.05◦
2.19◦

ball
1.98◦
4.03◦

that the optimization of light sources works well because the difference of the best case is smaller thanthat of the worst case.

5.4 Comparison with Ground TruthFirst, we compared the spectral reflectances of acolor checker estimated by using the straightforward
method and our proposed method with those measured by using a spectrometer. Figure 5 shows theimage (a) and the estimated spectral reflectances (b)and normals (c)(d)(e) of the color checker. In (b), thered-solid lines, green-dotted lines, blue-dotted lines,
and magenta-dotted lines stand for the measurement,
the straightforward method, and our proposed methodwith the best and the worst sets of light sources. Here,we could not estimate the spectral reflectances and

310

500

600

700

0
400

500

600

700

0
400

500

600

700

0
400

1

1

1

1

0.5

0.5

0.5

0.5

500

600

700

0
400

500

600

700

0
400

500

600

700

0
400

1

1

1

1

0.5

0.5

0.5

0.5

500

600

700

500

600

700

500

600

700

500

600

700

500

600

700

500

600

700

(g)

Figure 4: The estimated spectral reflectances (b) at four
points on the wooden bread (a). The red-solid lines, greendotted lines, and blue-dotted lines stand for the straightforward method and our proposed method with the best
and the worst set of light sources. (c), (d), and (f) show
the normals estimated by using the straightforward method
and our method with the best and the worst sets of lightsources. (e) and (g) show the difference between (c) and (d)and the difference between (c) and (f), and demonstrate that
our method performs as well as the straightforward method
when the best set of light sources is used.

object
best
worst

(e)

1

0
400

(f)

(d)

1

0
400

(d)

(c)

1

0
400

500

600

700

0
400

500

600

700

0
400

500

600

700

0
400

1

1

1

1

0.5

0.5

0.5

0.5

0
400

500

600

700

0
400

500

600

700

0
400

500

600

700

0
400

1

1

1

1

0.5

0.5

0.5

0.5

0
400

500

600

700

0
400

500

600

700

0
400

500

600

700

0
400

1

1

1

1

0.5

0.5

0.5

0.5

0
400

500

600

700

0
400

500

600

700

0
400

500

600

700

0
400

(b)
Figure 5: The measured/estimated spectral reflectances (b)at each patch of the color checker (a). The red-solidlines, green-dotted lines, blue-dotted lines, and magentadotted lines stand for the measurement, the straightforward
method, and our proposed method with the best and theworst sets of light sources. The estimated spectral reflectances are consistent with the measured ones. (c), (d),and (e) are the normals estimated by using the straightforward method, and our proposed method with the best andthe worst sets of light sources.

normals at black areas including the top-right color
patch because they were too dark and treated as shadows. Table 2 shows the RMS errors of the spectral reflectances from 400 nm to 700 nm 6 estimated
6 It is known that the basis functions of spectral reflectances
are not necessarily accurate at short-wavelength range (Parkkinen et al., 1989). In addition, the measured spectral reflectances
are also not accurate in that range because the halogen bulb
used for our experiment is not bright enough.

SimultaneousEstimationofSpectralReflectanceandNormalfromaSmallNumberofImages

Table 2: The RMS errors of the estimated spectral reflectances of the color checker; the straightforward method,
our proposed method with the best set of light source, andthat with the worst set of light sources from top to bottom
in each field.

row\col.
1

2

3

4

5

6

1
0.054
0.059
0.060
0.051
0.072
0.070
0.040
0.039
0.039
0.044
0.041
0.039
0.068
0.081
0.081
0.025
0.030
0.030

2
0.080
0.078
0.078
0.075
0.070
0.070
0.049
0.057
0.055
0.060
0.065
0.063
0.061
0.058
0.058
0.068
0.067
0.067

3
0.044
0.043
0.044
0.060
0.107
0.101
0.123
0.109
0.110
0.088
0.080
0.081
0.048
0.042
0.041
0.037
0.027
0.027

+ (ii)+ (iv)

(i)
+

4
N/A
N/A
N/A
0.015
0.016
0.015
0.025
0.025
0.024
0.059
0.057
0.059
0.063
0.070
0.069
0.038
0.052
0.050

by using the straightforward method and by using
our method with the best and the worst sets of lightsources from top to bottom in each field. The averages of the RMS errors are 0.056, 0.058, and 0.058
respectively. This result quantitatively demonstrates
that our method can accurately estimate spectral reflectances even from a small number of images.
Second, we evaluated the estimated surface normals. Figure 6 shows the image (a) and the estimated spectral reflectances (b) and normals of awooden ball. (d), (f), and (h) are the normals estimated by using the straightforward method and our
proposed method with the best and the worst sets oflight sources. (e), (g), and (i) show the differences
between the ground truth (c) and the estimated ones(d)(f)(h). We assume that the shape of the ball is aperfect sphere although it looks a little distorted both
locally and globally to some extent. Therefore, a partof the errors common to the estimated surface normals
by using the straightforward method and our method
would be due to those distortions. In addition, we
can observe white spots caused by specular reflection
components. The average errors of normals estimatedby using the straightforward method and our methodwith the best and the worst sets of light sources are
5.11◦ , 5.52◦, and 7.43◦ respectively including the deviation of the ball from a perfect sphere and the errors
due to specular reflection components. Thus, we cansee quantitatively that our method can accurately estimate normals even from a small number of images.

+ (iii)

1

1

0.5

0.5

(i)
0
400
1

(ii)
500

0.5

700

0
400
1

500

600

700

500

600

700

0.5

(iii)
0
400

(iv)
500

(a)

(c)

600

600

700

0
400

(b)

(d)

(e)

(f)

(g)

(h)

(i)

Figure 6: The estimated spectral reflectances (b) at four
points on the wooden ball (a). The red-solid lines, greendotted lines, and blue-dotted lines stand for the straightforward method and our proposed method with the best andthe worst set of light sources. (d), (f), and (h) are the normals estimated by using the straightforward method and our
method with the best and the worst sets of light sources. (e),(g), and (i) show the differences between the ground truth(c) and the estimated ones (d)(f)(h), and demonstrates that
our method performs as well as the straightforward method
when the best set of light sources is used.

6

APPLICATION

To demonstrate the effectiveness of estimating boththe spectral reflectances and normals by using our
proposed method, we synthesized images under novel
illumination conditions. Figure 7 shows the synthesized images of the plaster relief and wooden ball under 9 different illumination conditions; three spectral
distributions times three light source directions. The
spectral reflectances and normals estimated by using
the straightforward method (top) and our proposed
method (bottom) are used. We can see that the synthesized images look quite natural, and that the bottom images are consistent with the top images. This
result demonstrates that our proposed method extends
the capability of spectral relighting (Park et al., 2007;

311

VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications

1

0.5

0
400
1

500

600

700

500

600

700

500

600

700

500

600

700

500

600

700

500

600

700

0.5

0
400
1

0.5

0
400

non-Lambertian surfaces. As mentioned in Section 5
and Section 6, the estimated spectral reflectances andnormals are sometimes contaminated by specular reflection components since we assume the Lambertianmodel. We will use the robust estimation for removing those components as outliers or model them byusing parametric or non-parametric representation in
the future.

1

0.5

0
400
1

ACKNOWLEDGEMENTS

0.5

0
400
1

A part of this work was supported by JSPS KAKENHI Grant No. 25280057.

0.5

0
400

Figure 7: The synthesized images under novel illumination
conditions. The spectral reflectances and normals estimatedby using the straightforward method (top) and our proposed
method (bottom) are used.

Han et al., 2013) so that one can deal with novel
light source directions. Note that some artifacts due
to specular reflection components are observed on
the ball because our method assumes the Lambertianmodel.

7

CONCLUSION AND FUTURE
WORKIn this study, by integrating multispectral imaging
and photometric stereo, we proposed a method for simultaneously estimating the spectral reflectance andnormal per pixel from a small number of images
taken under multispectral and multidirectional lightsources. In addition, taking attached shadows observed on curved surfaces into consideration, we derived the minimum number of images required for thesimultaneous estimation and proposed a method forselecting the optimal light sources in terms of noise.
Through a number of experiments using real images, we showed that our proposed method can estimate spectral reflectances without the ambiguity of
per-pixel unknown scales, and that, when the optimal set of light sources is used, our method using
only 9 images performs as well as the straightforward
method using a large number of images. In addition,
we demonstrated that estimating both the spectral reflectances and normals is useful for relighting undernovel light source spectral distributions as well as under novel light source directions.
One direction of future study is the extension to

312

REFERENCES
Ajdin, B., Finckh, M., Fuchs, C., Hanika, J., andLensch, H. (2012). Compressive higher-order sparseand low-rank acquisition with a hyperspectral lightstage. Technical report, Eberhard Karls UniversitätTübingen. WSI-2012-01.
Drbohlav, O. and Chantler, M. (2005). On optimal light
configurations in photometric stereo. In Proc. IEEE
ICCV2005, volume 2, pages 1707–1712.
Gat, N. (2000). Imaging spectroscopy using tunable filters:
a review. In Proc. SPIE Wavelet Applications VII, volume 4056, pages 50–64.
Gu, J. and Liu, C. (2012). Discriminative illumination:
per-pixel classification of raw materials based on optimal projections of spectral BRDF. In Proc. IEEE
CVPR2012, pages 797–804.
Han, S., Sato, I., Okabe, T., and Sato, Y. (2013). Fast spectral reflectance recovery using DLP projector. IJCV.
DOI 10.1007/s11263-013-0687-z.
Okabe, T., Sato, I., and Sato, Y. (2009). Attached shadow
coding: estimating surface normals from shadows under unknown reflectance and lighting conditions. In
Proc. IEEE ICCV2009, pages 1693–1700.
Park, J.-I., Lee, M.-H., Grossberg, M., and Nayar, S. (2007).
Multispectral imaging using multiplexed illumination.
In Proc. IEEE ICCV2007, pages 1–8.
Parkkinen, J., Hallikainen, J., and Jaaskelainen, T. (1989).
Characteristic spectra of munsell colors. JOSA A,
6(2):318–322.
Schechner, Y. and Nayar, S. (2002). Generalized mosaicing:wide field of view multispectral imaging. IEEE Trans.
PAMI, 24(10):1334–1348.
Schechner, Y., Nayar, S., and Belhumeur, P. (2003). A
theory of multiplexed illumination. In Proc. IEEE
ICCV2003, pages 808–815.
Wellman, J. (1981). Multispectral mapper: imaging spectroscopy as applied to the mapping of earth resources.
In Proc. SPIE Imaging Spectroscopy, volume 268,
pages 64–73.

SimultaneousEstimationofSpectralReflectanceandNormalfromaSmallNumberofImages

Woodham, R. (1980). Photometric method for determining surface orientation from multiple images. OpticalEngineering, 19(1):139–144.
Yamaguchi, M., Haneishi, H., Fukuda, H., Kishimoto, J.,
Kanazawa, H., Tsuchida, M., Iwama, R., and N, O.
(2006). High-fidelity video and still-image communication based on spectral information: natural visionsystem and its applications. In Proc. SPIE-IS&T Electronic Imaging, volume 6062.

APPENDIX A
We give a counterexample and prove that 2 images (or2 light sources) are insufficient for illuminating every
point on an arbitrary convex surface at least once. Let
us consider a unit sphere illuminated by a single directional light source. The single light source illuminates
at most almost the half of the occluding boundary of
the sphere according to the assumption of illuminatedpixels in eq.(12). Then, the length of illuminated occluding boundary is (π − δ), where δ is a small number. Since the length of the entire occluding boundary is 2π and (π − δ) × 2 < 2π, we cannot illuminate
the entire occluding boundary at least once by using
2 light sources.

APPENDIX B
In a similar manner to Appendix A, we give a
counterexample and prove that 8 images (or 8 lightsources) are insufficient for illuminating every pointon an arbitrary convex surface at least 4 times, i.e. forsatisfying the condition (A) in subsection 4.3. Considering a unit sphere illuminated by a single directional light source, the length of illuminated occluding boundary is (π − δ). Since (π − δ) × 8 < 4 × 2π, it
is clear that we cannot illuminate the entire occluding
boundary at least 4 times by using 8 light sources.

313