Enhanced Photometric Stereo with Multispectral Images
Tsuyoshi Takatani∗†

Yasuyuki Matsushita‡

Yasuhiro Mukaigawa††Yasushi Yagi†‡

Osaka University{takatani,mukaigaw,yagi}@am.sanken.osaka-u.ac.jp

Abstract

Observed images at various wavelengths

We introduce a technique based on multispectral
images aimed at improving Lambertian photometricstereo. Many photometric stereo algorithms assume
Lambertian reflectance, but deviations from this ideal
produce errors in shape reconstruction. To alleviate
this problem, we exploit the wavelength-dependence ofmaterial reflectance. Based on the observation that reflectance at certain wavelengths for a given object ismore Lambertian than at others, we propose a method
for identifying such wavelengths by a matrix rank analysis, and use them to achieve more accurate photometric stereo. We merge reconstruction results fromdifferent wavelengths to produce the final surface normal map. Experimental results on synthetic and real
data demonstrate the greater accuracy of this method
compared to conventional photometric stereo based onbrightness images.

Photometric stereo, a technique originally proposedby Woodham [1] and Silver [2], is a traditional method
for estimating surface orientations from multiple images taken under different lighting conditions. A
key factor in this process is the surface reflectance,
which describes how the shading at each surface pointchanges in relation to the lighting direction and thesurface normal. By varying the light direction, thecorresponding changes in the shading are used to infer
the surface normals according to the reflectance model.
Surface reflectance is normally assumed to follow
Lambert’s law, whereby the intensity I of the reflected
light is proportional to the inner product of the lighting
vector l and the surface normal vector n:(1)where ρ(x) denotes the albedo, or intrinsic color, ofthe surface at point x. This Lambertian model of reflectance is frequently used in photometric stereo because of its simplicity and convenience. However, thesmall deviations from the Lambertian model displayedby many materials produces inaccuracies in shape reconstruction.
Various methods have been proposed for dealing
with non-Lambertian reflectance. Non-Lambertian
highlights are treated as outliers within a Lambertian
photometric stereo framework [3], or removed from images [4]. Some instances of photometric stereo areformulated for specific parametric reflectance models,∗ This work has been done while the author was a researchintern at Microsoft Research Asia.

wlen2

wlenm

wlen3

....

Photometric stereo using
observation at each wavelength
wlen1

wlen2

wlen3

Segmentation using
multispectral reflectancesegment1
segment2
segment3
segmentk

wlenm
....

+

....

Introduction

I(x) = ρ(x)nT l,

Microsoft Research Asia

{yasumat,stevelin}@microsoft.com

wlen1

1

Stephen Lin‡

Evaluation in each segment

Combination of normal maps estimated at
optimal wavelength in each segment
according to evaluation values

Figure 1. Overview of our algorithm for enhancing
the performance of photometric stereo using multispectral images.

such as the Torrance-Sparrow model [5], a combinationof isotropic Ward models [6], and m-lobed reflectancemaps [7]. Several work have been presented for an
arbitrary form for the reflectance [8, 9]. Some methods analyze color for photometric stereo, but mainly tominimize image acquisition with differently colored illuminations [10], to improve robustness by incorporating additional measurements [11], or to aid in removing
non-Lambertian reflectance components [4, 12].
Our algorithm is outlined in Fig. 1. Our input isa set of images captured at various wavelengths andfor different lighting directions. For each wavelength,
we apply Lambertian photometric stereo. In addition,
the multispectral image constructed from the set ofwavelengths is segmented into different multispectral
regions. In each of these, we evaluate the photometricstereo results for each wavelength, using matrix rank
analysis to identify which wavelength is most consistent with the Lambertian model. The surface normal
map computed for each wavelength are then mergedby selecting the optimal wavelength for each region.
We thus obtain the final normal map. This approachis validated with simulated images. Experiments onreal images support this multispectral approach, showing visible improvements in surface normal estimation
over conventional photometric stereo based on brightness images.

2

Photometric Stereo using Multispectral
Images

2.1

Wavelength-dependent Reflectance

We first examine the wavelength-dependence of reflectance and its effects on photometric stereo recon-struction. When accounting for a continuous rangeof wavelengths, the Lambertian reflectance model in
Eq. (1) can be expressed asI(λ, x) = σ(λ)ρ(λ, x)e(λ)n(x)T l,

(a)

(2)

where λ ranges over the measurable wavelengths of thesensor, σ is the spectral sensitivity of the camera, ρ isthe spectral reflectance at a given surface point, and eis the spectrum of the light source.
The diffuse reflection that is conventionally used
for photometric stereo arises from a physical process
in which light penetrates an object surface, scattersoff material particles, and is emitted out of the surface. According to the Lambertian model, this outgoing light is distributed uniformly in all directions, withan intensity given by Eq. (2).
In reality, the directional distribution of the outgoing light is determined by how the light is scattered
within the material. This subsurface scattering is physically determined with respect to the optical propertiesof the material, virtually all of which are wavelengthdependent. For example, the wavelength can affectthe number of scattering events. For details on the
wavelength-dependence of optical properties in materials, we refer the reader to [13].
This wavelength-dependence of optical properties
suggests that surface reflectance is also spectrumsensitive, thereby affecting the reconstruction qualityof photometric stereo. We demonstrate this in Fig. 2
with a simulated sphere rendered using real reflectancedata for red plastic, obtained from the MERL BRDF
dataset [14]. Forty images with different lighting directions were used for photometric stereo, and the color
spectrum was coarsely sampled in red, green, and blue
channels. The angular error maps in the reconstructions reveal different degrees of accuracy in the different color channels.

2.2

Region-based
Wavelengths

IdentificationofOptimalGiven a set of multispectral images captured witha stationary camera and different lighting directions,
our algorithm first computes Lambertian photometricstereo, performs a region segmentation, and identifies
the optimal wavelength for each region, which is thenused to merge the normal maps computed with different wavelengths. The details of the process are as
follows.
Region segmentation As mentioned previously,
the optimal wavelength for Lambertian photometricstereo may vary across the image because of varying
BRDFs. In particular, our investigations have shownthat regions of different color often have different optimal wavelengths. To improve the reconstruction accuracy, we therefore segment the scene by normalized
multispectral information, and determine the optimal
wavelength for each of the regions. We apply k-meansclustering for the segmentation, using an image where
the object is fully illuminated (i.e., no shadows).
Optimal wavelength identification in each region To find the optimal wavelength in each segmented region, we evaluate the data in each region.

(b)

Avg. 1.42
Var. 1.02(c)

Avg. 5.37
Var. 11.42(d)Avg. 13.43
Var. 44.13(e)Avg. 2.56
Var. 2.63(f)Figure 2. Effect of different wavelengths on photometric stereo. (a) One of 40 images rendered withdifferent lighting directions; (b) ground truth of thenormal map; normal maps and angular error maps
using the (c) red, (d) green, and (e) blue color channels for Lambertian photometric stereo; and (f) results with brightness images.

For a region with N pixels and M lighting conditions,
we can express Eq. (2) in matrix form as
I = g(λ)NL,(3)where g(λ) = σ(λ)ρ(λ)e(λ), I ∈ RN ×M is a matrixof image intensities, N ∈ RN ×3 is a matrix of surface
normals, and L ∈ R3×M is a matrix of lighting vectors.
As seen in Eq. (3), the Lambertian model predicts
that the rank of the observation matrix I must equalthat of the normal vector matrix N, given that the
sampled lighting directions are not co-planar (i.e.,rank(L) = 3). As rank(I) deviates further fromrank(N), the data becomes more poorly fitted by theLambertian model and then Lambertian photometricstereo should produce less accurate results as discussed
in [15].
In view of this effect, we evaluate the wavelengths
by determining how close the rank of its observation
matrix is to rank(N). The evaluation function used inthis work is
σk+1
E=
,
(4)
σk
where k = rank(N) and σi denotes the i-th eigenvalueof I. This function becomes large when the rank of I
falls below k (i.e., σk close to 0) or exceeds k (i.e., σk+1
becomes large). Thus, the wavelength that minimizes
E is that for which the rank of I is closest to rank(N).
A small additive term may be included in the denominator to avoid division by zero, but in practice we havefound σk never to be zero.
Assuming the rank of N to be three, Eq. (4) is evaluated with k = 3. Corresponding values of k should
be used in degenerate cases of lower rank. For eachsegmented region, the optimal wavelength minimizes
E, and to enhance Lambertian photometric stereo, weestimate the final normal map by combining the normal maps of each region that correspond to its optimal
wavelength.

3

Experimental Results

3.1

Simulation images

We first validate our method on a simulated scene
for which the ground truth surface normals are known.
Our scene consists of an egg-shaped surface renderedwith the reflectances of seven different materials from
the MERL BRDF dataset [14]. Forty images under

(a)

(b)

(c) Avg. 4.16; Var. 0.49(d) Avg. 5.88; Var. 0.73(e) Avg. 10.40; Var. 1.51(f) Avg. 4.29; Var. 0.37(g) Avg. 3.14; Var. 0.16

Figure 3. Normal map estimation for simulation images. (a) One of 40 images rendered under different
light directions; (b) ground truth of the normal map;
normal maps and angular error maps using the (c)red, (d) green, and (e) blue color channels; (f) withbrightness images; (g) with our method.

Table 1. Evaluation function values.
Material nameblue-acrylic
green-plastic
light-brown-fabricorange-paint
purple-paint
red-phenolic
yellow-matte-plastic

Red
0.960
0.840
0.687
0.767
0.348
0.403
0.434

Green
0.985
0.831
0.852
0.812
0.837
0.932
0.604

Blue
0.642
0.885
0.968
0.897
0.847
0.900
0.898

different lighting directions are generated. One of theseis shown in Fig. 3(a). The ground truth is exhibitedin Fig. 3(b). We note that the MERL BRDF dataset
provides reflectance data in terms of only red, green,
and blue, so our multispectral technique is performed
with three color channels in this instance.
Photometric stereo is first computed for each of thecolor channels separately, and also for the brightness
images obtained by summing the three color channels. The estimated normal maps, angular error maps,
and error statistics are shown in Fig. 3(c-f). Overall, the brightness images yield better results than theimages of individual color channels, probably becauseof a higher signal-to-noise ratio (SNR) resulting fromsumming multiple measurements. However, in someregions, using an individual color channel yields a better estimate than the brightness images (see the redrectangles in Fig. 3 for example).
The result of combining normal map regions fromdifferent color channels is shown in Fig. 3(g). Each region from the three color channel images is evaluatedusing Eq. (4), giving the values shown in Table 1. Pixels with sharp highlights detected by intensity thresholding were discarded from the evaluation by removing
their rows from the observation matrix. With this multispectral technique, we achieve reconstructions that
surpass those derived from brightness images, despite
our technique not using summed measurements to improve the SNR.

3.2

Real images

We conducted an experiment with a real scene
captured using a fixed camera (Point Grey Research
Grasshopper), a halogen lamp (OSRAM), and nine

narrow band filters at 450, 488, 580, 650, 694, 730,
780, 880 and 940 nm (Edmond Optics). A translation
stage (Sigma) was used to switch the narrow band filters automatically. Images were recorded under twelve
different lighting directions. We applied photometricstereo to the images under perspective projection [16]to estimate the surface normals.
The target object in this experiment is a plushie
shown in Fig. 4, made of fabric of different colors.
Normal estimation using each wavelength The
normal maps of the plushie estimated from different
wavelengths are shown in Fig. 4. Subtle but visible
differences are apparent in the results obtained fromthe different wavelengths. Long wavelengths tend toyield smoother reconstruction results, possibly because
they penetrate deeper into the surface. By contrast,
shorter wavelengths often yield sharper surface details.
At certain wavelengths, such as 650 nm, surface normal
discontinuities may appear, coinciding with sharp color
gradients on the surface. These effects may indicate
problems in fitting the Lambertian model to the dataat these wavelengths.
From the surface normal reconstructions for the different wavelengths, we next determine which wavelengths are most consistent within the Lambertianmodel with our segmentation and evaluation scheme.
Segmentation and region evaluation Segmentation is performed using the nine color channels of ourmultispectral images. The segmentation result of thetarget object is shown in Fig. 5(a). The plushie is divided into nine different region types. Disconnectedareas on the image with the same region index aretreated collectively as one region.
After segmentation, Eq. (4) is used to determine theoptimal wavelength for each region. The values of theevaluation function for the different wavelengths areshown in Fig. 5(c) for each region. The evaluation
curves tend to be even over the color spectrum, possibly because the fabric material is less transparent,and hence more similar degrees of light penetration
into the surface at different regions and for different
wavelengths.
Combination of normal maps using evaluation
function values The evaluation function minima in
Fig. 5(c) yield the optimal wavelength for each region,
displayed in the label map in Fig. 5(b). Long wavelengths in the visible range are often selected.
A reasonable supposition is that the optimal wavelength corresponds to the surface color of a region,
since these wavelengths produce the brightest image
and a high SNR. However, our results suggest otherwise. For example, the optimal wavelength is in the
red spectral range even for a green surface, because of
the subsurface scattering properties of this region.
The normal map computed by our method from thelabel map and the normal maps at different wavelengths is shown in Fig. 4. Though the ground truthis unavailable for this object, our normal map appears
qualitatively more correct than the normal map computed from brightness images. With our method, thenormal map is less sensitive to texture.

Plushie

450nm

488nm

580nm

650nm

694nm

730nm

780nm

880nm

940nm

Brightness

Ours

Region index

Figure 4. Picture and normal maps estimated using different wavelengths, brightness images, by applying our method.
450
488
580
650
694
730
780
880
940

0
1
2
3
4
5
6
7
8

(a)

(b)

(c)

Figure 5. Our algorithm illustrated. (a) Nine regions segmented by k-means clustering in multispectral space. (b) Optimal wavelength for each region
identified by Eq. (4). (c) Evaluation function values
versus wavelength for each region.

4

Conclusion

In this paper, we proposed a new technique basedon multispectral images to improve Lambertian photometric stereo. The resulting normal map is obtainedby combining different normal maps computed at an
optimal wavelength determined in each region of theimage. Optimal wavelength determination does not
require ground truth geometry, but is based insteadon matrix rank analysis. Our experiments demonstrated significant wavelength-dependence of various
materials in photometric stereo reconstruction. Our
method demonstrably shows improvements over traditional photometric stereo based on brightness images.
Some materials have reflectance properties that differ significantly from the Lambertian model for any
wavelength. To broaden the applicability of our technique, future work will extend our approach to otherparametric reflectance models used in photometricstereo. In addition, we intend to investigate thewavelength-dependent reflectance of various objects forthe purpose of material recognition.
Acknowledgments
This research is granted by the Japan Society for
the Promotion of Science (JSPS) through the “Funding Program for Next Generation World-Leading Researchers (NEXT Program),” initiated by the Council
for Science and Technology Policy (CSTP).

References[1] R. J. Woodham, “Photometric method for determining surface orientation from multiple images,” OpticalEngineering, vol. 19, no. 1, pp. 139–144, 1980.
[2] W. Silver, “Determining shape and reflectance using
multiple images,” tech. rep., Master thesis, MIT, 1980.
[3] S. Barsky and M. Petrou, “The 4-source photometricstereo technique for three-dimensional surfaces in thepresence of highlights and shadows,” IEEE Trans. on
Pattern Anal. Mach. Intell., vol. 25, no. 10, pp. 1239–
1252, 2003.
[4] S. P. Mallick, T. E. Zickler, D. J. Kriegman, and P. N.
Belhumeur, “Beyond lambert: Reconstructing specular surfaces using color,” in Proc. CVPR, vol. 2, pp. 619–
626, IEEE, 2005.
[5] A. Georghiades, “Incorporating the torrance and sparrow model of reflectance in uncalibrated photometricstereo,” in Proc. ICCV, pp. 816–823, IEEE, 2003.
[6] D. B. Goldman, B. Curless, and A. H. S. M. Seitz,“Shape and spatially-varying brdfs from photometricstereo,” in Proc. ICCV, vol. 1, pp. 341–348, IEEE,
2005.
[7] H. Tagare and R. de Figueiredo, “A theory of photometric stereo for a class of diffuse non-lambertian surfaces,” IEEE Trans. on Pattern Anal. Mach. Intell.,vol. 13, no. 2, pp. 133–152, 1991.
[8] N. Alldrin and D. Kriegman, “Photometric stereo with
non-parametric and spatially-varying reflectance,” inProc. CVPR, IEEE, 2008.
[9] T. Higo, Y. Matsushita, and K. Ikeuchi, “Consensus
photometric stereo,” in Proc. CVPR, pp. 1157–1164,
IEEE, 2010.
[10] R. Anderson, B. Stenger, and R. Cipolla, “Color photometric stereo for multicolored surfaces,” in Proc.
ICCV, IEEE, 2011.
[11] P. H. Christensen and L. G. Shapiro, “Three-dimensional
shape from color photometric stereo,” International
Journal of Computer Vision, vol. 13, no. 2, pp. 213–
227, 1994.
[12] M. Liao, X. Huang, and R. Yang, “Interreflection removal for photometric stereo by using spectrum-dependent
albedo,” in Proc. CVPR, pp. 689–696, IEEE, 2011.
[13] M. Bass, ed., Handbook of Optics: Optical properties ofmaterials, nonlinear optics, quantum optics. OpticalSociety of America, 2009.
[14] W. Matusik, H. Pfister, M. Brand, and L. McMillan,“A data-driven reflectance model,” ACM Trans. on
Graph., vol. 22, no. 3, pp. 759–769, 2003.
[15] L. Wu, A. Ganesh, B. Shi, Y. Matsushita, Y. Wang,
and Y. Ma, “Robust photometric stereo via low-rankmatrix completion and recovery,” in Proc. ACCV, 2010.
[16] A. Tankus and N. Kiryati, “Photometric stereo under perspective projection,” in Proc. ICCV, vol. 1,
pp. 611–616, IEEE, 2005.