Noname manuscript No.
(will be inserted by the editor)

Multispectral Photometric Stereo for Spatially-Varying
Spectral ReectancesHeng Guo · Fumio Okura · Boxin Shi · Takuya Funatomi ·Yasuhiro Mukaigawa · Yasuyuki Matsushita

Received: date / Accepted: date

Abstract Multispectral photometric stereo (MPS)synthetic and real captured data with spatially-varying

aims at recovering the surface normal of a scene mea-reectance demonstrate the eectiveness of our method

sured under multiple light sources with dierent wave-

and show the potential applicability for multispectral

lengths. While it opens up a capability of a single-shot

heritage preservation.

measurement of surface normal, the problem has been

Keywords Multispectral photometric stereo · Cul-

known ill-posed. To make the problem well-posed, existing MPS methods rely on restrictive assumptions,tural asset · Spatially-varying spectral reectancesuch as shape prior, surfaces having a monochromaticwith uniform albedo. This paper alleviates these restrictive assumptions in existing methods. We show that the

1 Introduction

problem becomes well-posed for surfaces with uniform
chromaticity but spatially-varying albedos based on our

Photometric stereo is eective for the detailed re-

new formulation. Specically, if at least three (or two)covery of three-dimensional (3D) surfaces. Classicalscene points share the same chromaticity, the proposed

photometric stereo methods, originally proposed bymethod uniquely recovers their surface normals with

Woodham [32] and Silver [29], use images captured

the illumination of no less than four (or ve) spectral

from a xed camera under varying lighting directions,lights in a closed-form. In addition, we show that a morewhich are commonly obtained at dierent timestamps.

general setting of spatially-varying both chromaticities

Since conventional photometric stereo methods stack

and albedos can become well-posed if the light spectra

grayscale or RGB images

and camera spectral sensitivity are calibrated. For thistarget surface has to be static during the multiple shots,

general setting, we derive a unique and closed-form so-and the spectral property of the material is omitted inlution for MPS using the linear bases extracted fromthe estimated reectance.

1

a spectral reectance database. Experiments on both

H. Guo, F. Okura, Y. Matsushita
Osaka University, Graduate School of Information Scienceand Technology
E-mail: {heng.guo, okura, yasumat}@ist.osaka-u.ac.jp
B. ShiPeking University, National Engineering Research Center ofVisual Technology, School of Computer Science and Institutefor Articial Intelligence, Beijing Academy of Articial Intellignece, & Peng Cheng Laboratory
E-mail: shiboxin@pku.edu.cn
T. Funatomi, Y. Mukaigawa
Nara Institute of Science and Technology, Graduate Schoolof Science and Technology
E-mail: {funatomi, mukaigawa}@is.naist.jp

withtime-multiplexing, theWith multispectral photometric stereo (MPS) [19],detailed 3D shapes and the corresponding spectral reectances can be jointly recovered from a one-shot
multispectral image via

spectral-multiplexing. It is use-

ful not only for obtaining the object's shape for digital

heritagepreservation

[22]

but

also

for

studiesbased on spectral analysis, such as artwork material identication [26] and revealing underdrawings ofoil paintings [12]. However, unlike conventional timemultiplexing photometric stereo, MPS remains an illposed problem even with a Lambertian assumption. In

1 When RGB images are used in conventional (single-band)
photometric stereo, they are turned into grayscale images forcomputing surface normal.

2

Heng Guo et al.

Input

Multispectral photometric stereo setup

Output

LED 1

LED 3
LED 4

LED 2

425nm
White
LED array
LED 5

LED 6

Camera

LED 9

LED 10

485nm

505nm

Surface normal

Light direction distribution

LED 8

LED 7

450nm

520nm

550nm

590nm

635nm

675nm

705nm

730nm

Recovered shape

LED 11

LED 12

Hardware setup

Wavelength [nm]650nmLight spectra distribution

Spectral image observations

425nm

440nm

…

730nm

Spatially-varying spectral reflectanceFig. 1: Our multispectral photometric stereo setup with 12 narrow-band spectral LEDs placed at dierent locations.
Taking the spectral image observations as input, our method outputs a closed-form unique solution of both surface
normal and spatially-varying spectral reectance for heritage preservation.

this paper, we propose a method to make the prob-ectance regions is not only cumbersome but also fragile

lem tractable and show that a unique solution canto the outliers, such as shadows and specular highlights.

be obtained even for scenes with spatially-varying re-In this paper, we make MPS to work well underectances. With the proposed method, we assess its po-

spatially-varying spectral reectances. Given a multi-

tential applicability to digital heritage preservation.

spectral image under calibrated lighting directions, werst provide a closed-form MPS solution for surfaces

An input image for MPS encodes observations un-with uniform chromaticity but spatially-varying albe-

der dierent lighting directions in dierent spectral

dos (SRT III in Fig. 2), without relying on any addi-

bands, conveying the information about the surface

tional priors. We further extend our method to deal

normals and spectral reectances. Figure 1 shows ourwith the surface with spatially-varying chromaticities

MPS setup, which contains a camera and 12 narrow-and albedos (SRT IV in Fig. 2) by additionally calibrat-band spectral light sources located at dierent posi-

ing the light spectra and camera spectral sensitivity.

tions. From the input spectral image observations, ourSpecically, for SRT III surfaces, we treat the esti-

goal is to estimate both object shape and spectral re-mation of spectral reectance and surface normal as a

ectance simultaneously. However, under the illumina-

bilinear optimization problem. We show that the prob-

tion of f spectral lights, there are f +2 unknowns (f forlem can be turned into a homogeneous system of linear

the reectance of the spectral bands, and 2 for the sur-equations, where the surface normal and spectral re-face normal). Since only f observations for each sceneectances are jointly estimated. Given observations ofpoint are given, MPS is inherently under-constrained.

SRT IV surfaces and calibrated light spectra and cam-

To make the problem tractable, existing methods
use additional priors,

e.g ., initial shape [1, 2], trainedneural networks [1618], or local smoothness regularization [23]. However, these priors are rather restrictive and may not always comply with the actual scene.
Without these priors, existing methods [5, 25, 29] provide a unique solution for MPS by assuming the surface spectral reectance types (SRT) to be gray chromatic or monochromatic with uniform albedo (SRT I
and II in Fig. 2). However, these spectral reectanceassumptions are also restrictive for real-world scenes.
As shown in [5, 25], incorrect surface normals are estimated at surface regions with roughly constant chromaticity but continuously changing albedos. Previous
methods [5, 25] also investigated MPS for spatiallyvarying reectance (SRT IV) with the relaxation ofera spectral sensitivity, we show that closed-form solutions for both surface normal and spectral reectanceare given in a per-pixel manner. We achieve this byexpressing the spectral reectance with linear bases,
which are extracted from a material database of bidirectional reectance distribution functions (BRDFs) [9].
Unlike previous methods that are restricted to 3 spectral channels [1, 2, 5, 18, 25], our method allows the useof arbitrarily many spectral channels. As a side-bonusof this input property, we can also rely on the o-theshelf four or more source photometric stereo methods todeal with outliers, such as shadow and specular highlights, making our methods for both SRT III and IV
more robust than existing RGB-based MPS methods.
To summarize, the primary contributions of ourwork are as follows.

piece-wise constant chromaticities and albedos. How- We show that MPS for monochromatic surfaces with

ever, the spatial clustering of the uniform spectral re-spatially-varying albedos (SRT III) can be solved

Multispectral Photometric Stereo for Spatially-Varying Spectral Reectances

Targetsurface𝐫𝑖 : [𝑅𝑖 (𝜆1 ), … , 𝑅𝑖 (𝜆𝑓 )]…

Light 𝑓Ci (λ) = Ck (λ) = C̃(λ)

𝑅𝑖 𝜆 = 𝜌𝑖
𝑅𝑘 𝜆 = 𝜌𝑘
𝜆1 𝜆2 𝜆3 𝜆4

⋯

𝜆𝑓

𝜆

Wavelength

𝑅𝑖 𝜆 = 𝑅𝑘 𝜆= 𝜌෥ 𝐶෩ (𝜆)𝜆1 𝜆2 𝜆3 𝜆4

⋯

𝜆

𝜆𝑓

SV-chromaticity:

SV-albedo: ρi , ρk ∈ R+

Ci (λ) = Ck (λ) = C̃(λ)

Ci (λ), Ck (λ) : {R+ → R+ }SV-albedo: ρi , ρk ∈ R+

Spectral response

Uniform albedo: ρi = ρk = ρ̃

SRT IV

Mono-chromaticity:

Spectral response

Ci (λ) = Ck (λ) = 1
SV-albedo: ρi , ρk ∈ R+

𝑘Light 2SRT IIIi ∈ (1, p)𝑖Light 1SRT II

Mono-chromaticity:

Spectral response

Ri (λ) = ρi Ci (λ),

SRT I

Gray-chromaticity:

Spectral response

Spectral Reectance Type (SRT)3

𝑅𝑘 𝜆 = 𝜌𝑘 𝐶෥ (𝜆)𝑅𝑖 𝜆 = 𝜌𝑖 𝐶෥ (𝜆)⋯𝜆1 𝜆2 𝜆3 𝜆4

𝜆𝑓

𝜆

𝑅𝑘 𝜆 = 𝜌𝑘 𝐶𝑘 (𝜆)𝑅𝑖 𝜆 = 𝜌𝑖 𝐶𝑖 (𝜆)𝜆𝑓𝜆Wavelength

Wavelength

Wavelength⋯

𝜆1 𝜆2 𝜆3 𝜆4Fig. 2: Visualization of four spectral reectance types (SRT) categorized by the spatial distribution of the chromaticity C(λ) and the albedo ρ. The color maps provide spatial distribution examples of chromaticities and albedos
for each SRT in the RGB space. Solid and hollow dots show the spectral reectances of two scene points at f wavelengths r = [R(λ1 ), · · · , R(λf )]. This paper presents unique and closed-form solutions for both SRT III and IV.

in a closed-form without introducing any external

Table 1: Comparison of MPS methods. OursIII provides

priors, and we derive the minimal conditions baseda unique solution for a relatively general spectral re-on the number of spectral lights and scene points

ectance (SRT III) without additional priors. OursIV

for the problem to have a unique solution.

solves MPS for general spatially-varying spectral re- We introduce a basis representation for the spectral reectance and present a closed-form MPS so-

ectance with less restrictive calibration.

SRT Method

Input # Lights Additional priors

I
II
II
II
II

[29]
[19]
[8]
[13]
[5, 25]

MSI1
RGB
RGB
RGB
RGB

f ≥3
f =3
f =3
f =3
f =3

III

[30]

RGB

f =3

estimation thanks to that our method can take ar-IV[5, 25]RGB

f =3

bitrary many spectral channels as input.

IV [3, 1618] RGB
f =3
IV[1, 2]
RGBD3 f = 3

lution for surfaces with spatially-varying chromaticities and albedos if the light spectra and camera
spectral sensitivity are calibrated.

 Our methods for both SRT III and SRT IV are robust to outliers, such as shadows and specular highlights, because of its capability of applying robust

IV

[23]

MSI

f ≥3

IV
III

[10]
OursIII

MSI
MSI

f ≥5
f ≥4

for surfaces with a uniform chromaticity but spatially-IV

OursIVMSI

f ≥4

varying (SV) albedos (SRT III) without additional pri-

1

The preliminary version of this work appeared in
Guo

et al . [11] (denoted as OursIII ), which solves MPS

ors. However, this spectral reectance type is still limited to handle the general spectral reectance in thereal scene. Therefore, this paper extends [11] by pro-Multispectral image

2

None
Surface integrability
Surface integrability
Irradiance-normal mapping2
None
Initial coarse shape
Pixels with uniform albedo
Reectance quantization
Piece-wise constant reectance
Fixed lighting direction
Piece-wise constant chromaticity
Reectance smoothness
Surface normal smoothness
Spectral reectance basis1
None
Calibrated light and camera spectrum
Basis expression of the reectance

Scene-dependent calibration

3

RGB + depth

2 Related works

viding a unique and closed-form MPS solution (denoted as OursIV ) for surfaces with more general SV-

As described in previous works [14, 30], the material

chromaticities and albedos (SRT IV). To demonstratespectral reectancethe eectiveness of our new approach, additional ex-

posed into two parts:

periments on both synthetic and real data are also pre-R(λ) : R+ → R+ can be decomChromaticity C(λ) : R+ → R+
and albedo ρ ∈ R+ , such that R(λ) = C(λ)ρ, where λsented. Specically, in Sec. 4, we present a new for-represents wavelength. As shown in Fig. 2, based on

mulation to make MPS under SRT IV well-posed andthe spatial distribution of chromaticity and albedo forconvex by introducing a linear basis representation ofa surface, we categorize 4 dierent surface spectral re-the inverse spectral reectances. In Sec. 5, we updateectance types (SRT) and order them in a way fromthe experiments on synthetic data rendered with realis-

simple to complex. In this section, we introduce exist-

tic reectances to demonstrate the eectiveness of ouring methods based on their assumptions on SRT andmethods on both SRT III and SRT IV surfaces. In Sec.

list their properties for the comparison in Table 1.

6, we evaluate our methods on real captured images ofstatues and reliefs. In this way, we explore the poten-

SRT I If the surface has gray chromaticity, i.e ., thetial applicability of our MPS method of both SRTs on

chromaticity remains constant w.r.t. varying wave-

heritage preservation.

length, MPS is identical to classical photometric stereo.

4Heng Guo et al.

Therefore, given 3 or more spectral bands, a closed-form

Some recent methods directly take an RGB image

solution for surface normal can be obtained without am-

as input and apply deep neural networks to predict thebiguity [29].

surface normal [3, 16, 18]. However, the lighting directions are required to be consistent between the training

SRT II For monochromatic surfaces with uniform
albedo, i.e ., all the scene points share the common chromaticity C̃(λ) and albedo ρ̃, previous methods [8, 19]
show that the surface normal can be estimated from asingle RGB image up to a rotation ambiguity. The correct rotation was approximated by imposing an additional integrability condition. Hernández

et al . [13] es-tablish a one-to-one linear mapping between pixel measurements and surface normals to reconstruct the deformable cloth shape. This unknown linear mapping iscalibrated via a planar board with a cloth sample xedin the center. If the crosstalk between spectral channels
is negligible, existing methods [5, 25] provide a uniquesolution for surface normals. However, their methods
are restricted to RGB 3-channel input and cannot be
expanded to more channels (see the appendix).

and test procedures. Miyazaki

et al . [23] recover surface

normals from a multispectral image with more than
three channels. However, their recovered shape tends
to be over-smoothed due to the spatial smoothness assumption on both surface normal and the reectance.
Fye

et al . [10] assume the spectral reectance lies in a

low-dimensional space and represent it with a statistical basis set. However, their spectral reectance bases
are scene-dependent and need to be calibrated with the
known surface normal and reectance pairs. Besides,
the optimization of this method is non-convex and requires a good initialization.

Our method Taking a multispectral image with an arbitrary number of channels as input, we rst formulate
MPS for monochromatic surfaces with spatially-varying
albedos (SRT III) as a well-posed problem, and esti-

SRT III Few methods focus on the monochromatic

mate surface normal without introducing external priors [11]. We further show that MPS under SRT IV cansurfaces with spatially-varying albedos, which is com-be made tractable if the light spectra and camera spec-

monly seen in natural objects (tral sensitivity are calibrated. Dierent than existingand human skins. Vogiatzis

e.g ., wood and rocks)

et al . [30] assume the spec-

works [2, 5, 25], we avoid both the piece-wise uniform

tral reectance of the human face follows SRT III andspectral reectance restriction and the reectance clus-

obtain detailed reconstructions of faces in real-time.

tering steps by introducing a basis representation of the

However, their surface normal estimation results rely

per-pixel spectral reectance. Compared with Fyeeton the accuracy of initial geometry and detection of

al . [10], our formations for both SRT III and SRT IV are

equal-albedo pixels.

convex, and our extracted spectral reectance bases areshown to be scene independent based on the real data

SRT IV If the chromaticity and albedo are bothexperiments.

spatially-varying, MPS from a single multispectral image is ill-posed. Existing methods apply additional regularizations and provide numerical solutions for MPS.

3 MPS for surfaces with SRT III

et al . [5] and Ozawa et al . [25] relax theGiven a multispectral camera with a linear radiomet-

spatially-varying spectral reectance to be piece-wise

ric response and f (geometrically but not spectrally)

3-

calibrated spectral directional lights, we capture a mul-channel RGB inputs, they discretize the spectral re-tispectral image of p scene points on a Lambertian sur-ectance in a 3D space to cluster pixels with equal

face by turning on all the spectral lights. If the crosstalk

chromaticities and albedos so that they can turn the

between spectral bands is negligible,

problem into a set of SRT II subproblems. The normaltion under each spectral light is only observed in its

map is then estimated in each surface region that iscorresponding camera channel, observations mi ∈ R+

predicted as having the same spectral reectance. The

for the i-th pixel can be written as follows

Chakrabarti

constant. Since their methods are both based on

method by Anderson

et al . requires a coarse shape from

depth map [1] or stereo pairs [2] and uses it to guidei.e ., the observaf

mi = diag(ti ){Lni }+ ,the chromaticity segmentation and the surface normal

where ni ∈ S

estimation. Similar to [5, 25], the piece-wise constantvector,2(1)3

spectral reectance assumption restricts the exibility⊂ R represents the unit surface normal
L ∈ Rf ×3 stacks all the light directions. We
use diag(·) as a diagonalization operator and {·}+ asof the target surface's reectance. The normal estima-

a non-negative operator, which accounts for attachedtion accuracy is also inuenced by the errors introduced

shadows. For simplicity, we omit this operator {·}+ inby the reectance clustering step.

the following explanation. Here, ti ∈ R+ is related tofMultispectral Photometric Stereo for Spatially-Varying Spectral Reectances
the camera spectral sensitivity, light source spectra and5

3.1 Unique solution for SRT III

the surface spectral reectance at f spectral bands. Its
Suppose a surface with p scene points sharing the sameelement follows

Z
tij =

Ej (λ)Ri (λ)Sj (λ)dλ,

(2)

λ∈Ωj
where Ωj is the wavelength range of the j -th spectral
band, Ej (λ) : R+ → R+ denotes the spectra of the j th light, Sj (λ) : R+ → R+ denes the camera spectral
sensitivity at j -th channel, and Ri (λ) : R+ → R+ is thematerial spectral reectance of the i-th scene point. The
problem of general MPS is to estimate f + 2 unknowns
including t and surface normal n from f -element measurement vector m, which is unfortunately an ill-posed
problem.
We turn the MPS to be well-posed by assuming
the surface following SRT III: The material spectral response can be decomposed into a uniform chromaticityC̃(λ) and spatially varying albedos ρi , such thatRi (λ) = ρi C̃(λ).

chromaticity, by representing all pixels and lighting directions in a matrix form, we rewrite Eq. (4) as

M = QLN⊤ P,(6)

Q = diag(q) is an f × f diagonal matrix,
f ×p
M ∈ R+
records the image observations of p scene
p×3
points under f lights, N ∈ R
stacks all the surface
normals in a row-wise manner, P is a p × p diagonal
wherematrix with its diagonal element dened by pixel-wisespatially-varying albedos.
The above spectral image formation model has
a similar structure with semi-calibrated photometricstereo (SCPS) [6]. However, the task and physical image formation model between SCPS [6] and our method
for SRT III are dierent. SCPS [6] denotes q as light intensities and aims at solving conventional photometric(3)

stereo without calibrating the light intensity, whereas

Combing Eqs. (2) and (3), we rewrite the spectral image

ours focuses on the use of relatively general reectanceobservations for a scene point of the SRT III surface asassumption (SRT III) and multispectral image cues tomi = diag(q)ρi Lni ,(4)q ∈ Rf+ is the uniform reectance devoid ofwhere

spatially-varying albedos, whose elements are

Ej (λ)C̃(λ)Sj (λ)dλ.

tional priors. The unknown q in our method encodes
the integral of the light spectra, camera spectral sensitivity, and the chromaticity shared by the scene points,Z
qj =formulate MPS as a well-posed problem without addi-

(5)

λ∈Ωj
With the uniform chromaticity C̃(λ), q remains constant over the surface since both light spectra and camera spectral sensitivity are independent of the scene
points. With the surfaces of SRT III, we found the minimum conditions to yield a unique MPS solution foras shown in Eq. (5), which is dierent from the lightintensity notation in SCPS [6].
Given image observations

void of albedos Q, surface normal N, and albedo P byminimizing the following energy function:

2{Q∗ , N∗ , P∗ } = argmin M − QLN⊤ P F ,

(7)

Q,N,P

surface normal are as follows.

Theorem 1 Given f spectral observations under varying lighting directions of p scene points known to sharethe same chromaticity C̃(λ), their surface normals canbe uniquely determined if either one of the minimal conditions for the number of lightings and pixels is satised:

M and the calibrated

lighting directions L, we recover uniform reectance de-∥ · ∥F denotes the Frobenius norm. We dene
B = P⊤ N ∈ Rp×3 as albedo-scaled surface normals.
Here, Q is invertible since its diagonal elements are
where

non-zero. Then we rewrite Eq. (6) as

Q−1 M − LB⊤ = 0.

 Minimal pixel condition (MPC): p = 2, f ≥ 5,
 Minimal lighting condition (MLC): f = 4, p ≥ 3.

After vectorizing the unknown parameters

In other words, if two scene points share the same chro-(Ip ⊗ L)vec(B⊤ )(8)

Q−1 and⊤B , we obtain

maticity but varying surface normals, their surface nor-

− [diag(m1 ) · · · diag(mp )]⊤ Q−1 1 = 0,mals can be uniquely determined given 5 or more light-(9)vec(·) and ⊗ represent vectorization and Kro-ing directions. On the other hand, if we know 3 or more

wherescene points sharing the same chromaticity and theirnecker product operators, respectively. Ip ∈ R

surface normals are non-coplanar, we can recover theiridentity matrix, 1 is a all-one f -dimensional vector, mi

p×p

is an

normal directions with 4 or more spectral light sources.

is the i-th column vector of the image observations M,

In the following subsections, we present the unique so-indicating the measurement at the i-th pixel position.

lution for SRT III and provide the proof for minimalsolvable conditions MPC and MLC.

By concatenating all unknowns of Eq. (9) into avector, we obtain a homogeneous system of linear equa-6Heng Guo et al.

tions:

show that the MPS under this SRT IV is tractable if vec(B⊤ )⊤−Ip ⊗ L|[diag(m1 )| · · · |diag(mp )]= 0, (10)
−1
|
{z
} Q 1
|
{z
}
D

the light sources' spectra E and camera spectral sen-xwhere D ∈ R

pf ×(3p+f )the dimension of 3p + f . If D has 1d right nullspace, theS are calibrated in the form of a vector of⊤

their products e = [E1 (λ1 )S1 (λ1 ), · · · , Ef (λf )Sf (λf )]for, and the unknown vector x has

solution of x is obtained up to a scale via a factorizationofsitivityf distinct spectral bands. By denoting the ma-

terial reectances of corresponding spectral bands as

r = [R(λ1 ), · · · , R(λf )]⊤ , then the image formation
model for a pixel under f lights can be written as

D by singular value decomposition (SVD). Basedon the prior knowledge that surface normal has a unit
norm, we normalize albedo-scaled surface normals B inx to nally obtain a unique surface normal estimation.

m = diag(e)diag(r)Ln.

e, we compute the normalized image observations m̂ for a pixel by m̂ = m ⊘ e,
where ⊘ denotes element-wise division. Then MPS for
Given the calibrated

the SRT IV surface can be formulated as a bilinear op-

3.2 Minimal conditions for a unique solution

timization of per-pixel surface normal n and material

As discussed before, to obtain a non-trivial solution of
the homogeneous system in Eq. (10), the right nullspace

spectral reectance r:2{n∗ , r∗ } = argmin ∥m̂ − diag(r)Ln∥2 .

pf ≥ 3p + f − 1.
solvablecondition

(11)
can

be

interpreted

in

(15)

n,r

of D should be one dimension. Therefore, we have

This

(14)

an-

p pixels observed under f spectral
bands, the total number of measurements is pf . Sinceother way. Given

The problem still has

f constraints with f + 2 un-

knowns. We now show how this can be further madetractable by introducing the basis representation of thematerial reectances in the next section.

we assume a monochromatic surface with spatiallyvarying albedos, we only need to know the uniform

4.1 Unique solution for SRT IV

q for one pixel, whose
number of unknowns is f . For the remaining (p − 1)

To reduce the number of unknowns in Eq. (15) andpixels, we need to know albedos with the number ofmake the problem well-posed, a more compact repre-(p − 1). Besides, for each pixel, the surface normal has 2 degrees of freedom. There arethus 2p unknowns for surface normal. Totally, thenumber of unknowns is f + (p − 1) + 2p = 3p + f − 1.

sentation for the spectral reectance r is needed.

Since the number of measurements needs to be no lessone pixel satisfy

reectance devoid of albedos

unknowns

than the number of unknowns, we obtain the minimalsolvable condition of Eq. (11).
To further analyze the minimal requirement for the

We assume the spectral reectancer is non-zero

anywhere and dene an inverse spectral reectance asr̂ = 1 ⊘ r. Then the normalized image observations fordiag(m̂)r̂ − Ln = 0.

(16)

In this expression, the inverse spectral reectancer̂

number of lighting directions and pixels, we rewrite

lies in a f -dimensional space. We approximate it with

Eq. (11) ask (< f ) independent linear basis to reduce the number(f − 3)(p − 1) ≥ 2.

(12)

Therefore, the minimal requirements for the number of
input lighting directions and pixels to obtain a uniqueof unknowns,

i.e .,r̂ = Bc,(17)

where B ∈ R

k

f ×k

is a basis matrix stacking k basis vec-

solution for SRT III surfaces are

tors, c ∈ R

Eqs. (16) and (17), we formulate the bilinear optimiza-p = 2, f ≥ 5,
f = 4, p ≥ 3,

(13)

which correspond to MPC and MLC in Theorem 1.

4 MPS for surfaces with SRT IV

is the unknown basis coecients. Combingtion of Eq. (15) as a homogeneous linear system, n[−L|diag(m̂)B]
= 0,
|
{z
} c
|{z}
A

(18)

y

where A ∈ R

f ×(3+k), and y has the dimension of 3 + k .

A has one-

As discussed in the previous sections, general MPS for

Similar to OursIII discussed in Sec. 3, if

a surface with spatially-varying reectance (both chro-dimensional right nullspace, we can obtain a unique so-maticities and albedos) is ill-posed. In this section, we

lution y up to a scale by SVD. The estimated y are

Multispectral Photometric Stereo for Spatially-Varying Spectral Reectanceschosen as the right-singular vector corresponding to theLight distribution

7
Surface normal

Material: paper_yellow

smallest singular value of A. By incorporating the unit
norm constraint for the surface normal, we can nallyresolve the scale ambiguity and uniquely obtain the estimation of per-pixel surface normal and spectral reectance.

4.2 Spectral reectance basis extraction
Previous methods conduct linear analysis on MERLBRDF dataset [20] and express the reectances by thesmall number of coecients associated with the basis

Spectral reflectance R(𝜆)2Chromaticity: 𝐶(𝜆)vectors. However, their extracted bases are not suitablefor MPS as the spectral information is omitted. In thisLED 1 (400nm)

from a spectral BRDF database.
Dupuy

et al . [9] provided a measured spectral

BRDF dataset for 62 materials with 195 equi-spaced
spectral bins covering the360 ∼ 1000 [nm] range.

LED 11 (550nm)

LED 23 (735nm)

Observations

paper, we provide spectral reectance bases extracted

For each material, spectral responses for 8192 incidentoutgoing direction samples are provided. Since we assume the Lambertian model, the spectral reectances ofFig. 3: Synthetic multispectral image rendering of a

8192 directional samples for one material are treated asthat of 8192 Lambertian materials independently. By

Bunny surface. The measured spectral BRDF pa-

stacking the spectral response of all materials at one

response R(λ) under varying groups of surface normal

wavelength as a row vector, we build a spectral mate-

and light directions can be represented by a common

rial database G ∈ R

195×507904(=62×8192)

per_yellow roughly follows SRT III since its spectral

chromaticity C(λ) with varying scales (albedos).

.

With the wavelengths of f spectral lights calibrated,
we obtain the corresponding spectral material databaseG̃ ∈ Rf ×507904 by sampling the rows of G. To extract bases for the inverse spectral reectance r̂, weremove the materials with near-zero spectral responses
at any of the f wavelengths in G̃ and conduct SVD on

Ĝ = 1 ⊘ G̃ as

We calculate the numerical rank of

A following thethreshold strategy suggested in [31] and iteratively increase the number of bases in B from 1 to f − 3 until A
satises the rank requirement. Since our basis extraction is based on measured spectral BRDF dataset [9]containing various spectral reectance candidates in the⊤Ĝ = UΣV ,(19)

where U and V are the left and right orthogonal singular vectors, and Σ is a f × f diagonal matrix containing

real world, the obtained basis B is expected to t diverse scenes, as we will demonstrate it in the real dataexperiments.

the singular values in a descending order. The column
vectors of U provide orthogonal bases for the inverse
spectral reectance r̂.

5 Experiments on synthetic data
We here introduce experimental results on synthetic

Determining the number of bases Following

thedatasets. We rst describe the details of synthetic datathe rst k
f ×kcolumns of U as the basis matrix B ∈ R
to approx-

creation and the baseline settings. Then we compare

EckartYoungtheorem[15],

weselectOursIII and OursIV with the existing MPS methods.

imate the inverse reectance r̂. To obtain a non-trivial
solution of Eq. (18), the number of independent basis
vectors k should be selected to make A ∈ R

f ×(k+3)

has

a one-dimensional right nullspace. Therefore, the rankof A should satisfy

rank(A) = k + 2 < f.

5.1 Experimental settings

Synthetic dataset In our previous work [11], we haveveried that OursIII can accurately recover the sur-(20)

face normal on synthetic surfaces rendered with ideal

8Heng Guo et al.
Material distribution

Material appearance◦

than 45 . The light spectra of the LEDs are narrowband with the central wavelengths distributed evenly
in the range between 400 ∼ 750 [nm].
To render the reectance with SRT III, we choosea measured spectral BRDF paper_yellow [9], whose
appearance is visualized under a natural illumination in

Image observation

paper_yellow

Fig. 3. As shown in the middle row of the gure, we plot

acrylic_felt_purple

part of the spectral reectance curves R(λ) of the material under varying groups of surface normals and light
directions. It is clear that most reectance curves can beapproximated by scaling the thick yellow curve labeled
as chromaticity C(λ), except for a few curves. ThereSpectralon

fore, surfaces rendered with paper_yellow roughly

Colodur_kalahari_2ahave a uniform chromaticity but spatially-varying albedos (SRT III). Following the above rendering setting, weFig. 4: Synthetic rendering for the SRT IV surface. The

generate a synthetic multispectral image with 24 chan-

spectral reectance contains 4 materials as labeled bynels. The observations under LEDs 1, 11, and 23 are

the material distribution mask. The material appear-visualized in the bottom row.

ances are visualized under natural illumination [9].

OursIII

OursIV

OS18 [25]

CS16 [5]The material distribution labels in the left-top indicate

Est. normal

OursIII (f4 )To render the reectance with SRT IV, we select 4
dierent measured spectral BRDFs as shown in Fig. 4.
which BRDF to be applied to the regions on the Bunny
surface. We render a synthetic multispectral image under the 24 lights and visualize it by concatenating the

3.58

5.05

9.31

spectral channels illuminated by LEDs 1, 11, and 23, as

10.87

Error map

5.63

20

shown at the left-bottom of the gure.

0

Baselines As the baseline of the experiments, we selected two state-of-the-art MPS methods: CS16 [5] andFig. 5: Surface normal estimation results for an SRT III

OS18 [25], where we implemented OS18 [25] and usedsurface shown in Fig. 3.

released code of CS16 [5] for evaluation. Since bothmethods take a 3-channel (OursIV

OS18 [25]i.e ., RGB) image as input,

we selected 3 out of 24 spectral observations to mimic

CS16 [5]Est. normal

OursIIIthe 3-channel input image, as shown in Fig. 3. To verify the MLC, we tested our method for SRT III surfaces
by assigning the spectral channels recording the obser-

4.26

Error map

7.99

vations under LEDs 1, 11, 21, and 23, which cover the
19.67

observations used in OS18 [25] and CS16 [5] for compar-

25.37

40

ison. The number of piece-wise constant chromaticities
need to be set manually in CS16 [5]. To make a faircomparison, we set the number of chromaticities to be0

1 and evaluate their method, OursIII , and the SRT II
module of OS18 [25] in the experiments of SRT II and

Fig. 6: Surface normal estimation comparison on the

III surfaces. When making comparisons on SRT IV sur-SRT IV surface shown in Fig. 4.

faces, we use the default number of chromaticity clusters to 100 in CS16 [5], and compare it with OursIV andthe SRT IV module of OS18 [25]. Besides, in the syn-SRT III reectances. This paper gives a more realistic

thetic experiments, we remove the materials used in thesynthetic dataset with measured spectral reectances.

test data from the spectral reectance database when

Similar to the synthetic shape and lighting direction

extracting the bases for OursIV .

distribution in [11], we choose Bunny as our target

In the following, OursIII and OursIV are given ob-

shape and regularly sampled 24 synthetic light direc-

servations under all 24 lights by default. OursIII (f4 ) de-tions on a hemisphere with the elevation angle larger

notes our method for SRT III surfaces under MLC.

Multispectral Photometric Stereo for Spatially-Varying Spectral Reectances
5.2 Surface normal estimation under SRT III

9tributions. Our setup consists of 12 narrow-band spectral light sources and a monochromatic camera (FLIR

Using the ground-truth surface normal, we evaluatedBlacky S). The light sources are xed on a metal frame

surface normal estimation accuracy by mean angular

rig and distributed uniformly around the camera's op-

errors (MAE) in degree. Figure 5 shows the results oftical axis to avoid biased light distributions. We cali-surface normal estimation for a synthetic SRT III sur-brated the light directions with a monochromatic mir-face. OursIII achieves the smallest angular error com-ror ball following the method by Shi

pared to the other methods. The estimation errors oftral wavelength of our spectral light sources uniformly

OS18 [25] and CS16 [5] are mainly caused by theirspans in the range of

SRT II assumption and shadows. Also, the local polyno-

measured by a spectrometer Sekonic C-800. To verify

mial shape regularization used in CS16 [5] additionally

our method without the inuence of crosstalk acrossbrings in errors in regions with large surface normal

wavelength channels, we captured multiple images withvariations. OursIII (f4 ) under MLC is less accurate thana monochromatic camera by turning on each spectral

OursIII due to the inuence of shadows. However, com-light source one after another. Spectral observations

pared with OS18 [25] and CS16 [5], OursIII (f4 ) achieves

under LEDs 2, 4, and 10 with the central wavelength

higher accuracy with only one additional spectral ob-

faces. In this setting, OursIV is less accurate compared

450nm, 550nm, and 650nm are selected to mimic the
RGB input for existing 3-channel MPS methods. We
used 4 spectral observations under the illumination ofLEDs 2, 4, 9, and 11 to verify the MLC of our method

to OursIII due to its exible representation power for

for SRT III surfaces (OursIII ).

servation appended to the input. This result demonstrates the eectiveness of our method on SRT III sur-this restricted setting.

et al . [28]. The cen-

400 ∼ 750 [nm], and they are

To obtain the baseline surface normal (we call it the
ground-truth (GT) surface normal hereafter), we addi-

5.3 Surface normal estimation under SRT IV
Figure 6 shows the surface normal estimation resultsof a surface with spatially-varying spectral reectance(SRT IV). OursIV can handle spatially-varying chromaticities and albedos, therefore producing more accurate surface normal recovery compared to OursIII

tionally put an LED board that contains 256 white lightsources sharing the same spectrum, in a similar manner to CS16 [5]. The GT surface normal is estimatedusing a conventional Lambertian least-squares photometric stereo [32], and we use it for quantitatively assessing the MPS results.

that assumes the uniform chromaticity. Compared toSpectral calibration For our method for SRT IV sur-

OS18 [25] and CS16 [5], OursIV obtains the smallest

faces (OursIV ), light sources' spectraangular error since we do not assume piece-wise con-camera spectral sensitivity S1 , . . . , Sf need to be cal-stant spectral reectances and require no reectanceibrated in the form of a vector of their products

E1 , . . . , Ef andclustering. From the error map shown in Fig. 6, thee = [E1 (λ1 )S1 (λ1 ), · · · , Ef (λf )Sf (λf )]⊤ . For the cali-

error distribution of OursIV is more uniform and has

bration, we use a MacBeth ColorChecker board [21]less correlation to the material distribution comparedconsisting of 24 patches of uniform spectral reectances
R1 , . . . , R24 . Based on the image formation modelof Eq. (14), the ratio of the vector e's elements at neigh-to the other methods. This result shows the strength of
OursIV on surfaces with spatially-varying reectances.

boring spectral channels follows

6 Real-world experiment
To assess the eectiveness of the proposed methods, webuilt a multispectral photometric stereo setup to conduct experiments on real data. To verify the applicability of our methods on e-Heritage, we choose reliefs andstatues shown in Figs. 8 and 9 with diverse spectralreectances.

ej+1
mj+1 R(λj ) l⊤
j n
=
.
⊤
ej
mj R(λj+1 ) lj+1 n

For a scene point on the ColorChecker board, theR(λj )R(λj+1 ) under dierent wavelengths is known from measured spectral reectancespectral reectance ratio

curves [24]. The surface normal n of the ColorChecker
board can be estimated by the detected image corners
and camera intrinsics [35]. With calibrated lighting directions

6.1 Hardware setup(21)

L and the multispectral observations m, weestimate the elements of e up to scale by solving thehomogeneous system of equations derived from Eq. (21)24 monochromatic patches of the Cole up to

Figure 1 (left) shows our multispectral photometricusing all thestereo setup, lighting direction and light spectra dis-orChecker board. Since we can only recover10

Heng Guo et al.
Centralized albedo

GT chromaticity

× 1𝑒4 distribution
𝜎 = 0.18

SRT Ⅱ

Chromaticity distribution
MDS coord. 2

GT albedo

Frequency

Image observation GT surface normal

SRT Ⅲ

Frequency

× 1𝑒3
𝜎 = 1.02

MDS coord. 2

HEAD-RELIEF

LOVE-RELIEF

SRT ⅣFrequency

𝜎 = 1.13

BUDDHA-RELIEF

MDS coord. 2

× 1𝑒3

Centralized albedo

MDS coord. 1

Fig. 7: Ground-truth surface normal, chromaticity, and albedo of three real objects: Head-relief, Love-relief
and Buddha-relief, where the chromaticity is visualized by mapping 450nm, 550nm and 650nm responses to
BGR color channels, respectively. The spectral reectances for the three reliefs can be categorized as SRT II to
IV from top to bottom, as seen by their centralized albedo histograms and the distributions of the chromaticities
projected to the 2D space via MDS [7].

scale, the spectral reectance estimation by OursIV nat-with the image observations shown in the rst col-urally has a scale ambiguity, but that does not inuence

umn. On the other hand, the chromaticity distribu-

the recovery of surface normals.

tion of Buddha-relief is more diverse than those of

Head-relief and Love-relief, which indicates thespatially-varying chromaticity distribution in Buddha-

6.2 Real data setup

relief. As such, the spectral reectances of the three

Based on our hardware setup, we capture a variety ofobjects for real data experiments. Prior to the experiment, we examine the SRTs of the scenes by analyzing their spectral reectance distributions, as shownin Fig. 7. With calibrated e, known light directions L
and the ground-truth surface normaln, we computethe spectral reectance r based on the spectral image
formation model shown in Eq. (14). The estimated reectance r is further decomposed into the albedo andreal reliefs roughly follow SRT II, III, and IV.
We also observed that piece-wise constant spectralreectance assumption used in [2, 5, 25] is relatively unpractical to approximate the general SRT IV surfaces.
Although Buddha-relief seems to contain only threepiece-wise constant chromaticity regions from the image observation under natural illumination, it actually
has diverse chromaticities, making the monochromatic
region clustering [2, 5, 25] unstable.

chromaticity by taking its norm as albedo and its direction as chromaticity as depicted in Fig. 7 as GT albedo
and GT chromaticity, respectively. The chromaticity is

6.3 Surface normal estimation results on real datavisualized by mapping the responses at 450nm, 550nm,

Surface normal estimation under SRT III As shownand 650nm to BGR color channels, respectively.

in Fig. 8, we compare our methods with baselines on

The last two columns of Fig. 7 show the histogram

three objects: Head-relief, Love-relief, and Moaiof centralized albedo by subtracting the mean value,

statue. The Head-relief scene follows SRT II, andand low-dimensional visualization of chromaticity dis-Love-relief and Moai statue follow SRT III. Since

tributions via multidimensional scaling (MDS) [7], re-both existing methods [5, 25] and our methods (OursIII ,spectively. The Head-relief has a relatively uniform

OursIV ) can handle SRT II, the accuracy of recovered

albedo compared to the Love-relief and Buddha-

surface normals are comparable.

relief since its standard derivation σ of albedos is

We observed large normal estimation errors bysmaller than the other two. This is also consistent

CS16 [5] and OS18 [25] on the Love-relief and theMultispectral Photometric Stereo for Spatially-Varying Spectral Reectances(f4 )OursIII(f12 )

OursIV

OS18 [25]

CS16 [5]

6.24

7.95

7.70

Est. normal

OursIII

Head-relief

7.87

7.03

20

Error map

SRT IIGT surface normal

11

Love-relief

6.36

4.81

5.76

20.05

14.22

40
Error map

SRT III

Est. normal

0

Moai statue

14.16

12.61

11.61

35.22

33.83

40
Error map

SRT III

Est. normal

00Fig. 8: Surface normal estimation results for real-world objects with SRT II (Head-relief) and SRT III (Love-relief and Moai statue).

Moai statue, since the spatially-varying albedos vio-

OursIV provides comparable results with OursIII onlate the assumptions made in their methods. The error

both SRT II and III. However, OursIV requires the spec-

maps of CS16 [5] and OS18 [25] on the Love-relief

tral calibration of both lights and camera as well as thehighlight the error regions due to the non-uniform

spectral reectance bases. Therefore, it is preferred toalbedo distribution. On the other hand, OursIII yields

apply OursIII for monochromatic surfaces.

more accurate surface normal estimation results, whichveries our method's strength on SRT III surfaces. Under minimal solvable lighting conditions (MLC), theestimation errors of OursIII (f4 ) increase compared tousing all the

12 lights (OursIII (f12 ), which is mainly

caused by the shadows at the concave regions.

Surface normal estimation under SRT IV Figure 9shows surface normal estimation results of three SRT IV
surfaces: Buddha-relief, Lion, and Puppy. CS16 [5]and OS18 [25] assume the surface contains a limited
number of regions with uniform spectral reectances.
However, based on the distribution of albedos and chromaticities shown in Fig. 7, such assumption is invalid

12Heng Guo et al.
OursIII

OursIV

OS18 [25]

CS16 [5]

13.82

6.02

20.78

29.13

Est. normal

GT surface normal

Buddha-relief

Error map

40

Est. normal

0

22.94

Lion

11.24

29.65

20.64

Error map

40

Est. normal

0

30.38

Puppy

14.56

30.41

22.17

Error map

400Fig. 9: Surface normal estimation results for surfaces with spatially-varying chromaticities and albedos (SRT IV):

Buddha-relief, Lion, and Puppy.

Multispectral Photometric Stereo for Spatially-Varying Spectral Reectances
Surface normal

Image observations

Material

Shadings

13

OursIII (f4 )OursIII (f24 )OursIII (r)OursIV (f4 )OursIV (f24 )OursIV (r)CS16 [5]OS18 [25]

19.26

17.19

10.20

20.94

14.30

9.62

23.73

21.61

30

0
Fig. 10: Robustness against specular highlights. Top two rows show 7 sampled spectral images and their shadingsof a cow surface [28] covered by material cc_green_malachite [9], where yellow and green boxes indicates regionswith specular highlights and shadows. Surface normal estimates from existing methods and ours, the corresponding
angular error distributions, and the mean angular error values are shown in the bottom two rows.

in the Buddha-relief. Also, it is dicult to infer theput is restricted to 3-channel RGB image and cannot

number of distinct albedos and chromaticities in thetake more bands (see appendix). On the other hand,

Lion and Puppy from the image observation. There-

our methods for both SRT III and SRT IV surfaces canfore, both methods results in inaccurate surface normal

handle multispectral images withestimates for these scenes.

channels. This capability of taking many spectral chan-

OursIII cannot handle spatially-varying chromatici-

nels allows us to use a

4 or more spectral

robust estimation approach inties and outputs large errors on both scenes as well. On

MPS, in a similar spirit to four or more source photo-

the other hand, the proposed method OursIV achieves

metric stereo methods [4, 27, 33], to make our method

accurate results because it explicitly accounts for therobust against shadows and specular highlights. Intu-

SRT IV surfaces. From the error map, it is seen thatitively, having more spectral channels allows us to dis-inaccurate surface normal estimates are mainly locatedcard some of them that are corrupted by outliers.

at the regions where shadows are observed, and the surface normal estimation accuracy is not inuenced by thespatially-varying reectances in the results of OursIV .

To demonstrate this capability, we use a per-pixel
thresholding strategy used in [27, 28] to discard outliers
from the input observations. Specically, for each pixel,
we sort the observations under varying lights based on

7 Discussion

the brightness, and discard shadows and specular highlights as outliers that correspond to dark and bright

In this section, we discuss our method's robustness

observations (top and bottom 25%). The surface nor-against outliers and applicability to dynamic scene re-mal and the spectral reectance can then be estimatedconstruction.

using the inlier image observations. In the following,
we denote the robust versions of our SRT III method

7.1 Robustness against outliers

as OursIII (r), and our SRT IV method as OursIV (r).
In Fig. 10, we test the robust estimation meth-

Although previous methods [5, 25] provide a unique

ods in comparison to our non-robust versions andsolution for SRT II without external priors, their in-previous methods on a cow scene [28] with its re-14Heng Guo et al.
OursIII (f4 )OursIII (f12 )OursIII (r)OS18 [25]

CS16 [5]OursIV (f4 )OursIV (f12 )OursIV (r)OS18 [25]

CS16 [5]

DOG

Image
observation

SHELL

Image
observation

Fig. 11: Shape estimation results of two shiny objects, where Dog is monochromatic and Shell has spatiallyvarying chromaticities. Even rows show estimated surface normals. Odd rows provide reconstructed surfaces integrated from the surface normal maps. Closed-up views show the artifacts caused by the specular highlights.

BRDF

timation strategy, more accurate surface normals arecc_green_malachite [9]. With a few spectral chan-estimated by OursIII (r) and OursIV (r), illustrating thenels (3 spectral bands for CS16 [5] and OS18 [25],benet brought by our method's capability of taking

4 spectral bands for OursIII (f4 ) and OursIV (f4 )) inarbitrarily many channels as input.

ectance

assignedbyameasured

spectralthe multispectral image input, recovered surface nor-

We further evaluate our robust estimation methodmals are less accurate based on the mean angular er-

on real objects with shiny surfaces: Dog and Shellror values. From the error distributions, inaccurate sur-shown in Fig. 11. Since the reectances of the two

face normal estimates are closely related to the dis-

objects signicantly deviate from the Lambertian re-tributions of specular highlights and shadows, as in-ectance, we cannot trust the surface normal estimateddicated by the yellow and green boxes. By adding

from conventional least-squares photometric stereo [32]more spectral bands, the recovered surface normals

as the ground truth. Therefore, instead of comparing

from our methods (OursIII (f24 ), OursIV (f24 )) are im-

the surface normal maps, we applied a surface normal

proved. The accuracy of our method for SRT IV is

integration method [34] to reconstruct 3D shapes from

relatively better near the shadow areas even without

estimated surface normals for a qualitative comparison.

the robust strategy, since attached shadows are inherently embedded in the spectral BRDF database [9],from which we extract the BRDF bases. However, both
OursIII (f24 ) and OursIV (f24 ) still suer from the inuence of specular highlights. By further removing specularities and shadows as outliers using the robust es-

As shown in Fig. 11, the recovered surface shapefrom few spectral image observations is heavily inuenced by specular highlights. We also observe shape
distortions at the middle region of Shell in the result of OS18 [25] and CS16 [5]. These are causedbytheinaccurate

chromaticity

clustering

for

the

Multispectral Photometric Stereo for Spatially-Varying Spectral Reectances

15

520nm

590nm

635nm

OursIII

OursIV

OS18 [25]

CS16 [5]Inte. surface(side view)Inte. surface(font view)Est. normal

Spectralobservations

485nm

Fig. 12: Dynamic shape recovery of a deforming surface with SRT IV. The rst row shows the image observationsof a multispectral video frame at varying bands. The last three rows provide the estimated surface normals andintegrated surface visualizations at varying viewpoints. Close-up views highlight surface shape details.

spatially-varying reectances. By adding more spec-

relatively at due to the inuence of spatially-varying

tral bands under varying lighting directions as input

reectances, as shown in the side view of integrated(OursIII (f12 ) and OursIV (f12 )), shape recovery becomessurfaces. Also, the shape details are lost in CS16 [5]more plausible. However, artifacts caused by speculari-

due to the polynomial local shape constraint, as high-ties still remain. By further discarding outlier of spec-

lighted in the close-up views. On the other hand, theular highlights, more convincing shape reconstruction

surface normal estimates of OursIV are unaected byresults are obtained from the robust version of our

the spatially-varying spectral reectance. As a result,method (OursIII (r) and OursIV (r)).

OursIV achieves more reasonable dynamic shape recovery results on the deformable SRT IV surface.
The dynamic shape recovery from our method has

7.2 Dynamic shape recovery

a potential to capture 3D movement and gesture of thehuman body, which may benet the preservation of in-

We further test the applicability of our method to dy-tangible cultural heritages such as traditional dances.

namic scenes using an industrial multispectral camera
2

IMEC-SM-VIS , with which image observations at different spectral bands are obtained at once in one shot.
As shown in Fig. 12, we estimate the dynamic shapeof a deformable SRT IV surface in motion

3and com-pare the result with OS18 [25] and CS16 [5]. We choose
four pairs of spectral lights and camera channels having
the strongest response at 480nm, 520nm, 590nm, and635nm to obtain the multispectral input. Three out of
the four channels at 480nm, 520nm, and 635nm are usedas the input for OS18 [25] and CS16 [5]. The recovered
shapes (cushion) of OS18 [25], CS16 [5] and OursIII are

2 https://www.imec-int.com/en. Retrieved Mar. 11, 2021
3 Please refer to the supplementary video.

8 Conclusion
In this paper, we show that MPS can be turned intoa well-posed problem and provide unique solutions forsurface normals under two general spectral reectancetypes. Specically, if the surface has uniform chromaticity but spatially-varying albedos (SRT III), we showthat surface normal can be uniquely determined from4+ spectral observations without introducing external
priors. By further calibrating the light spectra and thecamera spectral sensitivity, we present a closed-form
solution of surface normal and spectral reectance for16Heng Guo et al.

surfaces with spatially-varying chromaticities and albe-

U-Shaped CNN. In: Proc. of IEEE Conference on

dos (SRT IV), using a low-rank basis representation ofComputer Vision and Pattern Recognition Work-the spectral reectance. Since our methods can takeshops (CVPRW), pp. 481483 (2019) 3, 4

more than 4 spectral channels, our method can rely on

4. Barsky, S., Petrou, M.: The 4-source photomet-

outlier rejection strategies in the MPS setting to eec-

ric stereo technique for three-dimensional surfaces

tively remove shadows and specular highlights. Fromin the presence of highlights and shadows.

the experiments on real objects containing statues andTransactions on Pattern Analysis and Machine In-

reliefs, we demonstrate the potential applicability of our
method to e-Heritage.

IEEE

telligence pp. 12391252 (2003) 13
5. Chakrabarti, A., Sunkavalli, K.: Single-image rgb
photometric stereo with spatially-varying albedo.
In: International Conference on 3D Vision (3DV),

8.1 Future work

pp. 258266 (2016) 2, 3, 4, 8, 9, 10, 11, 12, 13, 14,
15, 16, 17, 18

To obtain a surface shape from a single-shot image, we

6. Cho, D., Matsushita, Y., Tai, Y.W., Kweon, I.S.:

encode image observations under dierent illuminations

Semi-calibrated photometric stereo. IEEE Transac-

at dierent spectral bands. Compared to the setting

tions on Pattern Analysis and Machine Intelligencein CPS, this setting requires a negligible crosstalk ef-

pp. 232245 (2018) 5

i.e ., each spectral channel only records the

7. Cox, M.A., Cox, T.F.: Multidimensional scaling. In:

image measurement under the corresponding spectral

Handbook of data visualization, pp. 315347 (2008)

fect [5, 25],

light. From a practical viewpoint, it is wanted to deal

10with the non-negligible crosstalk eect, which alleviates

8. Drew, M.S., Kontsevich, L.L.: Closed-form attitude

the requirement of the hardware setting in MPS. Our

determination under spectrally varying illumina-

MPS method is based on Lambertian reectance as-tion.

sumption and treats specular highlights as outliers. It

Vision and Pattern Recognition (CVPR), pp. 985

is interesting to explore the MPS solution method un-

In: Proc. of IEEE Conference on Computer

990 (1994) 3, 4

der general non-Lambertian spectral reectances. Due

9. Dupuy, J., Jakob, W.: An adaptive parameteriza-

to the inaccessibility to actual heritage objects, we in-tion for ecient material acquisition and rendering.

stead veried our method's applicability to e-Heritage

ACM Trans. on Graph.

by real-world objects (Buddha relief, lion and Moai

37, 114 (2018) 2, 7, 8, 13,

14statues) that have similar appearances to the heritage

10. Fye, G., Yu, X., Debevec, P.: Single-shot photo-

objects. We are interested in applying our method tometric stereo by spectral multiplexing. In: Proc. ofreal heritages as soon as we have a chance in the future.

IEEE International Conference on Computatiosnal
Photography (ICCP), pp. 16 (2011) 3, 4

9 Acknowledgments
This work was supported by Japan Science and Technology Agency CREST Grant Number JPMJCR1764,
and National Natural Science Foundation of China under Grant Number 62136001, 62088102, and 61872012.

11. Guo,H.,

Okura,

F.,

Shi,

B.,

Funatomi,T.,

Mukaigawa, Y., Matsushita, Y.: Multispectral photometric stereo for spatially-varying spectral reectances: A well posed problem?

In: Proc. ofIEEE Conference on Computer Vision and Pattern
Recognition (CVPR) (2021) 3, 4, 7, 8
12. Hain, M., Bartl, J., Jacko, V.: Multispectral analysis of cultural heritage artefacts. Measurement Sci-

References
1. Anderson, R., Stenger, B., Cipolla, R.: Augmenting depth camera output using photometric stereo.
Proc. of IAPR Conference on Machine Vision andApplications (MVA) pp. 369372 (2011) 2, 3, 4
2. Anderson, R., Stenger, B., Cipolla, R.: Color photometric stereo for multicolored surfaces. In: Proc.
of International Conference on Computer Vision(ICCV), pp. 21822189 (2011) 2, 3, 4, 10
3. Antensteiner, D., Stolc, S., Soukup, D.: Single image multi-spectral photometric stereo using a split

ence Review
13. Hernández,

3, 912 (2003) 1
C.,

Vogiatzis,G.,

Brostow,

G.J.,

Stenger, B., Cipolla, R.: Non-rigid photometricstereo with colored lights. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition(CVPR), pp. 18 (2007) 3, 4
14. Hernández, C., Vogiatzis, G., Cipolla, R.: Overcoming shadows in 3-source photometric stereo. IEEE
Transactions on Pattern Analysis and Machine Intelligence pp. 419426 (2010) 3
15. Johnson, R.M.: On a theorem stated by eckart andyoung. Psychometrika

28(3), 259263 (1963) 7

Multispectral Photometric Stereo for Spatially-Varying Spectral Reectances

17

16. Ju, Y., Dong, X., Wang, Y., Qi, L., Dong, J.:

29. Silver, W.M.: Determining shape and reectanceA dual-cue network for multispectral photometricusing multiple images. Master's thesis, MIT (1980)

stereo. Pattern Recognition p. 107162 (2020) 2, 3,

1, 2, 3, 4
30. Vogiatzis, G., Hernández, C.: Self-calibrated, multi-

4
17. Ju, Y., Qi, L., He, J., Dong, X., Gao, F., Dong, J.:

spectral photometric stereo for 3d face capture. In-

MPS-Net: Learning to recover surface normal for

ternational Journal of Computer Vision pp. 91103

multispectral photometric stereo. Neurocomputing(2012) 3, 4
31. William, H., Teukolsky, S.A., Saul, A., Vetterling,pp. 6270 (2020)
18. Ju, Y., Qi, L., Zhou, H., Dong, J., Lu, L.: Demultiplexing colored images for multispectral photometric stereo via deep neural networks. IEEE Access

W.T., Flannery, B.P.: Numerical recipes: The artof scientic computing (2007) 7
32. Woodham, R.: Photometric method for determining surface orientation from multiple images. Op-

pp. 3080430818 (2018) 2, 3, 4
19. Kontsevich, L., Petrov, A., Vergelskaya, I.: Recon-tical Engineering pp. 139144 (1980) 1, 9, 14

struction of shape from shading in color images.

33. Wu, L., Ganesh, A., Shi, B., Matsushita, Y., Wang,

Journal of the Optical Society of America pp. 1047

Y., Ma, Y.: Robust photometric stereo via low-rank

1052 (1994) 1, 3, 4

matrix completion and recovery. In: Proc. of Asian

20. Matusik,W.:A

data-driven

reectancemodel.

Ph.D. thesis, Massachusetts Institute of Technol-

Conference on Computer Vision (ACCV), pp. 703
717. Springer (2010) 13
34. Xie, W., Zhang, Y., Wang, C.C., Chung, R.C.K.:

ogy (2003) 7
21. McCamy, C.S., Marcus, H., Davidson, J.G., et al.:Surface-from-gradients: An approach based on dis-A color-rendition chart. J. App. Photog. Eng

crete geometry processing. In: Proc. of IEEE Con-2(3),ference on Computer Vision and Pattern Recogni-

9599 (1976) 9
22. Miyazaki, D., Hara, K., Ikeuchi, K.: Median photometric stereo as applied to the segonko tumulus

tion (CVPR), pp. 21952202 (2014) 14
35. Zhang, Z.: A exible new technique for camera cal-and museum objects. International Journal of Com-ibration.

puter Vision

and Machine Intelligence86, 229 (2010) 1

IEEE Transactions on Pattern Analysis

22, 13301334 (2000) 9

23. Miyazaki, D., Onishi, Y., Hiura, S.: Color photometric stereo using multi-band camera constrainedby median lter and occluding boundary. Journalof Imaging p. 64 (2019) 2, 3, 4
24. Mohammadi, M., Nezamabadi, M., Berns, R.: A
prototype calibration target for spectral imaging.
Association Internationale de la Couleur (AIC)International Color Association (2005) 9
25. Ozawa, K., Sato, I., Yamaguchi, M.: Single color
image photometric stereo for multi-colored surfaces. Computer Vision and Image Understanding
pp. 140149 (2018) 2, 3, 4, 8, 9, 10, 11, 12, 13, 14,
15, 16, 17, 18
26. Picollo, M., Cucci, C., Casini, A., Stefani, L.:
Hyper-spectral imaging technique in the cultural
heritage eld: new possible scenarios.

20(10), 2843 (2020) 1

Sensors

27. Shi, B., Tan, P., Matsushita, Y., Ikeuchi, K.: Bipolynomial modeling of low-frequency reectances.
IEEE Transactions on Pattern Analysis and Machine Intelligence pp. 10781091 (2014) 13
28. Shi, B., Wu, Z., Mo, Z., Duan, D., Yeung, S.K., Tan,
P.: A benchmark dataset and evaluation for nonlambertian and uncalibrated photometric stereo.
IEEE Transactions on Pattern Analysis and Machine Intelligence pp. 271284 (2019) 9, 13

Appendix: Limitation of 3-band multispectral
photometric stereo methods
As described in Sec. 2, existing methods [5, 25] provide a
unique solution for the monochromatic surface with a uniform
albedo (SRT II) using 3-channel RGB images. These methods are limited to 3 channels, and it is not straightforward
to extend them to take more channels as input. However,the capability of taking more channels is favorable because
it allows us to use robust estimation techniques, eectively
neglecting outliers such as shadows and specular reections.
Here we show the reason why the existing methods are limited
to 3-channel input.
Without loss of generality, we x the common albedo ρ̃
for all the scene points on SRT II surface to be 1 and denea diagonal matrix Q as Q = diag(q). Following Eq. (4), theimage observation for a pixel can be represented as
m = QLn.

Dening MoorePenrose inverse matrix K ∈ R3×f
K = (QL)† , the surface normal is then calculated by
n = Km.

(22)
as
(23)

The existing methods [5, 25] use a unit norm constraint about
a surface normal asn⊤ n = m⊤ K⊤ Km = 1.

(24)

18

Heng Guo et al.

As shown in Eq. (25), by dening E = K⊤ K ∈ Rf ×f , eachone of the p scene points provides an equation about E as
 ⊤m0 Em0 = 1, m⊤ Em1 = 1,

1





..
.

(25)

m⊤
p Emp = 1.

Dening m ⊗ m = vec(mm⊤ ), we rewrite Eq. (25) in a matrix form,
m0 ⊗ m0
 m1 ⊗ m1   vec(E) = 1,..

| {z }.
ymp ⊗ mp
{z
}
|


(26)

G

where ⊗ represents the Kronecker product, and H forms a
p × f 2 matrix. Since E ∈ Rf ×f is symmetric, y only has
at most f (f2+1) distinct elements. We extract the elementsof y that correspond to the upper triangle elements from Ef (f +1)
as z ∈ R 2
and the corresponding columns from H asf (f +1)Ĥ ∈ Rp× 2 . Then we rewrite Eq. (26) as(27)

Ĥz = 1.

The necessary condition to obtain a unique approximate solution for z is Ĥ to have full-rank, i.e ., assuming p ≥ f (f +1),
rank(Ĥ) =f (f + 1)
.
2

(28)

On the other hand, since the image observations for all
the scene points under Lambertian reectance has the rank of
3, we can represent any irradiance measurements with three
independent basis {e1 , e2 , e3 ∈ Rf }, i.e .,m = c1 e1 + c2 e2 + c3 e3 .

(29)With this expression, we can represent m ⊗ m asm ⊗ m = (c1 e1 + c2 e2 + c3 e3 ) ⊗ (c1 e1 + c2 e2 + c3 e3 )= c21 (e1 ⊗ e1 ) + 2c1 c2 (e1 ⊗ e2 ) + c22 (e2 ⊗ e2 )(30)

+ 2c1 c3 (e1 ⊗ e3 ) + 2c2 c3 (e2 ⊗ e3 ) + c23 (e3 ⊗ e3 ).

It indicates that m ⊗ m can be represented by at most 6
independent f -dimensional basis vectors ei ⊗ ej . Since H in
Eq. (26) is a stack of m ⊗ m, the rank of H should satisfy
rank(H) ≤ 6.

(31)

Together with the necessary condition in Eq. (28) for solving
Eq. (27), it leads to the following inequality,f (f + 1)= rank(Ĥ) ≤ rank(H) ≤ 6,2(32)

which indicates that the number of spectral channels f of the
input multispectral image should be no more than 3. Therefore, these existing method [5, 25] cannot be adapted to multispectral images with more than three bands. On the other
hand, our method is free from this restriction.