Deep Photometric Stereo Network
Hiroaki Santo∗1 , Masaki Samejima†1 , Yusuke Sugano‡1 , Boxin Shi§2 , and Yasuyuki Matsushita¶1
1Graduate School of Information Science and Technology, Osaka University
2
Artiﬁcial Intelligence Research Center, National Institute of AIST

Abstract

it is known difﬁcult to directly work with general nonparametric BRDFs in the context of photometric stereo. To
ease the problem, there have been studies to use parametric representations to approximate BRDFs. However, sofar, known parametric models have been only accurate fora limited class of materials, and the solution methods suffer from unstable optimization, which prohibits obtaining
accurate estimates. Thus, it is needed to develop a photometric stereo method that is both computationally tractable
and capable of handling diverse BRDFs.
To achieve this goal, we propose an end-to-end learning
approach to photometric stereo using a deep neural network(DNN). The proposed method, which we call a deep photometric stereo network (DPSN), uses a DNN for establishinga ﬂexible mapping from reﬂectance observations to surface
normal. To make the DPSN applicable to diverse real-world
materials, a database of measured BRDFs (MERL BRDF
database [14]) has been used for training the network. In
addition, we propose a shadow layer that accounts for the
non-local shadowing effect using a dropout strategy.
In our method, we assume that the light directions are
pre-deﬁned and remain the same between training and prediction phases, which is the case in many photometric stereo
apparatuses. The DPSN operates in a per-pixel manner, by
taking reﬂectance observations of a surface point (corresponding to a certain pixel) under varying light conditions,and infers the surface normal of the point. The result showsthe effectiveness of our method validated using both simulation and real-world images.

This paper presents a photometric stereo method basedon deep learning. One of the major difﬁculties in photometric stereo is designing can appropriate reﬂectance modelthat is both capable of representing real-world reﬂectancesand computationally tractable in terms of deriving surface normal. Unlike previous photometric stereo methods that rely on a simpliﬁed parametric image formation
model, such as the Lambert’s model, the proposed method
aims at establishing a ﬂexible mapping between complex
reﬂectance observations and surface normal by the use of adeep neural network. As a result we propose a deep photometric stereo network (DPSN) that takes reﬂectance observations under varying light directions and infers the corresponding surface normal per pixel. To make the DPSN applicable to real-world objects, a database of measured bidirectional reﬂectance distribution functions (MERL BRDF
database) has been used for training the network. Evaluation using simulation and real-world scenes shows effectiveness of the proposed approach over previous techniques.

1. Introduction
Photometric stereo estimates surface normal of an object
from a set of measurements that are observed under different light conditions. The basic idea of photometric stereo
was introduced in 1980s by Woodham [25] and Silver [22]based on the Lambertian reﬂectance assumption [13]. To
make photometric stereo applicable to real-world objects, it
is of interest to use a more ﬂexible reﬂectance function, forwhich in a general form it is represented by bidirectional
reﬂectance distribution functions (BRDFs).
While the image formation model with a BRDF representation has greater ﬂexibility and representation power,

2. Related work
Conventional photometric stereo [25, 22] is based on
the Lambert’s reﬂectance model. Because the Lambert’s
model is an ideal reﬂectance model that may not represent
well real-world reﬂectances, extending photometric stereo
to work with non-Lambertian surfaces has been of interestfor its practical use. Existing studies on non-Lambertian
photometric stereo can be classiﬁed into three categories.
The ﬁrst category is methods based on robust estima-∗ santo.hiroaki@ist.osaka-u.ac.jp† samejima@ist.osaka-u.ac.jp‡ sugano@ist.osaka-u.ac.jp§ boxin.shi@aist.go.jp¶ yasumat@ist.osaka-u.ac.jp

1

tion, where the non-Lambertian reﬂectances are treated as
outliers. They assume that the majority of reﬂectance observations obeys, or is close to, the Lambert’s model sothat the non-Lambertian reﬂectances, such as specular reﬂections, can be regarded as anomalies. Wu et al. [26] formulate the robust estimation problem as rank minimization.
They exploit the fact that the Lambertian observations forma low-rank subspace [2] and treat the non-Lambertian reﬂectances as sparse outliers. Mukaigawa et al. [16] use therandom sample consensus (RANSAC) scheme for discarding outliers, which essentially approximates the 0 residual
minimization. Other robust estimation methods, such as expectation maximization [27], taking the median values [15],1 residual minimization and sparse Bayesian learning [10],are also shown effective for dealing with sparse outliers.
Since the robust estimation methods are built upon statistical outlier rejection, they generally require a lot of input
images, e.g., 40 images in [26], recorded under distinct light
directions.
The second category is methods based on more sophisticated reﬂectance models than the Lambertian model to better approximate non-Lambertian reﬂectance observations.
Georghiades [4] uses the Torrance-Sparrow model [24], andRuiters et al. [18] use Cook-Torrance model [3] in photometric stereo. More recently, Shi et al. [20] proposea bi-polynomial BRDF model, which is capable of representing low-frequency non-Lambertian reﬂectances, and it
shows greater accuracy in surface normal estimation. Holroyd et al. [8] propose another approach for generalizing
reﬂectance properties based on the reﬂective symmetry of
the halfway vector across the normal-tangent and normalbinormal planes, which does not require estimating a surface reﬂectance model, and performs well on anisotropic
reﬂectance surfaces.
The third category is example-based methods, which determines surface normal with reference objects. Hertzmann
and Seitz [6] propose an example-based method using a reference sphere that has the same reﬂectance as the target object. From the observations that are consistent between thetarget and reference objects, their method determines thesurface normal of the target object by simply mapping thecorresponding one from the reference object. The examplebased method naturally avoids solving a complex optimization problem, but it requires a reference object, of which
the shape is known and reﬂectance is the same as the target
object.
Our method is somewhere between the second and third
categories. As with the methods in the second category, ourmethod is able to deal with diverse BRDFs. Instead of estimating both the BRDF parameters and surface normal in theprevious approaches, our method directly establishes mappings from reﬂectance observations to surface normal using
a deep learning framework. Our DPSN is trained using a

database of measured BRDFs of various materials (MERL
BRDF database [14]); therefore it shares the spirit of theexample-based methods in the third category, while DPSN
does not require a reference object to be placed together
with the target object.

3. Preliminaries
When a Lambertian surface with albedo-scaled surface
normal n ∈ R3 is illuminated by a directional light l ∈ R31
the measurement m ∈ R+ can be described as
m = l n.
For a vector of measurements m ∈ Rf+ observed under fdistinct light directions, the above equation can be writtenwith a light matrix L = [l1 , . . . , lf ] ∈ R3×f as
m = L n.
The conventional photometric stereo method [25, 22] determines the surface normal n using the above image formation model byn∗ = L−1 m,when f = 3 and rank(L) = 3, or, with more than three
distinct observations, a least-squares approximate solutionn∗ can be obtained by a pseudo-inverse of L as
−1

Lm.
n∗ = LL
Unfortunately, a pure Lambertian surface rarely exists in thereal world; therefore, making photometric stereo work with
non-Lambertian surfaces is one of the major interests for its
practical use.
With a BRDF function ρ, the appearance of a surface under a local illumination model can be described more ﬂexibly. The appearances of a surface observed from a ﬁxed
viewing direction v under varying distant light directions L
can be written as
m = b ◦ (L n),
where b ∈ Rf+ is a vector of reﬂectances sampled from the
BRDF function ρ as b = ρ(L, n, v), and the◦ rep
 operator
resents element-wise multiplication, and L n represents
the irradiances at the surface point under the corresponding
light directions.
The above equation assumes a shadow-free world, whilein the real-world the surface patches facing away from thelighting direction are in attached shadow and light path being occluded causes cast shadow. Such shadowing processes can be written as(1)
m = s ◦ b ◦ max(L n, 0) ,1 Throughout this paper, we assume l = 1 and the input image has
2been normalized by its corresponding light intensity.

Shadow layer

Fully connected layers

݉ଶ
݉ଷ

݊௫
䞉䞉䞉

݉ସ

݊௬
呍呍呍

Dropout呍呍呍呍呍呍Measurement vector

݉ଵ

݊௭

݉௙
Figure 1. Overview of the proposed network. It consists of two
components, the shadow layer and fully connected layers. In the
shadow layer, some of the measurement vector elements are randomly dropped to simulate the cast shadow effect. In this ﬁgure,m1 and m4 are dropped and corresponding values of the input
vector are set to 0.
fwhere s ∈ {0, 1} is a boolean vector with 0 indicating
observations in cast shadows and 1 otherwise. The effectof attached shadow is accounted by the element-wise max
operator.

4. Proposed method
The proposed DPSN is a differentiable multi-layer neural network, which learns a mapping from a measurement
vector m ∈ Rf obtained under f different light directions to the surface normal n ∈ R3 . It operates in a perpixel manner for both training and prediction. As statedin the introduction, we assume that the light directions
L = [l1 , . . . , lf ] ∈ R3×f are known and consistent between the training and prediction phases. Our method uses
simulated observations that are generated using diverse surface normals rendered with the MERL BRDF database [14],which stores BRDFs of 100 different real-world materials.
In what follows, we explain the structure of the proposed
network, and training and prediction procedures.

shadow. Different from attached shadow, cast shadow is due
to a global illumination effect, which cannot be modeled bya local illumination model regardless of the representation
ability of a BRDF model. To simulate the cast shadow effect in the training phase, we introduce a shadow layer thatis based on a variant of the dropout scheme [23]. Dropout
is a technique, which randomly drops units from the network during training (or could be used for testing as well)to prevent learned weights from excessive adaptation. Our
shadow layer applies dropout to input nodes and randomly
drops a part of the input measurement vector, namely setting them to 0, so that the dropped nodes can be regarded
as shadowed observations. By training the network with theshadow layer, the proposed DPSN effectively learns mapping from observations to surface normal with accounting
for diverse BRDFs and cast shadow.
While in conventional dropout, output from the dropout
layer is scaled by 1/(1−r) with a dropout rate r ∈ [0.0, 1.0)
to avoid shrinkage of the output magnitude, our shadow
layer does not apply the scaling but simply sets the selected
elements of the measurement vector to 0 to mimic the shadowing effect. The dropout parameter r corresponds to theratio of shadowed observations in our context. Obviously,
the parameter r depends on the object shape and the light
distribution, which is inaccessible in general; therefore, we
use varying values of r for training. Speciﬁcally, we ﬂuctuate the dropout rate by sampling from a binomial distribution r ∼ B(f, p), where the probability of each observation
being shadowed p is set to p = 0.05.
The DPSN structure is summarized in Table 1. The
DPSN consists of 7 layers, shadow layer and 6 dense layers. The each dense layers include ReLU and Dropout during training. The DPSN is trained with the following loss
function
L = ||n − n̂||22 ,(2)

where n is the ground truth of normal vector and n̂ is thepredicted normal vector by the network. L is minimized
using Adam [12] with the suggested default settings.

4.1. Network architecture
The proposed DPSN learns the mapping from a measurement vector m of a pixel to the corresponding surface
normal n at that pixel using a fully connected deep neuralnetwork. The DPSN takes as input a measurement vectorm ∈ Rf+ , in which each element corresponds to an observation under a certain light direction, and outputs the prediction of the surface normal n ∈ R3 . The measurement
vector m is linearly normalized so that ||m||2 = 1.0 before
being fed to the network. While the MERL BRDF dataset
contains three color channels, we treat each color channel
independently.
One of the major challenges in photometric stereo is cast

4.2. Training
The training set for DPSN consists of pairs of an observation vector and the corresponding surface normal, i.e.,{(m, n)}. Instead of collecting real-world observations,
we generate the observation vectors {m} using the MERLBRDF dataset [14] for a diverse set of surface normal {n}viewed under pre-deﬁned light directions L. The MERLBRDF dataset consists of measured BRDFs of 100 different
materials, and we form reﬂectance vectors {b} in Eq. (1)from the set of surface normal {n} and light directions L.
We ignore the cast shadows when generating a training set,
which means the shadow mask s in Eq. (1) is set to 1.

Table 1. The DPSN structure. The number in the parentheses represents the number of nodes in each layer. The shadow layer isused in training, but not for prediction. Dropout rates for training
the Dense layers are set to all 0.5. The input and output dimensions of the shadow layer is 96(= f ) in our setting.

Rendered
image Normal map

Layer
1
2
3
4
5
6
7
blob01

Shadow Layer
Dense-(4096), ReLU, Dropout
Dense-(4096), ReLU, Dropout
Dense-(2048), ReLU, Dropout
Dense-(2048), ReLU, Dropout
Dense-(2048), ReLU, Dropout
Dense-(3)blob02

blob03

white-paint silver-metallic-paint red-plastic
Figure 2. Examples of rendered images that are used for training. blob01 to blob03 are a part of shapes from Blobby Shape
Dataset [11], “white-paint”, “silver-metallic-paint” and “redplastic” are the material names in the MERL BRDF database [14].
Here, three are shown out of 100 different materials. As seen in
the ﬁgures, the rendered images contain specularity and attachedshadows.

While any distributions of surface normal can be used
for generating the surface normal set {n}, for this work,
we used the Blobby Shape Dataset [11], which consists ofvarious shapes. The shapes are rendered with the MERL
BRDFs under light directions L, and observations at eachpixel location of the rendered images for each color channel form a measurement vector m. Figure 2 shows the sample images of training data generated under the certain light
directions. As shown in the ﬁgure, the rendered images
contain complex reﬂectance components that do not obey
a simple parametric model.

gether by taking the mean vector of the normalized surface
normal estimates as


nr
ng
nb
1
+
+
.
n̄ =
3 ||nr ||2||ng ||2||nb ||2
Finally, the merged surface normal n̄ is further normalized
to obtain the ﬁnal surface normal estimate n̂ as
n̂ =

n̄
.
||n̄||2

5. Experiments
We evaluate the experimental result of the proposed
method using both simulation and real-world datasets. We
will explain the training data and implementation details forboth experiments before showing the results.
Training data and implementation details For experiments, eight shapes out of ten in the Blobby Shape Dataset
are used for generating the training data, and we kept theremaining two shapes for testing. The shapes are rendered
under pre-deﬁned light directions for each of 100 BRDFs
from MERL dataset. The pre-deﬁned light directions are
the same as the 96 light directions deﬁned in the DiLiGenT dataset [21]. As a result, we generated 96 images
for each object and material; therefore, and in total we rendered 8 × 100 × 96 = 76800 images. The resolution of each
image is 256 × 192, and the total number of the training set{(m, n)} becomes about 3.9 × 107 .
DPSN was implemented using TensorFlow2 , and trainedfor 5000 steps with the batch size 1000. The model which
achieves the highest accuracy for test data is used for evaluation.
Evaluation procedure We compare the proposed DPSN
with Lambertian photometric stereo based on conventional2 residual minimization (L2) [25] and that with 1 residual
minimization (L1) [10]. For these methods, surface normal
n for each pixel is computed byminimizen ||m − L n||2F ,4.3. Prediction

andIn the prediction phase, given a set of observations under
the light directions L, DPSN estimates surface normal in a
per-pixel fashion. In a similar manner to the training phase,color channels are treated independently. For an RGB image, the DPSN estimates three surface normals per-pixel
and merges them to obtain the ﬁnal estimate. Namely, say
nr , ng , and nb are the surface normal estimates from the
RGB color channels that are independently estimated, we
merge them by the following procedure. We put them to-from observations m and known lighting directions L, respectively. For real-world scenes, we also assess the performance of our method using the DiLiGenT benchmark thatcovers the state-of-the-art methods of non-Lambertian photometric stereo. To see the effect of shadow layer, we compare DPSN without a shadow layer (denoted as “Proposed”)
and with a shadow layer (“Proposed W/ SL”).

minimizen ||m − L n||1

2 TensorFlow: https://www.tensorflow.org

Experimental result for synthetic dataset To generatescenes with unseen BRDFs that are not used for training,
we synthesize new BRDFs by applying nonlinear transformation over a linearly combined BRDFs that are sampledfrom the MERL BRDF database. Speciﬁcally, a new BRDF
ρ̃(L, n, v) is generated as the following:ρ̃(L, n, v) = (αρ1 (L, n, v) + (1 − α)ρ2 (L, n, v))γ ,(3)

where ρ1 and ρ2 are BRDFs sampled from the MERLBRDF database, and α and γ are constant parameters. In
this experiment, ρ1 and ρ2 are selected randomly fromthe database, and the parameters are set to α = 0.5 andγ = 0.8. For target shapes, we used three objects, sphere,
blob02, and blob08, which have not been used for training. We used the same light directions as the training for
generating the input data.
The evaluation results are summarized in Fig. 3. The
two material names used for ρ1 and ρ2 are shown in the topof the ﬁgure. “Lambert” means the Lambertian reﬂectance,
for which the reﬂectance function ρ is a constant. Since
the generation of synthetic scenes neglects rendering of castshadows, our method without a shadow layer (“Proposed”)
shows superior performance to the one with the shadow
layer (“Proposed W/ SL”) in most scenes.
Prediction using real-world dataset Then we evaluate
the performance of our method using the DiLiGenT dataset.
The DiLiGenT benchmark includes the evaluation of thefollowing eight methods: WG10[17], IW12 [10], GC10 [5],AZ08 [1], ST12 [19], HM10 [7], ST14 [20], and IA14 [9].
We add our results in addition to these eight methods, as
shown in Fig. 4 and Table 2.
Figure 4 shows the estimated normal maps and corresponding error maps for “Proposed”, “Proposed W/ SL”,
“L2”, and “L1”. Here, we show 6 objects out of 10 in the
DiLiGenT dataset (buddha, cow, goblet, harvest,
pot2 and reading). buddha, pot2 and reading
are objects made of pottery material, whose reﬂections aremostly Lambertian except for some specular outliers; therefore, “L2” and especially “L1” based on methods work
well. For objects like cow and goblet, which are madeof metallic materials and exhibit strong specular reﬂection
in a wide area, the estimation error becomes rather large forLambertian based methods such as “L2” and “L1”, while
the proposed method can yield highly accurate estimates.
For harvest, the estimation accuracy is poor in all methods due to the complexity of the shape of the object. For
objects with complicated shapes, various reﬂection phenomenon besides cast shadow such as interreﬂections occur.
We discuss more about this issue in following section.
The effect of the shadow layer Figure 5 shows the difference map of error map between “Proposed” and “Pro-

posed W/ SL”. Here, we pick up 4 objects (ball, pot2,
goblet and harvest) as examples. For all these objects,
the accuracy is generally improved in boundary areas where
shadows often occur. The accuracy improves in larger areas
for ball and pot2 than for goblet and harvest. For
metallic objects with strong interreﬂections like goblet
and harvest, the measurement value is larger than 0 for
shadows, as the example shown in Fig. 6. Shadow layer
assumes that measurement values in the shadowing parts
become 0, so it shows degraded performance when strong
interreﬂections exist.
Discussion on shadowing probability p In the above
section, we use the result of shadowing probability p =0.05, however, the optimal p depends on the shape of object.
In this section, we discuss the effect of shadowing probability p. We show the results of p = 0.05 and p = 0.9 on ballin Fig. 7. Comparing (a) and (b), we can see that although(b) improves more in the peripheral parts of sphere (shadowing parts) than (b), it deteriorates in the central part.
With shadow layer, the model is optimized for the
shadow, and considers that the accuracy deteriorates in thearea where the shadow does not exist. In the case of
p = 0.05, since the dropout rate r sampled in the manner of Sec. 4.1 can be 0, the estimation accuracy does notdeteriorate in areas without shadow. On the other hand, inthe case of p = 0.9, dropout would be applied to all inputs,
so the accuracy deteriorates in the area where the shadow
does not exist. Based on such observation, we use p = 0.05
in the evaluation.
Benchmark comparison In this section, we compare
proposed methods (“Proposed” and “Proposed W/ SL”)
with benchmark results shown in [21]. Table 5 shows the
Mean Angular Error (MAE) in degree of each method foreach objects and the average of them. Green color represents good result and Red color represents bad result. For
each objects, the best result is highlighted using bold font.
For most objects, the best result is obtained by existing
methods, but “Proposed W/ SL” signiﬁcantly improves theaccuracy of harvest and obtains the best result of the average over all objects. This results show that DPSN generally achieves high accuracy for objects consisting of various
materials (real BRDF), which are not seen during training(synthetic BRDF). The high accuracy is partially due to thatDPSN an handle the cast shadow effectively, which is notaddressed by most existing methods.

6. Discussion
This paper proposed a photometric stereo method basedon deep learning. The proposed method uses deep neural network for establishing a ﬂexible mapping from shading observations to surface normal. Inspired by the suc-

blue-fabriccherry-235++
silver-metallic-paint natural-209(blob02)(blob02)

beige-fabric black-soft-plastic++
yellow-paint blue-metallic-paint(blob08)(blob08)

Lambert(blob02)

0.87

0.76

1.33

1.70

1.20

3.21

1.44

1.31

1.50

1.29

1.22

3.31

1.48

0.41

18.68

12.67

3.26

19.76

L2

Proposed W/ SL

ProposedGT

Lambert(sphere)L1

180 deg.

0.47

0.05

15.54

8.52

3.10

18.25

0 deg.

Figure 3. Experimental result for synthetic scenes. In each row, a normal map is shown on top of an error map. The numbers represent
Mean Angular Error (MAE) in degree. In the top row, GT means the ground truth, the images below the normal maps are examples ofobservation images. On the top, material and shape names that are used for synthesizing the data are displayed.

cow

gobletharvestpot2

reading

13.30

8.49

12.35

16.81

8.80

17.47

12.68

8.01

11.28

16.86

7.86

15.51

14.92

25.60

18.49

30.60

14.65

19.50

L2

Proposed W/ SL

ProposedGT

buddhaL1

180 deg.

12.19

23.77

16.00

28.32

13.11

14.01

0 deg.

Figure 4. Estimation result for real-world scenes from DiLiGenT [21]. In each row, a normal map is shown on top of an error map. The
numbers represent Mean Angular Error (MAE) in degree. GT means “ground truth” and below ﬁgures are one of observation images. The
contrast of observation images is adjusted for easy viewing.

Table 2. Comparison with benchmark [21].

Proposed
Proposed W/ SL
ST14
IA14
WG10
AZ08
HM10
IW12
ST12
GC10
BASELINE

ball
3.44
2.02
1.74
3.34
2.06
2.71
3.55
2.54
13.58
3.21
4.10

cat
7.21
6.54
6.12
6.74
6.73
6.53
8.40
7.21
12.34
8.22
8.41

pot1
7.90
7.05
6.51
6.64
7.18
7.23
10.85
7.74
10.37
8.53
8.89

bear buddha cow
7.20 13.30 8.49
6.31 12.68 8.01
6.12 10.60 13.93
7.11 10.47 13.05
6.50 10.91 25.89
5.96 12.54 21.48
11.48 13.05 14.95
7.32 11.11 25.70
19.44 18.37 7.62
6.62 14.85 9.55
8.39 14.92 25.60

goblet harvest pot2 reading AVG.
12.35 16.81 8.80 17.47 10.30
11.28 16.86 7.86 15.51 9.41
10.09 25.44 8.78 13.63 10.30
9.71 25.95 8.77 14.19 10.60
15.70 30.01 13.12 15.39 13.35
13.93 30.50 11.03 14.17 12.61
14.89 21.79 16.37 16.82 13.22
16.25 29.26 14.09 16.17 13.74
17.80 19.30 9.84 17.17 14.58
14.22 27.84 7.90 19.07 12.00
18.50 30.62 14.65 19.80 15.39
32 (worse)

0 [deg.]ball

gobletharvestpot2-32 (better)

Figure 5. Improvement by Shadow Layer. We show difference map of error map between “Proposed” and “Proposed W/ SL”. Pixels whose
normal estimation accuracy is improved by shadow layer are shown in blue, otherwise in red.

gobletharvestFigure 6. We adjust the left half of red box area with the gamma
correction. The shadowing parts in the box shows larger value than0 due to strong interreﬂections.

32 (worse)

0 [deg.](a)(b)-32 (better)

Figure 7. Comparison shadowing probability p on “Ball”. (a) is
p = 0.05 (“Proposed W/ SL”) and (b) is p = 0.9. Both are difference maps of error map with “Proposed”. Pixels whose normal
estimation accuracy is improved by shadow layer are shown inblue, otherwise in red.

cess of example-based methods to deal with challenging
reﬂectance, we use modern neural network structures andtrain it end-to-end with simulated observations rendered by
the MERL BRDF database [14] to replace the functionality of reference objects. In addition, we proposed a shadow
layer that simulates the non-local shadowing effect using
the dropout strategy. Evaluation using real-world scenes
shows more accurate estimation of the surface normal incomparison to previous techniques.
Limitations One of limitations in our method is the assumption that light directions are pre-deﬁned and remain
the same between training and test phases. Our methodneeds to train models for each lighting condition, because
our training requires simulated observations rendered under a speciﬁc lighting condition. Our method is particularly
useful if the photometric stereo data capture uses a device
with ﬁxed light sources and camera, so that we only need toperform the training for that device. It is our future work todeal with lighting condition which is not pre-deﬁned, suchas the data capture using a randomly waving light source.

Acknowledgement
This work was supported by JSPS KAKENHI Grant
Number JP16H01732.

References[1] N. Alldrin, T. Zickler, and D. Kriegman. Photometric stereo
with non-parametric and spatially-varying reﬂectance. In
Proceedings of IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), pages 1–8. IEEE, 2008. 5[2] R. Basri and D. Jacobs. Lambertian reﬂectance and linear
subspaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(2), 2003. 2[3] R. L. Cook and K. E. Torrance. A reﬂectance model forcomputer graphics. ACM Transactions on Graphics, 1(1):7–
24, 1982. 2[4] A. Georghiades. Incorporating the torrance and sparrowmodel of reﬂectance in uncalibrated photometric stereo.
pages 816–823, 2003. 2[5] D. B. Goldman, B. Curless, A. Hertzmann, and S. M. Seitz.
Shape and spatially-varying brdfs from photometric stereo.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(6):1060–1071, 2010. 5[6] A. Hertzmann and S. M. Seitz. Example-based photometric stereo: Shape reconstruction with general, varying brdfs.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(8):1254–1264, 2005. 2[7] T. Higo, Y. Matsushita, and K. Ikeuchi. Consensus photometric stereo. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1157–
1164. IEEE, 2010. 5[8] M. Holroyd, J. Lawrence, G. Humphreys, and T. Zickler. A
photometric approach for estimating normals and tangents.
ACM Transactions on Graphics (Proceedings of SIGGRAPH
Asia 2008), 27(5):133, 2008. 2[9] S. Ikehata and K. Aizawa. Photometric stereo using constrained bivariate regression for general isotropic surfaces.
In Proceedings of IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), pages 2179–2186, 2014. 5[10] S. Ikehata, D. Wipf, Y. Matsushita, and K. Aizawa. Robust
photometric stereo using sparse regression. In Proceedingsof IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 318–325. IEEE, 2012. 2, 4, 5
[11] M. K. Johnson and E. H. Adelson. Shape estimation innatural illumination. In Proceedings of IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), pages
2553–2560, 2011. 4[12] D. Kingma and J. Ba. Adam: A method for stochastic
optimization. Proceedings of International Conference on
Learning Representations (ICLR), 2014. 3[13] J. Lambert. Photometria. Augustae Vindelicorum, 1760. 1[14] W. Matusik, H. Pﬁster, M. Brand, and L. McMillan. A datadriven reﬂectance model. ACM Transactions on Graphics,
22(3):759–769, July 2003. 1, 2, 3, 4, 8
[15] D. Miyazaki, K. Hara, and K. Ikeuchi. Median photometric stereo as applied to the segonko tumulus and museum
objects. International Journal of Computer Vision, 86(2–3):229–242, 2010. 2[16] Y. Mukaigawa, Y. Ishii, and T. Shakunaga. Analysis of photometric factors based on photometric linearization. JOSA A,
24(10):3326–3334, 2007. 2[17] T. Papadhimitri and P. Favaro. A closed-form, consistent androbust solution to uncalibrated photometric stereo via local
diffuse reﬂectance maxima. International journal of computer vision, 107(2):139–154, 2014. 5[18] R. Ruiters and R. Klein. Heightﬁeld and spatially varying
brdf reconstruction for materials with interreﬂections. Computer Graphics Forum, 28(2):513–522, Apr. 2009. 2[19] B. Shi, P. Tan, Y. Matsushita, and K. Ikeuchi. Elevation angle
from reﬂectance monotonicity: Photometric stereo for general isotropic reﬂectances. Proceedings of European Conference on Computer Vision (ECCV), pages 455–468, 2012.
5[20] B. Shi, P. Tan, Y. Matsushita, and K. Ikeuchi. Bi-polynomial
modeling of low-frequency reﬂectances. IEEE Transactions
on Pattern Analysis and Machine Intelligence, 36(6):1078–
1091, 2014. 2, 5
[21] B. Shi, Z. Wu, Z. Mo, D. Duan, S.-K. Yeung, and P. Tan.
A benchmark dataset and evaluation for non-lambertian anduncalibrated photometric stereo. In Proceedings of IEEE
Conference on Computer Vision and Pattern Recognition(CVPR), 2016. 4, 5, 7, 8
[22] W. M. Silver. Determining shape and reﬂectance using multiple images. Master’s thesis, Massachusetts Institute of Technology, Cambridge, MA, U.S., 1980. 1, 2
[23] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and
R. Salakhutdinov. Dropout: A simple way to prevent neuralnetworks from overﬁtting. The Journal of Machine Learning
Research, 15(1):1929–1958, 2014. 3[24] K. E. Torrance and E. M. Sparrow. Theory for off-specular
reﬂection from roughened surfaces. JOSA, 57(9):1105–
1114, 1967. 2[25] R. J. Woodham. Photometric method for determining surface orientation from multiple images. Optical engineering,19(1):139–144, 1980. 1, 2, 4[26] L. Wu, A. Ganesh, B. Shi, Y. Matsushita, Y. Wang, and
Y. Ma. Robust photometric stereo via low-rank matrix completion and recovery. Proceedings of Asian Conference on
Computer Vision (ACCV), pages 703–717, 2011. 2[27] T.-P. Wu and C.-K. Tang. Photometric stereo via expectation
maximization. IEEE Transactions on Pattern Analysis andMachine Intelligence, 32(3):546–560, 2010. 2