This is the authorsʼ version of the work. It is posted here by permission of IEEE for personal use. Not for redistribution.
The final authenticated publication is available online at DOI: 10.1109/TPAMI.2020.3005219.
TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

1

Deep Photometric Stereo Networks for
Determining Surface Normal and Reflectances
Hiroaki Santo, Masaki Samejima, Yusuke Sugano, Boxin Shi, and Yasuyuki Matsushita
Abstract—This paper presents a photometric stereo method based on deep learning. One of the major difficulties in photometricstereo is designing an appropriate reflectance model that is both capable of representing real-world reflectances and computationallytractable for deriving surface normal. Unlike previous photometric stereo methods that rely on a simplified parametric image formation
model, such as the Lambert’s model, the proposed method aims at establishing a flexible mapping between complex reflectanceobservations and surface normal using a deep neural network. In addition, the proposed method predicts the reflectance, which allows
us to understand surface materials and to render the scene under arbitrary lighting conditions. As a result, we propose a deep
photometric stereo network (DPSN) that takes reflectance observations under varying light directions and infers the surface normal
and reflectance in a per-pixel manner. To make the DPSN applicable to real-world scenes, a dataset of measured BRDFs (MERL
BRDF dataset) has been used for training the network. Evaluation using simulation and real-world scenes shows the effectiveness ofthe proposed approach in estimating both surface normal and reflectances.
Index Terms—Photometric stereo, surface normal, bidirectional reflectance distribution functions (BRDFs), deep learning

F

1

I NTRODUCTIONPHOTOMETRIC stereo estimates surface normal of a scene froma set of measurements that are collected under different light
conditions. The basic idea of photometric stereo was introduced in
1980’s by Woodham [1] and Silver [2] based on the Lambertian [3]reflectance assumption. To make photometric stereo better applicable to real-world objects, it is of interest to use a more flexible
reflectance model, for which in a general form it is represented
by bidirectional reflectance distribution functions (BRDFs). In
addition to determining surface normal, it is desired to estimate
the reflectance function and its associated parameters rather thanassuming that they are known a priori.
While an image formation model with a general reflectance
representation based on BRDFs has great flexibility and descriptive power, it is difficult to directly work with general nonparametric BRDFs in the context of photometric stereo. This isdue to that the photometric stereo is an inverse problem, in whichsurface normal is determined based on image measurements as
input. The image formation is governed by reflectance functions,and it is desirable that the reflectance function is represented bya simple model so that the inverse estimation problem becomes
tractable while retaining the representation power. Along with thisdirection, there have been studies to use parametric representations
that are more flexible than the Lambert’s model, e.g., TorranceSparrow model [4], [5], microfacet-based model [6], and bipolynomial model [7], to better approximate complex real-world•••

H. Santo, M. Samejima, and Y. Matsushita are with Graduate School ofInformation Science and Technology, Osaka University, Osaka, Japan.
E-mail: {santo.hiroaki, samejima, yasumat}@ist.osaka-u.ac.jp
Y. Sugano is with Institute of Industrial Science, The University of Tokyo,
Japan.
E-mail: sugano@iis.u-tokyo.ac.jp
B. Shi is with the National Engineering Laboratory for Video Technology,
Department of Computer Science and Technology and Institute for Artificial Intelligence, Peking University, Beijing, China.
E-mail: shiboxin@pku.edu.cn

Manuscript received September 2, 2018; revised May 12, 2020.

reflectances. However, so far, parametric models have been only
accurate for a limited class of materials, and the solution methods suffer from unstable non-convex optimization for estimatingreflectance parameters, which prohibits obtaining accurate surface
normal. Indeed, estimation of reflectance is an important task inphotometric stereo because it is tightly coupled with the estimationof surface normal. In addition, correctly estimated reflectanceprovides important information for understanding the materialsof the scene, and opens up a capability of re-rendering the scene
under arbitrary light conditions that are unseen. Thus, it is neededa photometric stereo method that can recover surface normal of ascene with diverse reflectances as well as estimating the spatiallyvarying reflectances of the scene.
To achieve this goal, we propose an end-to-end learning approach to photometric stereo using a deep neural network (DNN).
The proposed method, which we call a deep photometric stereo
network (DPSN), uses a DNN for establishing a flexible mappingfrom observations to surface normal and reflectances. To make
DPSN applicable to diverse real-world materials, a dataset ofmeasured BRDFs (MERL BRDF dataset [8]) has been used fortraining the network. It allows the method to accurately estimate
surface normal of a scene with complex reflectances and also torecover the reflectance in a per-pixel manner. We use a flexible
expression of reflectances compared to the existing parametricmodels, which is represented as a linear combination of bases
derived from the MERL BRDFs, allowing us to recover a full
BRDF table for each pixel. Further, to deal with the non-local castshadowing effect, we propose a shadow layer that is based on aconventional dropout strategy.
In this paper, we assume that the light directions are predefined and remain the same between training and prediction
phases, which is the case in many photometric stereo apparatuses. DPSN operates in a per-pixel manner, by taking intensity
observations of a surface point under varying light directions, andinfers the surface normal and reflectance of the surface point. The
result shows the effectiveness of the proposed method on bothThis is the authorsʼ version of the work. It is posted here by permission of IEEE for personal use. Not for redistribution.
The final authenticated publication is available online at DOI: 10.1109/TPAMI.2020.3005219.
TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

simulation and real-world images for estimating surface normal
and reflectances.
The preliminary version of this work appeared in [9], and we
extend our previous work to achieve the simultaneous estimationof surface normal and reflectances while the previous method
was limited to the surface normal estimation. To enable this new
capability of reflectance estimation, we propose a new branch in
the network trained by a rendering loss function.

2

R ELATED WORKIn this section, we first describe previous works of photometricstereo. After that, we describe the reflectance modeling andmethods for simultaneous estimation of both surface normal andreflectances based on physics-based vision. Finally, we describe
the DNN-based approaches in photometric stereo and discuss their
distinctions from this work.
2.1

Photometric stereo

Conventional photometric stereo [1], [2] is based on the Lambert’s reflectance model. Because the Lambert’s model is an
ideal reflectance model that may not well represent real-worldreflectances, extending photometric stereo to work with nonLambertian surfaces has been of interest for its practical use.
Existing studies on non-Lambertian photometric stereo can beclassified into three categories.
The first category includes methods based on robust estimation, where the non-Lambertian reflectances are treated as outliers.
They assume that the majority of reflectance observations obeys,
or is close to, the Lambert’s model so that the non-Lambertian reflectances, such as specular reflections, can be regarded as anomalies. Wu et al. [10] formulate the robust estimation problem as arank minimization problem. They exploit the fact that the Lambertian observations form a low-rank subspace [11] and treat the nonLambertian reflectances as sparse outliers. Mukaigawa et al. [12]use the random sample consensus (RANSAC) scheme [13], which
essentially approximates the `0 residual minimization for discarding outliers. Other robust estimation methods, such as expectation
maximization [14], taking the median values [15], `1 residual
minimization and sparse Bayesian learning [16], are also shown
to be effective for dealing with sparse outliers. Since the robust
estimation methods are built upon statistical outlier rejection, theygenerally require many input images, e.g., 40 images in [10],recorded under distinct light directions.
The second category includes methods based on moresophisticated reflectance models than the Lambertian model
to better approximate non-Lambertian reflectance observations.
Georghiades [17] uses the Torrance-Sparrow model [18], andRuiters et al. [19] use the Cook-Torrance model [20] along thisdirection. More recently, Shi et al. [21] propose a bi-polynomial
BRDF model, which is capable of representing low-frequency
non-Lambertian reflectances, and it shows greater accuracy insurface normal estimation. Holroyd et al. [22] propose anotherapproach for generalizing reflectance properties based on the reflective symmetry of the halfway vector across the normal-tangent
and normal-binormal planes, which does not require estimating
a surface reflectance model, and performs well on anisotropic
reflectance surfaces. All the above methods estimate the parameters of the reflectance model simultaneously with estimating thesurface normal.

2

The third category includes example-based methods, which
determines surface normal by the use of reference objects. Hertzmann and Seitz [23] propose an example-based method using areference sphere that has the same reflectance as the target object.
From the observations that are consistent between the target andreference objects, their method determines the surface normalof the target object by simply mapping the corresponding one
from the reference object. The example-based method naturally
avoids solving a complex optimization problem, but it requires areference object, of which the shape is known and reflectance isthe same as the target object. More recently, Hui et al. [24] proposed an example-based photometric stereo without any reference
objects. They obtain a mapping from measurements to surface
normal by rendering virtual spheres using given light conditionsand various BRDFs instead of recording a reference sphere.
Our method is rooted somewhere between the second andthird categories. As with the methods in the second category, ourmethod is able to deal with diverse BRDFs. Instead of estimating
both BRDF parameters and surface normal like in the previous
approaches, our method directly establishes mappings from reflectance observations to surface normal using a deep learning
framework. Our DPSN is trained using a dataset of measured
BRDFs of various materials; therefore, it shares the spirit of theexample-based methods in the third category, while DPSN does
not require a reference object to be placed together with the target
object.
2.2

Reflectance estimation

The reflectance estimation is one of the important problems inphysically-based computer vision with a wide range of applications in computer graphics. Previously, Tominaga et al. [25]presented a reflectance estimation method based on the Phongmodel [26] from a single RGB image of a cylindrical object
taken under a point light source. They use the known 3D shapeinformation to estimate the parameters of the Phong model. Unlike
their method, Sato et al. [27] developed a method for estimating
parametric reflectances from a general shape object using theshape information obtained by laser scanning.
While these methods assume a given shape information forreflectances estimation, Goldman et al. [28] proposed the simultaneous estimation of shape and reflectance from multiple
images taken under different illuminations. Their method models
reflectances by an isotropic parametric reflectance model, namely
the Ward model [29], and estimates its parameters via iterative
alternating optimization. Wang et al. [30] also proposed an estimation method for surface reflectances and a classification method forthe surface materials using multiple images taken under varying
light conditions. To achieve the segmentation based on materials,
they classify materials using support vector machines (SVMs)
based on the observations under varying lightings.
Unlike these methods, Wang et al. [31] used light field images
for the reflectance estimation. They formulate the reflectanceestimation task as material classification using DNN, which takes
as input a 4D light-filed image and outputs the score prediction
for 12 classes of materials. Zhou et al. [32] also proposed to use
multi-view images taken under varying light directions for shapeand reflectance estimation. They used a flexible representationof reflectance using a data-driven BRDF model rather than aparametric reflectance model. The data-driven BRDF model uses
discrete look-up tables indexed by light and viewing directions

This is the authorsʼ version of the work. It is posted here by permission of IEEE for personal use. Not for redistribution.
The final authenticated publication is available online at DOI: 10.1109/TPAMI.2020.3005219.
TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

3

Input images

Normal estimation block
!
"
#

$

Shared block
Shadow
+
Layer

,,,

,,,

&' ( ) #BRDF table

BRDF estimation block%,,,*' ( ) +*' - .+ / 012

Fig. 1. Overview of the proposed deep photometric stereo network (DPSN). It consists of three components: shared, normal estimation, and BRDF
estimation blocks.
The shared> block is formed by a shadow layer and dense layers. The shared block takes as input the per-pixel measurement
vector m = m1 , . . . , mf . The normal estimation and BRDF estimation blocks consist of dense layers, which yield prediction of a surface normal
vector n∗ and reflectance parameters c∗ , respectively. From the reflectance parameters c∗ , a full BRDF table is reconstructed based on Eq. (2).

sampled by a certain interval. It has an excellent representation capability of reflectances, but it comes with a difficulty for estimation
because of the large number of parameters. To make the estimation
tractable, it has been shown a linear basis representation of BRDFs
where reflectances are represented by a linear combination of a
few basis vectors. This representation has been employed in [32].
Nielsen et al. [33] proposed a method for BRDF estimation froma few observations, which also used a linear basis representationof BRDFs. They use a log-relative mapping for reflectances todeal with the high dynamic range of reflectances. Xu et al. [34]also worked on the BRDF estimation from a few observations (asfew as two images) using the same BRDF representation as [33].
Similarly, our method also represents reflectance by a linear combination of basis BRDFs. Unlike these methods, our DNN-based
approach allows the use of many principal components without
restriction and achieves a flexible representation of BRDFs.
2.3 Deep learning-based approaches for normal andreflectance predictions
Early works that use neural networks in the context of photometricstereo can be found in the 1990’s. Iwahori et al. [35] used a neuralnetwork for determining surface normal. While effective, their
method requires the pre-training with a reference sphere painted
by the same material as the target object, akin to an examplebased method. There are other methods that use neural networks
afterwards, but they are restricted to Lambertian reflectances [36],[37] or specialized imaging setup [38]. In contrast to these earlyworks, our work [9] is the first attempt to use a modern DNN
architecture in the context of photometric stereo.
More recently, convolutional neural network (CNN) is shown
to be effective for photometric stereo [39], [40]. Chen et al.’s
method [39], called PS-FCN, extracts features from each input
image and the corresponding light direction vector, and estimatesa surface normal map from the feature map aggregated by maxpooling. With the feature map aggregation by pooling, PS-FCN
can deal with an arbitrary number of photometric stereo images.
Ikehata [40] proposed another approach, called CNN-PS, to allow
the network to deal with an arbitrary number of inputs. It uses anintermediate representation of observed images, called observation
map, to represent per-pixel observations under varying lightings.
While these methods [39], [40] consider only the surface normal
prediction, the proposed method aims to estimate both surface normal and reflectances. Unlike the supervised approaches [9], [39],[40], Taniai and Maehara [41] proposed an unsupervised learning
approach based on a reconstruction loss, which is defined by thepredicted surface normal, reflectances, and known light sources.
Their method is able to accurately estimate surface normal and aslice of BRDF without requiring pre-training. While our method
requires supervised training, it can estimate full BRDFs, not aslice of them, in the form of a linear combination of BRDF bases,
which allows us to re-render the target scene under a new unseen
illumination condition.
Along with the recent development of DNNs, methods for
estimating surface normal and reflectance from a single RGB
image have been proposed. Rematas et al. [42] proposed a
convolutional neural architecture to estimate reflectance maps ofspecular materials in natural lighting conditions from a single
RGB image. They treat the lighting conditions to be unknown
and estimate a reflectance map, which is a slice of a BRDF, andsurface normal simultaneously. Janner et al. [43] introduced animage decomposition method that factors a single input image
into the reflectance, shape, and lighting condition by a convolutional encoder-decoder architecture. Deschaintre et al. [44] andLi et al. [45] also proposed DNN-based estimation methods fordetermining reflectances from a single image. They focus on
the specular reflection on a near-planar surface and estimate thesurface normal, diffuse albedo, and roughness using an encoderdecoder architecture. Their methods use the rendering loss basedon the estimated parameters for training. Since these methods takeonly a single image as input, the accuracy of the surface normal
prediction is rather limited. In addition, while they only consider
the reflectance under a certain illumination condition, our method
can estimate a full BRDF in a per-pixel manner.
In addition to an RGB image, depth images are useful forthe purpose of reflectance estimation. Kim et al. [46] proposeda reflectance estimation method from RGBD images using DNN.
They use the isotropic Ward BRDF model [29] for reflectances,which is characterized by three parameters. Unlike their method,
we use the data-driven BRDF representation like [33], [34], which
has a greater expression power.

3

P RELIMINARIESWhen a Lambertian surface with albedo-scaled surface normaln ∈ R3 is illuminated by a directional light1 l ∈ S 2 ⊂ R3 , themeasurement m ∈ R+ can be described as

m = l> n.
1. Throughout this paper, we assume klk2 = 1, and the input image
intensities are normalized by the corresponding light intensity.

This is the authorsʼ version of the work. It is posted here by permission of IEEE for personal use. Not for redistribution.
The final authenticated publication is available online at DOI: 10.1109/TPAMI.2020.3005219.
TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCEf
For a vector of measurements m ∈ R+ observed under f distinct

light directions, the above equation can be written with a lightmatrix L = [l1 , . . . , lf ] ∈ R3×f as

m = L> n.
The conventional photometric stereo method [1], [2] determines
surface normal n using the above image formation model byn∗ = L−> m,when f = 3 and rank(L) = 3, or, with more than three
distinct observations, a least-squares approximate solution n∗ canbe obtained by a pseudo-inverse of L as−1n∗ = LL>
Lm.
Unfortunately, a pure Lambertian surface rarely exists in thereal world; hence, making photometric stereo work with nonLambertian surfaces is one of the major interests in practice.
With a general BRDF function, the appearance of a surface
under a local illumination model can be described more flexibly.
The appearances of a surface observed from a fixed viewing
direction v under varying distant light directions L can be writtenasm = b ◦ (L> n),fwhere b ∈ R+ is a vector of reflectances sampled from the BRDF
function ρ as b = ρ(L, n, v), the operator ◦ represents elementwise multiplication, and L> n is the irradiance at the surface under
the corresponding light directions.
The above equation assumes a shadow-free world, while in
the real-world the surface patches facing away from the lighting
direction are in attached shadow, and the light path being occluded
causes cast shadow. Such a shadowing process can be written ashim = s ◦ b ◦ max(L> n, 0) ,(1)fwhere s ∈ {0, 1} is a boolean vector with 0 indicating observations in cast shadows and 1 otherwise. The effect of attachedshadow is accounted by the element-wise max operator.

4

P ROPOSED METHOD

The proposed DPSN is a differentiable multi-layer neural network,
which learns a mapping from a measurement vector m obtained
under different light directions to the surface normal n andreflectance. It operates in a per-pixel manner for both training
and prediction. As stated in the introduction, we assume that thelight directions L are known and consistent between the training
and prediction phases. Our method uses simulated observations
as training data that are generated using diverse surface normals
rendered with the MERL BRDF dataset [8] that contains BRDFs
of 100 different real-world materials. In what follows, we explain
the representation of reflectance, the structure of the proposed
network, and training and prediction procedures.
4.1

Reflectance representation

A non-parametric representation of BRDFs has high accuracy andexpressiveness, while it comes with the cost of high storage requirement. In the context of photometric stereo, where we wish to
determine the reflectance for the purpose of estimating surface normal, the direct use of a non-parametric BRDF is computationally4intractable due to the large number of unknowns (e.g., if using a
MERL BRDF sampling rate, it becomes 90×90×180 = 1458000
for each color channel).
We follow the procedure of linear analysis in Matusik et al. [8]to extract the basis from the MERL BRDF dataset so that the
BRDFs can be expressed by the small number of coefficientsassociated with the basis vectors. The MERL BRDF dataset [8]stores BRDF tables for 100 different materials in 3 color channels.
We treat each color channel independently and construct a BRDF
matrix X ∈ R300×1458000(100 materials in 3 color channels with+
1458000 bins) by stacking each BRDF table as a row vector. Let
X̄ ∈ R+ donate the mean of all the elements in X, and X̃ be
X − 1X̄ computed by subtracting the mean X̄ from X for eachelement, where 1 is an all-ones matrix. We compute a singularvalue decomposition (SVD) of X̃ and obtain U, Σ, and V, where
U and V are left and right orthonormal singular vectors, and Σ
is a diagonal matrix with singular values. As with a standard lowrank approximation, by using the top k singular vectors, Uk and
Vk , and corresponding singular values Σk , X̃ is approximately
reconstructed as
X̃ ' Uk Σk Vk> .
To make it easy for a deep neural network to learn, we apply
normalization by the standard deviation σ of elements in Uk andregard Uσk as coefficient vectors and Wk = σΣk Vk> as the basis
matrix of the BRDF tables. With the basis representation, we can
express a BRDF table bt ∈ R1458000 as a linear combination of
the basis Wk with k -dimensional coefficients c ∈ Rk as>b>t = c Wk + 1X̄,

(2)

where 1 is a vector whose every element is one. The proposed
network estimates the coefficients c from the measurement vector
m. The number of singular vectors k is one of the hyperparameters in our method and is set to 300, i.e., we use all the
singular vectors without compression, because the dimensionality
reduction did not exhibit advantage in the estimation via DNN.
Once we obtain the full BRDF table bt , given a set of light
directions {l} forming a light matrix L, and a fixed surface normaln and viewing direction v, we can sample BRDFs to generate afcompact BRDF vector b ∈ R+ for the measurement setting froma full BRDF table bt as

b = ρ(L, n, v; bt ),(3)

using the function ρ that subsamples elements from bt . The
sampled BRDF b can then be used as a part of the image formation
model expressed in Eq. (1).
To deal with the high dynamic range of reflectances, previous
methods [33], [34] used a log-relative mapping for reflectance X.
In our case, the log-scaling requires the exponential function torecover the original values when assessing the reconstruction loss.
We have observed that it causes a negative effect on convergencein our preliminary experiments; therefore, we decided not to
use it. Instead, we effectively neglect specular spikes that shows
significantly larger values using `1 residual loss (more discussion
in Sec. 4.2).
4.2

Network Architecture

The proposed DPSN learns a mapping from a measurement vectorm ∈ Rf+ at a pixel to the corresponding surface normal n ∈ R3
and BRDF parameters c ∈ Rk of the pixel using a fully connectedThis is the authorsʼ version of the work. It is posted here by permission of IEEE for personal use. Not for redistribution.
The final authenticated publication is available online at DOI: 10.1109/TPAMI.2020.3005219.

deep neural network. Figure 1 shows the overview of the proposed
network architecture. DPSN consists of three blocks: Shared,
surface normal estimation, and BRDF estimation blocks. The
shared block takes as input a measurement vector m, in which
each element corresponds to an observation under a certain light
direction, and outputs the extracted feature vectors. The normal
estimation and BRDF estimation blocks take the feature vectors as
input and output the prediction of the surface normal n and BRDF
parameters c, respectively. The measurement vector m is linearlynormalized by a global scaling so that the maximum measurement
value in the all f images becomes 1.0.
One of the major challenges in photometric stereo is castshadow. Differently from attached shadow, cast shadow is caused
by a global illumination effect, which cannot be modeled by alocal illumination model regardless of the representation ability ofa BRDF model. To simulate the cast shadow effect in the training
phase, we introduce a shadow layer (Fig. 2) that is based on avariant of the dropout scheme [47], which randomly drops units
from the network during training (or could be used for testing
as well) to prevent learned weights from excessive adaptation.
Our shadow layer applies dropout to input nodes to randomly
drop a part of the input measurement vector, namely setting theselected measurements to 0, so that the dropped nodes can be
regarded as shadowed observations. By training the network withthe shadow layer, the proposed DPSN effectively learns mappingfrom observations to surface normal and BRDF parameters withaccounting for cast shadow.
While, in conventional dropout, output from the dropout layer
is scaled by 1/(1 − r) with a dropout rate r ∈ [0.0, 1.0) to
avoid shrinkage of the output magnitude, our shadow layer does
not apply the scaling but simply sets the selected elements ofthe measurement vector to 0 to mimic the shadowing effect.
The dropout parameter r corresponds to the ratio of shadowed
observations in our context. Obviously, the parameter r depends
on the object shape and the light distribution, which is inaccessible
in general; therefore, we use varying values of r for training.
Specifically, we fluctuate the dropout rate by sampling from abinomial distribution r ∼ B(f, p), where the probability of eachobservation being shadowed p is set to p = 0.05.
The DPSN structure is summarized in Table 1. The shared
block consists of three layers: One shadow layer and two dense
layers. The normal estimation and BRDF estimation blocks respectively consist of three and four dense layers. The numberof output nodes is three for the normal estimation block, and k
for the BRDF estimation block. All the dense layers use ReLU
and dropout during the training. We make the BRDF estimation block deeper than the normal estimation block because thenumber of output nodes is significantly larger (k is set to 300
in our experiment) in the BRDF estimation block. The existing
methods for a similar task, such as shape estimation from a single
image [42], [43], [44], [45], use an encoder-decoder architecturewith CNNs rather than fully-connected layers to extract featuresfrom neighboring pixels. GeoNet [48] also proposed a CNN-based
network architecture for joint estimation of depth and surface
normal from a single image. Compared to these approaches,
our task is better conditioned, allowing the choice of a simpler
architecture of the network. Indeed, its computation cost is muchsmaller than that of CNNs. Generally, the disadvantage of fullyconnected layers is the greater number of parameters, but in ourcase, the number of parameters is less than 4.3M (in the sharedand surface normal estimation blocks) + 3.5M (in the BRDF

5

Measurement vector

TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
!

"
#
‫؞‬
‫؞‬
‫؞‬

Dropout

‫؞‬
‫؞‬
‫؞‬

‫؞‬
‫؞‬
‫؞‬

$

Fig. 2. Overview of shadow layer. Shadow layer randomly drops someof the measurement vector elements to simulate the cast shadow effect.
In this illustration, m1 and m3 are dropped and corresponding values of
the input vector are set to 0.

estimation block) ≈ 8M, which is regarded a small model size
compared to the modern CNN architectures such as AlexNet,
VGG-16 (used in GeoNet as backbone), and ResNet-50 [49].
In addition, the per-pixel estimation is useful for scenes withspatially-varying BRDFs, and applicable to arbitrary resolutionof the input images, without overly smoothing the surface normal
estimates.
The DPSN is trained with the following loss function L:

L = (1 − α)Ln + αLBRDF ,
where


 Ln
LBRDF

 m∗

(4)

= ||n − n∗ ||22 ,= ||mh− m∗ ||1 ,i= s ◦ ρ(L, n, v; c∗ > Wk + 1X̄) ◦ max(L> n, 0) .

n is the ground truth surface normal vector, n∗ is the predicted
normal vector, c∗ is the predicted BRDF parameters by the
network, and α is a constant weight for balancing surface normal
loss Ln and reconstruction lossLBRDF . m∗ is the reconstructedmeasurement vector with the predicted BRDF tables based on
Eqs. (1), (2), and (3). The light directions L are predefined, andthe viewing direction v is set to [0, 0, 1]> . The rendering loss has
been used in previous methods [44], [45]. These methods render
the target scene under new light directions in addition to the light
direction of the input to constrain BRDFs. Unlike these methods,
since we have the photometric stereo images taken under multiple
light conditions, we simply render using these light directions toensure the consistent appearance. We also tested the log-scaled
loss for the rendering loss LBRDF in Eq. (4) and compared
the estimation accuracy of BRDFs. However, it did not exhibit
improvement in either quantitative or qualitative evaluations. Thisis because we use `1 residual in the rendering loss, with whichlarge but sparse reflectances are naturally neglected, and as aresult, the log-scaled rendering loss became ineffective.
4.3

Training

The training set for DPSN consists of pairs of an observation
vector and the corresponding surface normal, i.e., {(m, n)}.
Instead of collecting real-world observations, we generate theobservation vectors {m} using the MERL BRDF dataset [8] anda diverse set of surface normal {n} rendered under pre-defined
light directions L. The MERL BRDF dataset consists of measured
BRDFs of 100 different materials, and we form reflectance vectors{b} in Eq. (1) from the set of surface normal {n} and light
directions L.

This is the authorsʼ version of the work. It is posted here by permission of IEEE for personal use. Not for redistribution.
The final authenticated publication is available online at DOI: 10.1109/TPAMI.2020.3005219.
TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

6

TABLE 1
The DPSN structure. The number in the parentheses represents thenumber of nodes in each layer. The shadow layer is used for training,
but not for prediction. Dropout rates for training the dense layers are all
set to 0.5. The input and output dimensions of the shadow layer is
96 (= f ) in our setting.
Layer
1
2
3
1
2
3
4

[Shared block]Shadow Layer
Dense-(1024), ReLU, Dropout
Dense-(1024), ReLU, Dropout[Normal estimation block]
Dense-(1024), ReLU, Dropout
Dense-(1024), ReLU, Dropout
Dense-(3)[BRDF estimation block]
Dense-(1024), ReLU, Dropout
Dense-(1024), ReLU, Dropout
Dense-(1024), ReLU, Dropout
Dense-(k)silver[Normal map] white-paint metallic-paint red-plastic

Fig. 3. Examples of rendered images and the corresponding normal map
that are used for training. “white-paint”, “silver-metallic-paint”, and “redplastic” are the material names in the MERL BRDF dataset [8]. Here,three are shown out of 100 different materials. As seen in the figures,the rendered images contain specularity and attached shadows.

While any distributions of surface normal can be used forgenerating the surface normal set {n}, for this work, we useda simple sphere, which includes all surface normal directions
on the scene surface. A sphere is rendered with MERL BRDFs
under light directions L, and observations at each pixel location of
the rendered images for each color channel form a measurement
vector m. Figure 3 shows some of the images of training data
generated under certain light directions. As shown in the figure,the rendered images contain complex reflectances that do not obey
a simple parametric model.
In the loss function for training, the weight parameter α in
Eq. (4) is set to 0.1 for our experiments based on the result of thepreliminary experiments. The rendering loss LBRDF uses an `1 loss
defined in the rendered image intensity space to neglect outliers.
The loss function L is minimized using Adam [50] with a learning
rate set to 5 × 10−5 and other parameters to the suggested default
settings (β1 = 0.9 and β2 = 0.999). The learning rate isdecreased with 0.96 every 1000 steps.
After the above training, we perform fine-turning with thefollowing cosine similarity loss (denoted as FT: fine-tuning loss):

LFT = 1 −n> n∗,||n||2 ||n∗ ||2(5)

to directly optimize based on the angular errors of surface normal.
In this step, only the parameters in the normal estimation layers
are updated while the parameters in the other branches are fixed.
The loss function LFT is minimized in the same setting with L,
but the learning rate decay is not used in this phase.
In the experiments, we trained the model with the loss function
L for 30, 000 steps with the batch size set to 200 and α to 0.1.
After that, it is further trained with LFT for 5, 000 steps with thebatch size set to 1, 000. We choose the model for the evaluation,
which had the best performance in the validation data.

4.4

Prediction

In the prediction phase, given a set of observations under different
light directions L, DPSN estimates surface normal n and BRDF
coefficients c in a per-pixel fashion. In a similar manner to thetraining phase, color channels are treated independently.
In the normal estimation, for an RGB image, DPSN estimates
three surface normals per-pixel and consolidates them to obtainthe final estimate. Namely, nr , ng , and nb are the surface normal
estimates from the RGB color channels that are independently
estimated, we take the mean vector of the normalized surface
normal estimates as


1
nr
ng
nb
n̄ =
.
+
+
3 ||nr ||2||ng ||2||nb ||2
Finally, the merged surface normal n̄ is further normalized toobtain the final surface normal estimate n∗ as n∗ = n̄/||n̄||2 .
For BRDF estimation, the DSPN outputs k dimensional coefficient vector c per pixel per color channel, and we obtain the full
BRDF table in a per-pixel manner for each color channel based on
Eq. (2).

5E XPERIMENTS

We evaluate the proposed method using both simulation and realworld datasets. We will explain the training data and implementation details for the experiments before discussing the result.
Training data and implementation details: For training, we
generate a synthetic dataset rendered under pre-defined light
directions for each of 100 BRDFs from the MERL dataset. The
pre-defined light directions are the same as the 96 light directions
defined in the DiLiGenT dataset [51]. As a result, we render
100 × 96 × 3 (RGB) = 28, 800 grayscale images in total.
The sphere scene consists of 31, 400 valid pixels with a 100pixel radius, resulting in 31, 400 distinct surface normals n. Eachsurface normal is associated with 100 × 3 measurement vectors{m} with the length of 96.
The DPSN is implemented using TensorFlow2 . The training
takes approximately five hours using two NVIDIA Tesla P100
GPUs. The training data generation can be performed concurrently
with the training, i.e., we generate a subset of data for creatinga mini-batch, and during the training using the mini-batch, we
prepare for the next subset of the training data. We set thebatch size to 200, where one training data consists of a single
measurement vector m and the surface normal n. One iterationof training over a batch takes approximately 0.5 seconds. For theprediction, it takes about four seconds for an image with 612×512
pixels (same as the DiLiGenT setting) under 96 lights. Comparedto CNN-PS [40], our method can predict surface normals and
BRDFs about four times faster thanks to the simple networkarchitecture.
Baseline methods: We compare the proposed DPSN with Lambertian photometric stereo based on conventional `2 residual minimization (L2) [1] and that with `1 residual minimization (L1) [16].
For these methods, a surface normal n∗ for each pixel is computedby argmin ||m − L> n||22 ,∗nn = argmin ||m − L> n||1 ,n

2. TensorFlow: https://www.tensorflow.org

This is the authorsʼ version of the work. It is posted here by permission of IEEE for personal use. Not for redistribution.
The final authenticated publication is available online at DOI: 10.1109/TPAMI.2020.3005219.
TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
!" #

7

!"

6
$&7
< ,<

01

96
< ,<

$.7
01

6
$.7
< , <

1

3

1

'

1

&

1

%

01

8< $,$&'&

8< $,$$.%$%

8< $,$%$2
#$&&

$

1

#$#

'

1

&

#$%&

5
5$,%

1
$)(

#$#

#$'(

3

$()

1

#$ &

8< $,$%--

&$

#($&

# $(

# $(%

'$ (

#*$%(

##$('

#

#$&

)$ %

#)$)&

#)$)&

$)

'$#

# $ )

+' ,

-$.

,

-$.

*$#

Fig. 4. Experimental result of synthetic scenes. In each row, a normal
map is shown on top of the corresponding error map. The numbers
represent Mean Angular Error (MAngE) in degree. In the top row, GT
means the ground truth, the images below the normal maps are examples of observation images. On the top, material and shape names areshown. MERL1 and MERL2 correspond to “black-oxidized-steel” and“white-fabric2”, respectively. Material A to D are the synthetic BRDFs
that are created by linearly combining a pair of BRDFs in MERL BRDF
dataset: (“blue-fabric”, “silver-metallic-paint”), (“cherry-235”, “natural209”), (“beige-fabric”, “yellow-paint”), and (“black-soft-plastic”, “bluemetallic-paint”), respectively.

from observations m and known lighting directions L, respectively. For real-world scenes, we also assess the performance ofour method using the DiLiGenT benchmark that covers the stateof-the-art methods of non-Lambertian photometric stereo. To see
the effect of shadow layer, we compare DPSN without a shadow
layer (denoted as “Proposed w/o SL”) and with a shadow layer(“Proposed”).
5.1

Evaluation using synthetic dataset

To assess the performance for unseen BRDFs, we split the MERLBRDF dataset into a training set and a testing set. MERL BRDF
dataset includes 100 BRDFs and we randomly pick up 20 BRDFs
as testing BRDFs. For test shapes, we use three objects, sphereand two shapes (blob02 and blob08) from Blobby shapedataset [52]. We use the same light directions as the training for
generating the input data.
In addition to the testing set of BRDFs, we also evaluate usingsynthesized unseen BRDFs. We synthesized new BRDFs by applying nonlinear transformation over a linearly combined BRDFs
that are sampled from the MERL BRDF dataset. Specifically, a
new BRDF ρ̃(L, n, v) is generated as:γρ̃(L, n, v) = (βρ1 (L, n, v) + (1 − β)ρ2 (L, n, v)) ,(6)8< $,$%2&

8< $,$$42.

$

Fig. 5. Rendered results with the estimated BRDFs and the ground truth
normal maps for synthetic scenes. We show the images rendered undereight different light conditions. “Train light 1” to “Train light 4” are selectedfrom the 96 light conditions that are used for training, while “Test light 1”
to “Test light 4” are from the test light conditions that are not includedin the training data. We sampled 50 test light directions uniformly ona hemisphere. In “Multiple (blob08)”, we apply Gamma correction with
γ = 0.8 for visualization. “GT” and “Est.” mean the ground truth andestimated respectively and are rendered in the same manner to thegeneration of training data. MAE is the mean absolute error calculated in
the grayscale space with the intensities scaled in the range of [0.0, 1.0].
MAE for train light conditions is the mean of f (= 96) images, and fortest light conditions is the mean of 50 images.

where ρ1 and ρ2 are BRDFs sampled from the MERL BRDF
dataset, and β and γ are constant parameters. In this experiment,
ρ1 and ρ2 are selected randomly from the dataset, and theparameters are set to β = 0.5 and γ = 0.8. Furthermore, we used
the object consisting of multiple materials (denoted as “Multiple”
as the material name). We set the area of each materials randomly
with combining lines and circles so that each material occupies acertain area.
The surface normal prediction results are summarized inFig. 4. From testing BRDF sets, we show the result of “blackoxidized-steel” (MERL1) and “white-fabric2” (MERL2), which
are the names in MERL BRDF dataset. “Lambert” indicates theLambertian reflectance, for which the reflectance function ρ is aconstant. “Material A” to “Material D” are the synthetic BRDFs.
As the mean angular errors in the figure indicate, our method
consistently yields accurate estimates of surface normal acrossdifferent BRDFs. Since the generation of synthetic scenes neglects
rendering of cast shadows, our method without a shadow layer(“Proposed w/o SL”) shows superior performance to the one withthe shadow layer (“Proposed”) in some scenes.
Figure 5 shows the rendered images with the predicted BRDFs
with the ground truth surface normal (“Rendered” columns) together with the ground truth appearance (“GT” columns). Here, weshow the results for “Material B”, “Material C”, and “Multiple”.
For both GT and Rendered, we did not render cast shadows.
Along the rows, “Train light 1” to “Train light 4” indicate thelight directions that are used during the training, and “Test light1” to “Test light 4” are the new unseen light directions thatare not included in the training phase. For assessing the mean

This is the authorsʼ version of the work. It is posted here by permission of IEEE for personal use. Not for redistribution.
The final authenticated publication is available online at DOI: 10.1109/TPAMI.2020.3005219.
TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

8

Fig. 6. Plots of estimated BRDFs. The plots show the reflectance ρ(n, l, v) varying with the lighting direction l. We use the polar coordinate system
for the lighting direction l and viewing direction v as [lθ , lφ ] and [vθ , vφ ], where θ and φ represent the polar and azimuthal angles, respectively. We
fix the normal direction to n = [0, 0, 1]> , vφ = 0◦ , and lφ = 180◦ and vary lθ from 0◦ to 90◦ . In each row, from left to right, it shows a scene
image, and reflectance plots with three viewing directions vθ = {0◦ , 30◦ , 60◦ }. The dashed lines are the ground truth, and red and blue lines are
the estimated BRDFs at pixels marked by red and blue stars in the leftmost figure. We show the grayscale reflectance by taking the average of threecolor channels. Note that we cannot predict the absolute values of reflectances from images, and there is a scaling ambiguity between the groundtruth and estimated BRDFs. We align them by normalization based on the median of each BRDF table.

To qualitatively assess the prediction performance of BRDFs,
we show the rendered results under natural environment lightings.
Figure 7 shows the rendered sphere with estimated BRDFs undernatural lighting environment. We use “Uffizi Gallery, Italy” from
High-Resolution Light Probe Image Gallery 3 as the environment
map and the physically based renderer, Mitsuba 4 . The appearanceof rendered images shows good agreement to the ground truth.
5.2
Fig. 7. Rendering of a sphere scene with estimated BRDFs under thenatural lighting environment. The right-top figure shows the environment
map. We also show the lighting condition by showing a rendered mirror
sphere in the right-bottom. GT means the rendering with the groundtruth BRDFs.

absolute error (MAE), we scale the intensity range to [0, 1].
“Test light 1” and “Test light 2” cases are particularly difficult
ones, because their light directions are near perpendicular to theviewing direction, that are not covered by the training data. As
a result, we can observe that specular highlights in the left-mostexample in “Test light 1” and “Test light 2” are not successfully
rendered. Nevertheless, as seen in the figure, the renderings areplausible in most of the cases, and MAEs stay low. The resultof the surface normal estimation and re-rendering of the multiple
materials scene (“Multiple” column) in Figs. 4 and 5 demonstrates
the our method’s capability of per-pixel estimation as well.
Figure 6 shows the plots of estimated BRDFs by varying
the light direction lθ and viewing direction vθ , which are thepolar angles of light and viewing directions respectively in thepolar coordinate system. We show the estimated BRDFs of themetallic material “Material B” with blob02 and diffuse material“Material C” with blob08. While we achieve reasonably good
performance for the viewing directions near vθ = 0◦ , it shows
lower accuracy in steeper angles, such as vθ = 60◦ and lθ > 60◦ ,due to that the training light directions do not cover these regions.

Evaluation using real-world dataset

For the evaluation with real-world images, we use the DiLiGenT
dataset [51], which contains observations of ten different objects
under 96 light directions and the ground truth surface normal maps
measured by a laser scanner.
Figure 8 shows the estimated normal maps and corresponding
error maps for “Proposed w/o SL”, “Proposed”, “L2”, and “L1”.
Here, we show five objects out of ten in the DiLiGenT dataset(buddha, goblet, harvest, pot2, and reading). Amongthem, buddha, pot2, and reading are pottery objects, whosereflections are mostly Lambertian except for some sparse specular
highlights; therefore, L2-, and especially L1-based methods work
well. For an object like goblet, which is made of metallic
materials and exhibit strong specular reflection in a wide area,
the estimation error becomes larger with the Lambertian based
methods, i.e., L2 and L1, while the proposed method produces
highly accurate estimates. For harvest, the estimation accuracy
is poor for all methods due to the complexity of the shape ofthe object, particularly due to the large cast shadow region in the
middle.
Figure 9 shows the rendered images with the predicted BRDFs
and the ground truth surface normal map under light directions
that are used for training. In the figure, we show two objects:
One is mostly diffuse (buddha), and the other is highly specular
3. High-Resolution
Light
Probe
http://gl.ict.usc.edu/data/highresprobes/
4. Mitsuba: https://www.mitsuba-renderer.org/

Image

Gallery:This is the authorsʼ version of the work. It is posted here by permission of IEEE for personal use. Not for redistribution.
The final authenticated publication is available online at DOI: 10.1109/TPAMI.2020.3005219.
TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

"

#

9

!"

without using them as a part of training data. Using the BRDF
coefficients predicted from 90 input images, we render images
with the predicted BRDF parameters under the test light directions
and compare with the ground truth.
Figure 11 shows the rendered results of four objects; buddha,
goblet, cat, and reading. Here, we show results under
three light directions out of six for each object, and below eachsubfigure, MAE for the 90 training data (Train MAE) and six
test data (Test MAE) are respectively shown. The accuracy of
the test data shows is almost equivalent to that of training data,
which shows the the accuracy of our BRDF prediction. The pixels
with strong and sharp specular highlights, e.g. observed at theshoulder parts of reading, have less accurate estimates becausesuch strong and sharp reflections cannot be expressed by a linear
combination of MERL BRDFs. For cat scene, there is a large
error around the foreleg. This is caused by the cast shadow, while
our rendering does not account for the cast shadow.

$

%

"&

%

"&

Fig. 8. Surface normal estimation result for real-world scenes from
DiLiGenT [51]. In each row, a normal map is shown on top of an error
map. The numbers represent Mean Angular Error (MAngE) in degree.
GT means the ground truth, and figures under GT are examples ofobserved images. The contrast of observation images is adjusted forbetter visualization.

(goblet). For both cases, it can be seen that our method canrelight the scenes well because of the accurate estimation of
BRDFs. We also show a qualitative evaluation of the re-renderingresults under novel light directions in Fig. 10 that are not includedin the training dataset. The light directions (a) and (c) are moredifficult conditions than (b) since they are off from the training
light conditions. Even under such conditions, our method works
well supported by the prior knowledge of MERL BRDFs andproduces faithful predictions including specularities for the mostof the parts. The accuracy is limited at some regions due to thelack of observations, where we can observe unnatural predictions;for example, a strong specular highlight in goblet–(a).
Hold-out validation for BRDF estimation in DiLGenT
To evaluate the BRDF estimation, we performed the light source
hold-out validation using DiLiGenT dataset. We randomly choose
six light directions out of 96 and keep them as test light directions

Benchmark comparison
We compare the surface normal estimation accuracy of the
proposed methods (“Proposed w/o SL” and “Proposed”) with
DiLiGenT benchmark results shown in [51], which includes theevaluation of the following ten methods: WG10 [53], IW12 [16],GC10 [28], AZ08 [54], ST12 [55], HM10 [56], ST14 [21],IA14 [57], CH18 [39], and SI18 [40]. Table 2 shows the Mean
Angular Error (MAngE) in degree of each method for each objects
and the average performance over all the scenes. Green color
represents preferable results and red indicates poorer results. For
each object, the best result is highlighted with a bold font. It canbe seen that our method can achieve at a similar level of accuracy
compared to other methods. Recent DNN-based methods [39],[40] exhibit smaller angular errors, which shows the benefit of a
DNN-based approach. While our result is slightly less accurate
than the newest ones, our method is able to predict BRDFs unlike
other techniques.
5.3

Effect of the shadow layer

To observe the effect of shadow layer, we prepare Fig. 12 for
depicting the difference of error maps in the surface normal
prediction with and without the shadow layer. Here, we use four
objects (ball, goblet, harvest, and pot2) as examples. For
all these objects, the accuracy is generally improved in boundary
areas where shadows often occur. In some areas, the accuracy does
not improve for goblet and harvest. For metallic objects
with strong interreflections like goblet and harvest, themeasurement becomes greater than 0 even inside shadow, e.g.,
Fig. 13. In such a case, the shadow layer may actually degrades
the accuracy due to the wrong assumption that the measurement
in shadow becomes 0.
The effect of shadowing probability p may depends on the
shape of the target object. Figure 14 shows the variation of result incases where p = 0.05 and p = 0.9 on ball. Comparing (a) and(b), we can see that although (b) improves more in the peripheral
parts of the sphere (shadowed area) than (b), it deteriorates in thecentral part. With the shadow layer, since the model is optimized
for shadow, the accuracy becomes lower in the area where shadow
is never observed. In the case of p = 0.05, since the dropout
rate r sampled in the manner of Sec. 4.2 can be 0, the estimation
accuracy does not deteriorate in areas without shadow. On theother hand, in the case of p = 0.9, dropout would be applied toThis is the authorsʼ version of the work. It is posted here by permission of IEEE for personal use. Not for redistribution.
The final authenticated publication is available online at DOI: 10.1109/TPAMI.2020.3005219.
TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

<

01

<

'

&

%

01

10

3

5$,%
5$

8< $,$%/

8< $,$$.$$

Fig. 9. Rendered results with the estimated BRDFs and the ground truth normal maps for DiLiGenT. buddha is the object with non-specular materials
and goblet is made of metal materials. We apply Gamma correction with γ = 0.8 to the ground truth and rendered images for visualization. The
light conditions Light 1 to Light 4 are from 96 DiLiGenT light conditions.

6 7

6 7
6 7

6 7

6 7

6 7

6 7

6 7

6 7

Fig. 10. Qualitative evaluation of re-rendering under novel light directions. We show the three images for each object rendered with the new light
directions that are not included in the training data. Leftmost plot illustrates the light directions. Gray points represent the 96 light directions used in
the training data, and red, green, and blue points are the new light directions that corresponds to images (a), (b), and (c), respectively.
TABLE 2
Evaluation results using the DiLiGenT Benchmark [51].

Proposed w/o SL
Proposed
SI18
CH18
ST14
IA14
WG10
AZ08
HM10
IW12
ST12
GC10
BASELINE

ball
2.63
2.49
2.20
2.80
1.74
3.34
2.06
2.71
3.55
2.54
13.6
3.21
4.10

bear
7.22
7.05
4.10
7.60
6.12
7.11
6.50
5.96
11.5
7.32
19.4
6.62
8.39

buddha
14.3
13.8
7.90
7.90
10.6
10.5
10.9
12.5
13.1
11.1
18.4
14.9
14.9

cat
7.10
7.05
4.60
6.20
6.12
6.74
6.73
6.53
8.40
7.21
12.3
8.22
8.41

cow
7.96
7.92
7.90
7.30
13.9
13.1
25.9
21.5
15.0
25.7
7.62
9.55
25.6

goblet
11.2
10.9
7.30
8.60
10.1
9.71
15.7
13.9
14.9
16.3
17.8
14.2
18.5

harvest
19.2
18.4
13.9
15.9
25.4
26.0
30.0
30.5
21.8
29.3
19.3
27.8
30.6

pot1
8.91
8.73
5.40
7.10
6.51
6.64
7.18
7.23
10.9
7.74
10.4
8.53
8.89

pot2
9.6
9.73
6.00
7.30
8.78
8.77
13.1
11.0
16.4
14.1
9.84
7.90
14.7

reading
16.7
15.8
12.6
13.3
13.6
14.2
15.4
14.2
16.8
16.2
17.2
19.1
19.8

AVG.
10.5
10.2
7.20
8.40
10.3
10.6
13.4
12.6
13.2
13.7
14.6
12.0
15.4

This is the authorsʼ version of the work. It is posted here by permission of IEEE for personal use. Not for redistribution.
The final authenticated publication is available online at DOI: 10.1109/TPAMI.2020.3005219.
TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

<

01

<

1

'

1

&

1

%

01

11

8< $,$%2/
8< $,$%23

1
18< $,$$/$
8< $,$$/%

1

&

1

%

1
1

1

'

5$,%
5$

$
1
1

8< $,$'&
8< $,$'%

1
1

8< $,$$/2
8< $,$$2.

Fig. 11. Result of the hold-out validation of light directions. Among 96 light directions in DiLiGenT, we use 90 directions for training and keep theremaining six for testing. We train the model with the number of light directions f = 90 and estimated the BRDFs. Here, we show the three images
for each objects rendered under test light directions. For buddha, goblet, and reading, we apply Gamma correct with γ = 0.8 for visualization.
Train MAE is the mean absolute error for 90 light directions, and Test MAE is for the six test light directions.

Fig. 12. Improvement by shadow layer. We show a difference map of the error maps of “Proposed w/o SL” and “Proposed”. Pixels whose surface
normal estimation accuracy is improved by the shadow layer are colored in blue, otherwise in red.

This is the authorsʼ version of the work. It is posted here by permission of IEEE for personal use. Not for redistribution.
The final authenticated publication is available online at DOI: 10.1109/TPAMI.2020.3005219.
TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

12!!

gobletharvestFig. 13. Effect of interreflections. We adjust the intensities in the left halfin the red box by gamma correction. The shadowed area in the box
exhibits measurements greater than 0 due to strong interreflections.

'"# $ %

30 (worse)&

0 [deg.](a)(b)-30 (better)

Fig. 14. Comparison of shadowing probability p on ball scene. (a) and(b) are difference of the error maps of “Proposed” and “Proposed w/o
SL” in case p = 0.05 and p = 0.9, respectively. Pixels whose normal
estimation accuracy is improved by the shadow layer are shown in blue,
otherwise in red.

almost all the inputs, so the accuracy drops in the area where theshadow does not exist.
Including the cast shadow rendering directly in the training
dataset generation is another option to deal with the cast shadow
effect. Here, we use the dataset with the cast shadow rendering
for training and study the effect of the shadow layer. We use aphysically based renderer, Mitsuba, to render the training datasetwith cast shadow and also prepare the test dataset including castshadow. For this dataset, we use 10 shapes in the Blobby shapedataset instead of a sphere. We leave out 2 shapes (blob02
and blob08) and use the remaining 8 shapes for the training.
Figure 15 shows the surface normal prediction with and without
the shadow layer (SL) both trained with dataset with cast shadow
rendering. For the test shape, we use blob08 which is not used intraining dataset and has the most complex shape in Blobby shapedataset. We use “Lambert” and “Multiple” for the reflectances. As
shown in “Diff map”, the shadow layer improves the estimation
accuracy around the shadowing area although the network has
been training with cast shadows. It shows the effectiveness of the
shadow layer in this setting as well. Our shadow layer effectively
simulates the cast shadow effect and bypasses the preparation ofdataset with cast shadow rendering. And it improves the robustness
in both cases with and without the cast shadow rendering in the
training dataset. It is of interest to extend the shadow layer tosimulate a structured cast shadow, i.e., neighboring pixels arelikely shadowed together as the work of [58].

6

D ISCUSSION

We proposed a method that uses deep neural networks for establishing a flexible mapping from shading observations to surface
normal and BRDFs. Since the proposed method can simultaneously estimate surface normal and BRDFs, it can be applied toFig. 15. Comparison of with and without the shadow layer trained withthe dataset including the cast shadow rendering. Each row corresponds
to one material, and the estimated normal maps are shown on top of thecorresponding error map. GT means the ground truth, and one of theobservation images is shown below it. “Diff map” shows the differenceof the error maps of “w/o SL” and “w/ SL”. In “Diff map”, pixels whose
surface normal estimation accuracy are improved by the shadow layer
are colored in blue, otherwise in red. The numbers show the mean
angular error in degree.

a wide range of applications such as a material prediction andrelighting of target objects for virtual reality.
While early works that use neural networks in the context ofphotometric stereo can be found in 1990’s, they have difficulties
in applicability and accuracy. Even though neural networks havebeen recognized since then as a solution technique for the photometric stereo problem, no effective methods have been introduced
until today. The factors that realized our DPSN are (1) availabilityof the BRDF dataset [8] collected from the real-world objects,
which we used for generating training data, and (2) flexible
expression power of modern DNNs.
Evaluations show the accurate estimation of the surface normal
and BRDFs with our method. Indeed the surface normal estimation
accuracy is comparable to the other tailored state-of-the-art methods based on the benchmark evaluation. In addition to estimating
surface normal, our method is capable of predicting BRDFs thatis expressed by a flexible linear basis representation, which allows
us to re-render the scene under a new lighting condition.
One of the limitations of our method is the assumption thatlight directions are pre-defined and remain the same between
training and test phases. It is designed for a photometric stereo
device that has fixed light sources and a camera, so that we only
need to conduct the training once for the device. Fortunately, many
photometric stereo apparatuses use a fixed lamps with respect tothe camera, allowing the use of our method. On the other hand,more recent works [39], [40] handle the input images taken underthe (known) arbitrary light directions using the techniques such as
the aggregated feature map or observation map. One of our future
directions is to combine these techniques into our simultaneous
estimation for surface normal and reflectances. Moreover, while
our method and recent works [39], [40] still assume the given
light directions, future venues include simultaneous estimation of
the light conditions in addition to surface normal and reflectances
so that it can deal with arbitrary light directions that are not predefined and potentially unknown, such as the data captured undera hand-held light source.

This is the authorsʼ version of the work. It is posted here by permission of IEEE for personal use. Not for redistribution.
The final authenticated publication is available online at DOI: 10.1109/TPAMI.2020.3005219.
TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCEACKNOWLEDGMENTS
This work was supported by JSPS KAKENHI Grant Number
JP19H01123. Hiroaki Santo is grateful for support through a
JSPS research fellowship for young scientists by the Japan Society for the Promotion of Science (JP19J10326). Boxin Shi
is supported by the National Natural Science Foundation ofChina under Grants 61872012, National Key R&D Program ofChina (2019YFF0302902), and Beijing Academy of Artificial
Intelligence (BAAI).

R EFERENCES[1]

R. J. Woodham, “Photometric method for determining surface orientationfrom multiple images,” Optical engineering, vol. 19, no. 1, pp. 139–144,
1980.
[2] W. M. Silver, “Determining shape and reflectance using multiple images,”
Master’s thesis, Massachusetts Institute of Technology, Cambridge, MA,
U.S., 1980.
[3] J. Lambert, “Photometria,” Augustae Vindelicorum, 1760.
[4] A. S. Georghiades, “Incorporating the torrance and sparrow modelof reflectance in uncalibrated photometric stereo,” in Proceedings ofInternational Conference on Computer Vision (ICCV), 2003, pp. 816–825.
[5] K. Torrance and E. Sparrow, “Theory for off-specular reflection fromroughened surfaces,” Journal of the Optical Society of America, vol. 57,
pp. 1105–1114, 1967.
[6] L. Chen, Y. Zheng, B. Shi, A. Subpa-Asa, and I. Sato, “A microfacetbased reflectance model for photometric stereo with highly specularsurfaces,” in Proceedings of International Conference on Computer
Vision (ICCV), 2017, pp. 3181–3189.
[7] B. Shi, P. Tan, Y. Matsushita, and K. Ikeuchi, “Bi-polynomial modelingof low-frequency reflectances,” IEEE Transactions on Pattern Analysis
and Machine Intelligence, vol. 36, no. 6, pp. 1078–1091, 2014.
[8] W. Matusik, H. Pfister, M. Brand, and L. McMillan, “A data-driven
reflectance model,” ACM Transactions on Graphics, vol. 22, no. 3, pp.
759–769, Jul. 2003.
[9] H. Santo, M. Samejima, Y. Sugano, B. Shi, and Y. Matsushita, “Deepphotometric stereo network,” in Proceedings of the International Workshop on Physics Based Vision meets Deep Learning (PBDL) in Conjunction with IEEE International Conference on Computer Vision (ICCV),
2017, pp. 501–509.
[10] L. Wu, A. Ganesh, B. Shi, Y. Matsushita, Y. Wang, and Y. Ma, “Robust
photometric stereo via low-rank matrix completion and recovery,” Proceedings of Asian Conference on Computer Vision (ACCV), pp. 703–717,
2011.
[11] R. Basri and D. Jacobs, “Lambertian reflectance and linear subspaces,”
IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25,
no. 2, 2003.
[12] Y. Mukaigawa, Y. Ishii, and T. Shakunaga, “Analysis of photometricfactors based on photometric linearization,” JOSA A, vol. 24, no. 10, pp.
3326–3334, 2007.
[13] M. A. Fischler and R. C. Bolles, “Random sample consensus: A paradigm
for model fitting with applications to image analysis and automatedcartography,” Communications of the ACM, vol. 24, no. 6, pp. 381–395,
Jun. 1981.
[14] T.-P. Wu and C.-K. Tang, “Photometric stereo via expectation maximization,” IEEE Transactions on Pattern Analysis and Machine Intelligence,
vol. 32, no. 3, pp. 546–560, 2010.
[15] D. Miyazaki, K. Hara, and K. Ikeuchi, “Median photometric stereo as
applied to the segonko tumulus and museum objects,” International
Journal of Computer Vision, vol. 86, no. 2–3, pp. 229–242, 2010.
[16] S. Ikehata, D. Wipf, Y. Matsushita, and K. Aizawa, “Robust photometricstereo using sparse regression,” in Proceedings of IEEE Conference on
Computer Vision and Pattern Recognition (CVPR). IEEE, 2012, pp.
318–325.
[17] A. Georghiades, “Incorporating the torrance and sparrow model ofreflectance in uncalibrated photometric stereo,” 2003, pp. 816–823.
[18] K. E. Torrance and E. M. Sparrow, “Theory for off-specular reflection
from roughened surfaces,” JOSA, vol. 57, no. 9, pp. 1105–1114, 1967.
[19] R. Ruiters and R. Klein, “Heightfield and spatially varying BRDF
reconstruction for materials with interreflections,” Computer Graphics
Forum, vol. 28, no. 2, pp. 513–522, Apr. 2009.
[20] R. L. Cook and K. E. Torrance, “A reflectance model for computer
graphics,” ACM Transactions on Graphics, vol. 1, no. 1, pp. 7–24, 1982.

13[21] B. Shi, P. Tan, Y. Matsushita, and K. Ikeuchi, “Bi-polynomial modelingof low-frequency reflectances,” IEEE Transactions on Pattern Analysis
and Machine Intelligence, vol. 36, no. 6, pp. 1078–1091, 2014.
[22] M. Holroyd, J. Lawrence, G. Humphreys, and T. Zickler, “A photometricapproach for estimating normals and tangents,” ACM Transactions on
Graphics (Proceedings of SIGGRAPH Asia 2008), vol. 27, no. 5, p. 133,
2008.
[23] A. Hertzmann and S. M. Seitz, “Example-based photometric stereo:
Shape reconstruction with general, varying BRDFs,” IEEE Transactions
on Pattern Analysis and Machine Intelligence, vol. 27, no. 8, pp. 1254–
1264, 2005.
[24] Z. Hui and A. C. Sankaranarayanan, “Shape and spatially-varying
reflectance estimation from virtual exemplars,” IEEE Transactions on
Pattern Analysis and Machine Intelligence, vol. 39, no. 10, pp. 2060–
2073, 2017.
[25] S. Tominaga and N. Tanaka, “Estimating reflection parameters from asingle color image,” IEEE Computer Graphics and Applications, vol. 20,
no. 5, pp. 58–66, 2000.
[26] B. T. Phong, “Illumination for computer generated pictures,”Communications of the ACM, vol. 18, no. 6, pp. 311–317, Jun. 1975.
[Online]. Available: http://doi.acm.org/10.1145/360825.360839[27] Y. Sato, M. D. Wheeler, and K. Ikeuchi, “Object shape and reflectancemodeling from observation,” in Proceedings of annual conference on
Computer graphics and interactive techniques. ACM Press/AddisonWesley Publishing Co., 1997, pp. 379–387.
[28] D. B. Goldman, B. Curless, A. Hertzmann, and S. M. Seitz, “Shape andspatially-varying BRDFs from photometric stereo,” IEEE Transactions
on Pattern Analysis and Machine Intelligence, vol. 32, no. 6, pp. 1060–
1071, 2010.
[29] G. J. Ward, “Measuring and modeling anisotropic reflection,” in ACM
SIGGRAPH Computer Graphics, vol. 26, no. 2. ACM, 1992, pp. 265–272.
[30] O. Wang, P. Gunawardane, S. Scher, and J. Davis, “Material classification
using BRDF slices,” in Proceedings of IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), June 2009.
[31] T.-C. Wang, J.-Y. Zhu, E. Hiroaki, M. Chandraker, A. A. Efros, and
R. Ramamoorthi, “A 4D light-field dataset and cnn architectures for material recognition,” in Proceedings of European Conference on Computer
Vision (ECCV). Springer, 2016, pp. 121–138.
[32] Z. Zhou, Z. Wu, and P. Tan, “Multi-view photometric stereo with spatiallyvarying isotropic materials,” in Proceedings of IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), 2013, pp. 1482–1489.
[33] J. B. Nielsen, H. W. Jensen, and R. Ramamoorthi, “On optimal, minimal BRDF sampling for reflectance acquisition,” ACM Transactions on
Graphics, vol. 34, no. 6, p. 186, 2015.
[34] Z. Xu, J. Boll Nielsen, J. Yu, H. Wann Jensen, and R. Ramamoorthi,“Minimal BRDF sampling for two-shot near-field reflectance acquisition,” ACM Transactions on Graphics, vol. 35, pp. 1–12, 11 2016.
[35] Y. Iwahori, R. J. Woodham, H. Tanaka, and N. Ishii, “Neural network
to reconstruct specular surface shape from its three shading images,”in Proceedings of International Joint Conference on Nueral Networks,
1993, pp. 1181–1184.
[36] W.-C. Cheng, “Neural-network-based photometric stereo for 3D surface
reconstruction,” in Proceedings of International Joint Conference on
Nueral Networks, 2006, pp. 404–410.
[37] D. Elizondo, S.-M. Zhou, and C. Chrysostomou, “Surface reconstruction techniques using neural networks to recover noisy 3D scenes,” inProceedings of International Conference on Artificial Neural Networks,
2008, pp. 857—-866.
[38] Y. Ding, Y. Iwahori, T. Nakamura, R. J. Woodham, L. He, and H. Itoh,“Self-calibration and image rendering using RBF neural network,” inProceedings of International Conference on Knowledge-Based and Intelligent Information and Engineering Systems, 2009, pp. 705––712.
[39] G. Chen, K. Han, and K.-Y. K. Wong, “PS-FCN: A flexible learning
framework for photometric stereo,” in Proceedings of European Conference on Computer Vision (ECCV), 2018.
[40] S. Ikehata, “CNN-PS: CNN-based photometric stereo for general nonconvex surfaces,” in Proceedings of European Conference on Computer
Vision (ECCV), 2018.
[41] T. Taniai and T. Maehara, “Neural inverse rendering for general reflectance photometric stereo,” in Proceedings of International Conference
on Machine Learning (ICML), 2018, pp. 4864–4873.
[42] K. Rematas, T. Ritschel, M. Fritz, E. Gavves, and T. Tuytelaars, “Deepreflectance maps,” in Proceedings of IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), 2016.

This is the authorsʼ version of the work. It is posted here by permission of IEEE for personal use. Not for redistribution.
The final authenticated publication is available online at DOI: 10.1109/TPAMI.2020.3005219.
TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE[43] M. Janner, J. Wu, T. D. Kulkarni, I. Yildirim, and J. Tenenbaum, “Selfsupervised intrinsic image decomposition,” in Proceedings of Advancesin Neural Information Processing Systems (NIPS), 2017, pp. 5936–5946.
[44] V. Deschaintre, M. Aittala, F. Durand, G. Drettakis, and A. Bousseau,“Single-image SVBRDF capture with a rendering-aware deep network,”
ACM Transactions on Graphics (TOG), vol. 37, no. 4, p. 128, 2018.
[45] Z. Li, K. Sunkavalli, and M. Chandraker, “Materials for masses:
SVBRDF acquisition with a single mobile phone image,” in Proceedingsof European Conference on Computer Vision (ECCV), 2018, pp. 72–87.
[46] K. Kim, J. Gu, S. Tyree, P. Molchanov, M. Nießner, and J. Kautz, “Alightweight approach for on-the-fly reflectance estimation,” in Proceedings of the IEEE International Conference on Computer Vision, 2017,
pp. 20–28.
[47] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” The Journal of Machine Learning Research, vol. 15, no. 1, pp.
1929–1958, 2014.
[48] X. Qi, R. Liao, Z. Liu, R. Urtasun, and J. Jia, “GeoNet: Geometric neuralnetwork for joint depth and surface normal estimation,” in Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition,
2018, pp. 283–291.
[49] V. Sze, Y.-H. Chen, T.-J. Yang, and J. S. Emer, “Efficient processing ofdeep neural networks: A tutorial and survey,” Proceedings of the IEEE,
vol. 105, no. 12, pp. 2295–2329, 2017.
[50] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
Proceedings of International Conference on Learning Representations(ICLR), 2014.
[51] B. Shi, Z. Wu, Z. Mo, D. Duan, S.-K. Yeung, and P. Tan, “A benchmark
dataset and evaluation for non-lambertian and uncalibrated photometricstereo,” in Proceedings of IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), 2016.
[52] M. K. Johnson and E. H. Adelson, “Shape estimation in natural illumination,” in Proceedings of IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), 2011, pp. 2553–2560.
[53] T. Papadhimitri and P. Favaro, “A closed-form, consistent and robust
solution to uncalibrated photometric stereo via local diffuse reflectancemaxima,” International journal of computer vision, vol. 107, no. 2, pp.
139–154, 2014.
[54] N. Alldrin, T. Zickler, and D. Kriegman, “Photometric stereo with nonparametric and spatially-varying reflectance,” in Proceedings of IEEE
Conference on Computer Vision and Pattern Recognition (CVPR). IEEE,
2008, pp. 1–8.
[55] B. Shi, P. Tan, Y. Matsushita, and K. Ikeuchi, “Elevation angle fromreflectance monotonicity: Photometric stereo for general isotropic reflectances,” Proceedings of European Conference on Computer Vision(ECCV), pp. 455–468, 2012.
[56] T. Higo, Y. Matsushita, and K. Ikeuchi, “Consensus photometric stereo,”in Proceedings of IEEE Conference on Computer Vision and Pattern
Recognition (CVPR). IEEE, 2010, pp. 1157–1164.
[57] S. Ikehata and K. Aizawa, “Photometric stereo using constrained bivariate regression for general isotropic surfaces,” in Proceedings of IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), 2014,
pp. 2179–2186.
[58] J. Li, A. Robles-Kelly, S. You, and Y. Matsushita, “Learning to minify
photometric stereo,” in Proceedings of IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), 2019.

Hiroaki Santo received his B.S. and M.S. degrees in computer science from Osaka University, Japan, in 2016 and 2018, respectively,
where he is currently working toward the Ph.D.
His research interests include physics-based vision and machine learning.

14

Masaki Samejima received the master’s andPh.D. degrees in information science from Osaka University, Japan, in 2007 and 2008, respectively. He was with Hitachi, Ltd., Japan, from2007 to 2009. He is currently an Assistant Professor with the Department of Multimedia Engineering, Graduate School of Information Science and Technology, Osaka University. His current research interests include applied machinelearning and mathematical optimization.

Yusuke Sugano is an associate professor at
Institute of Industrial Science, The University ofTokyo. His research interests focus on computer
vision and human-computer interaction. He received his Ph.D. in information science and technology from the University of Tokyo in 2010. He
was previously an associate professor at Graduate School of Information Science and Technology, Osaka University, a postdoctoral researcher
at Max Planck Institute for Informatics, and a
project research associate at Institute of Industrial Science, the University of Tokyo.

Boxin Shi received the BE degree from the
Beijing University of Posts and Telecommunications, the ME degree from Peking University, and
the PhD degree from the University of Tokyo, in
2007, 2010, and 2013. He is currently a Boya
Young Fellow Assistant Professor and Research
Professor at Peking University, where he leads
the Camera Intelligence Group. Before joining
PKU, he did postdoctoral research with MIT Media Lab, Singapore University of Technology and
Design, Nanyang Technological University from2013 to 2016, and worked as a researcher in the National Institute ofAdvanced Industrial Science and Technology from 2016 to 2017. He
won the Best Paper Runner-Up award at International Conference on
Computational Photography 2015. He has served as an editorial board
member of IJCV and an area chair of CVPR.

Yasuyuki Matsushita received his B.S., M.S.
and Ph.D. degrees in EECS from the Universityof Tokyo in 1998, 2000, and 2003, respectively.
From April 2003 to March 2015, he was with
Visual Computing group at Microsoft Research
Asia. In April 2015, he joined Osaka University
as a professor. His research area includes computer vision, machine learning and optimization.
He is/was an Editor-in-Chief for International
Journal of Computer Vision and on editorial
board of IEEE Transactions on Pattern Analysis
and Machine Intelligence (TPAMI), The Visual Computer journal, IPSJ
Transactions on Computer Vision Applications (CVA), and Encyclopediaof Computer Vision. He served/is serving as a Program Co-Chair of
PSIVT 2010, 3DIMPVT 2011, ACCV 2012, ICCV 2017, and a General
Co-Chair for ACCV 2014 and ICCV 2021. He is a senior member ofIEEE.