Learning to Minify Photometric Stereo
Junxuan Li1,2

Antonio Robles-Kelly3

Shaodi You1,2

Yasuyuki Matsushita41Australian National University, College of Eng. and Comp. Sci., Acton, ACT 2601, Australia
2
Data61-CSIRO, Black Mountain Laboratories, Acton, ACT 2601, Australia
3
Deakin University, Faculty of Sci., Eng. and Built Env., Waurn Ponds, VIC 3216, Australia
4
Osaka University, Graduate School of Information Science and Technology, Osaka 565-0871, Japan

Abstract
Photometric stereo estimates the surface normal given
a set of images acquired under different illumination conditions. To deal with diverse factors involved in the image
formation process, recent photometric stereo methods demand a large number of images as input. We propose amethod that can dramatically decrease the demands on thenumber of images by learning the most informative ones under different illumination conditions. To this end, we use adeep learning framework to automatically learn the critical illumination conditions required at input. Furthermore,
we present an occlusion layer that can synthesize cast shadows, which effectively improves the estimation accuracy. We
assess our method on challenging real-world conditions,where we outperform techniques elsewhere in the literature
with a significantly reduced number of light conditions.

GT

Ours

PS-FCN

CNN-PS

Figure 1. Performance with only 8 inputs for our method, PSFCN [3] and CNN-PS [8] on the “pot1” from DiLiGenT [17]. Note
we outperform the alternatives.

ages so as to minify the photometric stereo input. We approach this problem by learning the relative importance ofdifferent illuminant directions, which opens up a possibilityof optimally selecting input illuminant directions for lightweight data acquisition.
However, reducing the number of input illuminant directions without loss in performance is a non-trivial task because the surface radiance is determined by the illumination
intensity, illuminant direction, surface normal and BRDF
function. For instance, Argyriou et al. [2] use a sparse representation of illuminations for reducing the required number of lights to 5 under the assumption of Lambertian surfaces. The task is further complicated by the presence of
cast shadows, specularities, and inter-reflections.
In this paper, we propose a deep learning approach,which applies a connection table that can select those illuminant directions that are most relevant to the surface normal prediction process. To this end, we employ a connection table so as to introduce a trainable input mapping. This
mapping is trained by making use of the `1 -norm and asparsity-loss. Once the model is trained, it can efficiently
estimate the surface normal only given a much-reduced
number of input images without loss in accuracy. For thesake of the scalability of our method, here we select the illuminant directions making use of an observation map [8]and estimate the surface normal in a pixel-wise fashion.

1. Introduction
Photometric stereo is a technique that estimates the surface normal of an object from numbers of photographs
taken under different illumination conditions from a fixed
viewpoint. Due to the diverse appearances of real-world
materials, a method for dealing with general reflectances isdesired. Recent methods toward this direction improve theestimation accuracy by robust estimation at the cost of increasing the number of images, such as [21]; however, it
complicates the data acquisition setup and calibration procedure.
Recently, deep learning-based methods appeared in thecontext of photometric stereo [16, 8, 3]. These methods areshown effective for surfaces with diverse reflectances, indicating that the mapping from input images to surface normal
can be well established even with the diversity. Unlike thesemethods that primarily focus on the estimation accuracy, we
study the problem of reducing the number of required im1

Furthermore, to account for the effect of global illumination, we tackle the problem of cast shadows explicitly.
Specifically, by regarding cast shadows as a localized zeropattern on the observation map, we introduce an occlusion
layer into the network to deal with cast shadows. In summary, our contributions are:and Sankaranarayanan [7] use a virtually rendered sphereinstead of a physical one for normal estimation.
Deep learning: These methods are the ones that, recently, have achieved the best performance in photometricstereo. Santo et al. [16] propose the first deep networkbased method for pixel-wise estimation. In their model,
they assume that the light directions are all known andconsistent between the training and prediction phases.
Nonetheless effective, this assumption on the consistency of
the training and testing set greatly limits the generalization
capabilities of the method and makes it overly sensitive tothe training set. Ikehata [8] has proposed a deep net which
employs an observation map for unstructured photometricstereo inputs. This map is intended to account for every
illuminant direction at each surface pixel. Despite effective, the author assumes a dense illumination map, making
the method prone to corruption with a lesser amount of illuminant directions at input. In a different approach, Taniai
and Maehara [18] propose an unsupervised network that cantake the whole images at input and predict the dense normal
in an end-to-end fashion. This method has the drawback,however, of being computationally very costly. More recently, Xu et al. [23] propose a deep net to learn the optimal
samples for image-based relighting.

1. We propose a connection table in the network input together with a suitable loss function and a rank selection
process so as to select those illuminant directions thatare the most relevant to the surface normal prediction
process.
2. We propose an occlusion layer that can simulate object
cast shadows. This occlusion layer can be applied on
both augmented and real-world datasets, and enhance
performance on shadowed areas.
3. Our end-to-end deep neural network for photometricstereo can predict the surface normal with a muchreduced number of light source directions. This isconsistent with our results on the DiLiGenT benchmark [17], where we can predict the surface normal
with as few as 8 input images.

2. Related Work
The literature of photometric stereo is vast, it can be divided, however, into the following groups:
Least-squares methods: Introduced by Woodham [20],least-squares approaches aim at solving the problem under the Lambertian assumption, i.e., the pixel intensity is
proportional to the cosine of the angle between the illuminant direction and the surface normal. Moreover, they often
assume the surface is convex and devoid of cast shadowswith a uniform, Lambertian reflectance. The Lambertianassumption is important since it allows for the image formation model to be cast into a linear system of equations
that can be solved in a closed-form manner.
Robust approaches: These approaches can cope with
specularities and cast shadows on the object under study byviewing non-Lambertian regions as outliers. Wu et al. [21]add an additional term to the image formation model so as tonaturally represent the error, which accounts for those pixels that deviate from the Lambertian assumption and solve
the photometric stereo problem using rank minimization.
Many other techniques are also used for solving the problem under the assumptions above, such as RANSAC [14],expectation maximization [22], sparse regression [9] andvariational optimization approaches [15].
Example based methods: These approaches take exemplars of materials and surface geometry as the referenceto estimate the surface normal of an unknown object withthe same or similar BRDF function which are in the example set. Hertzmann and Seitz [5] propose a method thatfirst clusters the materials on the object. More recently, Hui

3. Preliminaries
In this section, we commence by providing the basic underpinnings and notations regarding photometric stereo.

3.1. Reflected Irradiance
Assume that a light source illuminates a surface pointfrom the direction l ∈ S 2 (the space of 3-dimensional unit
vectors) and its irradiance is e. And, respectively, the reflected radiance of this surface is denoted as r. The surface
normal is denoted as n ∈ S 2 . The BRDF function is thendenoted as ρ(l, v, n), i.e., a function of the incident and reflected radiance vectors l, v and the surface normal n.
Due to the variation of the normals and light source directions, shadows may appear at the reflecting surface. If
the normal is opposite to the light direction, i.e., l · n ≤ 0,
the attached shadow occurs. If the surface is occluded by
the object itself, the cast shadow also occurs, as shown in
Fig. 2. As a result, the observed reflected radiance r(v) canbe written asr(v) = Q (eρ(l, v, n) max(l · n, 0)) ,(1)
where Q ∈ {0, 1} is a binary variable with a value of 0 at
cast shadows, assuming that there are no inter-reflectionsin shadowed areas. Hence, as long as the illumination is
occluded by objects, the indicator variable Q will be set tozero regardless.
Following the conventional calibrated photometricstereo problem, we assume that the light direction l and2Light source

Cameralationship between the reflectance properties of the object
under different illuminant directions. As shown in Fig. 2,
each value on the map naturally encodes the information of
the surface reflectance r, illuminant direction l and intensity e. Moreover, the map can also be viewed as a group offunctions of the normal n, the BRDF ρ and the cast shadow
indicator variable Q.

Observation map

Cast shadow

Attached shadow

3.3. illuminant direction Selection
We now turn our attention to the learning of those illuminant directions that are the most relevant to the photometric stereo process so as to greatly minimize the number ofviews required to compute the surface normal. Mathematically, if we have k different illuminations with directions
L = [l(1) , l(2) , · · · , l(k) ]> ∈ Rk×3 , and for each directionof the lights the relative intensities on a surface point are
given by m = [m(1) , m(2) , · · · , m(k) ]> ∈ Rk . Then foreach surface point we have a system of equations given by

Figure 2. Shadowed area on “harvest” from the DiLiGenT [17]dataset. Note cast shadows are caused by occlusions, while attached shadowed areas are facing back with respect to the lightsource. On the right-hand side of the figure, we show two observation maps obtained from cast shadows. The red dash line on theobservation map indicates the shape that occludes the lights.

intensity e are given at each illumination condition. Also,
we assume that the light source is in distance over the images. In order to simplify the ensuing equations, we consider the camera to be in orthographic position and, hence,v is [0, 0, 1]. Finally, here we use, without any loss of generality, the relative intensity m = re to rewrite the Eq. (1) as
followsm = Q ρ(l, v, n) max(l · n, 0) .
(2)

m = Q ◦ ρ(l, v, n) ◦ vmax(Ln, 0),(3)where Q ∈ {0, 1}k is a vector encoding the cast shadow
information under different illuminations, ◦ denotes the
Hadamard product, i.e., an element-wise multiplication operator, and similarly, vmax is an element-wise max operator. Our goal is hence to solve this system of equation withk being as small as possible.

4. Proposed Method

3.2. Observation Map Computation

Here, we propose a neural network that can take sparse
illuminations as inputs and predict the surface normal witha marginal loss in accuracy. As shown in Fig. 3, the observation map can encode different lighting directions into afeature map, making the selection over the input space feasible. Then an occlusion layer renders the system robust to
cast shadows. For the network itself, we have used a variation of the DenseNet [6] architecture.
To the effect of selecting the relevant illuminant directions, we employ a learnable connection table at the input ofour deep network. As the network undergoes training, ourmethod can predict the surface normal accurately by taking
at input a significantly less number of illuminant directions
than other methods elsewhere in the literature.

We follow Ikehata [8] and adopt the use of an observation map. Recall that an observation map is a 2-D projection of all the lighting directions in a 3-D hemisphere.
More formally, assume that a light direction is given in thex, y, z -coordinate system by the vector l = [lx , ly , lz ]> .
In contrast with Ikehata [8], we do not use the Cartesian coordinates directly, but rather compute the observation map by first projecting the Cartesian vector l into apolar coordinate system whose components
q are given bypθ = arcsin(lx / lx2 + lz2 ), ϕ = arcsin(ly / ly2 + lz2 ). Here,
we have used a different size of observation maps as compared to Ikehata [8], and have noted that the performanceof the polar coordinates outperforms Cartesian ones for thissize of observation maps.



ϕ+b
of the observaThe entry indexed int θ+b∆θ , int ∆ϕ

4.1. Connection Table

w×w

tion map O ∈ Rwith a window size w can be then
set to m. In the indexes above, b is the upper and lower
bound of the range of θ and φ and int(·) is an operator therounding operator that delivers the closest integer to its argument, and ∆θ and ∆ϕ are the gap between each of the2b
map indexes given by ∆θ = ∆ϕ = w−1.
In this manner, the observation map converts un-ordered
inputs into a meaningful feature map which captures the re-To the best of our knowledge, there are no learning based
approaches aiming to find the relevant input feature maps
for photometric stereo. However, as related to deep networks, Alvarez et al. [1] use sparse constraints for regularizing the training process of a deep neural network. Koray et al. [11] reduce the number of parameters and, hence,the complexity of deep networks using a binary connection table to “disconnect” a random set of feature maps.
3

Objects under
different illuminations

Illumination
directions

Observationmap

Occlusion
layer

Connection
table
CNN blocks

OutputObservation map
computation

Figure 3. Overview of our network. Given the images of objects under different illuminations, we can compute its corresponding observation map at each pixel. Then we apply an occlusion layer which operates as a binary filter mask so as to zero out an area of the observation
map. We then employ a learned connection table to weigh the feature map at input. After training, only a few illuminations corresponding
to the non-zero weights of the connection table are necessary for our network to predict the surface normal.
2bfined by the angular intervals i.e., ∆θ = ∆ϕ = w−1
withrespect to the illuminant direction. This is an important observation since the connection table can also be interpreted
as relevance for the corresponding range of the illuminantdirection, which renders the method more robust to small
variations in the illuminant direction.

Wei et al. [19] employ a one-dimensional connection tableto perform band reduction in imaging spectroscopy. These
contrast with our method, which aims at recovering a twodimensional sparse input connection table with non-binary
weights over the observation map.
4.1.1

Sparse Table Computation
4.1.2

Let the input connection table be denoted by C. This table
is applied to the observation map after an occlusion layer.
For now, we concentrate on the connection table and will
elaborate further on the occlusion layer later in the paper.
This table effectively acts as an input selector whereby theconnection weights are learned at training. Note that theconnection table has the same size as the observation map,C ∈ Rw×w , with all its parameters larger or equal to zero,Ci,j > 0. The sparse observation map is then yielded byan element-wise product with the observation map after it
has been “filtered” by the occlusion layer. This generates asparse observation map as the input for the network.
The loss function of the network is then given by two
terms. The first term accounts for the surface normal prediction, where we use the mean square error. The second
term is a regularization one over the connection table. These
yield the loss given by
L = kn − n0 k22 + λg(C),(4)

Connection Table Training

Here, to make the connection table sparse, we employ theregularization function given by
!
2
X
Ci,j
g(C) =
2Ci,j −
,
(5)
2α
i,j
where α is the maximum value of the map, i.e.,
α = max(Ci,j ) ∀ i, j.
Note that the function above is minimum as the connection table weights tend to zeros. Moreover, at each step of
the back-propagation, the connection table can be updated
in a straightforward manner using the derivative given by
∂g(C)
Ci,j
=2−
.
(6)
∂Ci,j
α
Since the connection table by definition is non-negative, thederivative ∂g(C)∂Ci,j takes values in the range of [1, 2].
With the connection table in hand, we can apply a selection strategy to select the k-most important inputs to the
network. To this end, we use a recurrent training scheme
and, at each training operation, we apply a rank selection
with exponential decay, i.e., k = int(c + τ e−βt ), wheret = {0, 1, 2, . . .} is the training operation index and c, βand τ are scalar parameters. It is worth noting in passingthat our choice of exponential decay in the number of table entries that are not null is reminiscent of the exponential
cooling strategy in annealing methods and rank selection
approaches elsewhere in the literature.

where n and n0 are the predicted and ground truth surface
normals, respectively, g(·) denotes the regularization function and λ is a hyper-parameter that controls the contribution of the regularize to the overall loss. Thus, a large λ will
enforce a more sparse connection table and, conversely, asmaller λ will yield a denser one.
Note that, in this manner, the connection table can beused to select the most relevant illuminant directions at input. These directions in observation map are effectively de4λ = 10!" MAE=6.88

λ = 10!# MAE=7.78

λ = 10!$ MAE=9.19

Figure 4. Observation map processed by an occlusion layer ascompared to being randomly zeroed [16]. Note that the occlusion layer effectively simulates the pattern in the observation map
induced by cast shadows.

λ = 10!% MAE=10.87

λ=1

λ = 104.2. Occlusion Layer

Figure 5. Visualization of the connection tables (cropped) and themean angular errors on the validation set as a function of λ.

Occlusion layer

Random zero

Recall that cast shadows are one of the most challenging
problems in photometric stereo. It is particularly true when
the number of input illuminant directions is small. Santo etal. [16], tackle this problem by making use of a shadow
layer in their network which randomly sets some of the inputs to zero.
Despite effective, their approach does not take into account that cast shadows are often consistent with respect tothe illuminant direction rather than occurring randomly. To
illustrate this, we show, in Fig. 2 the observation map obtained from a shadowed area in the “harvest” object. The
cast shadow is caused by the wrinkles on the cloth andshows a consistent pattern with a relatively sharp boundary. Locally, a smooth or flat occlusion results in a straightboundary while sudden changes in the object geometry often yield more abrupt boundaries.
Thus, here we employ an occlusion layer that can effectively simulate the cast shadows by randomly drawing aline across the observation map to obtain a mask that canthen be used to set the corresponding entries of the map tozero. Specifically, the layer randomly selects two sides ofthe map, and randomly picks a point on each side. Thenthe line connecting these two points divides the map into
two regions. The smaller of these two regions is then set
to zero. This is illustrated in Fig. 4. By training the network with this occlusion layer, our proposed method caneffectively learn the pattern of cast shadows and predict thesurface normal more robustly.

30

16

10

8

MAE=11.06

6!(#) !1-#$%&

55

MAE=11.03

Figure 6. Connection tables (cropped) after rank selection. Fromleft-to-right, each panel shows the table as the number of non-zerovalues decreases down to 6. The top row shows the maps yieldedusing an `1 -norm regularizer. The second row shows the maps
obtained using the regularization function in Eq. (5).

Pre-training For the connection table, as mentioned earlier, we perform rank selection as we recursively train the
network. To this end, we initialize the table to be all ones
and proceed to pre-train it. For the pre-training, we use analternative loss function given by
L = kn − n0 k22 + λf (C),(7)

where f (·) is given by the `1 -norm regularizer.
The reason for using the `1 -norm at pre-training and theregularizer g(·) later is due to the fact that, in contrast withthe behavior of g(·), the `1 -norm decreases as the magnitude of the connection table entries tends to zero. Thus, wehave employed the `1 -norm to provide a good initialization
for the training and employed the regularizer in Eq. (5) tofavor values of the table that are either close to unity or to
zero. Moreover, nonetheless λ can control the sparsity of
the connection table when the `1 -norm is used, it quickly
becomes ineffective as it increases. To illustrate this behavior, we show, in Fig. 5, the connection table obtained withdifferent values of λ at pre-training. We can see from thefigure that, as λ increases beyond 10−1 , the connection table shows virtually no change.

5. Implementation
Observation Map Parameters For the observation map,
we have set the window size w to 14. And the upper andlower bounds of θ and ϕ to π4 . With these parameters in
hand, we then randomly select 5% of the observation maps
to be put through our occlusion layer, which simulates theeffect of cast shadows. Besides, we also randomly set about
5% points of observation map to be zero. We discuss further the choice of this fraction of the observation maps inSec. 6.2.

Connection Table Rank Selection As mentioned above,
we commence by pre-training the network. Here, we haveset λ = 10−3 for the pre-training operation and then applied, rank selection to the connection table over a set ofsubsequent, recurrent training steps. As described earlier,5Inputs

Training
96

Methods
Ours-g(·)
Ours-L1
Ours-Random
PS-FCN [3]
CNN-PS [8]
L1-RES [9]Baseline [20]

16

10

8

6

9.66
10.29
10.50
9.37
10.49
15.47
16.65

10.02
11.06
11.39
10.51
14.34
16.37
17.31

10.39
11.33
11.85
11.42
19.50
16.84
17.47

12.16
12.57
12.58
12.54
30.28
18.45
18.60

Synthesis
8.43
8.39
7.20
14.08
15.39

Realworld

Methods
Occlusion layer
Random zeroingOcclusion layer
Random zeroing

0%
11.42
8.28

5%
10.42
10.75
7.99
8.33

10%
10.68
10.87
8.04
8.13

15%
10.57
10.94
7.85
8.37

Table 2. Comparison of our occlusion layer and random zeroing[16] on the DiLiGenT dataset for 10 illumination directions.

For the quantification of the error, in all our experiments
we have used the mean angular error (MAE) for surface normal evaluation. Here, we compare the results by our method
to those obtained using a number of alternatives [16, 3, 8, 9]that can be used with sparse inputs. We also illustrate theeffectiveness of our loss function and our occlusion layerand show results on pixel-wise normal estimation.

Table 1. Performance on the DiLiGenT dataset with different number of input light directions (see Sec. 6.1 for more details).

in this manner we select the top k largest weights, with ksteadily decreasing over the training process from all illuminations in the dataset (96) down to 6 following the sequence55, 30, 16, 10, 8, 6 in k.
In Fig. 6 we present the connection tables at each training step after the rank selection has been affected. In theleft-hand panel, we show the connection table obtained at
the first training step after the 55 most significant elementsof the learned table are selected and the rest are set to zero.
The second panel, from left-to-right, shows the table where
the 30 most significant entries have been selected after thesecond training operation and so on. For all training operations, we have set λ = 10−3 . Note this is consistent with
our pre-training operation.

6.1. Effectiveness of the Regularization Function
To illustrate the effectiveness of our regularizer g(·) inthe loss function, we now compare the performance of our
network when other regularization functions are used. In
Table 1, “Ours-g(·)” denotes the network trained using thefunction g(·) described in Eq. (5). “Ours-L1” corresponds
to the results yielded by our network when the `1 -norm regularizer is used as an alternative to g(·). Finally, the “OursRandom” corresponds to the case where no regularizer is
included and the loss function and the connection table israndomly initialized rather than pre-trained. All the othermethods are tested with randomly selected inputs with all
the results shown are averaged over 10 trails. As we can see
from the table, our model with the sparsity regularization
function g(·) performs the best and clearly outperforms the`1 -norm and random initialization.

5.1. Dataset
To train our network, we generate a synthetic dataset using the 10 shapes from the Blobby Shape Dataset [10] and100 reflectances from MERL BRDF dataset [13]. To dothis, we randomly select 8 BRDFs to repeatedly render eachobject using 144 randomly distributed light directions. For
the sake of consistency, we have used the same range oflight directions as that used in the DiLiGenT [17] dataset.
We choose 9 shapes for training and the remaining one
for validation. Thus, our synthetic dataset comprises approximately 9 × 104 × 8 = 720, 000 observation maps
for training and about 80, 000 maps for validation. For thetesting, we use the DiLiGenT [17] dataset, which is a realworld public benchmark containing the images of 10 distinct objects. Each object is observed under 96 different
known light directions, of which the horizontal direction
ranges from −45◦ to 45◦ and vertically from −30◦ to 30◦
over a hemisphere centered at the viewing direction. The
ground truth surface normals have been acquired using alaser scanner and are available for quantitative evaluation.

6.2. Evaluation of the Occlusion Layer
Now we turn our attention to the effects of the occlusion
layer. To this end, we have trained the network with both,our occlusion layer and that proposed by Santo et al. [16]and set k = 10, i.e., used 10 light directions at input. Here,
we have trained the network on both, our synthetic dataset
and DiLiGenT.
In Table 2, we show the effects of our occlusion layer andthat proposed by Santo et al. [16]. Note that in our synthetic
dataset there are no cast shadows. This contrasts with thereal-world objects in the DiLiGenT dataset, which exhibit adiversity of shadows and inter-reflections. The purpose of
the comparison shown here is not only showing the benefitsof our occlusion layer as compared to that in [16] but also
showing that it can benefit the training of real-world andsynthetic datasets alike.
Also, note that the percentages in the table have a different meaning for both approaches. For our occlusion
layer, the percentages indicate how many of the observation
maps are “masked” with our occlusion layer during training. While the percentages of randomly zeroing follow the

6. Experiments
For all our experiments we have trained our network using Adam [12] optimizer with the default setting. The network is implemented by Keras with Tensorflow as the backend on a workstation using Ubuntu-14 with an NVIDIAGTX 1080Ti 11G.
6

25 (Worse)

0 [deg.]-25 (Better)

10 Inputs

Figure 7. Performance comparison of our occlusion layer with respect to random zeroing [16]. The blue color in the panels corresponds tothe area where the occlusion layer outperforms the random zeroing. The converse applies to the red regions. We can see that, in the “Cat”
image, the occlusion layer is better in most of the shoulder region. In the “Buhhda” image, which is quite textured, our method outperforms
the random zeroing on all the cloth wrinkles. All of these areas correspond to cast shadows.
methods
Ours
PS-FCN[3]CNN-PS[8]L1-RES[9]Baseline [20]

ball
3.97
4.02
9.11
4.92
5.09

cat
6.69
8.30
11.71
9.68
9.66

pot1
7.30
10.14
13.23
10.69
11.32

bear
8.73
7.18
14.08
9.58
11.59

pot2
9.74
9.85
14.65
17.64
18.03

buddha
11.36
9.79
14.58
14.53
16.25

goblet
10.46
11.58
15.48
18.39
19.97

reading
14.37
15.03
16.99
18.23
19.86

cow
10.19
10.51
14.04
27.01
27.90

harvest
17.33
18.70
19.56
33.05
33.41

Avg.
10.02
10.51
14.34
16.37
17.31

8 Inputs

Ours
3.65
7.07
8.11
8.91
9.92
12.03
10.67
15.27
10.14
18.15
10.39
PS-FCN[3]
5.14
9.47
10.65
9.16
10.05
9.66
12.39
16.47
11.14
20.05
11.42
CNN-PS[8]22.74 21.06 18.87 16.89 19.18
19.95
18.62
17.77
16.04
23.90
19.50
L1-RES[9]4.70
9.57
11.96 10.10 17.33
15.62
19.07
18.96
28.23
32.82
16.84
Baseline [20]
5.50
9.89
12.01 11.57 17.49
16.53
19.73
20.41
28.64
32.96
17.47
Table 3. Results on the DiLiGenT dataset with 10 and 8 input illumination directions. All the alternatives are tested with randomly selected
inputs. All the mean angular errors shown correspond to the average performance over 10 trails.

ble 1. Note that in Table 1, all the methods except PSFCN [3] estimate surface normal in a pixel-wise fashion.
Since pixel-wise methods naturally discard the connection
between pixels and shape, a sparse input setting like theone used here is expected to decrease the performance dramatically. As we can see, the performance of CNN-PS [8]decreases from 7.20 to 30.28 at 6 inputs. In comparison,
our method do not exhibit such loss in performance, outperforming the other pixel-wise methods. We can see thatfrom 16 inputs down, our method starts to show its ability
to select the illuminant directions to avoid an overly loss
in performance. At 10 and 8 input directions, our method
shows a large improvement in performance with respect tothe other methods. Moreover, our method shows a decreasein performance as low as 1.25 degrees on MAE from 96
to 16 input directions and only 0.3 degrees from 16 to 10
inputs.
Also, note that PS-FCN [3] is a dense normal estimation method which takes the whole image into account.
Hence, PS-FCN [3] naturally has advantages on sparse inputs as it can utilize the structural information across pixels. Nonetheless, our method can perform better than PSFCN [3] in highly sparse illumination settings with 10 orless light source directions. In Table 3, we also show theperformance of our method and the alternatives on each object in DiLiGenT dataset for 10 and 8 inputs. In Fig. 8,
we show the surface normal and error maps of the “Goblet”
object from DiLiGenT.

setting in [16], indicating how many pixels at each input
map are randomly set to zero.
As we can see from the table, our occlusion layer performs better when 5% of the input maps are occluded. For
the random zeroing strategy, the best performance appears
to be also about 5%. Nonetheless, regardless of the percentage used, the lowest error is always achieved by our
occlusion layer. Finally, we show the error map for a number of sample objects in the DiLiGenT dataset in Fig. 7.
The error maps shown correspond to the difference between
the MAE yielded by our occlusion layer and random zeroing. Hence the negative areas, i.e., blue regions in Fig. 7 arethose where our occlusion layer outperforms the alternative.
From the figure, it is clear that the occlusion layer performs
much better at most of the rugged and wrinkled areas, where
cast shadows are stronger. This is consistent with the notionthat our occlusion layer can help the network cope with castshadows.

6.3. Benchmark Comparison Under Sparse Inputs
We commence by comparing our method with otherstate-of-art approaches using the DiLiGenT dataset. For the
results yielded by the alternatives in sparse illuminations,and in contrast with our method, we have randomly selected
input illuminant directions up to the k under consideration.
Note that this is in accordance with the sparse input setting
proposed by the authors.
The MAE for the dataset over 10 trails is shown in Ta7

Ours

PS-FCN

CNN-PS

L1-RES

Baseline

10 InputsGT

90 [deg.]

11.21

15.41

18.92

18.90

8 Inputs

10.46

0 [deg.]90 [deg.]

10.67

12.29

18.60

18.81

19.56

0 [deg.]Figure 8. Performance of “Goblet” in DiLiGenT dataset in only 10 and 8 inputs.
Light-Config
RandomOurs
Optimal [4]Ours

PS-FCN[3]CNN-PS[8]L1-RES[9]Baseline[20]

10.02

10.51
11.35
8.73

14.34
13.02
13.35

16.37
15.83
15.50

17.31
17.12
16.57

Table 4. Mean angular errors on the DiLiGenT benchmark using the same 10 lights directions obtained using random sampling(Random), our approach (Ours) and the method in [4].

InputsGT

Ours Err:13.77 PS-FCN Err:18.03

Figure 9. Performance of our method and PS-FCN for a spatiallyvarying BRDF when 8 input light source directions are shown.

Table 4 shows the mean angular errors for our and othermethods under similar light configurations. The first row(Random) shows the result when 10 lights are randomly
sampled from a uniform distribution over the hemisphere.
The second row (Ours) is the result under the light directions that correspond to the non-zero entries of the connection map learned by our method. The third row (Optimal)
correponds to the optimal light configuration proposed by
Drbohlav and Chantler [4]. Note that, these light directions
do not precisely match those provided in DiLiGenT, we use
the closest ones in the DiLiGenT dataset for testing. Our
method outperforms others except for the PS-FCN [3] under the optimal light configuration.

despite some noise, still recovers a surface normal in goodaccordance with the object itself, outperforming the PSFCN [3].

7. Conclusion
In this paper, we have proposed a deep learning approach
which applies a connection table that can select those illuminant directions that are most relevant to the surface normal prediction process. To this end, we have employeda connection table and showed how, once the model istrained, it can efficiently estimate the surface normal only
given a much-reduced number of input images without loss
in accuracy. Furthermore, we have tackled the problem ofcast shadows explicitly by introducing an occlusion layer
into the network. We have illustrated the utility of our
method for photometric stereo by comparing our resultswith those yielded by a number of alternatives.

Pixel-wise normal estimation. Finally, we present a
comparison between pixel-wise and dense normal estimation. In Fig. 9, we show the results yielded by our method
and PS-FCN [3] on a challenging synthetic object. In
Fig. 9, we use one object from Blobby [10] which has been
excluded from our training dataset and render it by randomly selecting a BRDF function from MERL [13] for eachpixel. We then test both methods using only 8 inputs. Notethat PS-FCN [3] suffers from the relationship between the
BRDF and the geometry of the object whereas our method,

Acknowledgments: Part of this work was supported by
JST CREST Grant Number JP17942373, Japan.
8References

photometric stereo under inaccurate lighting. In Computer
Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, pages 350–359. IEEE, 2017. 2[16] Hiroaki Santo, Masaki Samejima, Yusuke Sugano, Boxin
Shi, and Yasuyuki Matsushita. Deep photometric stereo network. In Computer Vision Workshop (ICCVW), 2017 IEEE
International Conference on, pages 501–509. IEEE, 2017. 1,
2, 5, 6, 7
[17] Boxin Shi, Zhipeng Mo, Zhe Wu, Dinglong Duan, Sai Kit
Yeung, and Ping Tan. A benchmark dataset and evaluation for non-lambertian and uncalibrated photometric stereo.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018. 1, 2, 3, 6
[18] Tatsunori Taniai and Takanori Maehara. Neural inverse rendering for general reflectance photometric stereo. In International Conference on Machine Learning, pages 4864–4873,
2018. 2[19] Ran Wei, Antonio Robles-Kelly, and José Álvarez. Context
free band reduction using a convolutional neural network.
In Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR), pages 86–96. Springer,
2018. 4[20] Robert J Woodham. Photometric method for determining
surface orientation from multiple images. Optical engineering, 19(1):191139, 1980. 2, 6, 7, 8
[21] Lun Wu, Arvind Ganesh, Boxin Shi, Yasuyuki Matsushita,
Yongtian Wang, and Yi Ma. Robust photometric stereo via
low-rank matrix completion and recovery. In Asian Conference on Computer Vision, pages 703–717. Springer, 2010. 1,
2[22] Tai-Pang Wu and Chi-Keung Tang. Photometric stereo via
expectation maximization. IEEE transactions on patternanalysis and machine intelligence, 32(3):546–560, 2010. 2[23] Zexiang Xu, Kalyan Sunkavalli, Sunil Hadap, and Ravi
Ramamoorthi. Deep image-based relighting from optimalsparse samples. ACM Transactions on Graphics (TOG),
37(4):126, 2018. 2[1] Jose M Alvarez and Mathieu Salzmann. Learning the number of neurons in deep networks. In Advances in Neural Information Processing Systems, pages 2270–2278, 2016. 3[2] Vasileios Argyriou, Stefanos Zafeiriou, Barbara Villarini,and Maria Petrou. A sparse representation method for determining the optimal illumination directions in photometricstereo. Signal Processing, 93(11):3027–3038, 2013. 1[3] Guanying Chen, Kai Han, and Kwan-Yee K Wong. Ps-fcn: A
flexible learning framework for photometric stereo. In European Conference on Computer Vision, pages 3–19. Springer,
2018. 1, 6, 7, 8
[4] Ondrej Drbohlav and Mike Chantler. On optimal light configurations in photometric stereo. In Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1,
volume 2, pages 1707–1712. IEEE, 2005. 8[5] Aaron Hertzmann and Steven M Seitz. Example-based photometric stereo: Shape reconstruction with general, varying
brdfs. IEEE Transactions on Pattern Analysis and MachineIntelligence, 27(8):1254–1264, 2005. 2[6] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Computer Vision and Pattern Recognition (CVPR),
2017 IEEE Conference on, pages 2261–2269. IEEE, 2017. 3[7] Zhuo Hui and Aswin C Sankaranarayanan. Shape andspatially-varying reflectance estimation from virtual exemplars. IEEE Transactions on Pattern Analysis and MachineIntelligence, 39(10):2060–2073, 2017. 2[8] Satoshi Ikehata. Cnn-ps: Cnn-based photometric stereo forgeneral non-convex surfaces. In European Conference on
Computer Vision, pages 3–19. Springer, 2018. 1, 2, 3, 6, 7, 8
[9] Satoshi Ikehata, David Wipf, Yasuyuki Matsushita, and Kiyoharu Aizawa. Robust photometric stereo using sparse
regression. In Computer Vision and Pattern Recognition(CVPR), 2012 IEEE Conference on, pages 318–325. IEEE,
2012. 2, 6, 7, 8
[10] Micah K Johnson and Edward H Adelson. Shape estimation in natural illumination. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages
2553–2560. IEEE, 2011. 6, 8
[11] Koray Kavukcuoglu, Pierre Sermanet, Y-Lan Boureau, Karol
Gregor, Michaël Mathieu, and Yann L Cun. Learning convolutional feature hierarchies for visual recognition. In
Advances in neural information processing systems, pages
1090–1098, 2010. 3[12] Diederik P Kingma and Jimmy Ba. Adam: A method forstochastic optimization. arXiv preprint arXiv:1412.6980,
2014. 6[13] Wojciech Matusik, Hanspeter Pfister, Matt Brand, andLeonard McMillan. A data-driven reflectance model. ACM
Transactions on Graphics, 22(3):759–769, July 2003. 6, 8
[14] Yasuhiro Mukaigawa, Yasunori Ishii, and Takeshi Shakunaga. Analysis of photometric factors based on photometriclinearization. JOSA A, 24(10):3326–3334, 2007. 2[15] Yvain Quéau, Tao Wu, François Lauze, Jean-Denis Durou,
and Daniel Cremers. A non-convex variational approach to9