What is Learned in Deep Uncalibrated
Photometric Stereo?
Guanying Chen1 Michael Waechter2 Boxin Shi3,4
Kwan-Yee K. Wong1 Yasuyuki Matsushita2
1

The University of Hong Kong 2 Osaka University
4
Peng Cheng Laboratory

3

Peking University

Abstract. This paper targets at discovering what a deep uncalibrated
photometric stereo network learns to resolve the problem’s inherent ambiguity, and designing an effective network architecture based on the
new insight to improve the performance. The recently proposed deep
uncalibrated photometric stereo method achieved promising results inestimating directional lightings. However, what specifically inside the
network contributes to its success remains a mystery. In this paper, weanalyze the features learned by this method and find that they strikingly
resemble attached shadows, shadings, and specular highlights, which
are known to provide useful clues in resolving the generalized bas-relief(GBR) ambiguity. Based on this insight, we propose a guided calibration network, named GCNet, that explicitly leverages object shape andshading information for improved lighting estimation. Experiments on
synthetic and real datasets show that GCNet achieves improved resultsin lighting estimation for photometric stereo, which echoes the findingsof our analysis. We further demonstrate that GCNet can be directly integrated with existing calibrated methods to achieve improved resultson surface normal estimation. Our code and model can be found athttps://guanyingc.github.io/UPS-GCNet.
Keywords: Uncalibrated photometric stereo, generalized bas-relief ambiguity, deep neural network

1

Introduction

Photometric stereo aims at recovering the surface normals of a scene from singleviewpoint imagery captured under varying light directions [50, 47]. In contrast
to multi-view stereo [41], photometric stereo works well for textureless surfaces
and can recover highly detailed surface geometry.
Following the conventional assumption, this paper assumes the scene isilluminated by a single light direction in each image. Most existing photometric stereo methods [46, 23, 22] require calibrated light directions as input.
Uncalibrated photometric stereo, on the other hand, simultaneously estimates
light directions and surface normals. In multi-view stereo, this problem of
auto-calibration (i.e., calibration from images of the scene without the use

2G. Chen et al.

of any explicit calibration targets) has been solved satisfactorily on “wild”
imagery such as those from Internet photo sharing sites [3]. Auto-calibration
for photometric stereo is without a doubt an important goal since it makes
photometric stereo applicable to wild data [43] and useful for amateurs who
know nothing about tedious calibration procedures [42, 1, 39]. Existing methods
for uncalibrated photometric stereo [4, 35] often assume a Lambertian reflectance model and their focus has been on resolving the generalized bas-relief(GBR) ambiguity [6]. Manifold embedding based methods [40, 29] can deal withsurfaces with general isotropic reflectances, but they rely on a roughly uniform
lighting distribution which is usually not satisfied in real-world datasets.
Despite the impressive results on complex reflectances reported by recent
deep learning methods for calibrated photometric stereo [38, 49, 10, 22, 27, 53],
not much work has been done on learning-based uncalibrated photometric stereo.
Recently, Chen et al. [8] introduced a deep uncalibrated photometric stereo network, called Lighting Calibration Network (LCNet), to estimate light directions
and intensities from input images, and a normal estimation network to predict
surface normals. Compared with UPS-FCN [10] which directly estimates surface
normals from images, Chen et al.’s two-stage approach achieves considerably
better results. However, the features learned by LCNet to resolve the ambiguity
in lighting estimation remain unknown.
This paper focuses on demystifying the problem of how deep uncalibrated
photometric stereo learns to resolve the GBR ambiguity, and how to improve itfor higher accuracy in lighting estimation. Our contributions are:
– We discuss the differences between the learning-based LCNet [8] and traditional uncalibrated methods, and analyze the features learned by LCNet toresolve the GBR ambiguity.
– We find that attached shadows, shadings, and specular highlights are key
elements for lighting estimation, and that LCNet extracts features independently from each input image without exploiting any inter-image information(“inter-image” means information shared by all images).
– Based on our findings, we propose a guided calibration network (GCNet)that explicitly utilizes object shape and shading information as guidances
for better lighting estimation.

2

Related WorkIn this section, we briefly review recent deep learning methods for calibrated
photometric stereo and existing methods for uncalibrated photometric stereo.
Readers are referred to [20, 2, 45] for more comprehensive surveys of photometricstereo algorithms. In the rest of this paper, we will use “lighting” to refer to light
direction and light intensity.
Deep calibrated photometric stereo Recently, deep learning methods havebeen proposed in the context of photometric stereo. Compared with traditional
methods that often adopt a simplified reflectance model, learning-based methods can directly learn the mapping from observations to surface normals andWhat is Learned in Deep Uncalibrated Photometric Stereo?

3

achieve state-of-the-art results on a real-world benchmark [45] with complex reflectances. Santo et al. [38] first introduced a fully-connected deep photometricstereo network to estimate pixel-wise surface normals from a fixed number ofobservations. To handle a variable number of input images in an order-agnostic
manner, Ikehata [22] proposed a fixed shape observation map representation,
while Chen et al. [9] adopted an element-wise max-pooling operation to fuse features stemming from multiple inputs. Li et al. [27] and Zheng et al. [53] focusedon reducing the number of required images while maintaining similar accuracy
under the framework proposed by Ikehata [22]. Different from the above supervised methods that require a synthetic dataset for training, Taniai and Maehara [49] proposed an unsupervised framework to estimate surface normals via
an on-line optimization process.
Uncalibrated photometric stereo Most existing uncalibrated photometricstereo methods are based on matrix factorization (e.g., singular value decomposition) and assume a Lambertian reflectance model. A Lambertian surface’s
normals can be recovered up to a 3 × 3 linear ambiguity when light directions
are unknown [19]. By considering the surface integrability constraint, this linear ambiguity can be reduced to a 3-parameter generalized bas-relief (GBR)
ambiguity [15, 6, 52, 26]. To further resolve the GBR ambiguity, many methods
make use of additional clues like inter-reflections [7], specularities [13, 16, 12],
albedo priors [4, 44], isotropic reflectance symmetry [48, 51], special light source
distributions [54], or Lambertian diffuse reflectance maxima [35].
Manifold embedding methods [40, 34, 29, 30] can handle surfaces with generalisotropic reflectance based on the observation that the distance between two surface points’ intensity profiles is closely related to their surface normals’ angulardifference. However, these methods often assume a uniform lighting distribution.
Other methods related to uncalibrated photometric stereo include exemplarbased methods [21], regression-based methods [32], semi-calibrated photometricstereo [11], inaccurate lighting refinement [37], and photometric stereo undergeneral lighting [5, 33, 18].
Recently, Chen et al. [8] introduced a Lighting Calibration Network (LCNet)to estimate lightings from images and then estimate surface normals based on
the lightings. This two-stage method achieves considerably better results than
the single-stage method [10]. It also has slightly better interpretability because
the lightings estimated in the first stage can be visualized. However, the features
learned by LCNet to estimate lightings remain unknown.

3

Learning for Lighting Calibration

In this section, we discuss the inherent ambiguity in uncalibrated photometricstereo of Lambertian surfaces, the fact that it can be resolved for non-Lambertiansurfaces, and the features learned by LCNet [8] to resolve such ambiguity.
Lambertian surfaces and the GBR ambiguity When ignoring shadows(i.e., attached and cast shadows) and inter-reflections, the image formation of a

4

G. Chen et al.
(d) sample images(a) normal (b) albedo (c) lighting(e) PF14(f) LCNet

0

1
0

1
0

1

Fig. 1. Row 1 is the true shape of a Sphere, while rows 2 and 3 are shapes under
two different GBR transformations. In column (c), the points’ positions and colors
indicate light direction and relative intensity, respectively. Columns (e) and (f) show
the lightings estimated by PF14 [35] and LCNet [8].

Lambertian surface with P pixels captured under F lightings can be written as
M = N> L,(1)

where M ∈ RP ×F is the measurement matrix. N ∈ R3×P is the surface normal
matrix whose columns are albedo scaled normals N:,p = ρp np , where ρp and np
are the albedo and unit-length surface normal of pixel p. L ∈ R3×F is the lighting
matrix whose columns are intensity scaled light directions L:,f = ef lf , where ef
and lf are the light intensity and unit-length light direction of image f .
By matrix factorization and applying the surface integrability constraint, N
and L can be recovered up to an unknown 3-parameter GBR transformation
G [6] such that M = (G−> N)> (GL). This GBR ambiguity indicates that there
are infinitely many combinations of albedo ρ, normal n, light direction l, andlight intensity e that produce the same appearance M (see Fig. 1 (a)-(d)):ρ̂ = ρ|G−> n|, n̂ =

G−> n
Gl
, l̂ =, ê = e|Gl|.
|G−> n||Gl|(2)Although the surface’s appearance remains the same after GBR transformation (i.e., ρ̂n̂> l̂ê = ρn> le, see Fig. 1 (d)), a surface point’s albedo will be scaled
by |G−> n|. As a result, the albedo of an object will change gradually and become spatially-varying. Because this kind of spatially-varying albedo distribution
resulting from GBR transformations rarely occurs on real world objects, someprevious methods make explicit assumptions on the albedo distribution (e.g.,constant albedo [6, 35] or low entropy [4]) to resolve the ambiguity.
PF14 [35], a state-of-the-art non-learning uncalibrated method [45], detects
Lambertian diffuse reflectance maxima (i.e., image points satisfying n> l = 1)
to estimate G’s 3 parameters. We will later use it as a non-learning benchmark
in our comparative experiments.
LCNet and the GBR ambiguity LCNet [8] is a state-of-the-art lighting calibration network for uncalibrated photometric stereo (see Fig. 2). Figure 1 (e)-(f)compare the results of LCNet and PF14 on surfaces that differ by GBR transformations. Since the input images are the same in all cases, LCNet estimates theWhat is Learned in Deep Uncalibrated Photometric Stereo?

fully-connected

𝒍$

20

𝑒$

256
64

fusion layer

conv(stride=2, k=3x3)

72

64

256

256

256

feature extractor

256

256

128

128

218

64

128

input images and mask

𝒍"

...

global featureC

conv(stride=1, k=3x3)

72

64

...

...

F inputs

max
pooling

𝑒"

256

256

256

256

256

256

128

128

128

64

128

C

20

64

local feature

5C

concatlighting estimation sub-network

Fig. 2. Network architecture of LCNet [8]. Each layer’s value indicates its output channel number. LCNet first extracts a local feature for each input with a shared-weightfeature extractor. All local features are aggregated by element-wise max-pooling to
produce the global feature, each local feature is concatenated with the global feature,and is fed into a shared-weight lighting estimation sub-network to estimate a 3D light
direction l and a scalar light intensity e for each image.
Table 1. Light direction estimation results of PF14 [35] and LCNet [8] ona Sphere rendered with different BRDF
types. Non-Lambertian BRDFs are takenfrom the MERL dataset [31]. Values indicate mean angular error in degree.

model Lambertian fabric plastic phenolic metallic avg.
PF14
LCNet

7.19
5.38

14.26 28.04
4.07 3.08

47.96
3.05

31.12 25.7
4.09 3.93

Table 2. Light direction estimation resultsof LCNet [8] trained with different inputs.
Values indicate mean angular error in degree.
model input

Sphere Bunny Dragon Armadillo

images

3.03

4.88

6.30

6.37

(a) attached shadows
3.50(b) specular component 2.53(c) shading
2.29

5.07
6.18
3.95

9.78
7.33
4.64

5.22
4.08
3.76

(a) + (b) + (c)

2.06

2.34

2.12

1.87

same lightings in all cases, namely the most likely lightings for the input images.
The same also applies to PF14. Although LCNet’s result does not exactly equal
the lightings for uniform albedo, we note that it learned from the training datathat GBR-transformed surfaces are unlikely.
Although uncalibrated photometric stereo has an intrinsic GBR ambiguity
for Lambertian surfaces, it was shown that GBR transformations do not preserve specularities [6, 16, 12]. Hence, specularities are helpful for ambiguity-freelighting estimation. However, traditional methods often treat non-Lambertian
observations as outliers, and thus fail to make full use of specularities for disambiguation [35]. In contrast, learning-based methods can learn the relation
between specular highlights and light directions through end-to-end learning.
As shown in Table 1, LCNet achieves good results for non-Lambertian surfaces
while PF14 completely fails when non-Lambertian observations dominate.
Feature analysis for LCNet To analyze the features learned by LCNet, wefirst visualize the learned local and global features. Figure 3 shows 3 representative features selected from 256 feature maps extracted by LCNet from images ofa non-Lambertian Sphere dataset1 . Comparing Fig. 3’s Column 2 with Column
1Please refer to our supplemental material for more visualizations.

6

G. Chen et al.
(a) input (b) highlight, shadow, shading

Fig. 3. Feature visualization of
LCNet on a non-Lambertian
Sphere. Column 1: 5 of the 96
input images; Columns 2–4: Specular highlight centers, attachedshadows, and shading renderedfrom ground truth; Columns 5–7: 3
of LCNet’s 256 features maps. The
last row shows the global features
produced by fusing local featureswith max-pooling. All features
are normalized to [0, 1] and color
coded.

(c) extracted local features0

fused features
1

5, Column 3 with Column 6, and Column 4 with Column 7, we can see thatsome feature maps are highly correlated with attached shadows (regions where
the angle ](n, l) ≥ 90◦ ), shadings (n> l), and specular highlights (regions wherel+v
n is close to the half angle h = |l+v|
of l and viewing direction v). As discussed
earlier, these provide strong clues for resolving the ambiguity.
To further verify our observations, we did the following. We computed (a) attached shadows, (b) the “specular components” (with a bit of concept abuse,
we denote h> n as specular component), and (c) shadings for the publicly available synthetic Blobby and Sculpture datasets [10] from their ground-truth light
directions and normals. We then trained 4 variants of the LCNet, taking (a),(b), (c), and (a) + (b) + (c), respectively, as input instead of regular images.
We compared these 4 variant networks with LCNet (i.e., the network trainedwith Blobby and Sculpture images) on a synthetic test dataset introduced inSec. 5.1. Similar to LCNet, the variant networks also took the object mask as
input. Table 2 shows that the variant models achieve results comparable to oreven better than the model trained on regular images.
We can see that shadings contribute more than attached shadows and specular components for lighting estimation. This is because shading information actually includes attached shadows (i.e., pixels with a zero value in the shading forsynthetic data), and can be considered as an image with a uniform albedo. The
uniform albedo constraint is a well-known clue for resolving the GBR ambiguity [6, 35]. In practice, attached shadows, shadings, and the specular components
are not directly available as input, but this confirms our assumption that they
provide strong clues for accurate lighting estimation.
As discussed before, LCNet learns to resolve ambiguity by assuming thata surface with a gradually changing albedo corresponding to GBR transformations rarely exists. However, we have not observed features apparently related
to albedo distribution. We hypothesize that the albedo distribution prior is implicitly employed to restrict the mapping space, since LCNet learns the mappingfrom extracted features to lightings.

What is Learned in Deep Uncalibrated Photometric Stereo?

47

Guided Calibration Network

We have analyzed the features learned by LCNet and discussed how it resolves
the ambiguity. In this section, we present the motivations for our guided calibration network (GCNet) and detail its structure.
4.1

Guided feature extraction

As we have seen, features like attached shadows, shadings, and specularities areimportant for lighting estimation, and a lighting estimation network may benefit greatly from being able to estimate them accurately. We further know thatthese features are completely determined by the light direction for each image aswell as the inter-image shape information derived from the surface normal map.
However, LCNet extracts features independently from each input image andthus cannot exploit any inter-image information during feature extraction. Thisobservation also indicates that simply increasing the layer number of LCNet’s
shared-weight feature extractor cannot produce significant improvement.
Surface normal as inter-image guidance Intuitively, if we can provide suchinter-image shape information as input to the network to guide the feature extraction process, it should be able to perform better. This, however, constitutes
a chicken-and-egg problem where we require normals and lightings for accuratefeature extraction but at the same time we require these features for estimating
accurate lightings. We therefore suggest a cyclic network structure in which we
first estimate initial lightings, and then use them to estimate normals as interimage information to guide the extraction of local (i.e., per-image) features toultimately estimate final lightings. An alternative idea might be directly estimating surface normals from images. However, previous work (UPS-FCN [10])shows that surface normals estimated directly from images are not accurate.
Shading as intra-image guidance Another advantage of first estimating initial lighting and surface normals is that we can easily compute coarse attachedshadows, shadings, or specular components as intra-image guidance for the feature extraction process (intra-image means the information is different for each
image). As shading information already includes attached shadows, and not allmaterials exhibit specular highlights, we only compute the shading for each image as the dot-product of the estimated lighting with the surface normals, anduse it as intra-image guidance. We experimentally verified that additionally including the specular component as network input does not improve results. The
computed shading, on the other hand, does improve results and can assist the
network to extract better features.
4.2

Network architecture

As shown in Fig. 4, the proposed GCNet consists of two lighting estimation
sub-networks (L-Net) and a normal estimation sub-network (N-Net). The first
L-Net, “L-Net1 ”, estimates initial lightings given the input images and object

8

G. Chen et al.
(a) L-Net
128
3

64

L2-normalize

128

128

...
128

128

128

64

32

16

20

64

conv
(stride=1, k=3x3)
conv(stride=2, k=3x3)
deconv(stride=2, k=4x4)
fully-connected
C concat
N normalize

𝒍"

𝒏 T𝒍

(c) GCNet
C

C
N

𝒍$

𝒏 T𝒍

C

L-Net2

...

N-Net

𝑒"&
𝒍"&
...

...

...

...

𝑒$

max
pooling

𝑒$

C

𝒍$

128

𝒍$

N

𝒍"

L-Net1

72

128

𝑒"

128

...
64

128

128

128

256

128

128

64

128

64

32

C

128

128

64

32

64

...

max
pooling(b) N-Net
16

72

𝒍"

128

20

64

128

128

128

256

128

128

128

64

64

32

C

𝑒"

𝒍&$
𝑒$&

Fig. 4. Structure of (a) the lighting estimation sub-network L-Net, (b) the normal
estimation sub-network N-Net, and (c) the entire GCNet. Values in layers indicate
the output channel number.

masks. The N-Net then estimates surface normals from the lightings estimatedby L-Net1 and the input images. Finally, the second L-Net, “L-Net2 ”, estimates
more accurate lightings based on the input images, object masks, the estimatednormals, and the computed shadings.
L-Net The L-Net is designed based on LCNet [8] but has less channels in
the convolutional layers to reduce the model size (see Fig. 4 (a)). Compared to
LCNet’s 4.4 × 106 parameters, each L-Net has only 1.78 × 106 parameters.
Following LCNet, we discretize the lighting space and treat lighting estimation as a classification problem. Specifically, L-Net’s output light direction andintensity are in the form of softmax probability vectors (a 32-vector for elevation,a 32-vector for azimuth, and a 20-vector for intensity). Given F images, the loss
function for L-Net is
Llight =1 X f(Lla + Lfle + Lfe ),
F

(3)

f

where Lfla , Lfle , and Lfe are the cross-entropy loss for light azimuth, elevation,and intensity classifications for the f th input image, respectively. For example,
Lfla = −32
X{yif = 1} log(pfi ),

(4)

i=1

where {·} is a binary indicator (0 or 1) function, yif is the ground-truth label (0or 1) and pfi is the predicted probability for bin i (32 bins in our case) for thef th image. The output probability vectors can be converted to a 3-vector light
direction lf and a scalar light intensity ef by taking the probability vector’s
expectation, which is differentiable for later end-to-end fine-tuning.

What is Learned in Deep Uncalibrated Photometric Stereo?

9

L-Net1 and L-Net2 differ in that L-Net1 has 4 input channels (3 for the image,
1 for the object mask) while L-Net2 has 8 (3 additional channels for normals, 1
for shading).
N-Net The N-Net is designed based on PS-FCN [10] but with less channels,
resulting in 1.1 × 106 parameters compared to PS-FCN’s 2.2 × 106 parameters(see Fig. 4 (b) for details). Following PS-FCN, the N-Net’s loss function is
1 X
1 − n>
Lnormal =(5)p ñp ,
P p
where P is the number of pixels per image, and np and ñp are the predicted
and the ground-truth normal at pixel p, respectively.
End-to-end fine-tuning We train L-Net1 , N-Net, and L-Net2 one after another until convergence and then fine-tune the entire network end-to-end using
the following loss
Lfine-tune = Llight1 + Lnormal + Lshading + Llight2 ,(6)where Llight1 and Llight2 denote the lighting estimation loss for L-Net1 and LP P>
2
Net2 . The shading loss term Lshading = F1P f p (n>p lf − ñp l̃f ) is included
to encourage better shading estimation, and lf and l̃f denote the light direction
predicted by L-Net1 and ground-truth light direction for the f th image.
Training details Following LCNet [8], we trained the networks on the publicly
available synthetic Blobby and Sculpture Dataset [10] which contains 85, 212
surfaces, each rendered under 64 random light directions.
First, we train L-Net1 from scratch for 20 epochs, halving the learning rate
every 5 epochs. Second, we train N-Net using ground-truth lightings and input
images following PS-FCN’s training procedure [10], and then retrain N-Net given
the lightings estimated by L-Net1 for 5 epochs, halving the learning rate every 2
epochs. Third, we train L-Net2 given the input images, object masks, estimatednormals, and computed shadings for 20 epochs, halving the learning rate every
5 epochs. The initial learning rate is 0.0005 for L-Net1 and L-Net2 , and 0.0002
for retraining N-Net. End-to-end training is done for 20 epochs with an initial
learning rate of 0.0001, halving it every 5 epochs.
We implemented our framework in PyTorch [36] and used the Adam optimizer [25] with default parameters. The full network has a total of 4.66 × 106
parameters which is comparable to LCNet (4.4 × 106 ). The batch size and the
input image number for each object are fixed to 32 during training. The input
images for all sub-networks are resized to 128 × 128 at both training and test
time.

5

Experimental Results

In this section, we evaluate our method on synthetic and real data. For measuring
estimation accuracy, we used mean angular error (MAE) for light directions andsurface normals, and scale-invariant relative error [8] for light intensities.

10

G. Chen et al.

0

Fig. 5. (a) Lighting distribution of thesynthetic test dataset. (b) Normal mapsof Sphere, Bunny, Dragon and Armadillo.

1(a) lightings(b) normal maps

Table 3. Lighting estimation results on the synthetic test dataset. The results areaveraged over 100 MERL BRDFs (bold fonts indicates the best).
ID modelSphereBunnyDragonArmadillo
direction intensity direction intensity direction intensity direction intensity

0 LCNet [8]

3.03

0.064

4.88

0.066

6.30

0.072

6.37

0.074

1
2
3
4

2.21
2.52
2.45
3.20

0.042
0.052
0.050
0.053

2.44
2.90
3.35
4.47

0.046
0.054
0.051
0.060

3.88
4.20
5.82
5.80

0.055
0.061
0.070
0.081

3.52
3.92
5.25
5.71

0.060
0.060
0.059
0.079

L-Net1 + N-Net + L-Net2 + Finetune
L-Net1 + N-Net + L-Net2(w/o normal)
L-Net1 + N-Net + L-Net2
L-Net1

5.1

Evaluation on synthetic data

To quantitatively analyze the effects of object shapes, biased lighting distributions, and spatially-varying BRDFs (SVRBDFs) on the proposed method,
we rendered a synthetic test dataset using the physically-based raytracer Mitsuba [24]. We rendered 4 different shapes (Sphere, Bunny, Dragon and Armadillo) using 100 MERL BRDFs [31], resulting in 400 test objects, each illuminated under 82 randomly sampled light directions. At test time, we randomly
generated relative light intensities in the range [0.2, 2.0] to scale the magnitudeof the images (see Fig. 5).
Ablation study To validate the design of the proposed network, we performed
an ablation study and summarized the results in Table 3. The comparison between experiments with IDs 2-4 verifies that taking both the estimated normals
and shading as input is beneficial for lighting estimation. The comparison between experiments with IDs 1 & 2 demonstrates that end-to-end fine-tuning
further improves the performance. We can also see that L-Net1 achieves resultscomparable to LCNet despite using only half of the network parameters, whichindicates that simply increasing the channel number of the convolutional layers
cannot guarantee better feature extraction. In the rest of the paper, we denote
the results of “L-Net1 + N-Net + L-Net2 + Finetune” as “GCNet”.
Table 4 shows that, as expected, the calibrated photometric stereo method
PS-FCN [10] can estimate more accurate normals given better estimated lighting.
Results on different lighting distributions To analyze the effect of biased
lighting distributions on the proposed method, we evaluated GCNet on the Armadillo illuminated under three different lighting distributions: a near uniform,
a narrow, and an upward-biased distribution. Table 5 shows that both GCNet
and LCNet have decreased performance under biased lighting distributions (e.g.,
the upward-biased distribution), but GCNet consistently outperforms LCNet.
Results on surfaces with SVBRDFs To analyze the effect of SVBRDFs,
we used two different material maps to generate a synthetic dataset of sur-

What is Learned in Deep Uncalibrated Photometric Stereo?
Table 4. Normal estimation results on thesynthetic test dataset. The estimated normals are predicted by PS-FCN [10] given
the lightings estimated by LCNet and
GCNet.
model

Table 5. Results on Armadillo under threedifferent lighting distributions. Light direction, intensity, and normal are abbreviated to “dir.”, “int.”, and “norm.”.
0

Sphere Bunny Dragon Armadillo

LCNet [8] + PS-FCN [10] 2.98
GCNet + PS-FCN [10]
2.93

4.06
3.68

5.59
4.85

6.73
5.01

11

1

model

near uniform
narrow
upward-biased
dir. int. norm. dir. int. norm. dir. int. norm.

LCNet [8] + PS-FCN 6.09 0.072 6.49 5.92 0.059 8.44 7.10 0.065 8.80
GCNet + PS-FCN
3.39 0.059 4.90 4.29 0.048 6.82 5.96 0.054 7.53

Table 6. Lighting estimation results on Bunny rendered with SVBRDFs. (a) Bunny
with uniform BRDF. (b) and (c) show the “ramp” and “irregular” material maps andtwo sample images of Bunny with the corresponding SVBRDFs.
model
LCNet [8]GCNet

uniform
ramp
irregular
direction intensity direction intensity direction intensity
4.88
2.44

0.066
0.046

6.09
4.16

0.066
0.043

6.00
3.68

0.075
0.050

(a) uniform(b) ramp(c) irregularfaces with SVBRDFs following Goldman et al. [17]. Specifically, we rendered
100 test objects by randomly sampling two MERL BRDFs and blended the
BRDFs for Bunny using “ramp” and “irregular” material maps shown in Table 6 (b) and (c). Table 6 shows that although both methods perform worse on
surfaces with SVBRDFs compared to uniform BRDFs, our method is still reasonably good even though it was trained on surfaces with uniform BRDFs. Thismight be explained by that although SVBRDFs may affect the feature extraction of some important clues such as shading, others such as attached shadowsand specular highlights are less affected and can still be extracted to estimate
reliable lightings.
Effect of the object silhouette Object silhouette can provide useful information for lighting calibration (e.g., normals at the occluding contour are perpendicular to the viewing direction). To investigate the effect of the silhouette, wefirst rendered the Bunny using two different types of BRDFs (alumina-oxide andbeige-fabric) under 100 lightings sampled randomly from the upper hemisphere,
and then cropped surface regions with different sizes for testing. Table 7 showsthat both LCNet and our method perform robustly for surface regions with orwithout silhouette, while our method consistently outperforms LCNet. This isbecause the training data for both methods was generated by randomly cropping image patches from the Blobby and Sculpture datasets [10], which contains
surface regions without silhouette.
Runtime The runtimes of LCNet and GCNet for processing an object (96images in total) from the DiLiGenT benchmark are ∼ 0.25 s and ∼ 0.5 s including
data loading and network feed-forward time, measured on a single 1080 Ti GPU.
Even though LCNet runs slightly faster, both methods are very fast and run
within a second.

12

G. Chen et al.
Table 7. Lighting estimation results on surface regions cropped from Bunny.

alumina-oxide
LCNet
GCNet
input dir.
int.
dir.
int.
(a)
(b)
(c)
(d)

4.29
3.83
3.75
4.04

0.054
0.050
0.042
0.047

beige-fabric
LCNet
GCNet
dir.
int.
dir.
int.

1.35 0.025 4.54
1.71 0.023 4.45
2.46 0.024 4.97
2.84 0.026 4.55

0.051
0.044
0.044
0.051

objectmask

2.29 0.026
2.00 0.029
3.13 0.025
3.46 0.025

surfacenormal
(a)

(b)

(c)

(d)

Table 8. Lighting estimation results on DiLiGenT benchmark.

model

Ball
dir. int.

Cat
dir. int.

Pot1
dir. int.

Beardir. int.

Pot2
dir. int.

Buddhadir. int.

Gobletdir. int.

Readingdir. int.

Cow
dir. int.

Harvestdir. int.

averagedir. int.

PF14 [35] 4.90 0.036 5.31 0.059 2.43 0.017 5.24 0.098 13.52 0.044 9.76 0.053 33.22 0.223 21.77 0.122 16.34 0.074 24.99 0.156 13.75 0.088
LCNet [8] 3.27 0.039 4.08 0.095 5.44 0.058 3.47 0.061 2.87 0.048 4.34 0.048 10.36 0.067 4.50 0.105 4.52 0.073 6.32 0.082 4.92 0.068
GCNet
1.75 0.027 4.58 0.075 1.41 0.039 2.44 0.101 2.81 0.059 2.86 0.032 2.98 0.042 5.47 0.048 3.15 0.031 5.74 0.065 3.32 0.052

5.2

Evaluation on real data

To demonstrate the proposed method’s capability to handle real-world nonLambertian objects, we evaluated our method on the challenging DiLiGenT
benchmark [45] and the Light Stage Data Gallery [14].
Results on lighting estimation We first compared our method’s lighting
estimation results with the state-of-the-art learning-based method LCNet [8]and non-learning method PF14 [35]. Table 8 shows that GCNet achieves thebest average results on the DiLiGenT benchmark with an MAE of 3.32 forlight directions and a relative error of 0.052 for light intensities. Although our
method does not achieve the best results for all objects, it exhibits the mostrobust performance with a maximum MAE of 5.77 and a maximum relative
error of 0.101 compared with LCNet (MAE: 10.36, relative error: 0.105) andPF14 (MAE: 33.22, relative error: 0.223). Figures 6 (a)-(b) visualize the lighting estimation results for the Pot1 and the Goblet. The non-learning method
PF14 works well for near-diffuse surfaces (e.g., Pot1 ), but quickly degenerates
on highly specular surfaces (e.g., Goblet). Compared with LCNet, our method
is more robust to surfaces with different reflectances and shapes.
Table 9 shows lighting estimation results on the Light Stage Data Gallery.
Our method significantly outperforms LCNet and PF14, and achieves an average
MAE of 9.20 for light directions and a relative error of 0.163 for light intensities,
improving the results of LCNet by 32.4% and 26.4% for light directions and lightintensities respectively. Figures 6 (c)-(d) visualize lighting estimation results for
the Light Stage Data Gallery’s Standing Knight and Plant.
Results on surface normal estimation We then verified that the proposed
GCNet can be seamlessly integrated with existing calibrated methods to handle uncalibrated photometric stereo. Specifically, we integrated the GCNet witha state-of-the-art non-learning calibrated method ST14 [46] and two learningbased methods PS-FCN [10] and IS18 [22]. Table 10 shows that these integrations
can outperform existing state-of-the-art uncalibrated methods [4, 44, 51, 29, 35,

What is Learned in Deep Uncalibrated Photometric Stereo?

13

Table 9. Lighting estimation results on Light Stage Data Gallery.

model

Helmet Side
dir.
int.

PF14 [35] 25.40
LCNet [8] 6.57
GCNet
5.33

object

GT

0.576
0.212
0.096

dir.

Plant
int.

Fighting KnightKneeling KnightStanding Knight Helmet Front
dir.
int.
dir.
int.
dir.
int.
dir.
int.

20.56 0.227 69.50 1.137 46.69 9.805 33.81
16.06 0.170 15.95 0.214 19.84 0.199 11.60
10.49 0.154 13.42 0.168 14.41 0.181 5.31

GCNet

LCNet

PF14

object

1.311
0.286
0.198

GT

81.60
11.62
6.22

0.133
0.248
0.183

GCNet

average
dir.
int.
46.26
13.61
9.20

LCNet

2.198
0.221
0.163

PF14

0
(a) Pot1

1.41
0.039

5.44
0.058

2.43
0.017

(b) Goblet

2.98
0.042

10.36
0.067

33.22
0.223

1
(c)

Standing
Knight

5.31
0.198

11.60
0.286

33.81
1.311

(d) Plant

10.49
0.154

16.06
0.170

20.56
0.227

Fig. 6. Visualization of the ground-truth and estimated lighting distribution for the
DiLiGenT benchmark and Light Stage Data Gallery.

28] by a large margin on the DiLiGenT benchmark. We can further see that
ST14, PS-FCN, as well as IS18 perform better with ours instead of LCNet’s predicted lightings: 10.8 vs. 12.1 for ST14, 8.7 vs. 9.7 for PS-FCN, and 9.6 vs. 16.5
for IS18. Figure 7 presents a visual comparisons on the Goblet from the DiLiGenT benchmark. Please refer to our supplemental material for more results.
5.3

Failure cases

As discussed in Sec. 3, LCNet [8] relies on features like attached shadows, shading, and specular highlights, which is also true for our method. For piecewise
planar surfaces with a sparse normal distribution such as the one in Fig. 8 (a),few useful features can be extracted and as a result our method cannot predictreliable lightings for such surfaces. For highly-concave shapes under directional
lightings, strong cast shadows largely affect the extraction of useful features.
Figure 8 (b) shows that GCNet erroneously estimates a highly-concave bowl to
be convex. Note that LCNet [8] and PF14 [35] also have similar problems.

6

Conclusions

This paper targeted discovering what is learned in deep uncalibrated photometricstereo to resolve the ambiguity. Specifically, we analyzed and discussed the behavior of the recent deep uncalibrated photometric stereo method LCNet. Basedon our findings, we then introduced the guided calibration network (GCNet) that
explicitly leverages inter-image information of object shape and intra-image information of shading to estimate more reliable lightings. Experiments on both

14

G. Chen et al.

Table 10. Normal estimation results on DiLiGenT benchmark. (∗ indicates the resultsof the calibrated method using ground-truth lightings as input.)
model

Ball

Cat

Pot1

Bear

Pot2

AM07 [4]SM10 [44]
WT13 [51]LM13 [29]PF14 [35]
LC18 [28]

7.3
8.9
4.4
22.4
4.8
9.3

31.5
19.8
36.6
25.0
9.5
12.6

18.4
16.7
9.4
32.8
9.5
12.4

16.8
12.0
6.4
15.4
9.1
10.9

49.2
50.7
14.5
20.6
15.9
15.7

32.8
15.5
13.2
25.8
14.9
19.0

46.5
48.8
20.6
29.2
29.9
18.3

53.7
26.9
59.0
48.2
24.2
22.3

54.7
22.7
19.8
22.5
19.5
15.0

61.7
73.9
55.5
34.5
29.2
28.0

37.3
29.6
23.9
27.6
16.7
16.3

LCNet + ST14
GCNet + ST14
ST14∗ [46]

4.1
2.0
1.7

8.2
7.7
6.1

8.8
7.5
6.5

8.4
5.7
6.1

9.7
9.3
8.8

11.6
10.9
10.6

13.5
10.0
10.1

15.2
14.8
13.6

13.4
13.5
13.9

27.7
26.9
25.4

12.1
10.8
10.3

LCNet + PS-FCN
GCNet + PS-FCN
PS-FCN∗ [10]

3.2
2.5
2.8

7.6
7.9
6.2

8.4
7.2
7.1

11.4
5.6
7.6

7.0
7.1
7.3

8.3
8.6
7.9

11.6
9.6
8.6

14.6
14.9
13.3

7.8
7.8
7.3

17.5
16.2
15.9

9.7
8.7
8.4

LCNet + IS18
GCNet + IS18
IS18∗ [22]

6.4
2.5
2.2

15.6
8.4
4.6

10.6
7.5
5.4

8.5
5.1
4.1

12.2
7.6
6.0

13.9
10.7
7.9

18.5
7.7
7.3

23.8
18.5
12.6

29.3
9.3
8.0

25.7
18.3
14.0

16.5
9.6
7.2

GT / object GCNet + IS18 LCNet + IS18

PF14 [35]Buddha Goblet Reading

Cow

Harvest average

GT / object GCNet + IS18 LCNet + IS18

PF14 [35]090(a) Pot1

MAE = 7.2 MAE = 10.6 MAE = 9.5 (b) Goblet MAE = 7.7 MAE = 18.5 MAE = 24.2

Fig. 7. Visual comparisons of normal estimation for Pot1 and Goblet in the DiLiGenT benchmark. We compared the normal estimation results of a calibrated method
IS18 [22] given lightings estimated by our method and LCNet [8].
Fig. 8. Failure cases. (a) Results on apiecewise planar surface with sparse normal distribution. (b) Results on a highlyconcave bowl. The estimated normals
are predicted by PS-FCN [8] given our
method’s estimated lightings.

objectGT Normal/Lighting Est. Normal/Lighting
0

(a)
1
0

(b)
1

synthetic and real datasets showed that our method significantly outperforms
the state-of-the-art LCNet in lighting estimation, and demonstrated that our
method can be integrated with existing calibrated photometric stereo methods
to handle uncalibrated setups. Since strong cast shadows affect our method’s feature extraction process and lead to unsatisfactory results, we will explore bettermethods to handle cast shadows in the future.
Acknowledgments Michael Waechter was supported through a JSPS Postdoctoral Fellowship (JP17F17350). Boxin Shi is supported by the National Natural Science Foundation of China under Grant No. 61872012, National Key R&D
Program of China (2019YFF0302902), and Beijing Academy of Artificial Intelligence (BAAI). Kwan-Yee K. Wong is supported by the Research Grant Council of Hong Kong (SAR), China, under the project HKU 17203119. Yasuyuki
Matsushita is supported by JSPS KAKENHI Grant Number JP19H01123.

What is Learned in Deep Uncalibrated Photometric Stereo?

15

References
1. Ackermann, J., Fuhrmann, S., Goesele, M.: Geometric Point Light Source Calibration. In: Vision, Modeling & Visualization (2013)
2. Ackermann, J., Goesele, M.: A survey of photometric stereo techniques. Foundations and Trends in Computer Graphics and Vision (2015)
3. Agarwal, S., Furukawa, Y., Snavely, N., Simon, I., Curless, B., Seitz, S.M., Szeliski,R.: Building Rome in a day. Communications of the ACM (2011)
4. Alldrin, N.G., Mallick, S.P., Kriegman, D.J.: Resolving the generalized bas-relief
ambiguity by entropy minimization. In: CVPR (2007)
5. Basri, R., Jacobs, D., Kemelmacher, I.: Photometric stereo with general, unknown
lighting. IJCV (2007)
6. Belhumeur, P.N., Kriegman, D.J., Yuille, A.L.: The bas-relief ambiguity. IJCV(1999)
7. Chandraker, M.K., Kahl, F., Kriegman, D.J.: Reflections on the generalized basrelief ambiguity. In: CVPR (2005)
8. Chen, G., Han, K., Shi, B., Matsushita, Y., Wong, K.Y.K.: Self-calibrating deep
photometric stereo networks. In: CVPR (2019)
9. Chen, G., Han, K., Shi, B., Matsushita, Y., Wong, K.Y.K.: Deep photometricstereo for non-Lambertian surfaces. TPAMI (2020)
10. Chen, G., Han, K., Wong, K.Y.K.: PS-FCN: A flexible learning framework forphotometric stereo. In: ECCV (2018)
11. Cho, D., Matsushita, Y., Tai, Y.W., Kweon, I.S.: Semi-calibrated photometricstereo. TPAMI (2018)12. Drbohlav, O., Chantler, M.: Can two specular pixels calibrate photometric stereo?
In: ICCV (2005)
13. Drbohlav, O., Šára, R.: Specularities reduce ambiguity of uncalibrated photometricstereo. In: ECCV (2002)
14. Einarsson, P., Chabert, C.F., Jones, A., Ma, W.C., Lamond, B., Hawkins, T.,
Bolas, M., Sylwan, S., Debevec, P.: Relighting human locomotion with flowed reflectance fields. In: EGSR (2006)
15. Epstein, R., Yuille, A.L., Belhumeur, P.N.: Learning object representations fromlighting variations. In: International Workshop on Object Representation in Computer Vision (1996)
16. Georghiades, A.S.: Incorporating the Torrance and Sparrow model of reflectancein uncalibrated photometric stereo. In: ICCV (2003)
17. Goldman, D.B., Curless, B., Hertzmann, A., Seitz, S.M.: Shape and spatiallyvarying BRDFs from photometric stereo. TPAMI (2010)18. Haefner, B., Ye, Z., Gao, M., Wu, T., Quéau, Y., Cremers, D.: Variational uncalibrated photometric stereo under general lighting. In: ICCV (2019)19. Hayakawa, H.: Photometric stereo under a light source with arbitrary motion.
JOSA A (1994)
20. Herbort, S., Wöhler, C.: An introduction to image-based 3D surface reconstruction
and a survey of photometric stereo methods. 3D Research (2011)
21. Hertzmann, A., Seitz, S.M.: Example-based photometric stereo: Shape reconstruction with general, varying BRDFs. TPAMI (2005)
22. Ikehata, S.: CNN-PS: CNN-based photometric stereo for general non-convex surfaces. In: ECCV (2018)
23. Ikehata, S., Aizawa, K.: Photometric stereo using constrained bivariate regression
for general isotropic surfaces. In: CVPR (2014)

16

G. Chen et al.

24. Jakob, W.: Mitsuba renderer (2010)
25. Kingma, D., Ba, J.: ADAM: A method for stochastic optimization. In: ICLR (2015)
26. Kriegman, D.J., Belhumeur, P.N.: What shadows reveal about object structure.
JOSA A (2001)
27. Li, J., Robles-Kelly, A., You, S., Matsushita, Y.: Learning to minify photometricstereo. In: CVPR (2019)
28. Lu, F., Chen, X., Sato, I., Sato, Y.: SymPS: BRDF symmetry guided photometricstereo for shape and light source estimation. TPAMI (2018)
29. Lu, F., Matsushita, Y., Sato, I., Okabe, T., Sato, Y.: Uncalibrated photometricstereo for unknown isotropic reflectances. In: CVPR (2013)
30. Lu, F., Matsushita, Y., Sato, I., Okabe, T., Sato, Y.: From intensity profile tosurface normal: Photometric stereo for unknown light sources and isotropic reflectances. TPAMI (2015)31. Matusik, W., Pfister, H., Brand, M., McMillan, L.: A data-driven reflectance model.
In: SIGGRAPH (2003)
32. Midorikawa, K., Yamasaki, T., Aizawa, K.: Uncalibrated photometric stereo by
stepwise optimization using principal components of isotropic BRDFs. In: CVPR
(2016)
33. Mo, Z., Shi, B., Lu, F., Yeung, S.K., Matsushita, Y.: Uncalibrated photometricstereo under natural illumination. In: CVPR (2018)
34. Okabe, T., Sato, I., Sato, Y.: Attached shadow coding: Estimating surface normals
from shadows under unknown reflectance and lighting conditions. In: ICCV (2009)35. Papadhimitri, T., Favaro, P.: A closed-form, consistent and robust solution touncalibrated photometric stereo via local diffuse reflectance maxima. IJCV (2014)
36. Paszke, A., Gross, S., Chintala, S., Chanan, G.: PyTorch: Tensors and dynamic
neural networks in Python with strong GPU acceleration (2017)
37. Quéau, Y., Wu, T., Lauze, F., Durou, J.D., Cremers, D.: A non-convex variational
approach to photometric stereo under inaccurate lighting. In: CVPR (2017)38. Santo, H., Samejima, M., Sugano, Y., Shi, B., Matsushita, Y.: Deep photometricstereo network. In: ICCV Workshops (2017)39. Santo, H., Waechter, M., Samejima, M., Sugano, Y., Matsushita, Y.: Light structure from pin motion: Simple and accurate point light calibration for physics-basedmodeling. In: ECCV (2018)
40. Sato, I., Okabe, T., Yu, Q., Sato, Y.: Shape reconstruction based on similarity inradiance changes under varying illumination. In: ICCV (2007)
41. Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison andevaluation of multi-view stereo reconstruction algorithms. In: CVPR (2006)
42. Shen, H.L., Cheng, Y.: Calibrating light sources by using a planar mirror. Journalof Electronic Imaging 20(1), 013002-1–013002-6 (2011)
43. Shi, B., Inose, K., Matsushita, Y., Tan, P., Yeung, S.K., Ikeuchi, K.: Photometricstereo using internet images. In: 3DV (2014)44. Shi, B., Matsushita, Y., Wei, Y., Xu, C., Tan, P.: Self-calibrating photometricstereo. In: CVPR (2010)
45. Shi, B., Mo, Z., Wu, Z., Duan, D., Yeung, S.K., Tan, P.: A benchmark dataset
and evaluation for non-Lambertian and uncalibrated photometric stereo. TPAMI(2019)
46. Shi, B., Tan, P., Matsushita, Y., Ikeuchi, K.: Bi-polynomial modeling of lowfrequency reflectances. TPAMI (2014)
47. Silver, W.M.: Determining shape and reflectance using multiple images. Ph.D.
thesis, Massachusetts Institute of Technology (1980)

What is Learned in Deep Uncalibrated Photometric Stereo?

17

48. Tan, P., Mallick, S.P., Quan, L., Kriegman, D.J., Zickler, T.: Isotropy, reciprocity
and the generalized bas-relief ambiguity. In: CVPR (2007)
49. Taniai, T., Maehara, T.: Neural inverse rendering for general reflectance photometric stereo. In: ICML (2018)
50. Woodham, R.J.: Photometric method for determining surface orientation frommultiple images. Optical Engineering (1980)
51. Wu, Z., Tan, P.: Calibrating photometric stereo by holistic reflectance symmetry
analysis. In: CVPR (2013)
52. Yuille, A.L., Snow, D., Epstein, R., Belhumeur, P.N.: Determining generative models of objects under varying illumination: Shape and albedo from multiple images
using SVD and integrability. IJCV (1999)
53. Zheng, Q., Jia, Y., Shi, B., Jiang, X., Duan, L.Y., Kot, A.C.: SPLINE-Net: Sparse
photometric stereo through lighting interpolation and normal estimation networks.
In: ICCV (2019)
54. Zhou, Z., Tan, P.: Ring-light photometric stereo. In: ECCV (2010)