NeIF: Representing General Reflectance as Neural Intrinsics Fields for Uncalibrated Photometric Stereo
Authors: ZONGRUI LI
Qian Zheng
Feishi Wang
Boxin Shi
Gang Pan
Xudong Jiang
Venue: NeurIPS 2022 Submitted (2022)

Abstract: Uncalibrated photometric stereo is challenging due to the general bas-relief ambiguity. Existing solutions alleviate this ambiguity by either building an explicit relationship between reflectance and lighting or resolving lighting information in a supervised manner before recovering surface normal, which suffers from poor generalization to unseen reflectance or data. In contrast, this paper builds the implicit relationship between general reflectance (specular, cast shadow) and lighting by representing the reflectance as several neural intrinsics fields, based on which we optimize the surface normal and lighting in an unsupervised manner. Specifically, the neural intrinsics fields include reflectance features (i.e., diffuse, specular, diffuse coefficient, specular coefficient, cast shadow) and shading features (i.e., surface normal, lighting information). The implicit relationship is achieved by feeding the lighting information to neural specular & shadow fields and optimizing all intrinsics through a rendering equation in an unsupervised manner, which facilitates the better generalization to unseen reflectance and data. Our method achieves a superior performance advantage over state-of-the-art uncalibrated photometric stereo methods on public datasets in terms of the surface normal & lighting estimation. 

NelF: Representing General Reflectance as Neural Intrinsics Fields for Uncalibrated Photometric Stereo
Anonymous Author(s)
Affiliation
Address
email
Abstract
Uncalibrated photometric stereo (UPS) is challenging due the inherent ambiguity brought by unknown light. Existing solutions alleviate the ambiguity by either explicitly associating reflectance light conditions or resolving light conditions a supervised manner. This paper establishes an implicit relation between light clues and light estimation and solves UPS in an unsupervised manner. The key idea represent the reflectance four neural intrinsics fields, i.e., position, light, specular, and shadow, based on which the neural light field implicitly associated with light clues of specular reflectance and cast shadow. The unsupervised optimization neural intrinsics fields can free from training data bias and fully exploits all observed pixel values for UPS. Our method achieves a superior performance advantage over state-of-the-ar UPS methods on public datasets and promising results under the challenging setting of sparse UPS. The code will be released soon.
11
13
Introduction
14
Photometric stereo (PS) (50) aims recovering the surface normal from several hight-varying images captured at a fixed viewpoint. As compared with other approaches (e.g., multi-view stereo [41], active sensor-bascl solutions (58)), photometric stereo is excellent at recovering iinedettiled surfaces and has been widely used for Hollywood movies [11), industrial quality inspection (49). and biometrics (55) Calibrating accurate lighting directions is crucial to the performance of photometric stereo methods (54). However, lighting calibration is often tedious, dramatically restricting the applicability in the real-world. To this end, researchers develop uncalibrated photometric stereo (UPS) methods that estimate surface normal with unknown lights.
16
18
20
22
23
Uncalibrated photometric stereo suffers from General Bas-Relief (GBR) ambiguity 6 for inte- grable surface. Early solutions address the ambiguity by explicitly associating reflectance to light, i.e., adopting analytic reflectance models (e.g., Lambertian reflectance [4) (37), parametric specular reflection [20). specular spikes (56). inter-reflection (12) or imposing priors from reflectance proper-
24
26
ties
25 44 Thus, due to the strong reliance reflectance assumption, these methods be
28
less effective for unknown reflectance. Besides, these methods ignore the clue of cast shadow and even fail in shadow regions due the shadow-free surface assumption. Further, most of them require the light intensity to be identical for robust estimation. Recently, deep learning-basee approaches address the ambiguity by estimating light direction and intensity before recovering surface normal [13]
30
31
32
40). They train a light estimation network using a large-scale amount of rendered data in supervised manner. However, the training data bias [33] hardly be eliminated and can produce
15
27
33
34
unexpected estimation for real-world data. Because rendered training data inevitably contains the domain gap from the real ones and scarcely cover all surfaces with different geometry and reflectance in the real-world. Besides, such a two-step solution can bring accumulating errors for surface normal
35
38
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

Position- Dependent
Light Dependent
kf
kg
Neural
Light
Neural Position Field
Neural Specular Field
Ligtt oroction
7
Shadow Field
of of
Light Field
Light drection
4
x
Osaamed
Figure 1: Illustration of our neural intrinsics fields. Left-top: the rendering equation. Left-bottom: our four neural intrinsic fields, i.e., from left to right: shadow, position, specular, and light fields, respectively. Each sub-figure the left-bottom illustrates the mutual information across dimensions of position-light, position, normal-light, and observed images, respectively. Left figure shows how the neural intrinsics fields are imposed to render a pixel. Right: a summary of our intrinsics w.r.. the dependence on position or light. The definition of notations can be found from Eq. ( and Eq.
37 estimation. Further, all these methods assume the light intensity distributed in pre-defined range
38 (i.e., [0.2, 2)), restricting their applicability.
39
To this end, we propose NelF, representing general reflectance as Neural Intrinsics Fields for uncalibrated photometric stereo. Our method differs from previous methods in three aspects: 1) it fully considers clues of specular reflectance and cast shadow from each observed pixel for light estimation so that it expected to produce accurate estimation for both light conditions and surface normal; 2) it does not make explicit assumptions about the reflectance or light so that it works with general surface reflectance and flexible light settings; 3) infers light and surface normal in an unsupervised manner so that it is free from training data bias and achieves stable performance for data from different sources.
40
41
42
43
44
45
46
47
Our key idea is to represent the general reflectance as four neural intrinsics fields (i.e., position, light, specular, and shadow, see Fig. []. implemented by four multi-layer perceptrons (MLPs). These four fields are connected based on the implicit relation (or dependence) of these intrinsics so that no explicit assumption is imposed, egg. take the estimated light as the input to recover specular reflectance and cast shadow instead of explicitly exploiting them for light estimation. These intrinsics fields are optimized reconstruct each pixel value from observed images, which fully exploit mutual information across different dimensions, as shown in Fig. [ The reconstruction eтror is backpropagated the neural light field through neural specular and shadow fields so that clues of specular and shadow can be implicitly and fully considered for light estimation. Our contributions are summarized as:
48
49
50
51
52
53
54
55
56
We represent general reflectance as four intrinsic neural fields implicitly associate per- pixel reflectance to light, which solves uncalibrated photometric stereo by fully considering clues of specular reflectance and cast shadow for light estimation.
58
60
We propose the NelF, an uncalibrated photometric stereo method trained in unsupervised manner, which works with general surface reflectance and flexible light settings, and free from training data bias.
61
62
83
We show that our method achieves superior performance over uncalibrated and unsupervised methods. We also demonstrate its excellent generalization capacity data from different sources and promising performance with the challenging setup of sparse uncalibrated photometric stereo.
64
85
86

---

67
2
Related Work
68
This section mainly reviews the latest works in neural reflectance representation, and related works on unsupervised PS methods and UPS methods. Readers may refer to [45] and [33] for more comprehensive summary.
70
2.1
Neural Reflectance Representation
72
Neural Radiance Fields (NeRFs) (34] focus on the 3D geometries without explicitly modeling the
73
interaction between light and objects via the image formation model. Many subsequent works
74
explore its application in various computer vision problems, such as relighting (16) 46). photometric
75
stereo 29 and multi-view stereo 7 9 60, 28 66]. These works require known lighting conditions 77
78
46
61 29). adopt simple reflectance model 159. or leverage multi-view information 7 60 28 61].
77
Different from these methods, our method considers general reflectance, cast-shadow effects, and
78
unknown light conditions for uncalibrated photometric stereo by taking images captured at a single
79
viewpoint.
80
2.2
Unsupervised Photometric Stereo
81 Classical methods solve the calibrated photometric stereo problem without knowing the ground truth surface normal. Therefore, we classify them as unsupervised methods. The least square- based algorithm [50] provides the simplest solution, which assumes the object to Lambertian. 83
82
84 It is generally served as baseline method due to its stability, but its strong assumption on the reflectance model makes fail for non-Lambertian surface. The following works either regard the or apply analytic 87 reflectance models including Torrance-Spp Sparroow (20). the Ward model 18) a mixture of multiple
85
86
non- Lon-rranbrriann reflectance components as the outliers [5]
18 36
51
26 52
Ward models [21], (T) etc. consider the nom-Lambertaan effects. However, the performance of those methods can only deal with limited types of materials. There are also more advanced methods that utilize the general refletance features such as reciprocity, isotropy (2). and monotonicity (24).
89
91 Those methods give a reliable estimation for objects with a broad range of materials.
92 With the progress of deep learning, many learning-baas based frameworks have been proposed for cali-
93 brated photometric stereo. Taniai el al. (47) proposed the first unsupervised learning- photomet- ric stereo method through rendering equation. However, their reflectance model does not separately consider shadow, specular highlights, and diffuse components.
94
95
96
We also train our method in an unsupervised manner. Different from previous methods, we address the challenging problem of UPS and separately model cast shadow, specular reflectance, and diffuse reflectance.
97
98
2.3
Uncalibrated Photometric Stereo
100
Previous works hold the Lambertian assumption and address the ambiguity brought by 3x3 transformation matrix. Belhumeur et al. (6) reduce the dimension of the transformation a three
101
102
parameters GBR transformation by considering integrability constraints. Based on that, extra clues from reflectance, such as half-vector symmetry (3T), albedo clustering (42), specular spikes (56). or assumptions of light source distribution, such as ring light [63), symmetry light (35), or uniform- distributed light sources [57] 4 42,53 37 31), are used to resolve the GBR ambiguity. However, all these methods require the light intensity to be identical, which is inapplicable in the real-world datasets such as DILIGENT [45). APPLE & GOURD (2). and LIGHT STAGE DATA GALLERY (IT).
103
105
106
107
Cho et al. [17] put up semi-calibrated method to deal with non-uniform light intensity, but they assume the light directions to be known. Quéau et al. (39] address the photometric stereo problem under inaccurate lighting calibration, while the accuracy can significantly drop when Lambertenn components become dominant.
109
111
112
Recently, many deep learning methods have been proposed for uncalibrated photometric stereo. Chen el al. [13) propose a supervised uncalibrated framework, SDPS-Net, which can simultaneously estimate the light conditions (intensity and direction) and the surface normal. They suggest treating
115
115
light estimation as classification problem and separating the normal and light prediction to reduce the complexity. Their following work, GC-Net [15), improves the performance of SDPS-Net by

---

117
adding shading as an extra channel to the input of the light estimation network. However, as common problem for all supervised methods, an over wver-fitting problem may occur due the training data bias (33]. In contrast, unsupervised methods do not have such concern. Another benefit is that there is no need to synthesize training sets for unsupervised network training. To utilize the advantage of unsupervised methods, Kaya er al. (27) propose compromised method that trains the light estimation network in a supervised manner (similar to (13). but estimate the surface normal in an unsupervised way (similar to [47]. However, they still suffer from training data bias during light estimation. Besides, all these methods make a strict assumption that the light intensity distributed in a pre defined range (i.e., [0.2, 21) and suffers from accumulating error due to their two-step frameworks. In contrast, our method neither makes a strict assumption on reflectance nor needs special light source distribution and jointly solves light conditions and surface normal in unsupervised manner.
118
119
120
122
124
126
3
Method
128
129
3.1
Problem Formulation
Given set of observations I _ (Jo, /1.....y of static surface, illuminated by f unknown directional illuminations distributing on the upper-hemisphere, uncalibrated photometric stereo aims at recovering light directions L å (4o, (o.,44.., ly), light intensities E P (eo. ci, ey), and surface normal N {nuli E IP). P is the set of all positions on the surface. The solution is achieved by solving the optimization problem,
130
131
132
134
#I' Σ Σ/ =1 D(may may),
arg min L
(1)
where mij E 1 is the observed pixel intensity at position i, #P is the number of elements in PT mij is the corresponding rendered pixel intensity, D(::-) is a metric describing their difference. Under an orthographic camera with linear radiometric response, mij is formulated as (simplified in per-pixel form),
135
136
137
138
mij = e;p(n; 1j, v) max(n, 1, = EyPij max(n, (,, 0),
(2)
139
where e = [0, 0, 1] is the view direction pointing toward the viewer, Pij describes the general reflectance, max(n, L,. 0) represents the attach shadow.
140
Unknown light brings two ambiguities when solving Eq. ( i.e., shape-light ambiguity, which is denoted as an invertable matrix G E R333,, and reflectance-light ambiguity, which is denoted as a non-zero scalar Cj E R,
142
143
mij = e (en(nn
144
3.2
Neural Intrinsics Fields
General reflectance decomposition. To exploit clues of specular reflectance and cast shadow for light estimation, we decompose the general reflectance as the cast shadow term Sij multiplying the bidirectional reflectance term,
146
147
Psy = +ky k(
(4)
148
where subscript 'i' and 'j" indicate position- (or normal-) and light-dependent factors, respectively; the cast shadow term $ig is either 0 or 1; p P represent the diffuse and specular reflectance, and ky and kn are coefficients that balance out the effects of specular and diffuse reflectance
150
151
Neural fields of specular reflectance and cast shadow. Previous methods identify specular features from specific pixels and associate explicit relation to light for light estimation. However, this scheme fails for the surface where the specular features are invisible or the explicit relation is violated.
152
153
154
Besides. leaving out the clue of cast shadow can obstruct producing competitive performance.
155
In contrast, we leverage both clues of specular and shadow for light estimation, which is achieved by building the neural specular field and neural shadow field, and associating the fields to light conditions.
156
Without loss of generality, we put '#' before a set symbol to represent its number of elements this paper. We think these coefficients of point will not change under different lights, while they can be different at different positions.

---

PositionNet
PE
LightNet
SpecularNet
ShadowNet
(x)))
Posilional Enzede
CNN Encoder
MLPs
my e/sh
( (fffff + kfof) max max(n! (,
114 + vll
The framework of the proposed NelF. PositionNet takes the input of positional code and outputs diffuse reflectance p.. surface normal n,, and coefficients ki kf. LightNet takes the observed image I as the input and outputs light intensity ej and direction 1,. SpecularNet takes v. h, and n h, the input and outputs specular reflectance Pl-- ShadowNet takes inputs of positional code and light direction I and output shadow indicator Sij- All the intrinsics are used to render the observed pixel value maj using Eq. (2) and Eq. 44.
Figure 2:
157
These neural fields make the utmost of all observed pixels, and exploit the mutual information across different normal-light (specular) and position-lighh (shadow) for light estimation. Two MLPs, namely SpecularNer and ShadowNer, implement these neural fields, respectively. we take the estimated light direction as the input of these networks to achieve their implicit association to light conditions. Since the specular reflectance and cast shadow are normal- and position-dependent we also feed the estimated surface normal and the positional code to them, respectively. SpecularNet and ShadowNet
158
159
180
161
162
163
output Pi and Siy. respectively, as shown Fig.
164
Neural light field. There is mutual information across different observed images, i.e., observed images with similar appearances expected to be illuminated by similar lights. To fully consider and exploit this mutual information, we build the neural light field by concatenating a CNN encoder an MLP, namely LightNet. The encoder extracts a light code from each observed image and the LightNet infers the corresponding light conditions (i.e., ej, 12) from the light code, as shown in Fig.
185
186
187
188
Neural position field. There is mutual information across different positions
surface, i.e., the consistency of shape and diffuse reflectance in the spatial domain. To fully consider and exploit this mutual information, establish neural position field, namely PositionNer, implemented by an MLP.
170
172
light-independent factors i.e., n. p. Rd kç. The PositionNet takes the positional code as the input, shown in Fig.
The neural position field outputs
173
174
3.3
Optimizing Neural Intrinsics Fields
We adopt the reconstruction loss function with the l metric to optimize our NeIF,
#P
Lree
muj - (ks + k(p),) max(n, I, 0)l.
(5)
Silhouette constraint. fre: cannot resolve the shape-light ambiguity in unsupervised manner due to the inherently severe ill- ill-posedness. Therefore, we introduce the silhouette constraint (similar to those in 23 [55) to stabilize the training of PositionNet. To be specific, we use polynomial fitting with moving window block to traverse and pre-compute the contour's 's normal of the given objects, represented as ri A {Ag E 2 k E S}, S is the point set of the contour. We consider Nrs can
177
178
179
180
We use light code instead of positional code because we experimentally find that the light code contains discriminative features of light intensity and direction.
*Since specular reflectance and cast shadow are both position-deed dependent and them requires the input of light conditions, which increases the complexity of the neural position field, Therefore. we do not estimate them the PositionNet, but predict them using SpecularNet and ShadowNet, respectively.
predicting
5

---

guide the prediction of the azimuth of boundary normal (at the same positions) and introduce the silhouette loss function,
182
#2
Lu
Nor(C(nn)) n
(6)
where ny E RIx3 represents the estimated surface normal at the positions silhouette from PositionNet, C(-) cuts off the 3rd dimension of n& (i.e., C(n2g) IR 12))) and Nor(-) is the vector
183
185
normalization operation.
186
Warm-up loss functions. To avoid local minimum and achieve faster convergence, we warm up the NelF in early-stage during training with three additional loss functions. We use the azimuth of lighting direction estimated by YS97 [57] to guide the training of LightNet,
187
188
Nor(C(())) Nor(C(()))
where 1%f are estimated light directions by 57 We adopt the gradient penalty [22) Lap to stabilize the training SpecularNet,
190
max( (-V
Vnn nz h, Pig, 0)2
0)
191 The intuition is from Blinn-Phong model (8)
where the specular reflectance is monotonically
increasing w.r.t the nz hjj i... nn, h 18g
0.
We also supervise the training of ShadowNet using
pseudo shadow maps Sy.j = 1,2, ...,....
193
Cabadow
1 #I
Σ Ø=1
fls
S5yl2
194
The pseudo shadow maps are obtained by binarizing the observed images, i.e., considering an observed pixel to be cast shadow if its intensity value is smaller than 0.2 the mean intensity value of this image. After etrly-stage training, we discard these loss functions for a broad range reflectance.
195
196
197
NelF training. We train NelF with the warm-up loss function in first 10 epochs.
Lwarmup Live + AghCa AazCaz + Agp + Ashadow Cshaddww
(10)
198
where Ag Auz = 0.1, Agp = 10, Ashadow = 10. We then train NeIF until 500 epochs or converging with the loss function,
C = Lise + Xahe + Ashadow CreeShadow
(11)
200
where FreeShadow is the another shadow map supervision loss train the ShadowNet. The loss function the same to Lxhadow- The only difference the calculation of 5,. For LreeShudow- we calculate S by rendering depth map with the estimated 1j. The depth map is reconstructed from the estimated surface normal map N by method [10]. LreeShadow is used to align outputs of ShadowNet to those of NositionNet.
201
203
205
Implementation details. We generate the positional code from the coordinate (in the image plane) by the same method in [34]. With the assumption of isotropic reflectance, we simplify the input of SpecularNet from {e L,, n,; n. LL } to (u hj n. h 15 31 for easier training. Similar to the
206
207
208
reduce the number of unknowns. The CNN decoder with declining channels processes the down-sampled images with a dimension of
Mork-Toreeece reflectance model [19], we assume ky¹ k
210
256 x 256. The LightNet takes flatten features to predict 1,, ej in two different branches. For the
ShadowNet, we also generate the positional code for 1, and concatenate it to the feature in the 9th
212
layer. The output of ShadowNet is either or which realized by a step function with a similar implementation in 38
hi is the bisector I, and v. h2 1,

---

Table 1: Quantitative comparison terms of mean angular error for surface normal on DILIGENT benchmark [45]. This table summarizes comparison methods. 'N.A." represents not applicable as calibrated PS is with known l and e. 'Semi' indicates certain method leverage partial information of light. 'v" (or 'X') represents that certain methods (do not) adopt supervised learning for the estimation of surface normal n, light direction L, or light intensity e. 'Identical' means certain methods require the light intensity of different illuminating to be identical.
PS/
Method
Supervision
Supervision
Supervision
BALL
Cow
GOBLET
Por1
Por2
AVG
18 50
PS
N.A.
N.A.
4.10
8.39
14.92
8.41
25. .01
18.50
30.62
8.89
14.05
19:30
15.39
TM
47
PÇ
N.A.
N.A.
147
5.79
10.36
5.44
6.32
11.47
22.90
6.09
7.76
11.03
8,8
UFS
39.12
02.94
AMB
UPS
X
Ideasical
7:27
16.81
32.81
31.45
54.72
46.34
61.70
18:37
49.16
53.65
57:25
SMI0
UPS
Idenscall
8.00
11.98
15.54
19.84
22.73
48.79
73.86
16.68
50.68
26,
20.59
UPS
Meetical
6.42
19:25
14.52
58.
23.93
UPS
Menrical
22.43
15:44
25.76
25.01
22.53
79.
32.82
20.57
27.63
M11
UPS
X
Monrical
4.77
9.07
14,92
9.54
19:53
29.93
29.21
9.51
15.90
24.
16.66
LC17
UPS
Mdeiicall
9.30
10.90
19.00
12,60
15.00
18.30
28.00
12.40
15.70
22,
16:35
Semi
Known
13.06
11.84
18.07
11.13
11.31
13.62
C1119
K
UPS
2.77
6.89
8.97
8,06
3,48
11.91
17.43
8.14
7.50
14.90
9.51
CW20
15
UPS
2.50
5.60
8.60
7,90
7,80
9.60
16.20
7,30
7.10
14.90
8.71
CM20
Semi
Known
2.78
13.38
8.05
26.90
18.18
33.35
9.47
19.58
14.19
15:40
UPS
10.85
10.17
11.62
SK32
UPS
3,46
5.48
10.00
8.94
6.04
9.78
17.97
7.76
7.10
15.02
9.15
Ouark
LIPS
1.15
4.41
8.78
5,08
6.14
9.49
17.68
7.34
6.12
11.82
7.66
4 Experiments
215
Training details. Our main framework is implemented in PyTorch, while the pre-calculations (method (57) and silhouette fitting) are implemented in MATLAB. We use Adam as the optimizer with a learning rate a x 10 to train our framework in 500 epochs for each scene separately, and the warm-up stage takes up 10 epochs. The last 100 epochs use lower learning rate 5 x 10 for fine-tunning. The batch size is 32 for lighting and 256 for spatially random sampling. After every epoch, the depth map is reconstructed according the predicted normal map by method (10). Each scene takes from hours to 6 hours on one RTX 2080Ti 12GB GPU, depending on the resolution of the objects.
217
219
221
223
Evaluation metrics. We adopt the same metric in 13 the scale-invariant relative error, to measure the accuracy of recovered light intensity as,
Em
(12)
where, 77 is calculated by solving arg min,, E/=1 (nee 22)² by least squares minimization. The metric measure the accuracy of the predicted light directions and surface normal is the widely used mean angle eтror (MAE) in degree.
225
226
227
228
4.1
Evaluation on Public Datasets
Since the proposed NelF is unsupervised uncalibrated photometric stereo method, we compare its performance with state-of-the-ar aht uncalibrated and unsupervised photometric stereo methods. Three
230
real-world datasets, including the DILIGENT benchmark dataset (45). APPLE & GOURD dataset (2), and LIGHT STAGE DATA GALLERY dataset [1]. are used for evaluation.
232
Quantitative comparison for normal map. Table I lists relevant works for a comprehensive surface normal estimation comparison. As summarized in Table [ our method is the only method that addresses UPS without the supervision of N, L. E. However, performance and maintains a considerable advantage over other competitors [13 15) (e.g., 7.86 for
234
method achieves the best
236
NelF vs. 8.71 for (15) Handling the objects in DILIGENT dataset (45) with different shapes and reflectance, numbers from our method are either best or competitive (about 1 as compared with the
238
best performing UPS method), which shows its good generalization capacity to general reflectance and various shapes. This because our NelF fully considers mutual information by building up intrinsics fields and the implicit modeling facilitates general reflectance modeling.
240
242
Visual quality comparison for the normal map. Fig. illustrates the visual quality comparison in terms of recovered surface normal maps and corresponding error maps DILIGENT dataset (45)
7

---

observed
CHrE
CIN20
GH19
7.00
6.32
11.62
14.80 14.90
19:03
Figure 3: Visual quality comparison in terms of normal map and error map on Cow (left) and READING (right) from DILIGENT 455 For each subfig, from left to right: ground truth of normal map observed image, normal map / error map from our method, CW20 (15). CH19 and TM18 47).
Object
Est Normal
Error
6
NEJEET STANDNN K.
Figure 4: From left to right: the reference image, estimated surface normal map from our method, the lighting direction error, light intensity error, the estimated shadow map, diffuse reflectance map, specular reflectance map, diffuse scaling coefficient map, and specular coefficient map. Top three objects APPLE, GOURD1, and GOURD2 from APPLE & GOURD dataset (2), and bottom two objects are STANDING KNIGHT and HELMET from LIGHT STAGE DATA GALLERY dataset
244
The comparison is conducted with two state-of-the-art UPS methods [13 [5] and state-of-the -he-thh-rrt art unsupervised PS method [47). Our method is less sensitive to spatially-varying albedo due to the per pixel manner (red boxes in the left subfigure of Fig. 3). However, although the positional code considers the global shape effect, this per-pixel manner less effective in modeling complex shape information as compared with the all-pixel one. It fails for regions with cast shadow (or overexposure) under most light directions (red boxes in the right subfigure of Fig. 3.. We also show the visual quality results for APPLE GOURD dataset (2) and the LIGHT STAGE DATA GALLERY dataset (IT) in Fig. Our method produces reliable estimation for most regions, thanks to the full exploitation of mutual information across different dimensions.
245
246
248
250
252
253
Quantitative comparison for light directions and intensities. As can be observed in Table 2
254
and Table 3] our method achieves a superior performance advantage over unsupervised methods (YS97 [7 and PF14 (37]) while maintaining competitive performance as compared with supervised methods (CH19 13) and CW20 (55). These supervised methods adopt two-step solutions and
255
256
suffer from accumulating lating error. Therefore, even though they achieve similar performance in terms of
258
light conditions accuracy, they are less effective on estimating surface normal as compared with our
method (see Table I).
8

---

Table 2: Quantitative comparison in terms of mean angluar error for light direction and scale- -nvar-invaai
error for intensity on DILIGENT benchmark [45]
BALL
BEAR
DUDUEA
CAP
Cow
GOBLET
IARVVSTT
PoT1
For2
READING
AVG
Model
dir
dir.
dir
dir
dir
dr.
int.
dir
dir
dir
Inr
PF14
22
5.24
0.053
16.34
0.044
0.223
0.17
0.44
21.77
13.75
CH119 17
3.27
0.009
3.67
0.06
4.34
0.048
4.08
0.065
4.52
0.073
10.36
0.067
6.32
0.022
5.44
0.088
2.87
0.048
4.50
0.105
4.92
0.068
CW20
15
1.75
0.827
2.44
0.101
2.86
0.152
4.58
0,075
3.15
01031
2,88
0,042
5.74
0.065
1,41
0.039
281
0.059
5,47
0.048
3,32
0.052
Ours
1,99
0,030
3,95
00810
1.73
0.032
292
0,81
4,88
00030
6.32
0,040
7,06
0,832
3.33
0.134
571
0,028
7,45
0.042
4.37
0.042
Table 3: Quantitative comparison in terms of mean angular enor for light direction and error for intensity on APPLE& & GOURD [2] and LIGHT STAGE DATA GALLERY [TT..
STANDING KNIGHT
APPLE
GOURDI
GOURD2
AVG
HELMET
AVG
Model
dir.
int.
dir.
int.
dir.
int.
dir.
int.
dir.
int.
dir.
int.
dir.
int.
YS97
37
25.71
0,400
22.23
0.329
29.30
0.347
25.75
0.359
37.48
0.533
34.43
0.476
35.96
0.505
PF14
6.68
0.109
21.23
0.096
25.87
0.329
17.92
0.178
33.81
1.311
25.40
0.576
29.61
0,944
CH19
9.31
0.106
4.07
0.048
7.11
0.186
6.83
0.113
11.60
0.286
6.57
0.212
9.09
0.249
CW20
15
10.91
0.094
4.29
0.042
7.13
0.199
7.44
0.112
5.31
0.198
5.33
0.096
5.32
0.147
Ours
2.65
0,011
1,76
0.029
3.21
0.230
2.54
0.090
13.38
0.189
8.12
0.082
10.75
0.135
280
4.2
Evaluation on Sparse Uncalibrated Photometric Stereo
261
As compared with other per-pixel PS methods (e.g.. ZJ19 62 and LL19 (30]. the proposed NelF exploits much more constraints to estimate the normal at point (i.e., #IP vs. f). Because it
263
learns fields instead of regressing intensity profile (43) to surface normal. Therefore, we investigate the effectiveness of our method under the challenging setting of sparse UPS.
264
265
We randomly select 10 16 images illuminated by different lights from DILIGENT dataset [45] and
266
test our method using these images. We repeat this process 30 times, similar to [62] We compare the
267
performance with that from the classical UPS method PF14 [37]. a deep learning based per-pixel approach ZJ19 [62] and an all-pixel approach CHI8 14]. Due to the fewer reconstruction terms to train our NeIF, we slightly increase the dimension of the positional encoding module from 4 to 6 that increases the frequency (48) to stabilize the training, i.e., strengthening the role of positional code to
269
271
reduce the variance of estimated intrinsics. Besides, we drop Ca during early stage warm-up because YS97 [57) fails for the sparse inputs.
273
can observed, our method achieves competitive performance on normal estimation compared with state-of-the-arr sparse PS methods with known light (i.e., 11.62 for NelF vs. 9.82 for ZJ19 (62) under 10 lights, and 10.38 for NelF vs. 9.00 for CHI8 [14) under 16 lights). Kindly note that we do not require any ground truth normal for supervision. As compared with the unsupervised UPS method (PF14 (37)) we achieve a superior performance advantage. The results validate the effectiveness of building up neural intrinsics fields that fully exploit the mutual information across different dimensions.
We report the mean results over 30 trails in Table 4
274
275
277
279
5
Conclusion
280
This paper proposes NelF for UPS. By representing the general reflectance as four neural intrinsics fields, it implicitly imposes the light clues of specular reflectance and cast shadow for light estimation, which facilitates solving UPS with general reflectance. The proposed NelF can fully exploit mutual information from all observed pixel values so that produces stable estimation for PS and sparse PS. The unsupervised training manner of NelF is beneficial to the generalization capacity of data from different sources.
282
284
286
287
Limitations. Although our method produces promising results for light conditions and surface normal estimation, it has several limitations. First, the estimated intrinsics, such as shadow, specular and diffuse reflectance, and their balancing coefficients are less accurate for objects with complicated geometries, as shown Fig. 4 Second, we need to balance the role of positional code and estimated intrinsics when applying our method for sparse UPS. However, how to find an optimal balancing strategy is unknown. Third, as shown in Fig. 3 our method is less effective on global effects, especially for regions with cast shadow (or overexposure) under most light directions due to our
288
289
290
292

---

Table Quantitative comparison terms of mean angular error for surface normal on DILIGENT dataset [45). We compare our method with supervised PS methods (CH18 (14) and ZJ19 62) ) and unsupervised UPS methods (PF14 (37) under different randomly selected lights. All of the selected methods are retested under the same light setting with us. In the table. "(z)" indicates lights are used in the certain methods; 'V' (or xx') represents that certain methods (do not) adopt supervised learning for the estimation of surface normal (note that light information is known in CH18 [14
and ZJ19 [62] ). Bold font indicates the best performance under 10 lights and 16 lights, respectively.
Model
UPS
PS/
Supervision
BALL
BEAR
BUDDHA
CAT
Cow
GOBLET ET
HARVEST
Part
Por2
READING
AVG
CHI 14
PS
3.87
6.35
9.20
K.47
9.95
10.68
18.40
9.01
8.97
13.29
10.02
ZJ19 (10) 6z
PS
4.38
5.79
9.60
7.13
7.87
10.00
18.35
8.41
11.20
15.45
9.82
PF14 3V
UPS
X
69.95
10.19
17.47
11.11
22.23
49.77
46.96
10.38
19.69
31.50
28.93
Ours (10)
UPS
2.49
4,90
11.58
7.47
10.21
14.45
27.30
11.41
6.57
19.85
11.62
CHI8 (16)
14
PS
3.27
5.98
8.48
7.21
8.58
9.48
17.04
8,06
8.15
13.73
9.00
PF14 (16)
UPS
63.76
9.15
15.24
9.13
20.30
41.52
33.58
9.78
17.08
26.40
24.60
Ours (16)
UPS
X
2.34
4,52
10.33
7.87
7.68
11.23
25.07
8.00
6.35
20.42
10.38
294 per- pixel manner. Besides, the inference of our method is inefficient due to the unsupervised training manner and the per-pixel estimation, which is inapplicable for real-time applications.
295
References
297
[I]
Jens Ackermann, Fabian Langguth, Simon Fuhrmann, and Michael Goesele. "Photometric stereo for outdoor webcams". In: Proc. Computer Vision and Pattern Recognition (CVPR). 2012.
298
299
300
[2]
Neil Alkdrin, Todd Zickler, and David Kriegman. "Photometric stereo with non-parametric and spatially-varying reflectance", In: Proc. Computer Vision and Pattern Recognition (CVPR).
301
2008.
303
[3]
Neil G Alldrin and David J Kriegman. "Toward reconstructing surfaces with arbitrary isotropic reflectance: A stratified photometric stereo approach". In: Proc. International Conference on Computer Vision (ICCV). 2007.
304
306
[4]
Neil G Alldrin, Satya P Mallick, and David Kriegman. "Resolving the generalized bas-relief ambiguity by entropy minimization". In: Pruc. Computer Vision and Pattern Recognition (CVPR). 2007.
308
309
[5]
Svetlana Barsky and Maria Petrou. "TTe 4-source photometric stereo technique for three- dimensional surfaces in the presence of highlights and shadows". In: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). 2003.
311
312
[6]
Peter N Belhumeur, David J Kriegman, and Alan L Yuille. "The bas-relief ambiguity". In: International Journal of Computer Vision (IJCV). 1999.
313
314
77]
Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. "Neural reflectance fields for appearance acquisition". In: arXiv preprint arXiv:2008. 03824. 2020.
316
317
[8]
James F Blinn. "Models of light reflection for computer synthesized pictures". In: International Conference on Computer Graphics and Interactive Techniques. 1977.
318
319
9
Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P.A. Lensch. "NeRD: Neural Reflectance Decomposition From Image Collections". In: Proc. International Conference on Computer Vision (ICCV). 2021.
320
321
322
[10]
Xu Cao, Boxin Shi, Fumio Okura, and Yasuyuki Matsushita. "Normal Integration via Inverse Plane Fitting With Minimum Point-to-Plane Distance". In: Proc. Computer Vision and Pattern
323
324
Recognition (CVPR). 2021.
325
[11]
Charles-Félix Chabert, Per Einarsson, Andrew Jones, Bruce Lamond, Wan-Chun Ma, Sebas- tian Sylwan, Tim Hawkins, and Paul Debevec. "Relighting human locomotion with flowed reflectance fields". In: EGSR. 2006.
326
327
[12] Manmohan Krishna Chandraker, Fredrik Kahl, and David J Kriegman. "Reflections on the generalized bas-relicf ambiguity". In: Proc. Computer Vision and Pattern Recognition (CVPR). 2005.
329
330
10

---

331
[13]
Guanying Chen, Kai Han, Boxin Shi, Yasuyuki Matsushita, and Kwan- Yee K Wong. "Self- calibrating deep photometric stereo networks". In: Proc. Computer Vision and Pattern Recog- nition (CVPR). 2019.
332
333
[14]
Guanying Chen, Kai Han, and Kean-Yee K. Wong. "PS-FCN: A Flexible Learning Framework for Photometric Stereo". In: Proc. European Conference on Computer Vision (ECCV). 2018. Guanying Chen, Michael Waechter, Boxin Shi, Kwan-Yee K Wong, and Yasuyuki Matsushita. "What is learned in deep uncalibrated photometric stereo?" In: Proc. European Conference on Computer Vision (ECCV). 2020.
335
336
[15]
337
338
339
[16]
Xingyu Chen, Qi Zhang, Xiaoyu Li, Yue Chen, Feng Ying, Xuan Wang, and Jue Wang. "Hallucinated Neural Radiance Fields in the Wild". In: Proc. Computer Vision and Puttern Recognition (CVPR). 2022.
340
[17]
Donghyeon Cho, Yasuyuki Matsushita, Yu-Wing Tai, and In So Kweon. "Semi-calibrated photometric stereo". In: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). 2018
343
345
[18]
Hin-Shun Chung and Jiaya Jia. "Efficient photometric stereo on glossy surfaces with wide specular lobes". In: Proc. Computer Vision and Pattern Recognition (CVPR). 2008.
348
[19]
Robert L. Cook and Kenneth E. Torrance. "A reflectance model for computer graphics". In: ACM TOG. 1982.
348
349
[0
Athinodoros S Georghiades. "Incorporating the torrance and sparrow model of reflectance in uncalibrated photometric stereo". In: Proc. International Conference on Computer Vision (ICCV). 2003.
350
351
352
[21]
Dan B Goldman, Brian Curless, Aaron Hertzmann, and Steven M Seitz. "Shape and spatially- varying brdfs from photometric stereo". In: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). 2009.
353
354
[22] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. "Improved training of wasserstein gans". In: Proc. Conference on Neural Information Process- ing Systems (NeurIPS). 2017.
356
[23]
Shuhei Hashimoto, Daisuke Miyazaki, and Shinsaku Hiura. "Uncalibrated photometric stereo constrained by intrinsic reflectance image and shape from silhoutte". In: International Confer- ence on Machine Vision Applications (MVA). 2019.
359
360
381
[24]
Tomoaki Higo, Yasuyuki Matsushita, and Katsushi Ikeuchi. "Consensus photometric stereo". In: Proc. Computer Vision and Pattern Recognition (CVPR). 2010.
362
[25]
Michael Holroyd, Jason Lawrence, Greg Humphreys, and Todd Zickler. "A photometric approach for estimating normals and tangents". In: ACM TOG. 2008
364
385
[26]
Satoshi Ikehata, David Wipf, Yasuyuki Matsushita, and Kiyoharu Aizawa. "Robust photometric stereo using sparse regression". In: Proc. Computer Vision and Pattern Recognition (CVPR). 2012.
366
387
388
[27]
Berk Kaya, Suryansh Kumar, Carlos Oliveira, Vittorio Ferrari, and Luc Van Gool. "Uncali- brated neural inverse rendering for photometric stereo of general surfaces". In: Proc. Computer Vision and Pattern Recognition (CVPR). 2021.
369
370
[28]
Berk Kaya, Suryansh Kumar, Francesco Sarno, Vittorio Ferrari, and Lue Van Gool. "Neural Ra- diance Fields Approach to Deep Multi- Vieti--iewwwww Photometric Stereo". In: Proc. Winter Conference on Application of Computer Vision (WACV). 2022.
372
374
[29]
Junxuan Li and Hongdong Li. "Neural Reflectance for Shape Recovery with Shadow Handling". In: Proc. Computer Vision and Pattern Recognition (CVPR). 2022.
375
376
[30]
Junxuan Li, Antonio Relles-Kelly, Shaodi You, and Yasuyuki Matsushita. "Learning to minify photometric stereo" In: Proc. Computer Vision and Pattern Recognition (CVPR). 2019.
377
378
[31]
Feng Lu, Xiaowu Chen, Imari Sato, and Yoichi Sato. "SymPS: BRDF symmetry guided photometric stereo for shape and light source estimation". In: IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI). 2017.
380
381
[32]
Feng Lu, Yasuyuki Matsushita, Imari Sato, Takahiro Okabe, and Yoichi Sato. "Uncalibrated Photometric Sterco for Unknown Isotropic Reflectances", In: Proc. Computer Vision and Pattern Recognition (CVPR). 2013.
382
383
[33]
Brian Mac Namee, Padraig Cunningham, Stephen Byrne, and Owen Corrigan. "The problem of bias in training data in regression problems in medical decision support". In: Artificial Intelligence in Medicine, 2002.
385
11

---

387
[34]
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. "Nerf: Representing scenes as neural radiance fields for view synthesis". In: Proc. European Conference on Computer Vision (ECCV). 2020.
388
[35]
Kazuma Minami, Hiroaki Santo, Fumio Okura, and Yasuyuki Matsushita. "Symmetric- Light Photometric Stereo". In: Proc. Winter Conference on Application of Computer Vision ( (WACV). 2022.
391
393
[36]
Daisuke Miyazaki, Kenji Hara, and Katsushi Ikeuchi. "Median photometric stereo as applied to the segonko tumulus and museum objects". In: International Journal of Computer Vision (IJCV). 2010.
394
395
396
[37]
Thoma Papadhimitri and Paolo Favaro. "A closed-form, consistent and robust solution to uncalibrated photometric stereo via local diffuse reflectance maxima". In: International Journal of Computer Vision (IJCV). 2014.
398
389
[38]
Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, and Nicu Sebe. "Binary neural networks: survey". In: Puttern Recognition. 2020.
400
401
[39]
Yvain Quéau, Tao Wu, François Lauze, Jean- Denis Durou, and Daniel Cremers. "A non-convex variational approach to photometric stereo under inaccurate lighting". In: Proc. Computer
402
403
Vision and Pattern Recognition (CVPR). 2017.
404
[40] Francesco Sarno, Suryansh Kumar, Berk Kaya, Zhiwu Huang, Vittorio Ferrari, and Luc Van Gool. "Neural Architecture Search for Efficient Uncalibrated Deep Photometric Stereo". In: Proc. Winter Conference on Application of Computer Vision (WACV). 2022.
405
406
407
[41]
Steven M Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski. "A comparison and evaluation of multi view stereo reconstruction algorithms". In: Proc. Computer
408
Vision and Pattern Recognition (CVPR). 2006.
[42]
Boxin Shi, Yasuyuki Matsushita, Yichen Wei, Chao Xu, and Ping Tan. "Self-calibrating photometric stereo". In: Proc. Computer Vision and Pattern Recognition (CVPR). 2010. Boxin Shi, Ping Tan, Yasuyuki Matsushita, and Katsushi Ikeuchi. "Bl-polynomial modeling of low-frequency reflectances". In: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). 2013.
411
412
[43]
413
414
[44]
Boxin Shi, Ping Tan, Yasuyuki Matsushita, and Katsushi Ikeuchi. "Elevation angle from reflectance monotonicity: Photometric stereo for general isotropic reflectances". In: Proc.
416
European Conference Computer Vision (ECCV). 2012.
[45]
Boxin Shi, Zhe Wu, Zhipeng Mo, Dinglong Duan, Sai-Kit Yeung. and Ping Tan. "A benchmark dataset and evaluation for non-lambertian and uncalibrated photometric stereo". In: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). 2016.
419
420
421
[46]
Pratul P. Srinivasan, Boyang Deng. Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T. Barron. "NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis". In: Proc. Computer Vision and Pattern Recognition (CVPR). 2021.
422
423
424
[47]
Tatsunori Taniai and Takanori Machara. "Neural inverse rendering for general reflectance photometric stereo"". In: Proc. International Conference on Machine Learning (ICML). 2018. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. "Attention is all you need". In: Proc. Conference on Neural Information Processing Systems (NeurIPS). 2017.
425
426
[48]
427
428
429
[49]
Eva Weigl. Sebastian Zambal, Matthias Stöger, and Christian Eitzinger. "Photometric stereo sensor for robot-assisted industrial quality inspection of coated composite material surfaces". In: International Conference on Quality Control by Artificial Vision (QCAV). 2015.
431
432
[50]
Robert J Woodham. "Photometric method for determining surface orientation from multiple images". In: Optical Engineering. 1980.
433
434
[11]
Lun Wu, Arvind Ganesh, Boxin Shi, Yasuyuki Matsushita, Yongtian Wang, and Yi Ma. "Robust photometric stereo via low-rank matrix completion and recovery". In: Proc. Asian Conference on Computer Vision (ACCV). 2010.
436
437
[52]
Tai-Pang Wu, Kam-Lun Tang, Chi-Keung Tang, and Tien-Tsin Wong. "Dense photometric stereo: A markov random field approach". In: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). 2006.
438
439
440
[53]
Zhe Wu and Ping Tan. "Calibrating photometric stereo by holistic reflectance symmetry analysis". In: Proc. Computer Vision and Puttern Recognition (CVPR). 2013.
441
12

---

442
[54]
Limin Xie, Zhan Song, Guohua Jiao, Xinhan Huang, and Kui Jia. "A practical means for calibrating an LED-based photometric stereo system". In: Optics and Lasers in Engineering. 2015.
443
[55]
Wuyuan Xie, Zhan Song. and Ronald C Chung. "Real-time three-dimensional fingerprint
446
acquisition via new photometric stereo means". In: Optical Engineering. 2013.
447
[56]
Sai-Kit Yeung, Tai-Pang Wu, Chi-Keung Tang, Tony F Chan, and Stanley J Osher. "Normal estimation of transparent object using video". In: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). 2014.
448
449
[57]
Alan Yuille and Daniel Snow. "Shape and albedo from multiple images using integrability".
451
In: Proc. Computer Vision and Pattern Recognition (CVPR). 1997.
452
[58]
Guangjun Zhang and Zhenzhong Wei. "A novel calibration approach to structured light 3D vision inspection", In: Optics & Laser Technology. 2002.
453
454
[59]
Jason Zhang, Gengshan Yang. Shubham Tulsiani, and Deva Ramanan. "NeRS: Neural re- flectance surfaces for sparse-view 3d reconstruction in the wild". In: Proc. Conference on Neural Information Processing Systems (NeurIPS). 2021.
455
456
457
[60]
Kai Zhang, Fujun Luan, Qianqian Wang. Kavita Bala, and Noah Snavely. "PhySG: Inverse Rendering With Spherical Gaussians for Physics- -Base Material Editing and Relighting". In: Proc. Computer Vision and Pattern Recognition (CVPR). 2021.
459
480
[61]
Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron. "Nerfactor: Neural factorization of shape and reflectance under an unknown illumination". In: ACM TOG. 2021.
461
462
[62]
Qian Zheng, Yiming Jia, Boxin Shi, Xudong Jiang, Ling- Yu Duan, and Alex C Kot. "SPLINE- Net: Sparse photometric stereo through lighting interpolation and normal estimation networks". In: Proc. International Conference on Computer Vision (ICCV). 2019.
464
[63]
Zhenglong Zhou and Ping Tan. "Ring-light photometric stereo". In: Proc. European Conference on Computer Vision (ECCV). 2010.
487
13


Reviews:
[{"review_id": "qR4yJzLyZVh", "reviewer": "Reviewer_yGxh", "summary": "This paper proposed a method that aims to solve the uncallibrated photometric stereo problem where the input light intensities and light directions are unknown. The key idea of this paper is to represent the surface normal, and surface reflectance, shadows, and lights as neural networks. And then jointly train those neural networks via the image reconstruction loss in an unsupervised manner. \n\nThe proposed method explicitly considers the clues from specular effects and cast shadows for solving this problem. \nSpecifically, the paper proposed: a PositionNet for outputting the surface normal, specular weights, and diffuse weights of the object; a SpecularNet that takes half angle and surface normal as input, outputting the specular component; a ShadowNet that explicitly predict the shadow map; and LightNet that takes RGB images as input, outputting the light intensities and directions.\n\nThe performance of the method was evaluated on DiLiGenT and some other real datasets and achieved lower normal estimation errors than other state-of-the-art algorithms.", "questions": "How will the number of input images affect the testing time of this method? e.g., how long does it take to test with 16 images at input?", "limitations": "- The rendering equation doesn't take inter-reflection into account. Hence, in theory, the proposed method will fail if the inter-reflection effects dominate the surface. \n- The photometric stereo images are usually taken in a dark room with limited lighting exposures, which may also introduce high-level image noise. I encourage the authors to discuss more about the robustness of this method when the image assumptions are violated in these aspects.\n\nSuggestion:\n- I encourage authors to list the testing time of other most recent inverse rendering-based methods [44,25,26] for comparison. ", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n- The paper is well written and easy to follow.\n- The evaluation of the proposed method is extensive and complete, and supports the claimed effectiveness of each component of the method.\n\n \n\nWeaknesses\n- The design of the PositionNet and SpecularNet is very similar to the design of [26] (\"Neural Reflectance for Shape Recovery with Shadow Handling\"). But I didn't find a proper reference to [26] to state the difference: both [26] and this work have a network outputting the surface normal, diffuse albedo, and specularity; both [26] and this work have a SpecularNet which take normal, half-angle, and view direction and outputting the specular effects of the surface; both [26] and this work explicitly compute the shadow map which is then used in image rendering; both [26] and this work use a warm-up loss for early-stage supervision of the shadow map. I encourage the authors to discuss the differences to help clarify the position of this work.\n\n- How does the SpecularNet work for objects with specular-SV-BRDF? Especially for a object with two different materials: soft and shiny specular effects? (such as a object make up with two different metals: one with high roughness, one with low roughness).\nThe input of SpecularNet is only the surface normal, half angle, and view direction. So for different positions over the surface, they will have the same SpecularNet, which means that it assumes all the surface points will have the same specular effects (same specular BRDF curve, or same specular roughness). But this isn't the case for most of the objects in the world: for some objects with spatially-varying BRDF where both soft specular effects and very shinny and sharp specular effects exist on the surface. \n\n- Unsupervised or weak-supervised? The method is claimed to be an unsupervised method, but as mentioned in line 194, the method uses the light estimation results from YS97 as a weak-supervision signal to train the LightNet. Besides, from Supplementary Table 9, it seems that without using YS97 as weak-supervision in the early-stage warm-up will cause a significant drop in accuracy.\n\n- For the design of LightNet, since we need to train the LightNet in a weak-supervised manner from YS97, why not just use light directions from YS97 as the initialization and directly optimize on these lights. (i.e. why not explicitly optimized the light directions instead of optimizing the parameters of LightNet?)\n\n- Since we need to train the LightNet in a weak-supervised manner, why don't we choose a better method as the \"teacher network\"? For example, why not use light direction and intensities from SDPS[12] as the guidance to guide the training of LightNet?\n\n- The ambiguity in ShadowNet. I believe that there is ambiguity in the design of the proposed ShadowNet. From equations (3) and (4), it seems that the authors aim to use ShadowNet to represent the cast shadows. In the object's appearance, both pixels that are in cast-shadows and attach-shadows have 0 intensity. \nBut their causes are completely different; the cast shadows are caused by objects self-occluding the light source; the attach-shadows are caused by the surface back-facing the light source. And surfaces with extremely dark materials may sometimes also have zero pixel intensity under certain light directions. And the proposed method will have ambiguity under the above situations. The downside of this ambiguity can actually be seen in Fig 4, 5th column, where the generated shadow map shows the inconsistent boundaries in many objects (e.g Standing K. and Gourd2, Apple).  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SgX7bi31p6v", "reviewer": "Reviewer_zHtR", "summary": "This paper introduces a novel training framework and model design for unsupervised photometric stereo, which is the problem of estimating surface normals of an object given a set of images with uncalibrated lighting positions. The model leverages Neural Intrinsic Fields, which are a set of MLPs which represent shadows, positions, specularity and light, as well as a CNN encoder aimed at aiding the representational power of the system. The training procedure uses a set of domain-specific loss functions and a warm-up strategy aimed at improving stability and generalization. Ablation studies are provided, which show the effectiveness of the model and the training procedure, and comparisons with previous work show quantitative improvements. ", "questions": "- On line 34, it is claimed that supervised UPS methods fail at generalizing to data points outside the training distribution. The authors thus present unsupervised learning as a better alternative for enhancing generalization. This argument is repeated throughout the paper, but, why is this the case? Why is unsupervised learning inherently better than learning from labels in this particular scenario? Please explain better or demonstrate quantitatively.\n- The paper could benefit from referring the reader to important papers that would help in contextualization and learning about this particular domain. For Neural Fields, I would suggest referring to (Neural Fields in Visual Computing and Beyond, Xie et al.), which should help the reader understand that this UPS paper is not that closely related to NeRF papers as the title and the related work suggest. For material models and intrinsic decomposition, I would suggest (A Survey on Intrinsic Images: Delving Deep Into Lambert and Beyond, Garces et al.). \n- Please discuss the limitations of the material model in terms of which effects are not modelled with the current image formation assumptions, as many effects cannot be represented by NeIF but this is not clear to the reader. \n- Please explain terminology that is not explained nor it is obvious to the reader. For example, GBR, Lambertian objects, etc.\n- Please include standard deviations in the tables, to better understand whether or not the improvements are significant. \n- Please double-check the references (eg line 185).\n", "limitations": "The paper does not include any negative impact analysis of their work. It is difficult to imagine any direct negative impact of surface normal estimation, but I would argue that this method is computationally very expensive at test time and it may be detrimental in terms of energy consumption. In terms of limitations, the paper includes an adequate limitation analysis. To complete it, I would suggest the authors to discuss the limitations of the material model that their method assumes for training, as it does not account for complex light phenomena, including sub-surface scattering, transmittance or iridescence. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Summary* \n\nThis paper provides valuable ideas for training methods for unsupervised photometric stereo, which is an important problems in computer vision and graphics. The proposed method is sound, ablation studies are provided that validate the effectiveness of the design choices of their method, and the model combines advances in machine learning with domain-specific ideas in a smart way. However, the paper is not well written, presentation is arguably poor and comparisons with previous work do not show a huge improvement. Given that the paper is hard to read and comparisons with previous method do not show significant improvements, it is difficult to recommend this paper for acceptance at its current state. \n\nStrenghts:\n\n- This paper introduces novel ideas for uncalibrated photometric stereo, combining advances in machine learning with ideas from more classical computer vision and graphics. \n- The proposed method is technically sound, and ablation studies show the impact of each component in the overall generalization capabilities of the model. Evaluation is somewhat exhaustive. \n- Comparisons with previous work are complete and exhaustive to the best of my knowledge. \n\nWeaknesses:\n\n- The paper needs revision in the way it is presented and written. It is hard to follow and some sentences are very difficult to understand. The paper would benefit strongly from a revision on this level. Further, it is presented in a way that assumes a lot of previous knowledge from the reader. Please see below for suggestions on references that may help the render understand better this paper and its context.\n- Previous work is not adequately studied and it is not clear where this paper sits on the literature. By reading the title and some parts of the paper, it would seem that this is a NeRF paper with applications for uncalibrated photometric stereo. This is not the case, and I believe the paper would benefit from a more clear distinction with Radiance Fields methods. \n- Comparisons with previous work could be presented better, in my opinion. For example, standard deviations should be shown next to averages (eg Table 1) and sometimes previous work shows better perfomance that the proposed method, but the authors decide not to highlight that (eg Table 3, CW20 is better in AVG dir but this is not in bold). It is not entirely clear how much better the proposed method is with respect to previous work, and, when it is not, why previous work is better. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "R1_opWsCKqL", "reviewer": "Reviewer_UR1t", "summary": "An unsupervised method is proposed for uncalibrated photometric stereo. It works by parameterising lighting, reflectance, normal map and shadows by neural nets, a.k.a. neural intrinsic fields a.k.a. NeIF. These networks are supervised by rendering loss, with additional regularisations on boundary normal directions, lighting directions, BRDF monotonicity and shadows. Experiments show overall superior results than unsupervised or uncalibrated methods.\n\n", "questions": "What is ours(16) and ours(19) in table 1,2? Are these results with sparse and all input views because I cannot find any explanation in texts?\n\n\n\n", "limitations": "I think there is room for improvement. See my comments above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The method is uncalibrated and unsupervised, meaning it does not require any additional information other than input images themselves.\n- Although implicit neural nets are not new to PS, the intrinsic decomposition paradigm makes sense and the method is technically sound.\n- Experiment results are convincing. \n\nWeakness:\n- I think the paper could benefit from being more upfront about its assumptions and limitations. For example, lighting is assumed to be distant, and BRDF being a bivariate one (with half angle and difference angle parameterisation) with a single specular lobe. \n- While the BRDF parameterisation does allow variations between specular and diffuse, one could argue the single specular lobe assumption is too restrictive for diverse, spatially varying materials, e.g. an object made with two specular materials with different BRDFs.\n- I need more convincing arguments about why and how the bas-relief ambiguity is resolved here. The boundary normal azimuth penalty in eq(6) does not fully disambiguate light/normal directions. However the experiments seem to suggest both normal and light directions are in fact successfully recovered. \n- One strength of this paper is that it deals with varying light intensities. However there is no mention to how much light intensity varies in experiments (e.g. diligent dataset). I wonder how the results would change if you apply a random scale to each input images.\n- Minor typos in reference style:\n    - line 25: [4], [34] -> [4,34]\n    - line 76 [bi2022neural]\n    - line 185 [8758025]\n    - missing citation numbers in figure 3", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FNGeqilN0--", "reviewer": "Reviewer_U25q", "summary": "This paper presents the neural uncalibrated photometric stereo method under the single directional lighting setup. Following recent neural-field-based algorithm, the method simultaneously predicts surface attributes and lighting parameters in a unified self-supervised learning method. Shadows are explicitly considered for filtering out its contribution in the optimization process. In the evaluation, the proposed method achieved state-of-the-art performance on the public DiLiGenT benchmark.", "questions": "- The concurrent work [1] also introduced the neural fields for the photometric stereo. This work and [1] seem very close to each other for the modeling of shadow and BRDF even though this work solves the uncalibrated task unlike [1]. Other than this difference, is there any critical difference between these works that makes this work more significant?\n\n- The proposed method certainly works in the environment assumed in the benchmark, but there is some question as to whether this research direction is the right for the photometric stereo field. Since many applications in photometric stereo are time-critical (i.e., surface defect inspection), it is questionable whether the small improvement in accuracy obtained with the neural field's approach is worth the sacrifice the efficiency in learning-based photometric stereo methods which only take less than a second to recover comparable results. In addition, I believe that data-driven methods can handle a more generic environment including the global illumination effects under a more complicated lighting environment than an inverse rendering that relies on a specific physically-based model. There is no point in simply improving benchmark scores if they are not immediate to real problems. Therefore, I would like to see a little more convincing motivation of this work.\n", "limitations": "The limitation is addressed in the paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The proposed method is very clearly written and the algorithms are easy to understand.\n- The benchmark study shows that the proposed method is superior to the competitor.\n- It is very interesting that the loss is propagated from the cast shadow since the shadow mapping is generally indifferentiable.\n- Ablation studies have been conducted for each of the technical components.\n- The application of neural fields to uncalibrated photometric stereo is new among published studies.\n\nWeakness:\n- It seems an overclaim that there are no explicit assumptions about the light and reflectance since the light is obviously constrained to the single directional lighting excluding near and natural lighting.  As for the reflectance, how to handle spatially-varying BRDF is not clear since SpecularNet is only queried by nh and vh without the location of the pixel. It seems that the proposed method only scales a single BRDF by the specular coefficient rho_k, which obviously limits its representability. The more proper manner seems to define multiple bases (neural networks) and compute multiple coefficients as was done in [1].\n\n- The behavior of ShadowNet is not clear to me. Since the gradient of binarization layer is static, the loss doesn’t seem to be propagated. Therefore, it seems to me that the shadow is not explicitly considered for recovering the geometry unlike the shape-from-shadow and it is only used to detect outliers to make the reconstruction loss robust to cast shadows as is similar with [1]. If so, it seems that simply detecting shadows statistically and reflecting them in the loss function would be sufficient. This work may need evidence that results from ShadowNet are superior to statistical shadow removal.\n\n- The physical parameters recovered by the proposed method have not been properly evaluated, so it is not possible to examine whether the formulation is appropriate. From the results presented in Fig17 of supplementary, it appears that the proposed method couldn’t successfully recover physical attributes such as shadows and reflectance, which makes the proposed method less useful.\n\n- All results are based on public evaluation data sets taken in carefully controlled environments. In addition to those results, I would have liked the authors to have shown the actual results taken by authors to demonstrate its availability in real use-case.\n\n[1] Li and Li, Neural Reflectance for Shape Recovery with Shadow Handling, CVPR2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject