S$^3$-NeRF: Neural Reflectance Field from Shading and Shadow under a Single Viewpoint
Authors: Wenqi Yang
Guanying Chen
Chaofeng Chen
Zhenfang Chen
Kwan-Yee K. Wong
Venue: NeurIPS 2022 Accept (2022)
Keywords: Single-view Reconstruction
Photometric Stereo
Shape-from-shadow
Neural Scene Representation

Abstract: In this paper, we address the "dual problem" of multi-view scene reconstruction in which we utilize single-view images captured under different point lights to learn a neural scene representation. Different from existing single-view methods which can only recover a 2.5D scene representation (i.e., a normal / depth map for the visible surface), our method learns a neural reflectance field to represent the 3D geometry and BRDFs of a scene. Instead of relying on multi-view photo-consistency, our method exploits two information-rich monocular cues, namely shading and shadow, to infer scene geometry. Experiments on multiple challenging datasets show that our method is capable of recovering 3D geometry, including both visible and invisible parts, of a scene from single-view images. Thanks to the neural reflectance field representation, our method is robust to depth discontinuities. It supports applications like novel-view synthesis and relighting. Our code and model can be found at https://ywq.github.io/s3nerf.

Se-NeRF: Neural Reflectance Field from Shading and Shadow under a Single Viewpoint
Wengi Yang
Guanying Chen FNii and SSE, CUHK-Shenzhen chenguanying@cuhk. edu. cn
The University of Hong Kong wqyangBcs hku. hk
Chaofeng Chen
Zhenfang Chen MIT-IBM Watson Al Lab
Nanyang Technological University chaofenghust@gmail.. com
(beenzhenfangg201@gmaallll com
Kwan-Yee K. Wong
The University of Hong Kong kykwongōcs. hku. hk
Abstract
In this paper, we address the "dual problem" of multi-view scene reconstruction in which we utilize single-view images captured under different point lights to learn a neural scene representation. Different from existing single-view methods which can only recover 2.5D scene representation (i.e., normal depth map for the visible surface), our method learns a neural reflectance field to represent the 3D geometry and BRDFs of scene. Instead of relying multi-view photo- consistency, our method exploits two information rich monocular cues, namely shading and shadow, to infer scene geometry. Experiments datasets show that our method is capable of recovering 3D geometry, including both visible and invisible parts, of scene from single-view view images. Thanks to the neural reflectance field representation, our method is robust to depth discontinuities. It supports applications like novel-view synthesis and relighting. Our code and model can be found at https: btttps///ye //wqq//wwwau/.. .iittuu............. io/s3nerf.
multiple challenging
Introduction
3D reconstruction from images is central and important problem in computer vision. Multi-view stereo methods, which capture a target scene from multiple viewpoints under a fixed lighting condi- tion [12, 24, 45, 46], are the most widely adopted approach for scene reconstruction. These methods, however, often assume surfaces with Lambertian reflectance and have difficulties in recovering high- frequency surface details.
An alternative approach to scene reconstruction is to utilize images captured from a fixed viewpoint but under different point light sources (see Fig. (a)). This setup is adopted by photometric stereo (PS) methods [15, 47, 56] where shading information is utilized to reconstruct surface details of non- Lambertian objects. Shadow another cue that has been exploited for shape recovery by shape-from- shadow methods [11, 61, 67]. However, existing single-view methods typically adopt single normal or depth map represent the visible surface, making them incapable describing back-facing and occluded surfaces (see Fig. (b)). Besides, methods relying on surface normal representation struggle
"Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Mullivriw
Single-ciew Varying Lighnings
Input
Normal
Depth
Neural Field (Oun)
Fixed Lighting
Iesialle Regions
(a) Different capturing setups
(b) Comparison of different scene representations
Figure I: (a) Difference between multi-view fixed lighting and single-view varying lighting setups. (b) Comparison the normal map, depth map, and neural field in representing a 3D scene. Obtaining accurate depth from normal integration is non-trivial [S], and depth map cannot describe the invisible regions. The adopted neural field is capable of modeling the complete scene geometry.
to deal with depth discontinuities [25]. It is desirable to obtain more complete scene reconstruction (including both visible and invisible parts) from single-view images. In this paper, we realize this by exploiting both shading and shadow
recover both visible and invisible parts of scene.
Recently, neural scene representations have achieved significant progress in multi-vie reconstruction and novel- synthesis [35, 48, 63]. These methods model a continuous 3D space (i.e., the scene) with multi-layer perception (MLP) which maps 3D points to scene properties (e.g., density and color in NeRF [35]). Despite its great success in multi-view scene modeling, neural scene representation has been less explored in single-view scene modeling.
In this paper. building on top of the recent advances in neural scene representation, we propose to optimize neural field using images captured from a single viewpoint under different point lights. Our method is fundamentally different from existing works [35, 48, 63) in that, instead of relying on multi-view photo-consistency, we exploit monocular shading and shadow cues to optimize our neural field for scene reconstruction (see Sec. A in supplementary for intuitive explanations on shadow cues).
straightforward idea would be condition the color MLP of NeRF [35] also on the point light directions. However, we find such a naïve solution fails to recover scene geometry and appearance. To make better use of the photometric stereo images, we explicitly model the surface geometry and BRDFs with reflectance field and adopt physics- rendering to obtain the 3D point color [2. 3]. The 2D pixel color of a sampled ray can then computed using volume rendering. Differentiable shadow computation is considered explicitly by tracing ray from a 3D point to the point light position check the light visibility [71]. As evaluating the light visibility all points sampled along a ray computationally expensive [49], we accelerate the computation by only evaluating the light visibility the expected surface point, making online shadow computation possible during optimization.
To summarize, our contributions are:
We address novel problem of 3D neural reflectance field optimization from single-view images captured under different point lights. Different from existing neural scene representation methods that rely multi-view photo-consistency. our method exploits monocular shading and shadow cues for neural field optimization.
Our method jointly recovers the geometry and BRDFs of scene, and adopts an efficient online shadow computation to fully exploit the information-ris -ich shading and shadow cues.
Experiments on multiple challenging datasets show that our method can faithfully reconstruct complete scene geometry from single-view images. Our method is robust depth discontinuities. It supports applications like novel-view synthesis and relighting.
Related Work
Photometric stereo (PS) methods can recover pixel-wise surface normals from images captured under different light directions [15, 56]. Traditional PS methods treat specular observations as

---

outliers [36, 57, 58] or fit sophisticated reflectance models [10, 18, 54] to handle non- non-Lambertian surfaces. Recent methods resort deep learning technique solve this problem. Supervised learning methods learn mapping from image observations to surface normals using synthetic dataset with ground-truth normals [7, 9, 17, 26, 32, 43, 73]. Self-supervised methods optimize the network parameters using an image reconstruction loss [22, 51]. The above methods assume directional lightings. For near-field PS problem, methods based on PDE [40] and deep learning [29, 31, 34, 44] have been proposed. More recently, Li et al. [25] proposes a coordinate-based MLP to represent the normal map of the visible surface assuming directional lights. In contrast, our method represents a scene with continuous volume and recovers the full 3D scene geometry under a hear-field setup.
Shape from shadow Shadow has been exploited to estimate shape information [11]. Yu and Chang optimized a height map from shadow cues using a graph-based representation [67]. Shadowcuts [6] explicitly considers shadow in Lambertian photometric stereo. Yamashita et al. [61] introduced
1D shadow graph accelerate the shadow computation. Recently, DeepShadow [21] models the
depth map of a scene by an MLP and optimizes the model with a shadow reconstruction loss. These
methods can only recover a height map of the visible surface. Besides, they require the detected shadow regions as input, but shadow detection is itself non-trivial problem.
Neural scene representation Neural scene representations have heen successfully applied in novel- view synthesis and multi-view reconstruction [38, 48, 52, 59, 63]. The popular neural radiance field (NeRF) [35] represents a continuous space with an MLP, which regresses the volume density and RGB color of 3D point from the point coordinates and view direction. Attracted by the photo-realistic rendering produced by NeRF, many follow-up works are introduced to improve the reconstructed surface quality [39, 55, 64], rendering speed [14, 30, 41], optimization speed [37, 50, 65], and robustness [1. 33, 68].
The above methods consider each 3D point an emitter, making them not able to model the surface materials and lighting separately. Inverse rendering methods have been proposed to jointly recover shape, materials, and lightings in casual capture setup [3, 4, 71, 72]. NeRV [49] explicitly models shadow and indirect illumination assuming a known environment map. NRF [2] and IRON [70] adopt a co-located camera-light setup to simplify the image formation model. PS-NeRF [62] utilizes multi-view and multi-light images to induce regularizations for more accurate surface reconstruction.
There are some attempts to reconstruct radiance field from single- image in a data-driven manner (e.g.. conditioning the MLP input with image features [13, 42, 66]), or utilizing depth image as shape prior [60]. However, due to the strong ambiguity, these methods struggle to achieve high-quality reconstruction. Compared with the above approaches, our method extends neural scene representation to reconstruct accurate shape and materials from single-view photometric stereo images.
3 Method
Given N images captured from a single viewpoint under different near point lights, our method targets at recovering the geometry and materials for the scene (see Fig. 2). Following existing near- field photometric stereo methods [34, 44], we assume calibrated perspective camera and known point light positions. Instead of representing the visible surface with normal / depth map like others [25, 34, 44), we adopt a 3D neural field representation [3, 35, 39] to describe the 3D scene.
3.1 Neural Reflectance Field Representation
Our method built on top of the recent neural radiance field (NeRF) [35]. Following UNISURF [39], we adopt an occupancy field instead of a density field to better represent the surface geometry. UNISURF uses an MLP to map a 3D point æ E R33 and a view direction d E R to occupancy o(x) E IR and color c(æ, d) E R An image can be generated through volume rendering in which the color of each pixel (or ray r) is calculated by
Nv
C(r) = Σ ((x)))
II - o (xz)) c(;))) (es,dd) d), HZl
where m; denotes a 3D point sampled along the ray r o + (d. with o E R being the camera center and d E 3R3 the ray direction specified by the pixel, and Nv R is the number of samples per ray.
3

---

Shading
MLP
Pt d Le
AcculV
MLP
E8
a Nv
Renderer
c
MLP
D s
d
shared-weight weight
Shadow
Pt
( (
MLP
1- Accu-l
v
Figure 2: Overview of the method. For each camera ray, first apply root-finding locate the surface intersection point sg Nv points on the camera ray are sampled within a relatively large interval around the surface the surface-to-light segment to calculate the light visibility, which is multiplied to the accumulated shading output the final RGB value.
generate accumulated shading values. Nz points are sampled on
Given multi-view images, radiance field can be optimized to reproduce the input images. However, applying SeRF-based methods to single-view images is non-trivial. A straightforward idea would be condition the color MLP of NeRF also light directions, but our experiments show that such a naive solution fails to produce reasonable reconstruction due to the lack of constraints on scene geometry.
To utilize shading information in photometric stereo images, we explicitly model the BRDFs of the scene and recover 3D point color with physics-based rendering [2, 3]. Observing that shadow provides strong cues for inferring the geometry both visible and invisible surfaces in a scene, we compute shadow in online manner by tracing ray from surface point the light position to determine its light visibility. Give point light located at P IR with emitted intensity Le R, Eq. (1) can be rewritten as
C(r)
Σ o(ar)) II o1 (m;)) (oPp: jZ1
Tue a:);
where the 3D point color c(æ;, d) is replaced by the product of light visibility (o x) and physics- based rendered color Se(d, Pt, Le)a),, the details of which are given in the following subsections.
3.2 Physics-basee Color Rendering
We consider non-Lambertian surfaces with cattially-vayyingg BRDFs. The rendering equation for surface point æ viewed from direction d under point light (pt: Le) be written
fold, Pi, Z) Line Lei x) fm (d, w. (p:::)): max (w: (p: (xxx), 0),
(3)
Light Intensity
BRDF Value
Shading
where Line(P:: Lei x) denotes the incident light (taking light falloff into account), w; (Prs x) the incident light direction, and fm (d, w. w.(P; w); x) the BRDF value æ. The normal can be derived from the gradient of the occupancy field as nx(x)= = Vo(=)/ Vo(æ)]12 [39].
Lighting model Following previous works [34, 44], we adopt the inverse-square law for point light attenuation where light intensity Line is proportional to the multiplicative inverse of the square of the distance s (i.e., Line & 1/s2). The incident light direction w; and light intensity Lint at a point x given by
D: x
Le
w; (p; 2:)
Line Le; a))
1p xx|12
|p
BRDF model Similar to [3, 69, 71], we adopt a BRDF model represented by a combination of diffuse color Pa and specular reflectance Ps. which is given by
fm (w;, wo: x)) = Pa + Ps (w, wo: x)
(5)

---

Following [16, 25], we model the isotropic specular reflectance by a weighted combination of Sphere Gaussian (SG) bases, which demonstrates better results modeling specular effects than the parametric Microfacet model [20]. The specular component Ps is hence written as w D(h, n.).
where (h,, n) G(h, n:; A) Axx (h"n'"),,,__ (a(h"
denotes the SG bases, with X_ 4 controls the specular sharpness. The diffuse component pa and SG weights estimated by two MLPs.
3.3 Online Shadow Computation
A 3D point æ is shadowed if there is any occluders in its line of sight for the light position P,. It follows that light visibility (P x) E [0. 1] for a 3D point æ can be computed by accumulating occupancies along this line (see Fig. 2), i.e.
So(Gw))
(e:)
II j<i
o (wz)),
(6)
where Nr. is the number of points sampled along the line.
However, calculating light visibilities for all Nv points sampled along the ray for pixel is computationally ex- pensive (i.e., ON N₂) MLP queries for each pixel ray
[x
(see Fig. (a)). To speed up shadow computation, pre- vious methods either adopt MLP to directly regress light visibility
[
point [49] to reduce the queries for each ray to (Nyv) (see Fig. 3 (b)), or pre- pre-extracts the surface points (assuming a fixed scene geometry) [71] to reduce the number of MLP queries to (NN). Instead, we first locate the expected surface points æ. along the ray by moot-finding [39] and caleulate its light visibility an online manner. Eq. (2) can be reformulated for effi- cient color rendering
(as) Pi
(a) N()
(b) O(Nv)
Figure 3: Alternative shadow modeling.
Ny
C(r)= = fo
Σ
(1-o(xx))) oxx;) II - (xx))) (a, d. Pt, Le).
(7)
jei
3.4 Optimization
Different from shape-from-shadow methods [21, 53], our method does not require direct supervision for shadow rendering. We rely on image reconstruction loss for optimization.
Volume rendering loss The first loss is the LI reconstruction loss between the volume rendered image C. (f.e., the computed C(r) in Eq. (7)) and the input image:
C = Σ C 7111-
(8)
Surface rendering loss UNISURF [39] proposes to combine the volume rendering and surface rendering by gradually shortening the sampling range in a ray refine the surface region. However, we empirically found that the model will start to degrade when the sampling interval is decreased as there is no multi-vie view information to constrain the non-sampled regions. We therefore propose to adopt a joint volume and surface rendering strategy. We additionally compute the surface rendering color Cr(r) using the expected surface point x, and calculate the LI loss, i.e.,
C. Σ IC, -I
C,(r) = f. (d, L.;
(10)
Normal smoothness loss Similar to [39]. we also include regularization loss to promote smoothness surface normal (e is small random perturbation):
Ln Σ |m(x)) nl e)1
(11)
5

---

Table 1: Comparison with neural field methods on relighting and normal estimation results.
BUDDHA
READING
BUNNY
CHAIR
LEGO
HOTDOG
Method
PSNR1
MAE]
PSNR1
MAE]
PSNR
MAE.
PSNR
MAE.
PSNRT
MAE]
PSNR1
MAEL
NeRF" [35]
38.57
70.12
39.50
72.60
37.41
68.35
35.25
88.46
35.56
91.09
39.80
72.07
UNISURF"
41.51
54.86
40.54
60.59
38.48
54.27
34.98
47.79
34.55
45.81
38.64
51.00
Ours
4342
2.44
43.13
2,03
40.43
1.72
36.33
1.83
35.54
6.49
38.01
2.50
Nearest Input
GT
Ours
UNISURF"
NeRF
GT
Ours
UNISURF
NeRF
Figure 4: Comparison with neural field methods relighting (left) and normal estimation (right).
Overall loss The overall loss function used for optimization is as follow with a set to 0.005:
L = Lv + L. + aLn.
(12)
Experiments
4.1 Implementation Details
Similar to UNISURF [39], we use an 8-layer MLP (256 channels with softplus activation) to predict the occupancy o and output 256-dimensional feature vector. Two additional 4-layer MI.s then take the feature vector and point coordinates as input to predict the albedo Pa and weights w of SG bases. We sample Ny 256 points along the camera ray and N₂ 256 points along the surface-to-light line segment. We use Adam optimizer [23] with an initial learning rate of 0.0002 which decays at 200 and 400 epochs. We train each scene for 800 epochs on one Nvidia RTX 3090 card, which takes about 16 hours to converge.
Evaluation Metries We adopt mean angular error (MAE) in degree for surface normal evaluation and L1 emrrr in em for depth assessment. PSNR is used to measure the quality of images rendered under novel view or novel lighting.
4.2 Datasets
Our method targets at recovering the complete scene by exploiting both shading and shadow. However, existing photometric stereo datasets are mostly interested in the object region and intentionally remove the influence of the background (e.g., cover the background with black cloth to avoid inter- reflections [47]), which makes the shadow and shading information invisible the background regions. Therefore, such datasets [34, 47] are not suitable evaluate the full potential of our method.
Instead, we evaluate our method on multiple synthetic datasets with complicated scene geometry and materials. Specifically, we used 10 3D objects for data rendering, where 5 objects from DiLiGent- Dataset [27] (namely, BEAR, BUDDHA, COW, POT2, and READING), 2 objects from the internet (namely, BUNNY and ARMADILLO), and 3 objects from NeRF's s blender dataset [35] (namely, LEGO, CHAIR, and HOTDOG). We rendered LEGO, CHAIR, and HOTDOG with Blender's Cycles pathtracer, and the other 7 objects with Mitsuba [19]. As our method does not explicitly model inter- reflections, we set the max bounces to during rendering. During rendering, we created scene by adding a horizontal and a vertical plane model the desk and wall, and objects are placed on the

---

Table 2: Comparison with single-view view normal depth estimation methods (only object regions).
BUDDHA
READING
BUNNY
CHAIR
LEG0
HOTDOG
Method
MAE]
Depth.11
MAE]
Depth
MARI
Depth1.11
MARL
Depch
Depch1.11
MAR]
Daych 1.1]
2L18 [28]
37.51
19.84
57.24
25.97
31,40
17.68
39.53
41.19
46.82
34.56
39.74
18.02
QY18
12.25
40.84
26.13
14.21
29.68
15.95
17.87
8.98
HS20 [44]
6.47
16.92
29.56
13.99
13:27
27.25
Ours
14:24
1.50
7.00
2.09
9.40
1.63
17.43
4.74
31.13
7.31
14.65
1.88
GT
Ours
ZL18 [28]
QY18[40]
HS20044]]
GT
Ours
ZL18(28]
HS20[44] QY18[40]
45
Figure 5: Comparison with single-view normal depth estimation baselines. Row and Row 2 show the normal and error maps. Row 3 shows the side-view of the reconstructed surfaces.
horizontal plane. Each scene was rendered under 128 uniformly sampled near point lights, and the rendered images are in linear space with resolution of 512 x 512.
4.3 Comparisons with Existing Methods
To justify the effectiveness of our method, we compare with three types methods, namely, neural field methods, photometric stereo methods, and single-image shape estimation methods.
Neural radiance field methods We first verify the design of our method by comparing it with two simple baselines (i.e., adapting NeRF [35] and UNISURF [39] for this problem by conditioning the color MLP on light direction). Table and Fig. 4 show the normal estimation and relighting results in the training view. Although the baseline methods can achieve reasonable rendering results in terms of PSNR, they fail to predict accurate cast-shadow and cannot reconstruct the geometry of the scene (with large average MAE of 17.1222.....
contrast, our method is able to accurately reconstruct the shape with an average MAE of 2.84, and achieves the best rendering results (average PSNR of 39.48). This result indicates that simply conditioning the color MIP on light direction does not provide sufficient constraint to regularize the scene geometry.
Single-view shape estimation methods We then compare with three state-of-the-ar single-view normal depth estimation methods, including two near-field PS methods (QY18 [40] and HS20 [44]) and one single-image shape estimation method (ZL18 [28]). QY18 [40] and HS20 [44] consider exactly the same setup as our method (multiple images captured under near point lights), so the input are the same as our method. ZLI8 [28] assumes an image captured under co-located flash light as input, so we choose the image illuminated by a point light that closest to the camera as its input. As these methods are designed to estimate the shape in the object region and have difficulty in dealing with the background, we only report the normal and depth estimation results on the object region for the training view in Table 2. Since ZL.18 [28] and HS20 [44] require depth alignment before evaluation, we align the estimated depth with the ground truth for all the methods for fair comparison. We can see that our method achieves the best results for both normal and depth estimation. Moreover, as shown in Fig. 5, our method can faithfully reconstruct both visible and invisible parts of the scene, which is not possible by methods that rely on the normal or depth representation.
4.4 Method Analysis
We next conduct ablation study for different components of our method, and evaluate our method on different setups to further analyze its behavior.
7

---

Table 3: Quantitative results for the ablation study,
Tran View
Novel Views
CHAIR
BUNNY
BUDDHA
CHAIR
BUNNY
BUDDHA
Method
MAEL
PSNR
MAEL
PSNKT
MAE
PSNR
MAE]
PSNR
MAEL
PSNRT
w/o shading
33.71
w/o shadow
3.39
30.81
2.26
33.45
3.33
34.30
12.24
22.31
11.93
24.43
16,60
23.27
w/o Ours
35.85
3.77
5.10
28.58
6.27
29.11
1.3
36.33
1.72
40.43
2.44
43.42
5.45
26.82
6.11
29.55
6.89
31.53
Impur/ / GT
w/o Shading
wlo Shadow
w/o C,
Ours
Input GT
w/o Shading
w/o Shadon
wio C,
Ours
45
Figure 6: Visual results for the ablation study. Row 1 the normal of train view, and row 2 shows its map compared with ground truth. Row 3 shows the normal of novel view.
Input
GT Normal and Side-view
Ours
ZL18 128]
HS20
144
Figure 7: Results on scenes with multiple occluding objects.
Joint shading and shadow modeling Our method exploits both shading and shadow information for scene reconstruction. To analyze the effect of both components in reconstructing the scene geometry, we trained two variant models where (a) "w/o shading" replaces the BRDF module with 4-layer MLP to directly predict RGB values with additional light location input; and (b) "w/o shadow" removes the shadow module and only output the shading. Results are summarized in Table 3 and Fig. 6. We evaluate the results for both trained view and novel views, and report MAE of normal maps and PSNR of rendered images. As the "w/o shading model fails estimate proper depth and the recovered surfaces totally deviate from the ground truth, we omit its results for novel views. Without shadow information, the model may still predict proper surface normal for the trained view. However, the model fails to predict the depth and shape of the object since there is no constraint on invisible regions. By exploiting both shading and shadow cues, our method can well reconstruct the full scene.
Joint volume and surface rendering We also analyze the effectiveness of the surface rendering loss L. by comparing our full model with the one without L.. Results in Table 3 and Fig. 6 show that surface rendering loss can effectively refine the surface normals of the object surface.
Effect of occlusion/ddicconiinnig and unseen region We further demonstrate the potential of our method in reconstructing the complete geometry of the scene, especially when there are occlusions or discontinuous surfaces. Figure shows the reconstructions for two scenes with multiple objects occluding each other. It is very difficult to identify the shape of the invisible regions just from the single-view images. However, by effectively leveraging the shadow information, method successfully predicts the shape (i.e., the occupancy field) of the invisible regions, which is not feasible for existing works.
We also investigate the performance of our method on surface with challenging invisible shapes. Note that the shape of invisible regions are mainly constrained by the shadow (which indicates the

---

Input View
GT
Ours
GT Novel
Ours Novel
Input View
GT
Ours
GT Novel
Ours Novel
Figure 8: Results two scenes with challenging invisible shapes.
Table 4: Analysis Light numbers.
Table 5: Analysis Light Range.
CHAIR
ARMADILLO
CHAIR
ARMADILLO
Lighsu
MAE.
Depth
PSNR
MAE]
Depth
PSNKT
Range
MAEL
Depth)
PSNRT
MAK]
Depth.
PSNKT
4
30.39
181.22
19.39
46.87
93.66
15.87
small
3.92
18.79
30.15
2.32
6.86
35,26
10.99
35.60
5.98
37.17
median
1.93
8.75
35.96
1.70
4.59
38.92
16
2.10
8.11
36.20
5.89
37.50
32
1.97
8.77
36.23
2.05
6.27
39.20
64
8.64
36.52
7.18
39.78
128
9.04
36.33
6.65
40.13
GT / Input
L
L
L
L 64
Small
Median
Broad
(a) Different Number of Lights
(b) Different Light Distributions
Figure 9: Analysis on different number of lights and light distributions.
occupancy along the light path). In Fig. 8, we show the reconstruction of READING object which is posed make the concave surface invisible. From the results of novel views, we that method can properly recover the invisible irregular surface, though some invisible regions are not fully consistent with the ground truth shape. This result demonstrates that shadow provides strong cue for shape recovery especially for unseen regions.
Effect of light distributions To analyze the robustness of our method on different light distribution, evaluate it on scenes illuminated by different number of lights different ranges (see supple- mentary material for visualization light distributions), and the numerical results are summarized in Table 4 and Table (depth errors calculated in object regions). The model fails reconstruct the scene with only 4 light inputs, but can reconstruct faithful shape of the scene with light inputs. With more lights used, the surface of the object is further refined (see Fig. 9). We also observe that method can still work for small range of light distributions to recover invisible regions. Overall, our method is robust to different number and different range of lights.
Results on real scenes To further demonstrate the practicality of our method, we evaluate three real scenes, which were captured using a fixed camera (with 28mm focal length) and a handheld cellphone flashlight (see Fig. 10). The object off all the environmental light sources and only kept the flashlight on, which was randomly moved around to capture images illuminated under different light conditions. For each object took around 70 images.
put on the table and close to the wall. We turned
Our setup does not require manual calibration of lights. Instead, we applied the sfate-of-the -te-art self- calibrated photometric stereo network (SDPS-Net [8]) for light direction initialization, and roughly measured the camera-object distance as initialization of light-object distance. After initialization, the position and direction of lights are jointly optimized with the shape and BRDF during training. Please refer to our supplementary materials for more training details. Sample inputs and results are

---

Capturing Setup
BOTANIST
GIRL
CAT
Figure 10: The data capturing setup and three testing objects.
Sample Input Images
Relighting
Albedo
Normal
Novel Views
Figure 11: Results on the real captured data. From top to bottom: BOTANIST, GIRL, CAT.
shown in Fig. 11. Even with this casual capturing setup and uncalibrated lights, our method achieved satisfactory results in normal prediction and full 3D shape reconstruction.
Conclusion
In this paper, have introduced method optimize neural reflectance field for nom-baabertian scene from single-view images captured under different near point lights. Our method jointly recovers the geometry (i.e., occupancy field) and BRDFs of the scene by fully utilizing the shading and shadow cues. Interestingly, our results on scenes with complicated shapes and materials show that the complete scene geometry be faithfully reconstructed just from single- photometric images. Moreover, comprehensive method analysis demonstrates that our method is robust to scenes with different geometry, materials, light number, and light distributions. Additionally, method supports applications like novel-view synthesis and relighting.
Limitation First, like existing pear-field PS methods [44], our method requires known light positions, which requires additional efforts for lighting calibration. Second, as our method relies on shadow cue invisible shape reconstruction, its performance may decrease the scene background is highly complicated as the background geometry will affect the appearance of shadows. Third, although the shape of the invisible parts can be well reconstructed, the reflectance those regions are not well constrained by shadow. Last, our method ignores inter-reflection effects in image formation. In the future, we will further extend our method to tackle these limitations.
Acknowledgements This work was partially supported by the National Key R&D Program of China (No.2018YP)) 1 1800800), the Basic Research Project No. HZQB- KCZYŽ-202 of Hetao Shenzhen-Ho S&T Cooperation Zone, NSFC-62202499, and the Research Grant Council of Hong Kong (SAR), China (project no. 17203119).
References
[1] Jonathan Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla and Pratul P Srinivasan. Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields. Proceedings of the JEFF/CVF International Conference on Computer Vision (ICCV), 2021.3
10

---

[2] Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Geof-Grr,rr David Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition. arXiv preprint arXiv:2008. 03824, 2020. 2, 3,4
[3] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch. NeRD: Neural reflectance decomposition from image collections. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pages 12684-12694, 2021. 2,
3, 3
[4] Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan Barron, and Hendrik Lensch. Neural-PIL: Neural pre-integrated lighting for reflectance decomposition. In Proceedings of the
Advances in Neural Information Processing Systems (NeurIPS), volume 34, 2021. 3
[5] Xu Cao, Boxin Shi, Fumio Okura, and Yasuyuki Matsushita. Normal integration via inverse plane fitting with minimum point-to-plane distance. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2382-2391, 2021. 2
[6] Manmohan Chandraker, Sameer Agarwal, and David Kriegman. Shadowcuts: Photometric stereo with shadows. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-8, 2007. 3
[7] Guanying Chen, Kai Han, and Kwan- Yee K. Wong. PS-FCN: A flexible learning framework for photometric stereo. In Proceedings of the European Conference on Computer Vision (ECCV), pages 3-18, 2018.
[8] Guanying Chen, Kai Han, Boxin Shi, Yasuyuki Matsushita, and Kwan- Yee K. Wong. Self- calibrating deep photometric stereo networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
2019. 9
[9] Guanying Chen, Kai Han, Boxin Shi, Yasuyuki Matsushita, and Kwan- Yee K. Wong. Deep photometric stereo for non-Lambertian surfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), (1:4:11199911) 2020.
[10] Hin-Shun Chung and Jiaya Jia. Efficient photometric stereo on glossy surfaces with wide specular lobes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2008.
[11] Michael Daum and Gregory Dudek. On 3-d surface reconstruction using shape from shadows. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 461 166, 1998. 1,
[12] Yasutaka Furukawa and Jean Ponce. Accurate, dense, and robust multiview stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 32(8):1362-1333 2009. 1
[13] Chen Gao, Yichang Shih, Wei-Sheng Lai, Chia-Kai Liang, and Jia-Bin Huang. Portrait neural radiance fields from a single image. arXiv preprint arXiv:2012.05903, 2020. 3
[14] Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fast- NeRF: High- figh-fideiity neural rendering at 200fps. In Proceedings the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11346-14355, 2021. 3
[15] Hideki Hayakawa. Photometric stereo under light source with arbitrary motion. JOSA A, 1994. 1,2
[16] Zhuo Hui and Aswin C "Sankaranarayanan Shape and spatially-varying reflectance estimation from virtual exemplars. IEEE Transactions on Pattern Analysis and Machine Intelligence (T- PAMI), 2017.
[17] Satoshi Ikehata. CNN-PS: CNN-based photometric stereo for general non-convex surfaces. In Proceedings of the European Conference on Computer Vision (ECCV), 2018. 3
[18] Satoshi Ikehata and Kiyoharu Aizawa. Photometric stereo using constrained bivariate regression for general isotropic surfaces. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2014. 3
11

---

[19] Wenzel Jakob. Mitsuba renderer, 2010. 6
[20] Brian Karis and Epic Games, Real shading in unreal engine 4. Proc. Physically Based Shading
Theory Practice, 1(3):1, 2013. 5
[21] Asaf Karnieli, Ohad Fried, and Yacov Hel-Or. Deepshadow: Neural shape from shadow. arXiv
preprint arXiv:2203. 15065, 2022. 3,5
[22] Berk Kaya, Suryansh Kumar, Carlos Oliveira, Vittorio Ferrari, and Luc Van Gool. Uncalibrated neural inverse rendering for photometric stereo of general surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3814, 2021. 3
[23] Diederik Kingma and Jimmy Ba. ADAM: A method for stochastic optimization. In Proceedings of the The International Conference on Learning Representations (ICLR), 2015. 6
[24] Vladimir Kolmogorov and Ramin Zabih. Multi-camera scene reconstruction via graph cuts. In Proceedings of the European Conference on Computer Vision (ECCV), pages 82-96, 2002.
[25] Junxuan Li and Hongdong Li. Neural reflectance for shape recovery with shadow handling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2,5
[26] Junxuan Li, Antonio Robles-Kelly, Shaodi You, and Yasuyuki Matsushita. Learning to minify photometric stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 3
[27] Min Li, Zhenglong Zhou, Zhe Wu, Boxin Shi, Changyu Diao, and Ping Tan. Multi-view photometric stereo: robust solution and benchmark dataset for spatially varying isotropic materials. IEEE Transactions on Image Processing (TIP), 2020. 6
[28] Zhenggin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Learning to reconstruct shape and spatially-varying reflectance from a single image. ACM Transactions on Graphics (TOG), 37(6):1-11, 2018. 7,8
[29] Daniel Lichy, Soumyadip Sengupta, and David W Jacobs. Fast light- near-field photo- metric stereo. arXiv preprint arXiv:2203. 16515, 2022. 3
[30] Lingjie Liu, Jiatao Gu Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 15651-15663, 2020. 3
[31] Fotios Logothetis, Ignas Budvytis, Roberto Mecca, and Roberto Cipolla. cnn based approach for the near-field photometric stereo problem. arXiv preprint a:X1v:2009.05792, 2020. 3
[32] Fotios Logothetis, Ignas Budvytis, Roberto Mecca, and Roberto Cipolla. Px-net: Simple and efficient pixel-wise training of photometric stereo networks. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 12757-12766, 2021.
[33] Ricardo BratinnBruuall,, Noha Radwan, Mehdi SM Sajjadi. Jonathan T Barron, Alexey Doso- vitskiy, and Daniel Duckworth. NeRF in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 7210-7219, 2021. 3
[34] Roberto Mecca, Fotios Logothetis, Ignas Budvytis, and Roberto Cipolla. Luces: A dataset for near-field point light source photometric stereo. arXiv preprint 23X1v:2104.13135, 2021.
[35] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV), pages 405-421, 2020. 2,
3, 6,
[36] Yasuhiro Mukaigawa, Yasunori Ishii, and Takeshi Shakunaga. Analysis of photometric factors based on photometric linearization. JOSA A, 2007. 3
12

---

[37] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. arXiv preprint arXiv:2201.05989 2022.
[38] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable vol- umetric rendering: Learning implicit 3d representations without 3d supervision. In Proceed- ings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3514-3515, 2020. 3
[39] Michael Oechsle, Songyou Peng. and Andreas Geiger. UNISURF: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3589-5599 2021. 3, 4, 6, 7
[40] Yvain Quéau, Bastien Durix, Tao Wu, Daniel Cremers, François Lauze, and Jean-Denis Durou. Led-based photometric stereo: Modeling, calibration and numerical solution. Journal of Mathematical Imaging and Vision, 60(3):3 2018. 3,7
[41] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. KiloNeRF: Speeding up neural radiance fields with thousands of tiny mips. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 14335-14345, 2021.
[42] Konstantinos Rematas, Ricardo Martin-Brualla, and Vittorio Ferrari. Sharf: Shape-conditioned radiance fields from a single view. Proceedings of the ACM International Conference on Machine Learning (ICML), 2021.
[43] Hiroaki Santo, Masaki Samejima, Yusuke Sugano, Boxin Shi, and Yasuyuki Matsushita. Deep photometric stereo network. In Proceedings of the IEEE International Conference Computer Vision Workshops (ICCVW), 2017. 3
[44] Hiroaki Santo, Michael Waechter, and Yasuyuki Matsushita. Deep near-light photometric stereo for spatially varying reflectances. In Proceedings of the European Conference on Computer Vision (ECCV), pages 137-152, 2020. 3, 4. 7, 8,8,7 10
[45] Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In Proceedings of the European Conference on Computer Vision (ECCV), pages 501-5188 2016. 1
[46] Steven M. Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski. A comparison and evaluation of multi-view stereo reconstruction algorithms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2006. 1
[47] Boxin Shi, Zhipeng Mo, Zhe Wu, Dinglong Duan, Sai- Kit Yeung, and Ping Tan. A benchmark dataset and evaluation for non-Lambertian and uncalibrated photometric stereo. IEEE Transac- tions on Pattern Analysis and Machine Intelligence (T-PAMIJ, 2019. 1, 6
[48] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous sd-structure-aware neural scene representations. Proceedings of the Advances in Neural Information Processing Systems (NeurlPS), 32, 2019. 2,
[49] Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron. NeRV: Neural reflectance and visibility fields for relighting and view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7495-7504, 2021. 3,
[50] Cheng Sun, Min Sun, and Hwann- Trong Chen. Direct Voxel Grid Optimization: Super-fast convergence for radiance fields reconstruction. arXiv preprint arXiv:2111.11215 2021. 3
[51] Tatsunori Taniai and Takanori Maehara. Neural inverse rendering for general reflectance photometric stereo. In Proceedings of the ACM International Conference Machine Learning (CCML),2018. 2018.
[52] Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Yifan Wang, Christoph I Lassner, Vincent Sitzmann, Ricardo vances in neural rendering, arXiv preprint arXiv:2111.0884 2021. 3
Stephen Lombardi, et al. Ad-
13

---

[53] Kushagra Tiwary, Tzofi Klinghoffer, and Ramesh Raskar. Towards learning neural representa- tions from shadows. arXiv preprint arXiv:2203. 2022. 5
[54] Silvia Tozza, Roberto Mecca, M Duocastella, and A Del Bue. Direct differential photometric stereo shape recovery of diffuse and specular surfaces. Journal of Mathematical Imaging and Vision, 2016. 3
[55] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), volume 34, 2021. 3
[56] Robert J. Woodham. Photometric method for determining surface orientation from multiple images. Optical Engineering, 1980. 1,2
[57] Lun Wu, Arvind Ganesh, Boxin Shi, Yasuyuki Matsushita, Yongtian Wang. and Yi Ma. Robust photometric stereo via low-rank matrix completion and recovery. In Proceedings of the Asian Conference on Computer Vision (ACCV), 2010. 3
[58] Tai-Pang Wu and Chi-Keung Tang. Photometric stereo via expectation maximization. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 2010. 3
[59] Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond. arXiv preprint urXiv:2111.I 11426, 3021.
[60] Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Humphrey Shi, and Zhangyang Wang. Sinnerf: Training neural radiance fields complex scenes from single image. arXiv preprint arXiv:2204.0m 009080,009..... 2022. 3
[61] Yukihiro Yamashita, Fumihiko Sakaue, and Jun Sato. Recovering 3d shape and light source positions from non-planar shadows. In Proceedings of the International Conference Pattern Recognition (ICPR), pages 1775-1778, 2010. 1,3
[62] Wenqi Yang, Guanying Chen, Chaofeng Chen, Zhenfang Chen, and Kwan- Yee K. Wong, Ps- nerf: Neural inverse rendering for multi-view photometric stereo. In European Conference on Computer Vision (ECCV), 2022. 3
[63] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Ronen Basri, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 2502, 2020. 2,3
[64] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34, 2021. 3
[65] Alex Yu, Sara Fridovich-Keil Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. arXiv preprins arXiv:211..0531,, 2021.
[66] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRF: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 3
[67] Yizhou Yu and Johnny T Chang. Shadow graphs and 3d texture reconstruction. International Journal of Computer Vision (IJCV), 62(1):35-60, 2005. 1,3
[68] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. NeRF++: Analyzing and improving neural radiance fields. arXiv preprint arXiv.2010.07492, 2020.
[69] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. PhySG: Inverse ren- dering with spherical gaussians for physics-based material editing and relighting. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5453-5462 2021. 3.
14

---

[70] Kai Zhang, Fujun Luan, Zhengqi Li, and Noah Snavely, Iron: Inverse rendering by optimizing neural sdfs and materials from photometric images. arXiv preprint arXiv:2204.02232, 2022.
[71] Xiumiigg Zhang, Pratul Srinivasan, Boyang Deng, Paul Debevec, William Freeman, and Jonathan T Barron, NeRFactor: Neural factorization of shape and reflectance under unknown illumination. ACM Transactions on Graphics (TOG), 40(6), 2021. 2, 4,5
[72] Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, and Xiaowei Zhou. Modeling indirect illumination for inverse rendering. arXiv preprint arXiv:2204.06833 2022. 3
[73] Qian Zheng, Yiming Jia, Boxin Shi, Xudong Jiang, Ling- Yu Duan, and Alex C. Kot. SPLINE- Net: Sparse photometric stereo through lighting interpolation and normal estimation networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5 Limitation.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discuss in supplementary materials.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See supplementary materials for training details.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See supplementary materials.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? (Yes)
(b) Did you mention the license of the assets? [Yes] See supplementary materials.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "aCxLvsYrVkh", "reviewer": "Reviewer_nudb", "summary": "This paper proposed a single-view scene reconstruction method under different point light conditions based on the neural reflectance field. It uses the inverse-square law and BRDF to model the shading and root finding to model the shadow. By explicitly utilizing both shading and shadow cues, the model can reconstruct the scene from single view, even for some invisible regions.\nThe authors evaluated the model with different objects and showed that the method can well reconstruct the geometry with images from single view but different point lights.\n", "questions": "How’s the normal smoothness loss affect your method? Is that similar to UNISURF? From your result, especially on LEGO, the loss of detail of studs and track seems very serious. Is that caused by normal smoothness loss or by more complicated light reflection at those areas?\n", "limitations": "Yes, the authors addressed different aspect of limitations of the work in the paper and supplementary materials. They also pointed out this method can recover some non-directly seen area, that may address some privacy issues. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nA novel NeRF-based method that faithfully reconstructs a scene from a fixed viewpoint with multiple single point light conditions by explicitly modeling the shading and shadow. By carefully modeling the single point light shading by inverse-square law and BRDF, and shadow by root-finding. This method gives an excellent result.\nThe paper clearly illustrated the method in detail. The authors compared their method with other methods quantitatively and qualitatively. They also evaluated their model with multiple occluding objects, showing it can reconstruct the unseen area with shading and shadow cues.\n \n### Weakness\n\nThe authors studied the contribution of different components (shading and shadow) of their method. But the impact of different loss terms was not included. Specially, at L171, how the value of $\\alpha$ for normal smoothness loss is found is not illustrated.\n   \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "X9Dek14yHtp", "reviewer": "Reviewer_H2aV", "summary": "This pape proposed a method that aims at solving the near-field photometric stereo via neural reflectance field representation. The core insight of this paper is to use both the shading and cast-shadows of the view as the cues to recover both the visible and invisible parts of a scene. \nThe paper adopts the NeRF-like neural field representation to represent the occupancy field, reflectance field of a scene. They then use lighting, BRDF modeling and shadow rendering to reconstruct the observed images. The networks are jointly optimized via volume rendering loss, smoothness loss and color loss. ", "questions": "Please see the quesitons in the Weaknesses part.", "limitations": "- Limitations are well discussed in the paper.\n- Suggestions: It will be great to isolate the invisible region of the scene and show quantitatively how the proposed shadow rendering improve the accuracy in these regions.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The proposed method can estimate the geometry of the invisible region of the scene from a single-view point. The paper is interesting in estimating invisible regions from the shadows. And unlike previous shadow-based methods, this paper doesn't require explicitly shadow detection as input.\n- The method presents state-of-the-art geometry reconstruction results compare to prior near-field photometric stereo methods. \nEach component of the method, such as shadow and shading, is validated in the ablation studies.\n- The paper is well writen and easy to follow. \n\nWeaknesses\n- Each part of the proposed method is not novel. The method consists of three major parts: neural reflectance field representation follows [38] in occupancy field estimation and normal computation; the color and BRDF modeling follow [15,24] to use the sphere gaussian bases, SG weights and diffuse albedo; the shadow computation is slightly different to [48], but it is similar to [24] where they both check the visibility of the surface point. Hence, in the proposed method and rendering procedure of this paper, the novelty is very limited.\n\n- It is suggested to evaluate this method in more challenging real and synthetic datasets. The method was only tested on two datasets: a simple synthetic dataset and a real dataset. For the synthetic dataset: the background is clean and uniform; how will this paper perform in backgrounds with more textures?\n\n- I am curious to see how the number of lights affects the accuracy of the geometry in the invisible part of the scene. The only cues for invisible regions are the casted shadows on the ground. If lights fail to illuminate the invisible regions from different angles, how will the accuracy be affected?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "N6NgMgnZjY3", "reviewer": "Reviewer_13bh", "summary": "This paper proposed a promising framework for near-field photometric stereo utilizing volume and surface rendering. The proposed method is tested on a synthesis dataset and a real dataset. Their results have reached state-of-the-art. \n", "questions": "Except for the problems listed above, I wonder how well the proposed method can perform under a more complicated background.\n", "limitations": "The authors have addressed parts of the limitations and potential negative social  impact of their work. However, the impacts of the background's complexity and the shadow's visibility are not stated. \n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "---Originality---\n1) The overall idea shows the originality, especially in how to combine surface and volume rendering.\n2) Imposing the shadow's constraint on the occupancy field in the 3D space is enlighting. \n---Quality---\n1) Eq.8: the author used $C_v$ to denote the volume-rendered image and $C_s$ to indicate surface rendered image. However, no formula illustrates how $C_v$ is calculated, which  confuses reading. \n2) Figure 2: two \"MLP\" blocks are shown in two separate branches, which makes me think there are two different MLPs estimating occupancy along the surface-to-light segment and the camera ray. However, according to Eq.6, the MLP for shadow handling is identical to the MLP for the occupancy field. Please try to redraw this part.\n---Clarity---\n1) Figure 2's caption, lines 2-3: How the points along the rays are sampled is unclear to me. The author mentions to sample $N_V$ points around the surface and $N_L$ points on the surface-to-light segment. I assume $N_L$ points are sampled uniformly, but without knowing the scene's scale, it is unclear how to ensure the sampled points along the camera ray are \"around the surface.\"\n2) Although the author tries to explain the necessity of the shadow clues for surface reconstruction in Section 4.4 and Section B in the supplementary material, it is still unclear to me why the shadow becomes a strong constraint on the model that can substitute the constraint from multi-view photo-consistency. There are two kinds of cast shadows in the synthesis dataset, including the foreground shadow on the object, and the background shadow. According to the results in section F, table S2, and section 4.2 table 2, the visibility of the background shadow is very important for depth estimation. The author should analyze the contribution of two different kinds of shadows separately.\n3) According to the presenting results, the proposed method has reached an ideal visual and quantitative result in their synthetic dataset. Even the background is well reconstructed with little artifacts. However, the author didn't directly explain why their method is not sensitive to the background (it is almost perfect) and can reconstruct an accurate background simultaneously. (e.g., UNISURF [A] applies NeRF++ [B] for the complex background and assumes black for the simple background. However, their reconstructed background still contains visible artifacts).\n\n[A] Michael Oechsle, Songyou Peng, and Andreas Geiger. UNISURF: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2021\n\n[B] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv.org, 2020.\n\n---Significance---\n\n1) There is no intuitive explanation for the role of modeling shadow for shape estimation. What is the problem aiming to solve by considering the shadow, the ambiguity, additional clue of global shape information, useful clue for depth estimation, or even light refinement? \n2) The evaluation is mainly conducted on the synthetic dataset built by the authors, which is less convincing. Although the paper shows the result of the LUCES dataset in the supplementary material, the depth estimation is not satisfactory (4.39 vs. 3.17 for the state-of-the-art). This evaluation could not provide strong support to the proposed method.\n3) The compared methods, NeRF and UNISURF, are implemented in a naive way. It is unclear whether the authors have also sampled points along the light ray in those methods. If not, it can be expected that the reconstruction results are not good because NeRF and UNISURF are mainly applied in multi-view stereo, which is constrained by the multi-view photo consistency that is lacking in single-view photometric stereo. Therefore, using them for comparison is less convincing. \n4) As said before, the background shadow's visibility greatly influences the performance. Therefore, this work has lots of pre-setting to the environment (the background is better to be the light color, and the intensity should be bright enough), which will limit the application scope of this method. Moreover, The background's complexity may influence the background shadow's visibility and the shading of the scenes, which further impact the performance of the methods. While the background in the synthesis dataset and LUCES is still an ideal case, there is a lack of such discussions on a more realistic scenario.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GqkNEe-O7U", "reviewer": "Reviewer_bpE7", "summary": "This paper is about using neural radiance field to reconstruction the geometry of an object lit by varying directional light sources, leveraging cues of cast shadow. The optimization uses an efficient way to trace rays for shadow computation.", "questions": "- to make the setup slightly more practical, how well does the method perform when shadow is softer, i.e., slightly larger light source, or small area lights?\n\n- line 149: has the proposed shadow generation method been validated over direct regression method using an MLP?\n\n- fig 4: should also show the nearest neighbor training samples", "limitations": "The calibrated lighting is more demanding than calibrated camera pose, which can be easily acquired, even though not entirely accurate, on most modern smartphones. The lighting needs to be real directional light source, which is difficult to acquire in real-world scenarios.\n\nIt would be also valuable to discuss potential applications for the proposed setup and algorithm.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The setup is interesting as in using shadow cues to reconstruct 3D geometry when multi-view captures are not available. The scope of the paper is quite limited, in terms of the problem setup and the practicality. The setup is demanding -- real directional light plus known lighting direction. This highly controlled lighting setup seems to only be feasible in a laboratory or a light stage setting. This may be the reason the paper only shows toy synthetic examples. Another 'contrived' setup is in the simple scene / background. Cast shadow depends on the object geometry, the lighting, and also the surface the shadow is cast on. The geometry of the surface and the geometry of the object can be entangled to cause ambiguity. This is more severe in real-world examples. The task can be unnecessarily (due to such setup being unlikely in real-world scenarios) challenging and ambiguous without any priors on the object geometry. The reconstructed geometry is not on par with multi-view settings, and degrades drastically as the number of lights decreases. My biggest concern about this paper is it makes the reconstruction task unnecessarily challenging and impractical.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept