A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis
Authors: Xingang Pan
Xudong XU
Chen Change Loy
Christian Theobalt
Bo Dai
Venue: NeurIPS 2021 Poster (2021)
Keywords: Generative adversarial networks
Deep implicit models
3D-aware image synthesis

Abstract: The advancement of generative radiance fields has pushed the boundary of 3D-aware image synthesis. Motivated by the observation that a 3D object should look realistic from multiple viewpoints, these methods introduce a multi-view constraint as regularization to learn valid 3D radiance fields from 2D images. Despite the progress, they often fall short of capturing accurate 3D shapes due to the shape-color ambiguity, limiting their applicability in downstream tasks. In this work, we address this ambiguity by proposing a novel shading-guided generative implicit model that is able to learn a starkly improved shape representation. Our key insight is that an accurate 3D shape should also yield a realistic rendering under different lighting conditions. This multi-lighting constraint is realized by modeling illumination explicitly and performing shading with various lighting conditions. Gradients are derived by feeding the synthesized images to a discriminator. To compensate for the additional computational burden of calculating surface normals, we further devise an efficient volume rendering strategy via surface tracking, reducing the training and inference time by 24% and 48%, respectively. Our experiments on multiple datasets show that the proposed approach achieves photorealistic 3D-aware image synthesis while capturing accurate underlying 3D shapes. We demonstrate improved performance of our approach on 3D shape reconstruction against existing methods, and show its applicability on image relighting. Our code is available at https://github.com/XingangPan/ShadeGAN.

A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis
Xingang Pan
Xudong Xu²
Chen Change Loy²
Christian Theobalt'
Bo Dai
Max Planck Institute for Informatics
2Th Chinese University of Hong Kong
Expan, theobaltt (anpllttttmpppmiie iinf--inf... mmp. de
xx018@ie cuhk. edu.. hk
S-Lab, Nanyang Technological University {ccloy, vo.dai)@ntu, edu.
Abstract
The advancement of generative radiance fields has pushed the boundary of 3D- aware image synthesis. Motivated by the observation that a object should look realistic from multiple viewpoints, these methods introduce multi-view constraint as regularization to learn valid 3D radiance fields from 2D images. Despite the progress, they often fall short of capturing accurate 3D shapes due the shape- color ambiguity, limiting their applicability downstream tasks. this work, we address this ambiguity by proposing novel shading thading-guided generative implicit model that is able to learn a starkly improved shape representation. Our key insight is that accurate 3D shape should also yield realistic rendering under different lighting conditions. This multi- lightil constraint realized by modeling illumina- tion explicitly and performing shading with various lighting conditions. Gradients are derived by feeding the synthesized images to a discriminator. To compensate for the additional computational burden of calculating surface normals, further devise an efficient volume rendering strategy via surface tracking, reducing the training and inference time by 24% and 48%, respectively. Our experiments on mul- tiple datasets show that the proposed approach achieves photorealistic 3D-aware image synthesis while capturing accurate underlying 3D shapes. We demonstrate improved performance of our approach on 3D shape reconstruction against existing methods, and show its applicability on image relighting. Our code will be released at
Introduction
Advanced deep generative models, egg., StyleGAN 1 2 and BigGAN 3. have achieved great successes in natural image synthesis. While producing impressive results, these 2D representation- based models cannot synthesize novel views of an instance in a 3D-consistent manner. They also fall short of representing an explicit 3D object shape. To overcome such limitations, researchers have proposed new deep generative models that represent 3D scenes as neural radiance fields 4 5]. Such 3D-aware generative models allow explicit control of viewpoint while preserving 3D consistency during image synthesis. Perhaps more fascinating merit is that they have shown the great potential of learning 3D shapes in an unsupervised manner from just collection of unconstrained 2D images. If we could train 3D-aware generative model that learns accurate 3D object shapes, it would broaden various downstream applications such as 3D shape reconstruction and image relighting.
tend to learn coarse 3D shapes that are inaccurate and noisy, shown in Fig (a). We found that such inaccuracy arises from inevitable ambiguity inherent in the training strategy adopted by these methods. In particular, a form of
Existing attempts for 3D-aware image synthesis 4 [
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

generalor
comera pose louttigsourre
z
D
radiance
3D "albedo" field
field
Fake
Take images
(a) pCGAN
(b) ShacEGA (OUns)
Figure I: Motivation. (a) Previous methods like pi-GAN 4 resort the "multi-view constraint", where the 3D representation is projected to different viewpoints as fake images to the discriminator. The extracted 3D meshes are often inaccurate due to the shape-color ambiguity. (b) The proposed approach ShadeGAN further adopts "multi-lighting constraint" ". which motivates the 3D represen- tation to look realistic under different lighting conditions. This constraint effectively addresses the ambiguity, giving rise more natural and precise 3D shapes.
regularization, which we refer to as "mudti-view constraint", ,, is used to enforce the 3D representation to look realistic from different viewpoints. The constraint commonly implemented by first projecting the generator's outputs (e.g., radiance fields [6] to randomly sampled viewpoints, and then feeding them discriminator as fake images for training. While such a constraint enables these models to synthesize images in 3D-aware manner, it suffers from the shupe-color ambiguity, ie., small variations of shape could lead similar RGB images that look equally plausible to the discriminator, as the color of many objects is locally smooth. Consequently, inaccurate shapes are concealed under this constraint.
In this work, we propose a novel shading-guider generative implicit model (ShadeGAN) to address the aforemennioned ambiguity. In particular, ShadeGAN learns more accurate 3D shapes by explicitly modeling shading, i.e., the interaction of illumination and shape. We believe that an accurate 3D shape should look realistic not only from different viewpoints, but also under different lighting conditions, i.e., satisfying the lighti-lighting constraint". This idea shares similar intuition with photometric stereo 77, which shows that accurate surface normal could be recovered from images taken under different lighting conditions. Note that the multi-lighting constraint is feasible as real-world images used for training are often taken under various lighting conditions. To fulfill this constraint, ShadeGAN takes relightable color field the intermediate representation, which approximates the albedo but does not necessarily satisfy viewpoint independence. The color field is shaded under a randomly sampled lighting condition during rendering. Since image appearance via such a shading process is strongly dependent on surface normals, inaccurate 3D shape representations will be much more clearly revealed than in earlier -haning-agnostic generative models. Hence, by satisfying the multi-lighting constraint, ShadeGAN is encouraged to infer more accurate 3D shapes shown in Fig (b).
The above shading process requires the calculation of the normal direction back pack-propagatton through the generator, and such calculation needs to be repeated dozens of times for pixel in volume rendering 5) introducing additional computational overhead. Existing efficient volume rendering techniques 88 10 mainly target static scenes, and could not be directly applied to generative models due to their dynamic nature. Therefore, to improve the rendering speed of ShadeGAN, we formulate efficient surface tracking network estimate the rendered object surface conditioned on the latent code. This enables us to save rendering computations by just querying points near the predicted surface, leading to 24% and 48% reduction of training and inference time without affecting the quality of rendered images.
Comprehensive experiments are conducted across multiple datasets to verify the effectiveness of ShadeGAN. The results show that our approach is capable of synthesizing photorealistic images while capturing more accurate underlying 3D shapes than previous generative methods. The learned

---

distribution of 3D shapes enables various downstream tasks like 3D shape reconstruction, where our approach significantly outperforms other baselines on the BFM dataset [3] Besides, modeling the shading process enables explicit control over lighting conditions, achieving image relighting effect. Our contributions can summarized as follows: 1) We address the shape-color ambiguity in existing 3D-aware image synthesis methods with a seading-guided generative model that satisfies the proposed multi-lighting constraint. In this way, ShadeGAN is able to learn more accurate 3D shapes for better image synthesis. 2) We devise an efficient rendering technique via surface tracking, which significantly saves training and inference time for volume rendering-basee generative models. 3) We show that ShadeGAN learns to disentangle shading and color that well approximates the albedo, achieving natural relighting effects in image synthesis.
2 Related Work
Neural volume rendering. Starting from the seminal work of neural radiance fields (NeRF) [ neural volume rendering has gained much popularity in representing 3D scenes and synthesizing novel views. By integrating coordinate -base neural networks with volume rendering, NeRF performs high-fidelity view synthesis in a 3D consistent manner. Several attempts have been proposed to extend further model illumination, and learn to disentangle reflectance with shading given well- well-aligned multi-view and multi-lighting images. Besides, many studies accelerate the rendering of static scenes from the perspective of spatial sparsity [8] [] architectural design [10] [T], or efficient rendering 12). However, it is not trivial to apply these illumination and acceleration techniques to volume rendering- generative models 4]. as they typically learn from unposed and unpaired images, and represent dynamic scenes that change with respect to the input latent codes.
or improve NeRF. For instance, [14 13 [6]
In this work, we take the first attempt to model illumination in volume rendering-basee bbsed generative models, which serves as a regularization for accurate 3D shape learning. We further devise an efficient rendering technique for our approach, which shares similar insight with (12), but does not rely on ground truth depth for training and it is not limited to small viewpoint range.
Generative 3D-aware image synthesis. Generative adversarial networks (GANs) 18] are capable of generating photorealistic images of high-resolution, but lack explicit control over camera viewpoint. In order to enable them to synthesis images in a 3D-aware manner, many recent approaches investigate how 3D representations could be incorporated into GANs [19
20
21
22
23
24
25
26
27
28
29 30 While some works directly learn from 3D data 19
30 in this work we focus on approaches that only have access unconstrained 2D images, which is more practical setting. Several attempts 223 24 25 adopt 3D voxel features with learned neural rendering. These methods produce realistic 3D-aware synthesis, but the 3D voxels are not interpretable, i.e., they cannot be transferred to 3D shapes. By leveraging differentiable renderer, [26] and (27) learn interpretable 3D voxels and meshes respectively, but (26) suffers from limited visual quality due to low voxel resolution while the learned 3D shapes of (27) exhibit noticeable distortions. The success of NeRF has motivated researchers to use radiance fields as the intermediate 3D representation in GANs 4 28]. While achieving impressive 3D-aware image synthesis with multi-view consistency, the extracted 3D shapes of these approaches are often imprecise and noisy. Our main goal this work to address the inaccurate shape by explicitly modeling illumination in the rendering process. This innovation helps achieve better 3D-awa image synthesis with broader applications.
20 21
22
Unsupervised 3D shape learning from 2D images. Our work is also related to unsupervised approaches that learn 3D object shapes from unconstrained, monocular view 2D images. While several approaches use external 3D shape templates or 2D key-points as weak supervisions to facilitate
learning [31 32, 33
35 36 37), in this work we consider the harder setting where only 2D images are available. To tackle this problem, most approaches adopt an
34
paradigm (3 39 40 Specifically, they design photo-geometric autoencoders to infer the 3D shape and viewpoint of each image with a reconstruction loss. While succeed in learning the 3D shapes for some object categories, these approaches typically rely on certain regularization to prevent trivial solutions, like the commonly used symmetry assumption on object shapes assumption tends to produce symmetric results that may overlook the asymmetric aspects of objects. Recently, GAN2Shape [41) shows that it is possible to recover 3D shapes for images generated by 2D GANs. This method, however, requires inefficient instance-specifie training, and recovers depth maps instead of full 3D representations.
33 40
32]. Such
3

---

Random lighting condition ~ Pr
I P
3D coordinate
(x, z)
X
real
P
go
D3
raal/fake
a xx,, d,
Latent
I
fake
Generator
Random camera pose
P
Discriminator
Figure 2: Method overview. Our generator Do models relightable color field conditioned on latent code ~ Pz- To synthesis an image, performs volume rendering under a random camera pose & ~ Pt. The rendering process also performs shading with randomly sampled lighting condition ~ Py- The discriminator learns to distinguish the synthesized images with real images from the training dataset, and the whole model trained with GAN loss. Although our model is trained from unconstrained 2D images, it allows explicit control over camera pose and lighting condition during inference.
The proposed 3D-aware generative model also serves as a powerful approach for unsupervised 3D shape learning. Compared with aforementioned autoencoder-baat based methods, our approach avoids the need to infer the viewpoint of each image, and does not rely on strong regulariza- tions. In experiments, we demonstrate superior performance over recent state-of sate-of-the-asd art approaches Unsup3d 39 and GAN2Shape [4]]
3 Methodology
We consider the problem of 3D-aware image synthesis by learning from collection of unconstrained and unlabeled 2D images. We argue that modeling shading, i.e., the interaction of illumination and shape, in a generative implicit model enables unsupervised learning more accurate 3D object shapes. In the following, we first provide some preliminaries on neural radiance fields (NeRF) [6]. and then introduce our shading-guie guidddd generative implicit model.
3.1 Preliminaries on Neural Radiance Fields
As deep implicit model, NeRF 6 uses an MLP network to represent a 3D scene as radiance field. The MLP fe (æz d) + (a,, c) takes 3D coordinate
R3 and viewing direction S as inputs, and outputs a volume density a E IR and color c E R To render an image under given camera pose, each pixel color C of the image obtained via volume rendering along its corresponding camera ray r(t) + td with near and far bounds t, and t as below:
C(r) ((r)=/(( (()) dddd, where T(I) exp( L( (r(s))dss.
(1)
In practice, this volume rendering is implemented with discretized form using stratified and hierarchical sampling. As this rendering process is differentiable, NeRF could be directly optimized via posed images of a static scene. After training, NeRF allows the rendering of images under new camera poses, achieving high-quality novel view synthesis.
3.2 Shading-Guided Generative Implicit Model
In this work, we are interested in developing a generative implicit model that explicitly models the shading process for 3D-aware image synthesis. To achieve this, we make two extensions to the MLP network in NeRF. First, similar most deep generative models, further conditioned on latent code z sampled from a prior distribution \(00, l)d Second, instead of directly outputting the color outputs a relightable pre-cosine color term R-3 which conceptually similar to albedo way that it could be shaded under given lighting condition. While albedo is viewpoint-independen in this work we do not strictly enforce such independence for order to account for dataset bias. Thus, generator ge (x, d, x) + (a, a) takes a coordinate viewing direction d, and latent

---

code = as inputs, and outputs volume density a and pre-cosine color a. Note that here o is independent of d. while the dependence of on is optional. To obtain the color C of a camera ray r(t) = o + ld with near and far bounds in and y,, we calculate the final pre-cosine color A via:
A(r, '"' T(t, (tt,)))(r(0) d, d)t,, where T(t, exp(- [ (()(s),))ds)) Jn
We also calculate the normal direction n with:
n x) = - fo(rrr)r///////) ( where ((r,))=- S."
where V (r(e)r(((t), is the derivative of volume density a with respect to its input coordinate, which naturally captures the local normal direction, and could be calculated via back-propagation. Then the final color C' is obtained via Lambertian shading as:
C(r, 2) A(r, 2): + komax(0, I ((rr)))),
(4)
where E 2 is the lighting direction, ka and kig are the ambient and diffuse coefficients. We provide more discussions on this shading formulation at the end of this subsection.
Camera and Lighting Sampling. Eq.(2 4 describe the process of rendering a pixel color given a camera ray r(t) and lighting condition f (1, kig, ka). Generating full image Ig Hx requires one to sample camera pose & and a lighting condition μ in addition to the latent code i.e., Ig = Gz E: 1t). In our setting, the camera pose & could be described by pitch and yaw angles, and is sampled from a prior Gaussian or uniform distribution Pt, as also done in previous works 55. Sampling the camera pose randomly during training would motivate the learned 3D scene to look realistic from different viewpoints. While this multi-view constraint is beneficial for learning valid 3D representation, it is often insufficient to infer the accurate 3D object shape. Thus, in our approach, we further introduce multi-lighting constraint by also randomly sampling lighting condition fu from a prior distribution Pn. In practice, Pa could be estimated from the dataset using existing approaches like [39]. We also show in our experiments that simple and manually tuned prior distribution could also produce reasonable results. As the shading process is sensitive to the normal direction due to the diffuse term k.max (0, I nr, 2)) in Eq.44,, this multi-lightinn constraint would regularize the model to learn more accurate 3D shapes that produce natural shading, as shown in Fig (b).
Training. Our generative model follows the paradigm of GANs 18, where the generator trained together with discriminator D with parameters @ in an adversarial manner. During training, the generator generates fake images I, = Gz & 14) by sampling the latent code z, camera pose & and lighting condition μ from their corresponding prior distributions ps, Ps. and Py- Let I denotes real images sampled from the data distribution pr. We train our model with a non-saturating GAN loss with R2 regularization 42]:
[[(DA(Go(2,E,1)) + Er~po (I(-D&())) + +IIVD
[(0, 6) = Eg-ps E pu
(5)
where fw) = - log(1 + exp((uu), and À controls the strength of regularization. More implementa- tion details are provided in the supplementary material.
Discussion. Note that in 2 4.. we perform shading after A and n are obtained via volume rendering. An alternative way is to perform shading at each local spatial point as c(r(t), d, = a(r(l), d, =ka + (0, 1. nr((1),, x))), where n(r(!), = (r()) 2)/ 2 (r()((()))() is the local normal. Then could perform volume ren- dering using =)) to get the final pixel color. In practice, we observe that this formulation obtains suboptimal results. An intuitive reason is that in this formulation, the normal direction is normalized at each local point, neglecting the magnitude of V (r()) 2), which tends to be larger near the object surfaces. We provide more analysis in experiments and the supplementary material.
The Lambertian shading we used is an approximation to the real illumination scenario. While serving as a good regularization for improving the learned 3D shape, it could possibly introduce an additional gap between the distribution of generated images and that of real images. To compensate
5

---

Predicted depth
Volume readered depth E
N. T(t, s)o(((())
r(t)
90
90
Z
Z-
Generator
5 Surface tracking network
8
C(. fx)
depth mimic loss
(a) Conventional volume rendering
(b) Proposed efficient volume rendering
Figure 3: (a) Conventional volume rendering samples dozens of points within predefined near and far bounds fn and t (b) We propose an efficient volume rendering technique via surface tracking. Before rendering, our surface tracking network Sy predicts an initial guess of the surface position conditioned on the latent code and camera pose &. Then we sample points near s, which requires fewer samples. Finally, we use the volume rendered depth d as the ground truth to train Sg. During training. Sy is able to predict depth s that well approximates the real surface depth d.
for such risk, we could optionally let the predicted a be conditioned on the lighting condition, i.e., a a(r(t), d. u, z). Thus. in cases where the lighting condition deviates from the real data distribution, the generator could learn to adjust the value of a and reduce the aforementioned gap. We show the benefit of this design in the experiments.
3.3 Efficient Volume Rendering via Surface Tracking
Similar to NeRF, we implement volume rendering with a discretized integral, which typically requires to sample dozens of points along camera ray, shown in Fig. (a). In approach, we also need to perform pack-propagation across the generator in Eq. 3 to get the normal direction for each point, which introduces additional computational cost. To achieve more efficient volume rendering,
natural idea is to exploit spatial sparsity. Usually, the weight T(t. 2 (r((t, 2) in volume rendering would concentrate on the object surface position during training. Thus, if we know the rough surface position before rendering, we could sample points near the surface to save computation. While for a static scene it is possible to store such spatial sparsity in sparse voxel grid g 91, this technique cannot be directly applied to our generative model, as the 3D scene keeps changing with respect to the input latent code.
To achieve more efficient volume rendering in our generative implicit model, we further propose a surface tracking network S that learns to mimic the surface position conditioned on the latent code. In particular, the volume rendering naturally allows the depth estimation of the object surface via:
( Jrn T(t, 2))(((t),, 2)) dt,
t, (T. 2)
where Ttt, 2) is defined the same way in Eq. Thus, given camera pose and latent code z, we could render the full depth map ta (z, E). As shown in Fig. (b), we mimic t, (z, {) with the surface tracking network Sg, which is a light- convolutional neural network that takes as inputs and outputs a depth map. The depth mimic loss is:
((w) = Ezups E-rps S + - t, 1) + (,,
where Prec is the perceptual loss that motivates S₂F to better capture edges of the surface.
During training, Sp is optimized jointly with the generator and the discriminator. Thus, each time after we sample a latent code z and a camera pose ę. we can get an initial guess of the depth map as S (2. Then for pixel with predicted depth s, we could perform volume rendering in Eq. 3 with near bound f, rendering that decreases as the training iteration i grows. Specifically, we start with a large interval Amaz and decrease to Amin with an exponential schedule. As A; decreases, the number of points used for rendering m also decreases accordingly. Note that the computational cost our efficient surface tracking network is marginal compared to the generator, the former only needs a single forward pass to render an image while the latter will queried for H x W m times. Thus, the reduction of would significantly accelerate the training and inference speed for ShadeGAN.
2 and far bound t,
+ / where A, is the interval for volume

---

Image
3D2 Mesh
Normal
Albedo
Diffuse shading
GRAFF
N/A
N/A
N/A
N/A
Ouass
GRAUP
N/A
N/A
IF-GAN
N/A
N/A
Oues
GRAFF
N/A
N/A
22 GAN
N/A
N/A
Outb
Figure 4: Qualitative comparison on BFM (top), CelebA (middle), and Cats (bottom) datasets. "Albedo" refers to the pre-cosine color that approximates albedo. Our approach synthesizes more accurate 3D shapes than pi-GAN and GRAF, and also learns to disentangle shading with albedo.
4 Experiments
In this section, we evaluate the proposed ShadeGAN on 3D-aware image synthesis. We also show that ShadeGAN learns much more accurate 3D shapes than previous methods, and in the meantime allows explicit control over lighting conditions. The datasets used include CelebA 433 BFM [3]
and Cats 44. all of which contain only unconstrained 2D RGB images.
Implementation. In terms of model architectures, we adopt SIREN-based MLP 45 as the generator and a convolutional neural network as the discriminator following 44. For the prior distribution of lighting conditions, we use Unsup3d [39] to estimate the lighting conditions of real data and subsequently fit multivariate Gaussian distribution of f (I. ki, ka) as the prior. A hand-crafted prior distribution is also included in the ablation study. In quantitative study, we let the pre-cosine color be conditioned on the lighting condition μl as well as the viewing direction d unless otherwise stated. qualitative study, we observe that removing view conditioning achieves slightly better 3D shapes for CelebA and BFM datasets. Thus, we show results without view conditioning for these two datasets in the main paper, and put those with view conditioning in Fig. of the supplementary material. Other implementation details are also provided in the supplementary.
7

---

izzaee
Mesh
Noraa
Albedo
[a) ShaleGAN
2
(b) Local normal
Manual prtor
Figure 5: Generated face images and their 3D meshes.
Figure 6: Qualitative ablation.
See the main text for discussions.
Table I: Comparisons on the BFM dataset. We report FID (128²) for image synthesis, and SIDE (x (x10-2) and MAD (deg.) for the accuracy
Table 2: Comparisons the CelebA and
Cats datasets. The image resolution is 128².
of 3D shapes. --" indicates not available. Results
Datasct
Method
FID
MAD
of pi-GAN and Ours averaged over 5 runs.
Method
FID
SIDE 1
MAD I
GRAF
43.0
30.48
CelebA
pi-GAN
15.7
27.22
Supervised
0,410
10.78
Ours
16.2
20.49
Unsup3d 39
0.793
16.51
GRAF
30.3
65.47
GAN2Shape
41
0.756
14.81
Cats
pi-GAN
10.7
33.48
GRAF [S]
53.4
1,857
26.60
Ours
10.3
25.47
pi-GAN
4
16.7+0.2
0.727+0.012
20.0940.25
Ours
17.7+0.2 0.607
14.52+0.11
Comparison with baselines. We compare ShadeGAN with two state-of-the-art generative implicit models, namely GRAF (S] and pi-GAN [4. Specifically, Fig. [innllues both synthesized images as well as their corresponding 3D meshes, which are obtained by performing marching cubes on the volume density a. While GRAF and pi-GAN could synthesize images with controllable poses, their learned 3D shapes are inaccurate and noisy. In contrast, our approach not only synthesizes photorealistic 3D-consistent images, but also learns much more accurate 3D shapes and surface normals, indicating the effectiveness of the proposed multi-lighting constraint as a regularization. More synthesized images and their corresponding shapes are included in Fig(5] Besides more accurate 3D shapes, ShadeGAN can also learn the albedo and diffuse shading components inherently. As shown in Fig. 4 although not perfect, ShadeGAN has managed to disentangle shading and albedo with satisfying quality, as such disentanglement a natural solution to the multi-lighting constraint.
The quality of learned 3D shapes is quantitatively evaluated on the BFM dataset. Specifically, we use each of the generative implicit models to generate 50k images and their corresponding depth maps. Image-depth pairs from each model are used as training data to train an additional convolutional neural network (CNN) that learns to predict the depth map of an input image. We then test each trained CNN on the BFM test set and compare its predictions to the ground-truth depth maps as a measurement of the quality of learned 3D shapes. Following [39,, we report the scale-invariant depth error (SIDE) and mean angle deviation (MAD) metrics. The results are included Tab. | where ShadeGAN significantly outperforms GRAF and pi-GAN. Besides, ShadeGAN also outperforms other advanced unsupervised 3D shape learning approaches including Unsup3d (39 and GAN2Shape [41). demonstrating its large potential in unsupervised 3D shapes learning. In terms of image quality, Tab. [ includes the FID [46] scores of images synthesized by different models, where the FID score of ShadeGAN is slightly inferior to pi-GAN BFM and CelebA. Intuitively, this is caused by the gap between approximated shading (i.e. Lambertian shading) and the real illumination, which can be potentially avoided by adopting more realistic shading models and improving the lighting prior.
In Tab. 2 we also show the quantitative results different models on CelebA and Cats. To evaluate the learned shape, we use each generative implicit model to generate 2k front-view images and their corresponding depth maps. While these datasets do not have ground truth depth, we report MAD obtained by testing pretrained Unsup3d models (39 these generated image-depth pairs as
8

---

Table 3: Ablation study on the BFM dataset.
Table 4: Training and inference time cost on
No.
Method
FID
SIDE 4
MAD 1
CelebA. The efficient volume rendering signifi-
cantly improves training and inference speed.
(I)
ShadeGAN
17.7
0.607
14.52
(2)
local shading
30.1
0.754
18.18
Method Train (h) Inference (s) FID
(3)
w/ light
19.2
0.618
14.53
ShadeGAN
92.3
0.343
16.4
w/o view
18.6
0.622
14.88
+efficient
70.2
0.179
16.2
(5)
manual prior
20.2
0.643
15.38
pi-GAN
56.8
0,204
15.7
(6)
+efficient
18.2
0.673
14.72
+efficient
46.9
0.114
15.9
Taage
Dedetel
dipói
Stotiog
(c) with specular Figure 8: Illminatioo--wwree image synthesis. ShadeGAN al- lows explicit control over the lighting. The pre-cosine color (albedo) is independent of lighting in (a) and is conditioned on lighting in (b). We show results of adding a specular term in (c).
w/o light condition
(b) with light condition
Figure 7: Visualization of depths predicted by our depth tracking network and those calculated via volume rendering.
reference. As observe, results CelebA and Cat are consistent with those on the BFM dataset.
Ablation studies. We further study the effects of several design choices in ShadeGAN. First, we perform local points-specific shading as mentioned the discussion of Sec. S.c.3.2 As Tab. No. and Fig. (b) show, the results of such local shading strategy are notably worse than the original one, which indicates that taking the magnitude of V into account is beneficial. Besides, the results of Tab. 3 (3.3 and (4. imply that removing a's dependence could lead to a slight performance drop. The results of using a simple manually tuned lighting prior provided in Tab. N5) and Fig. 6 (c), which are only moderately worse than the results of using a data-driven prior, and the generated shapes are still significantly better than the ones produced by existing approaches.
the lighting µ the viewpoint d
To verify the effectiveness the proposed efficient volume rendering technique, we include its effects on image quality and training/inference time in Tab. Babb3 No. and Tab. 4 It is observed that the efficient volume rendering has marginal effects on the performance, but significantly reduces the training and inference time by 24% and 48% for ShadeGAN. Moreover, in Fig. 7 we visualize the depth maps predicted by our surface tracking network and those obtained via volume rendering. is shown that under varying identities and camera poses, the surface tracking network could consistently predict depth values that are quite close to the real surface positions, so that we can sample points near the predicted surface for rendering without sacrificing image quality.
Illumination-wware image synthesis. As ShadeGAN models the shading process, by design allows explicit control over the lighting condition. We provide such illumination-aware image synthesis results in Fig g where ShadeGAN generates promising images under different lighting directions. We also show that in cases where the predicted is conditioned on the lighting condition μ, a would slightly change w.r.t. the lighting condition, e.g., it would be brighter in areas having a overly dim shading in order to make the final image more natural. Besides, could optionally add specular term m.maa h - n in Eq. (i.e., Blinn- Phong shading 47. where h is the bisector of the angle between the viewpoint and the lighting direction) to create specular highlight effects, as shown in Fig & (c).
GAN inversion. ShadeGAN could also be used to reconstruct a given target image by performing GAN inversion. As shown in Fig. such inversion allows us to obtain several factors of the image, including the 3D shape, surface normal, approximated albedo, and shading. Besides, we can further perform view synthesis and relighting by changing the viewpoint and lighting condition. The implementation of GAN inversion is provided in the supplementary material.

---

Real image
Reconstmation
3D Mesh
Normal
Albedo
Diffuse shading
View synthesis
Relighting
Figure 9: GAN inversion for real image editing.
Discussions. As the Lambertian shading we used is an approximation to the real illumination, the albedo learned by ShadeGAN is not perfectly disentangled. Our approach does not consider the spatially-varying material properties of objects as well. the future, we intend to incorporate more sophisticated shading models to learn better disentangled generative reflectance fields.
5 Conclusion
In this work, we present ShadeGAN, new generative implicit model for shape-accurate 3D-aware image synthesis. We have shown that the multi-lighting constraint, achieved in ShadeGAN by explicit illumination modeling, significantly helps learning accurate 3D shapes from 2D images. ShadeGAN also allows
control the lighting condition during image synthesis, achieving natural image relighting effects. To reduce the computational cost, we have further devised light- weighte surface tracking network, which enables efficient volume rendering technique for generative implicit models, achieving significant acceleration on both training and inference speed. generative model with shape- accurate 3D representation could broaden its applications in vision and graphics, and our work has taken a solid step towards this goal.
Aeknowledgment. We would like thank Eric Chan for sharing the codebase of pi-GAN. This study is supported under the ERC Consolidator Grant 4DRepLy (770784), This study is also supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
References
[1] T. Karras, S. Laine, and T. Aila, "A style-based generator architecture for generative adversarial networks," in CVPR, 2019.
[2] Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, "Analyzing and improving the image quality of stylegan," in CVPR, 2020.
[3] A. Brock, J. Donahue, and K. Simonyan, "Large scale gan training for high fidelity natural image synthesis," in ICLR, 2019.
[4] E. R. Chan, M. Monteiro, P. Kellnhofer, J. Wu, and G. Wetzstein, "pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis," in CVPR, 2021.
[5] K. Schwarz, Y. Liao, M. Niemeyer, and A. Geiger, "Graf: Generative radiance fields for 3d-aware image synthesis," in NewrIPS, 2020.
[6] B. Mildenhall, Representing scenes as neural radiance fields for view synthesis," in ECCV, 2020.
Srinivasan, M. Tancik, J. Barron, R. Ramamoorthi, and R. Ng, "Nerf:
[7] R. J. Woodham, "Photometric method for determining surface orientation from multiple images," Optical engineering, vol. 19, no. p. 191139 1980.
[8] I. Liu, J. Gu, K. 7aw Lin, T.-S. Chua, and C. Theobalt, "Neural sparse voxel fields," in NeurlPS, 2020.
[9] A. Yu, R. Li, M. Tancik, H. Li, R. Ng, and A. Kanazawa, "PlenOetrees for real-time rendering of neural radiance fields," in arXiv, 2021.
[10] C. Reiser, S. Peng, Y. Iiiao, and Geiger, "Kilonerf: Speeding up neural radiance fields with thousands of tiny mips," arXiv preprint arXiv:2 13744, 2021.
[11] S. J. Garbin, M. Kowalski, M. Johnson, J. Shotton, and J. Valentin, "Fastnerf: High-fidelity neural rendering 200fps," arXiv preprint arXiv:2103. 10380, 2021.
[12] T. Neff, P. Stadbbauer, M. Parger, A. Kurz, C. A. Chaitanya, A. Kaplanyan, and M. Steinberger, "Donerf: Towards real-time rendering of neural radiance fields using depth oracle networks." 2021.
arXiv preprint ar
10

---

[13] P. Paysan, R. Knothe, B. Amberg, S. Romdhani, and T. Vetter, "A 3d face model for pose and illumination invariant face recognition," in 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance, leee, 2009.
[14] M. Boss, R. Braun, V. Jampani, J. T. Barron, C. Liu, and H. Lensch, "Nerd: Neural reflectance decomposition from image collections," arXiv preprint arXiv:2012.03913, 2020.
[15] S. Bi, Xu, P. Srinivasan, B. Mildenhall, K. Sunkavalli, M. Hašan, Y. Hold-Geoffroy, D. Krieg- man, and R. Ramamoorthi, "Neural reflectance fields for appearance acquisition," " arXiv preprint Stfv:20008.08222 2020.
[16] Srinivasan, B. Deng, X. Zhang, M. Tancik, B. Mildenhall, and T. Barron, "Nerv: Neural reflectance and visibility fields for relighting and view synthesis," in CVPR, 2021.
[17] D. B. Lindell, J. N. Martel, and G. Wetzstein, "Autoint: Automatic integration for fast neural volume rendering," arXiv preprint arXiv:20/2.017
2020.
[18] I. Goodfellow, J. Pouget- Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial nets," in NIPS, 2014.
[19] Wu, C. Zhang, T. Xue, B. Freeman, and J. Tenenbaum, "Learning a probabilistic latent space of object shapes via 3d generative-adversarii modeling," in NIPS, 2016.
[20] H. A. Alhaija, S. K. Mustikovela, A. Geiger, and C. Rother, "Geometric image synthesis," in ACCV, Springer, 2018.
[21] X. Chen, D. Cohen-Or, B. Chen, and N. J. Mitra, "Towards a neural graphics pipeline for controllable image generation," Computer Graphics Forum, vol. 40, no. 2, 2021.
[22] J.-Y. Zhu, Z. Zhang. C. Zhang, J. Wu, A. Torralba, J. Tenenbaum, and B. Freeman, "Visual object networks: Image generation with disentangled 3d representations," in NeurIPS, 2018.
[23] T. Nguyen-Phane Phuyec, C. Li, L. Theis, C. Richardt, and Y.-L. Yang. "Hologan: Unsupervised learning of 3d representations from natural images," in ICCV, 2019.
[24] T. Nguyen-Phuoc, C. Richardt, L. Mai, Y.-L. Yang, and N. Mitra, "Blockgan: Learning 3d object-aware scene representations from unlabelled images," in NeurIPS, Nov 2020.
[25] Y. Liao, K. Schwarz, L. Mescheder, and A. Geiger, "Towards unsupervised learning of generative models for 3d controllable image synthesis," in CVPR, 2020.
[26] Henzler, N. J. Mitra, and T. Ritschel, "Escaping plato's 's cave: 3d shape from adversarial rendering." " in ICCV, 2019.
[27] A. Szabó, G. Meishvili, and P. Favaro, "Unsupervised generative 3d shape learning from natural images," arXiv preprint arXiv: 191.0091,, 2019.
[28] M. Niemeyer and A. Geiger, "Giraffe: Representing scenes as compositional generative neural feature fields," in CVPR, 2021.
[29] A. Tewari, M. Elgharib, G. Bharaj, F. Bernard, H.-P. Seidel, P. Pérez, M. Zollhofer, and
C. Theobalt, "Stylerig: Rigging stylegan for 3d control over portrait images," " in CVPR, 2020.
[30] Yenamandra, A. Tewari, Bernard, H.-P Seidel, M. Elgharib, D. Cremers, and C. Theobalt, "13dmm: Deep implicit 3d morphable model of human heads," in CVPR, 2021.
[31] S. Tulsiani, N. Kulkarni, and A. Gupta, "Implicit mesh reconstruction from unannotated image collections," arXiv preprint arXiv:2007.nm 008504 2020.
[32] S. Goel, A. Kanazawa, and J. Malik, "Shape and viewpoint without keypoints," " ECCV, 2020.
[33] L.. Tran and X. Liu, "Nonlinear 3d face morphable model," in CVPR, 2018.
[34] A. Kanazawa, S. Tulsiani, A. A. Efros, and J. Malik, "Learning cattegor--spccinne mesh recon- struction from image collections." in ECCV, 2018.
[35] A. Kanazawa, M. J. Black, D. W. Jacobs, and J. Malik, "End-to-end recovery of human shape and pose," in CVPR, 2018.
[36] S. Sanyal, T. Bolkart, H. Feng, and M. J. Black, "Learning to regress 3d face shape and expression from an image without 3d supervision," " in CVPR, 2019.
[37] Shang, T. Shen, Li, Zhou, M. Zhen, T. Fang, and I.. Quan, "Self-supervised monocular 3d face reconstruction by occlusion-aware multi-view geometry consistency," ECCV, 2020.
11

---

[38] M. Sahasrabudhe, Z. Shu, E. Bartrum, R. Alp Guler, D. Samaras, and I. Kokkinos, "Lifting autoencoders: Unsupervised learning of dislttiiiedt 3d morphable model using deep non-rigid structure from motion," in ICCV Workshops, 2019.
[39] S. Wu, C. Rupprecht, and A. Vedaldi, "Unsupervised learning of probably symmetric deformable 3d objects from images in the wild," in CVPR, 2020.
[40] X. Li, S. Liu, K. Kim, S. De Mello, V. Jampani, M.-H. Yang, and J. Kautz, "Self-supervised single-view 3d reconstruction via semantic consistency," ECCV, 2020.
[41] X. Pan, Dai, Liu, C. C. Loy, and P. Luo, "Do 2d gans know 3d shape? unsupervised 3d shape reconstruction from 2d image gans," in ICLR, 2021.
[42] L. Mescheder, A. Geiger, and S. Nowozin, "Which training methods for gans do actually converge?," in ICML, 2018.
[43] Z. Liu, P. Luo, X. Wang, and X. Tang, "Deep learning face attributes in the wild," in ICCV, 2015.
[44] W. Zhang, J. Sun, and Tang, "Cat head detection-how to effectively exploit shape and texture features," in ECCV, 2008.
[45] V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein, "Implicit neural representa- tions with periodic activation functions," in NeurlPS, 2020.
[46] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, "Gans trained by two time-scale update rule converge to a local nash equilibrium," NeurlPS, 2017.
[47] B. Phong, "Illumination for computer generated pictures," Communicutions of the ACM, vol. 18, no. 6, pp. 31 311-317, 1975.
[48] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville, "Film: Visual reasoning with a general conditioning layer," in AAAI, 2018.
[49] R. Liu, J. Lehman, P. Molino, F. P. Such, E. Frank, A. Sergeev, and J. Yosinski, "An intriguing failing of convolutional neural networks and the coordcony solution," in NeurIPS, 2018.
[50] K. He, X. Zhang, Ren, and Sun, "Deep residual learning for image recognition," in CVPR, 2016.
[51] Y. Wu and K. He, "Group normalization." " in ECCV, 2018.
[52] S. loffe and C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift," in ICML, 2015.
[53] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv preprint arXiv: 69800 2014.
[54]
[55] "Celeba."
[56] "Cats." https://ee.arcciive.c
203/
[57] Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha, "Stargan v2: Diverse image synthesis for multiple domains," in CVPR, 2020.
12


Reviews:
[{"review_id": "z7MQHa-UZ3", "reviewer": "Reviewer_YNf7", "summary": "The paper proposes a novel generative model for neural radiance field, where the key idea is to synthetically add shading effects based on random lighting configurations. Shading will make images unrealistic when the geometry is corrupted. Therefore, the system gets trained to generate accurate shapes in its implicit radiance field representation. The system makes convincing results both qualitatively and quantitatively.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of the paper is very simple and intuitive (adding lighting effects and utilizing discriminator to discover accurate geometry). I am not familiar with the complete literature but if no papers have shown this idea before, this idea could be an interesting one for the computer vision community but not for the machine learning community. I am curious if other reviewers spot any existing works with similar ideas. Given extensive GAN literature, there may exist similar ideas somewhere.\n\nEmpirical evaluations (both qualitative and quantiative) look convincing, where experimental results also support the claims qualitively and quantitatively. However, they are shown only for human faces.\n\nI am stuck on the overall recommendation but am leaning towards negative. The first reason is the merit/impact to the machine learning community. The strength of this paper is the idea which is for the computer vision community. The technical contribution is weak, as there is not much to learn from the paper. The second reason is that results are only presented for faces. This is very critical. We already have very good approaches that turn single image into 3D face models. This paper will not bring any impacts. The idea of this paper will really shine if this idea works on more variety of object categories. With those reasons, I will vote for rejection.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I did not find texts addressing the limitations and societal impact. The check-seat has a comment saying that \"see the supplementary document\" on this point, but I do not see relevanat texts.  Deep fakes are relevant societal impac.", "review_text": "", "extra_scores": {"ethical_concerns": "As described above, I wish to see some texts explaining potential malicious use of the presented technique (e.g., deep fakes).", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "aQuyDJ81KRJ", "reviewer": "Reviewer_8VoF", "summary": "This paper proposes a new method to train generative radiance fields that can achieve more accurate 3D shape reconstruction. The key idea is that instead of directly synthesizing the images from different viewpoints through volume ray tracing, which may suffer from color-shape ambiguity, it synthesizes albedo and normal and then renders the image with different lighting conditions. This new method requires the synthesized albedo and normal to be able to render realistic images under various lighting conditions, providing an extra regularization for shape reconstruction. To further improve the rendering speed, it uses an extra 2D CNN to predict a depth map from latent code and camera poses so that it only needs to sample near the surface in volume ray tracing. Experiments show that the proposed method greatly improves shape reconstruction with similar image synthesis quality compared to state-of-the-arts.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I will list the strengths of the paper and some minor suggestions that I believe may help improve the paper. My current evaluation of the paper is very positive. Authors may prioritize answering other reviewers' questions in the rebuttal. \n\nStrengths:\n\n1. The major contribution of the paper is clear. It proposes the multi-lighting constraint to help regularize the 3D shape reconstruction when training generative radiance fields, which is intuitively reasonable practically effective. In my opinion, this is an interesting idea and may inspire future research in the related fields.\n\n2.  The paper is clearly written, well-organized, and easy to follow. The citations is complete and appropriate. \n\n3. The experiments are comprehensive and convincing. It clearly supports the major contribution of the paper that the depth prediction quality can be improved by the multi-lighting constraint. Moreover, ablation studies also demonstrate that all minor design improvements such as depth prediction to reduce sampling number and lighting parameters as inputs can further improve the image synthesis quality. \n\n4. Authors clearly summarize the limitations of the current method, which may help researchers to build new frameworks based on current results. \n\nMinor suggestions:\n\n1. When first reading the paper, I was confused by how GAN inversion is done in practice. I later found the details in the supplementary but authors can consider adding a reminder pointing to the supplementary in the main paper. \n\n2. When discussing the intuition of adding the multi-lighting constraint, authors can consider connecting it to photometric stereo, which can achieve accurate normal reconstruction from images taken under different lighting conditions. This may make the argument more convincing and easy to understand. \n\n3. One way to improve the rendering quality is that instead of using sampled lighting conditions, authors can use captured real HDR environment maps. This may minimize the domain gap and render more realistic results. It may be even more useful if authors hope to reconstruct specular highlights in the future. \n\n4. One minor thing to check is if both the lighting and albedo are in the sRGB space or linear RGB space. If they are not in the color space, authors may consider using gamma correction to align them. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors mention in the discussion paragraph that one limitation is that the current method does not handle spatially varying material. I cannot see any other obvious limitations that need to be mentioned in the paper. I cannot see any potential negative societal impact either. ", "review_text": "", "extra_scores": {"ethical_concerns": "I cannot see any ethical concerns. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "HfDwKD0oL2H", "reviewer": "Reviewer_TVTi", "summary": "\nThis paper proposes a GAN capable of generating a relightable radiance field when trained on an unlabeled dataset of face images. The relightable radiance field generator is conditioned on the 3D location and a latent code and outputs a volume density field (à la NeRF) and an \"albedo\" field (I strongly advise against the use of this word; see below) that allows explicit control of the light location. The training is conducted in the GAN style, where the training signal comes from only a discriminator loss. The intuition is that the generator is better off generating a meaningful 3D volume that, when rendered from a random camera view, generates a photorealistic face image that falls into the training data distribution, than generating some intricately-designed volume that gives faces of different identities when viewed from different angles.\n\nThe authors demonstrate that the model is able to learn 3D shapes in an unsupervised fashion from just 2D images. The work is mainly compared against pi-GAN, which is shown to suffer from what the authors call \"color-shape ambiguities.\" For efficient rendering, the authors also propose an auxiliary network that predicts the surface location given the latent code and a viewing direction, such that the network can sample around the predicted surface to avoid expensive sampling of unoccupied space.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper explores a cute idea: by additionally modeling lighting, one learns better 3D shapes in an unsupervised way from unstructured 2D images. This gain seems free by exploiting the lighting variation that already exists in the datasets.\n\nThe model design is sensible and simple (in a positive sense). The observation that bad 3D shapes do not affect view synthesis in approaches that do not model lighting like pi-GAN is interesting. The shape results look convincing. The insight that the \"local shading\" alternative underperforms the adopted shading scene is appreciated. The paper is well-written and easy to follow.\n\nNow the drawbacks:\n\nI strongly advise against calling the generated radiance field the \"albedo field.\" Although the authors clarify in the text that this \"albedo\" depends on the viewing direction and lighting and hence is not the conventional albedo, this makes the paper really hard to read since any vision and graphics person is familiar with the concept that albedo does not depend on viewing directions or lighting. I had to keep reminding myself of the fact that this \"albedo\" is special. If the properties contradict the two fundamental properties of real albedo, why use it? Maybe something like \"pre-cosine radiance\" or \"unmodulated/demodulated radiance\"?\n\nRelated to this albedo complaint, the authors applied Lambertian shading to the \"albedo\" to allow explicit control of lighting. A simpler alternative is directly conditioning the albedo field generation on the light direction also, eliminating the need for a post-hoc Lambertian shading process. While I can see this alternative may not provide direct control over lighting, it warrants an ablation study. If this simpler alternative ends up working better, the authors can consider naming the field something like \"light transport field\"? See https://arxiv.org/pdf/2008.03806.pdf and https://arxiv.org/pdf/1911.11530.pdf. The current post-hoc Lambertian shading is a bit awkward, without a solid reason for its existence.\n\nIn terms of result quality, I'm concerned on the diversity of what the GAN is able to generate. The results shown seem to be suffering from mode collapse. This concern can be easily addressed by showing a video where we vary the latent code and in the meanwhile change the viewpoint and light direction. Then one would be able to judge if the generated results are diverse or not. Not asking for new experiments, but would love to see this in the revision or a future version of this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "FmkoePGg4Fg", "reviewer": "Reviewer_mEVj", "summary": "This paper aims to improve the density distribution of generative radiance field models by generating relitable reflectance fields and using different lightings to shade them during training. The core idea is that artifacts will arise when the generated density distribution (\"shape\") is unnatural and is getting shaded by different lightings, which should be detected and resolved by a discriminator. Experiments verify that this shading regularization can yield naturally looking shapes induced by the generated radiance fields.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "---------------UPDATED AFTER BURETTAL\n\nI feel the reponses have addressed my concerns. I recommend acceptance. My suggestion is to add the explanation for view-dependence (to address dataset biases, in addition to non-Lambertian shading) to the main paper.\n\n\n---------------ORIGINAL REVIEW\n\nStrengths:\n+ Simple and interesting idea to improve generative radiance fields.\n+ Novel in the context of generative radiance fields.\n+ Clearly written.\n+ Straightforward implementation with good qualitative results that achieve the goal.\n\nWeakness:\n- My major concern is on the unconstrained nature of the \"albedo\" fields. It is dependent on direction and lighting, making it unconstrained, and possibly unstable, to regularize the density distribution. The \"color-shape\" ambiguity is still not addressed. In the proposed formulation, the ambiguity can still be baked into the dependency on direction and lighting, i.e., instead of optimizing a good shape, the model might choose to optimize a sophisticated but working A(r,z,d,mu) function. For example, if a generated face shape has no nose, to have a plausible appearance that is multi-view consistent, the model can either (1) generate a nose in the shape and generate a constant albedo in A(z), or (2) keep the \"no nose face\" and hack the A(r,z,d,mu) function by giving high albedo to left nose when light source is on the left, and low albedo when light source is on the right. This is not seen in the shown images but this might happen to some extent, or might happen to some random seed. Showing results averaged over multiple runs could alleviate this concern and I might increase my rating.\n\n- I understand that the flexible A() function is used also to compensate the simplified lighting and reflectance model. But as for the dependence of view angle, why not use a simple specular part in the BRDF? Just like the test in Figure 8c.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed the mentioned issues to some extent, but not fully addressed them. Please see above comments.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3h"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission initially received mixed reviews. After the rebuttal, all reviewers became positive (though reviewer YNf7 didn't update the score).  The AC agrees with the reviewers that the proposed idea is neat and well-executed. The authors are strongly encouraged to address the remaining concerns in the camera-ready version. This includes adding the results in response to reviewers TVTi, mEVj, and YNf7, addressing the concern on view dependence, among others.
Decision: Accept (Poster)