Generative Occupancy Fields for 3D Surface-Aware Image Synthesis
Authors: Xudong XU
Xingang Pan
Dahua Lin
Bo Dai
Venue: NeurIPS 2021 Poster (2021)
Keywords: Generative model
Generative Radiance Fields
3D-aware image synthesis

Abstract: The advent of generative radiance fields has significantly promoted the development of 3D-aware image synthesis. The cumulative rendering process in radiance fields makes training these generative models much easier since gradients are distributed over the entire volume, but leads to diffused object surfaces. In the meantime, compared to radiance fields occupancy representations could inherently ensure deterministic surfaces. However, if we directly apply occupancy representations to generative models, during training they will only receive sparse gradients located on object surfaces and eventually suffer from the convergence problem. In this paper, we propose Generative Occupancy Fields (GOF), a novel model based on generative radiance fields that can learn compact object surfaces without impeding its training convergence. The key insight of GOF is a dedicated transition from the cumulative rendering in radiance fields to rendering with only the surface points as the learned surface gets more and more accurate. In this way, GOF combines the merits of two representations in a unified framework. In practice, the training-time transition of start from radiance fields and march to occupancy representations is achieved in GOF by gradually shrinking the sampling region in its rendering process from the entire volume to a minimal neighboring region around the surface. Through comprehensive experiments on multiple datasets, we demonstrate that GOF can synthesize high-quality images with 3D consistency and simultaneously learn compact and smooth object surfaces. Our code is available at https://github.com/SheldonTsui/GOF_NeurIPS2021.


Generative Occupancy Fields for 3D Surface-Aware Image Synthesis
Xudong Xut
Xingang Pan Dahua Lin't
Bo Dai
CUHK SenseTime Joint Lab, The Chinese University of Hong Kong Max Planck Institute for Informatics 8$ - Lab, Nanyang Technological University
[xx0188 dhlin]@ie. cuhk. edu. hk "xpan@mpi- -nnf mpg. bbo.dai doo.dai@tuu. eeu.t.uuuu..... sg
Abstract
The advent of generative radiance fields has significantly promoted the development of 3D-aware image synthesis. The cumulative rendering process in radiance fields makes training these generative models much casier since gradients are distributed over the entire volume, but leads diffused object surfaces. In the meantime, compared to radiance fields occupancy representations could inherently ensure deterministic surfaces. However, if we directly apply occupancy representations generative models, during training they will only receive sparse gradients located object surfaces and eventually suffer from the convergence problem. In this paper, we propose Generative Occupancy Fields (GOF), a novel model based on generative radiance fields that can learn compact object surfaces without impeding its training convergence. The key insight of GOF is a dedicated transition from the cumulative rendering in radiance fields to rendering with only the surface points as the learned surface gets more and more accurate. In this way, GOF combines the merits of two representations in a unified framework. In practice, the training-time transition of start from radiance fields and march to occupancy representations is achieved in GOF by gradually shrinking the sampling region in its rendering process from the entire volume to minimal neighboring region around the surface. Through comprehensive experiments on multiple datasets, we demonstrate that GOF can synthesize high-quality images with 3D consistency and simultaneously learn compact and smooth object surfaces. Our code available at htops:///itthue com wwt.cco///Seeeeeeeee wwt.m/hheloonssuiiim Neur IPS2021
1 Introduction
Deep generative adversarial networks 4 have demonstrated their superiority synthesizing photorealistic and striking images. However, these models are often constrained in the 2DD domain, struggling to generate 3D) consistent images, let alone grasping the underlying 3D) object shapes. 3D-aware image synthesis thus becomes appealing and promising choice it learns 3DD representation explicitly from a collection of unposed images. Consequently, it can not only synthesize 3D consistent images by manually controlling the rendering camera poses, but also pave the way for various downstream tasks such as shape editing and relighting.
Inspired by the success of neural radiance fields (NeRF) [5] in 3D scene modeling, recent 3D-aware generative models, referred to as generative radiance fields (GRAFs), have applied NeRF as the explicit 3D representation for image synthesis (6 7). With the help of NeRF, they capable of hallucinating photorealistic images in a 3D consistent manner. Moreover, since NeRF holds the superior ability for rendering translucent objects by compositing colored densities along each ray in its volume rendering process, it also significantly facilitates the training of GRAFs as gradients naturally distributed over the entire volume. However, they still incur an inevitable incapacity of
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

0.8
Image
Surface Rendering
Normal
Shape
GOF pi-GAN
Pi-GAN
0.6
GOF(ours)
0'0
-1.0
-05
0.0
0.5
1.0
(a) Cumulative Rendering Weights
(b) Performance Comparison
Figure 1: (a) The cumulative rendering weights (color weights) of our approach GOF more focus on the surface (y-axis) than previous methods like pi-GAN G which indicates our predicted volume densities more concentrate on the object surfaces. (b) Owing to the diffused volume densities, the preceding method pi-GAN captures messy surface normals and object shapes. Moreover, the image rendered only with the surface points is quite noisy. In contrast, more surface-centralized densities predicted by our method ensure compact and smooth object surfaces thus enable high-quality surface rendering during inference. (Zoom in for best view)
capturing an accurate and compact object surface. As shown in Fig. Ta), the state-of-ther the--rt GRAF model pi-GAN is prone to predict diffused object surfaces, as the volume densities are smoothly spread around the surfaces. Such diffused surfaces could significantly hamper the applications of GRAF models in downstream tasks such as shape recovery. Moreover, under different light conditions, the artifacts of surfaces could be amplified and inherited through the rendering process, resulting in synthesized images that are messy and faulty.
In this work, we propose Generative Occupancy Fields (GOF), novel GRAF-like image synthesis model that can learn compact object surfaces. GOF is inspired by the design of occupancy networks S that implicitly represents a 3D surface with the continuous decision boundary of a neural classifier. In this way, occupancy networks are capable of effectively locating surfaces via root- root-finding and encouraging the compactness of modeled surfaces inherently. However, GOF avoids directly applying such design to 3D-aware image synthesis. While occupancy networks require precise object 10.. a more crucial factor is that they rely on the surface points for differentiable rendering [II 12 9 1 A generative model equipped with occupancy representations will thus meet severe convergence problems during training due to the sparsity of gradients. To unify the merits of both NeRF and occupancy networks for 3D-aware image synthesis, GOF adopts the design of GRAFs and at the same time leverages a nontrivial transition from the cumulative rendering to rendering with only the surface points, i.e. start from radiance fields and march to occupancy representations. Specifically, GOF will reinterpret the alpha values in the cumulative rendering process as occupancy values, so that it can locate the learned surface via root-finding. Subsequently it can naturally encourage the compactness of learned surfaces by gradually shrinking the sampling region in the rendering process from the entire volume to a minimal neighboring region around the surface.
masks to train 9
Thanks to the unified integration of radiance fields and occupancy representations, GOF can benefit from the representation effectiveness of radiance fields while ensuring the compactness of learned object surfaces through the shrinking process. As presented in Fig. D a), in GOF the distribution of cumulative rendering weights concentrates more closely around object surfaces compared to that of pi-GAN, eventually resulting in a compact and smooth surface. Moreover, GOF can thus alternatively render an image only with points on the learned surfaces like occupancy networks as illustrated in Fig. [(b), (b). And during inference such a rendering scheme has the potential alleviate the burden of sampling a large number of points along each ray for synthesizing single image. Through exhaustive experiments on synthetic and real-world datasets, we demonstrate that GOF can achieve suue--o-the-as performance on 3D-awar image synthesis. Meanwhile, it is capable of capturing compact and accurate 3D shapes that empower its applications in various downstream tasks such as 3D shape reconstruction. We validate this point by quantitative results of 3D shape reconstruction on the Synface dataset. Finally, we have also verified the ability of GOF in rendering high-quality images with only the surface points, which is hardly achievable in previous approaches.

---

2 Related Work
has exploited neural implicit functions for 3D geometry modeling. Among these works, neural radiance fields (NeRF) [5] has attracted growing attention due to its compelling results on novel view synthesis. leverages MLP network to approximate the radiance fields of static 3D scenes. And by learning to reconstruct existing views, it is capable of capturing 3D geometric details from only 2D supervision. A series of succeeding variants of NeRF have been proposed to improve it, including utilizing the spatial sparsity to reduce its computational complexity [22 23], refining the rendering process to improve its efficiency [24 25], well as adopting reflectance decompositions to enhance its modeling capacity 226] 27. There are also works that capitalize on the differentiable rendering of neural implicit functions for 3D reconstruction [T] 12
Neural implicit function for 3D representations. A plethora of works 14 15
28 29]. Specifically, SDFDiff [12] relies on the interpolation of eight neighboring SDF samples around the surface intersection to obtain the derivatives, while Atzmon et al (28 use a sample network relate samples' positions to network parameters and thus achieve an improved generalization ability. More interestingly, by
B
adopting occupancy representations. DVR [1]] and IDR [ show volumetric rendering is inherently differentiable so that network parameters can be optimized directly with derived analytic gradients. Different from methods aforementioned above, GOF is a generative model for 3D-aware image synthesis that can learn 3D representations from a set of 2D images with unknown camera poses.
Generative 3D-aware image synthesis. In order to synthesize 3D consistent images, researchers have explored lot on how to incorporate 3D representations into the classical GAN model M. Some resort learning from 3D data directly, yet the requirement of 3D supervision limits their practical applicability. A more appealing alternative thus learn from unposed 2D images in an unsupervised manner. Preceding works along this line of research adopt voxels their intermediate 3D representations [31 and achieve explicit control over the pose of synthesized images. Inspired by the superior representation capacity of radiance fields over voxels, recent attempts have replaced voxels with neural radiance fields [5] to improve the fidelity of synthesized 3D consistent images. Despite the striking performance, these models, referred to as generative radiance fields (GRAFs), tend to predict diffused object surfaces, which impeees its applicability in various downstream tasks. In this work, GOF aims at resolving this problem of GRAFs by combining them with the perspective of occupancy networks [8] and recent successes of recovering smooth and
methods 30 32
7 36
accurate shapes from natural images 37:40 40 Recently, three concurrent works, UNISURF [TO],
NeuS [41 and VoISDF 422, also combine implicit surfaces and radiance fields in unified framework, sharing similar spirits with our proposed GOF but different in tasks and focuses. Specifically, They focus on multi-view 3D reconstruction and attempt to alleviate the requirement of training-time precise masks through the integration of radiance fields and occupancy representations. Nevertheless, they still require images with ground-truth poses for training. By contrast, GOF targets on the challenging task of 3D-aware image synthesis, where the synthesized images should be not only natural and vivid, but also consistent in the 3D space. By integrating radiance fields and occupancy representations, GOF is able to facilitate the convergence of GRAFs and ensure the compactness of learned object surfaces. Compared to existing GRAFs, the applicability of GOF is thus significantly broadened.
3 Methodology
We propose generative occupancy fields (GOF), a novel synthesis model, belonging to generative radiance fields and aiming to learn from unposed images. Conditioned on a latent code z Pa. our generator Je can generate a 3D radiance field R. from which we can render realistic image with a sampled camera pose ç ~ Pt and simultaneously recover smooth and compact object surfaces. In the following, we first present the background of neural radiance fields, and then introduce our proposed GOF model in detail.
3.1 Neural Radiance Fields
We adopt neural radiance fields (NeRF) as our explicit 3D representation for image synthesis, owing to its strong performance in novel view synthesis on complex scenes. NeRF represents static scene as per-point volume densities and view-dependent RGB colors. Given 3D point x E R33 in space and a view direction d 3 NeRF capitalizes multi-layer perceptron (MLP) to predict the
3

---

tf
agg ax(x,27 )
Dint
Generator
2
Co cx d, 2)
baf
surface
x: 3D point
2 ~ Py: latent code
o: camera origin
d
tm
d: camera direction
Camera pose { ~ Pe
Training
Inference
Figure 2: Shrinking process. During the training, the sampling interval is initially half of the distance between near f, and far bounds ł and shrinks gradually to pre-defined value Amin- For inference, we can use cumulative rendering by sampling points the minimal interval Amin and alternatively render only with the surface points.
volume density (x) E R and the emitted color c(x. d) RR3 To render novel view for the scene, NeRF leverages the classic volume rendering technique 45 to estimate the color of each pixel. starts by accumulating the colored densities of N points {x; far bounds [4n, by] along the camera ray r(() o + Id, where o stands for the camera origin. The integrated color then estimated via alpha composition follows:
t,d] sampled within near and
N
i-1 c(xi, d), where T. = =exp(- Σ (x) j=1
C(r) = Σ 11 exp-exp((2 ((x)))))) ( 1=1
(I)
where 8; = | 2z is the distance between adjacent points. Note that, equation is naturally differentiable and NeRF can be directly optimized through the reconstruction error of existing views.
3.2 3D Surface-Aware Image Synthesis via Generative Occupancy Fields
To apply NeRF the 3D representation, the proposed generative occupancy fields (GOF) incorporates an additional latent code z ~ ps into NeRF, such that synthesizing an image follows a reformulated cumulative rendering process:
N C(r,2) = Σ T (1-cpp (-oo(x, z)):) co (x; d, d),, where T; exp( Σ oo(x),2)0,) (2)
1=1
However, directly training GOF according to Eq. 2 fails to maintain the surface compactness as reported in previous approaches 6 77 Actually, such defect arises from inevitable "shape-color ambiguity" the cumulative rendering process, i.e., small perturbations on surfaces still lead to realistic RGB images which are enough to fool the discriminator.
Owing to the constrained range of poses seen training, the discriminator is less motivated to further 2)):) aforementioned in Fig. (a) on the
concentrate the color weights = (1 exact object surface. On the other hand, we observe that although leading to diffused surfaces at the end, color weights w; gradually concentrate around the object surface the training proceeds. Inspired by this observation, GOF we propose a training-ting time operation to facilitate the concentra- tion of color weights w;. The basic idea is gradually shrinking the sample region in the cumulative rendering process from the entire volume to narrow interval around the surface, so that color weights enforced continuously move towards the exact surface.
To enable the proposed training-time shrinking process, GOF is required to locate the surface by thresholding the predicted densities ap(x, 2), assuming points the surface have the largest densities, However, values of the densities predicted generative radiance fields could range from 50, making it hard to determine effective threshold during the whole training period. On the other

---

hand, in the cumulative rendering process shown in Eq. 2q. we found that the intermediate alpha values used for numerical stability inherently fall in fixed value range as:
ag (x:: 2) - 1 exp( (x; 2)))) E [0, 1].
(3)
More importantly these alpha values crg z) come close to 1 for points in the occupied space while approaching 0 for points in the free space, making them resemble the occupancy values [8] in both quantity and semantics. Inspired by the similarity, we thus propose to reformulate generative radiance fields by predicting alpha values 0gg (x, z) directly instead of volume densities 0x z). In the mean time, we reinterpret the alpha values as occupancy values, and subsequently locate surfaces with root-finding, more effective strategy originated in occupancy networks [IT] According to the above reformulation and reinterpretation, we thus dub our method as Generative Occupancy Fields.
As GOF estimates alpha values instead of volume densities, the original volume rendering process in Eq. 2 conditioned on the latent code z is reformulated as
N Σ
C(r,z)
ao (x z)
II (1 (x, cx) d,2), jei
where the value range of or (x;, z) is guaranteed with a sigmoid function. And to locate the surface via root-finding for specific ray r(t) + we will evenly sample M points {x, = ++ttd: k 1.... } that partition the entire volume [t.. ty] into M equally-saaced bins. After obtaining the corresponding alpha values {xg(Xe, z); k 1, ..., M) } by querying the generator ge. the surface S is
located in the k -th bin where ogg changes for the first time from free space (ov < 7) to occupied
space (as- < <)):
h = argmin (ag(xs. z) < (x(xx+1,,))
(5)
k
where is a pre-defined threshold. In practice, empirically set as 0.5. In order to find the surface point x, o + f.d more precisely, we further apply the above secant method iteratively for m, times, resulting in fine-grained bin [x,=, x1=+1l- It's worth noting that the M sampled points are only used for root-finding. Thus they do not require the computation of gradients in the implementation.
Based on the located surface S, we can thus successfully conduct the proposed shrinking process, which is schematically elaborated in Fig. 2 Specifically, when sampling N points for Eq.44) at each training step, we will only sample within a region neighboring the surface [t. A, t. +
2i -
2i N
} where = =1,2, N.
1 ~ 21
1/ A
A. 4,
A +
(6)
N
A is the sampling interval, which is set to Amitt = (t, - tx)/2 at the beginning, a half of the distance between near en and far bounds t,. And it will decrease monotonically with an exponential decay rate y until it drops to a pre-defined minimal value Amin- Formally, An max (Aiit exp(-7n), Amin) for n-th decaying step. Additionally, during training when the estimated t, is too close to the near or the far bound so that the sampling region [t. A.t, + A] exceeds the original range it,, ty), we will shift the region [t, - A, + A back to within [tn,, ty). As shown in Fig. 2 at the beginning of training, points sampled for Eq.(4) will cover the entire volume, leading to dispersed gradients which facilitate the convergence of GOF. And as the training goes, the predicted surface will become more and more accurate, which the outcome of gradually refining the sampling region, and in turn also makes the above shrinking operation valid.
Thanks to the dedicated shrinking process, the color weights w, can successfully concentrate on the object surface as illustrated in Fig. a As a result, GOF is capable of synthesizing high- fidelity images in a 3D-consistent manner and simultaneously capturing compact object surfaces. During inference, to synthesize an image under a random camera pose & ~ Pç. the generator go will fetch a truncated latent code z and sample N points {x,) on each ray within the minimal region [t. Amin, to + Amin] for the rendering as in Eq. An important benefit of learning a compact object surface that we can effectively reduce the number of sampled points for rendering, even using only one point on each ray, i.e. the surface point, As shown in Fig. [ (b). the image rendered with only the surface point almost indistinguishable from that with multiple points. Such equivalence can be guaranteed theoretically when A + 0, and we include the proof in the supplementary material.
5

---

3.3
Loss Functions
Instead of training on posed 2D images, the proposed GOF leverages corpus of unposed images for 3D-aware image synthesis, where multiple loss functions are adopted.
GAN Loss. Following pi-GAN [6]. GAN loss is used where GOF synthesizes fake images by randomly sampling camera poses [ from a dataset-related distribution P& and rendering according to (4. Denote as real image from the data distribution pD, the non-saturating GAN loss can be described as follows:
Corigin BG) -Exupss-pt [ (Do,, (Goa (z. (1))
+ Eippp [f(-Dop (1)) + X where f(u) = - log(1 + exp(-u)).
However, Cungin alone is not sufficient to guide the training, which may lead to messy images with mike artifacts. Therefore, two more regularizations are incorporated reduce artifacts and further smooth the learned surfaces.
Normal Regularization. The first regularization is prior the surface normal smoothness, which specially useful for learning from 2D real-world images [... GOF, this normal prior is only employed for the surface points E S to encourage natural and smooth surface:
Lnomal
Σ
||no(x,.)) nx(x. 2,2)]12,
x,ES
where e is a small random 3D perturbation and ng denotes the normal vector, which can be computed by n z) Vxag(x, / Vxa((x, 12-
Opacity Regularization. Since alpha values predicted in GOF can be regarded as occupancy values, ideally the entropy of them should be 0 so that or (x, z) values will equal for points in the occupied space and for points in the free space. We thus apply the second opacity regularization, aiming to reduce the entropy of predicted alpha values:
N Σ
Cupacity
log(ao(x;,))) 2)) + log(1 ax z)).
(9)
In summary, the final loss function for training GOF can be written as:
C(O, o) = Corigin (0, の) + Cnommal + Aopacity Copacity,
(10)
where Anomaal and Aopucity are both balancing coefficients.
Experiments
Implementation Details. Unless stated otherwise, in all experiments we set N, the number of points sampled for rendering, to 12, and set M, the number of bins used in root-finding, to 12. As discussed in Sec 3.2 apply iterative process in root-finding. In practice, the number of iterations is set to m, = 3 times. During inference, GOF requires M + m, + N queries to obtain the color of a pixel, while existing methods require 2N queries due to the of hierarchical sampling strategy. Recall it is sufficient for GOF to sample only the surface point to render an image, GOF is thus capable of using just M + M, queries, potentially speeding up the rendering process in off-line applications. More training and implementation details can be found in the supplemental material.
Datasets. To assess method comprchensively, conduct experiments on three datasets, namely CelebA 44) BFM (5). and Cats [46). Specifically, CelebA is a high-resolution face dataset containing 200, 000 diverse face images. Following pi-GAN (6). we crop all images in CelebA from the top of the hair to the bottom of the chin as a pre-processing step. As for the Cats dataset, it contains 6, 444 cat faces of size 128 128. Finally, BFM is synthetic face dataset rendered with Basel Face Model, where each face paired with a ground-truu depth map, making it a good benchmark for quantitatively evaluating the quality of learned surfaces.
Comparison with baselines. To validate the effectiveness of GOF, we compare with two represen- tative GRAF methods, namely GRAF [7] and pi-GAN. Firstly, Fig. demonstrates the qualitative

---

Table 1: Quantitative results (128 128 px) BFM, CelebA and Cats datasets, three metrics
Fréchet Inception Distance (FID), Inception Score (IS) and the weighted variance of sampled depth Σx, (x10-4).
BFM
CelebA
Cats
FID
1St
Σ
FID
1St
Ex,
FID4
ISt
Σ1,
GRAF pi-GAN 66
45.2
1.49
4.64
41.4
1.86
6.51
28.6
1.65
5.47
16.4
2.49
8.16
15.1
2.63
14.58
16.6
2.09
5.91
Ours (w/o L
15.6
2.85
2.66
17.0
2.29
7.15
16.1
1.92
6.06
Ours (w/o Lopacity)
17.1
2.61
3.37
14.4
2.91
4.94
14.3
2.35
4.42
Ours
15.3
2.89
2.54
14.2
2.87
4.58
14.1
2.48
4.28
Image
Normal
3D Mesh
GRAP
P4-GAN
GRAF
piGAN
GRAF
P-GAN
Figure 3: Qualitative comparison BFM (top), CelebA (middle), and Cats (bottom) datasets. Our method synthesizes realistic images while ensuring compact object surfaces.
comparison between these three methods, where we include the synthesized images, the learned surfaces in the form 3D meshes, as well as the corresponding normal maps. As can be observed, GRAF struggles to render good images, let alone estimate compact and reasonable underlying sur- faces. Compared to GRAF, pi-GAN can synthesize images and estimate corresponding surfaces with improved quality. However, messy parts can be clearly recognized on its learned surfaces and normal maps, indicating it incapable of capturing the compact 3D geometric details. In contrast both pi- Di-GAN and GRAF, the proposed GOF is shown to hallucinate realistic images with 3D consistency

---

Table Comparisons on the geometry proper- ties of learned surfaces. We report mean cur- vature(MC) ( (x10=) and mean geodesic dis- (anCe(MGD) between random points to assess the geometry properties of recovered surfaces.
Table 2: Comparisons on the compactness and accuracy of learned surfaces.
Method
SIDE
MAD 4
Supervised
0.412
10.84
Unsup3d 37
0.795
16.51
BFM
CelebA
Cats
GRAF 7
1.866
26.69
pi-GAN
16.84
25.94
34.05
MC 4
pi-GAN 6
0.727
20.46
Ours
12.25
23.13
30.14
GAN2Shape
38
0.759
14.94
pi-GAN 0.483
0,450
0.494
Ours
0.779
13.81
MGD +
Ours
0.226
0.231
0.317
Figure 4: Generated images and their 3D meshes on CelebA and Cats datasets.
and simultaneously learn smooth surface normals as well as compact object surfaces, which verifies the benefit of adopting the transition from radiance fields to occupancy fields. More qualitative results of synthesized images and corresponding surfaces are included Fig.
To quantitatively evaluate the quality of generated images, we report the Fréchet Inception Distance (FID) scores and Inception Score (IS) scores in T On these two metrics, GOF demonstrates substantial improvements over baseline methods. To further measure the compactness learned surfaces, the concentration of color weights w; as mentioned in Fig. (a) is also computed. Specifi- cally, We sample N = 36 equally-spaced points {x, o + {.d} within near and far bounds [f.,, ty] and calculate the corresponding color weights wi, i 1,2,..., N. Actually, the weighted variance of these samples' depth 4 reflects the concentration of color weights in a single image:
N
Σ (t, - f)2, where i = Σ wets / w.
Σ, = (N 1) W/
Intuitively, smaller variance implies the learned surface is more compact. Finally, for each method, the overall concentration of color weights is averaged over 1000 randomly synthesized images at the 256 x 256 resolution. The results terms of this new metric are also included Table
For the quality of learned surfaces, we first evaluate the compactness and accuracy of surfaces on the BFM dataset, since it contains ground-truth depth maps. Specifically, 50K images are generated by each method, together with their corresponding depth maps. For each method, we train a separate CNN these generated images and depth maps predict depths from images. Subsequently, measure the accuracy of learned surfaces by running the CNN on the test split of BFM and comparing its outputs the ground-truth depth maps using the scale-invariant depth error (SIDE) and the mean angle deviation (MAD). While MAD focuses more on the compactness of surfaces, SIDE emphasizes more on the accuracy of depth. As shown in Table GOF significantly outperforms baseline methods on the MAD metric and is comparable to strong baselines on the SIDE metric. Moreover, we also report mean curvature (MC) and mean geodesic distance (MGD) between random points assess the geometry properties of learned surfaces. The lower these two metrics, the smoother recovered object surfaces. Owing to the absence of such two metrics on ground-truth surfaces for reference, we consider the smoother surfaces better conform to ground-truth cases. The reported values on these two metrics are averaged over 100 randomly synthesized meshes. Quantitative comparisons in Table demonstrate our method GOF can preserve better geometry properties.
can
8

---

BFM
CelebA
Cats
Figure 5: Rendering with only surface points. Images (right) rendered only with surface points indistin-
guishable from those (left) obtained with cumulative rendering.
BFM
(b) CelebA
Figure 6: Qualitative ablation proposed priors (upper row w/o Copucity bottom row w/o L
Rendering only with surface points. As mentioned in Sec. 3.2 GOF is able to render an image using only the surface points. To verify this, showcase in Fig. images rendered by GOF using multiple points and only the surface point. As can be observed, images synthesized with these two strategies nearly indistinguishable from each other. Thus, GOF possesses the potential to significantly reduce the number of generator queries when synthesizing an image. To compare the efficiency straightforwardly. we estimate the rendering speed of 256 256 images for both pi-GAN and GOF on single Intel Xeon(R) CPU, On average, pi-GAN costs about 78s per image, while GOF takes about 56s, saving approximately 28% of the time. Owing the reduction in the burden of queries, GOF enables a light rendering scheme that is promising for applications on mobile devices.
Ablation studies. We here analyze the effects of the proposed regularizations Luormal and Lupucity Table Tincludes the quantitative ablation study on these priors. We also include qualitative samples in Fig. 6 which contains images synthesized by GOF without one regularization item. As shown in the BFM cases removing opacity prior leads to the smoke-like artifacts around the cheek part and the absence of normal regularization might degrade the quality of learned normal maps. While testing on the real-world dataset ()),, undesirable specular highlights emerge on the face and the hollows appear on the corresponding shapes if without the normal regularization. Moreover, we observe that removing opacity prior on CelebA dataset will make the face surfaces too flat and unnatural. It is worth noting that although the performance of GOF is deteriorated due the absence of these priors, images and surfaces produced by GOF are still of reasonable quality when compared to that from previous approaches, indicating the transition from radiance fields to occupancy fields is the main cause that leads to the success of GOF. Moreover, we showcase the degenerated results on BFM dataset if our model is trained without the shrinking process. As illustrated Fig. 7 despite realistic generated images, there emerges random noise on the corresponding normal maps and some nasty dents appear on the face shapes, which demonstrates that the combination of our proposed occupancy representation and the shrinking sampling procedure ensures the surface compactness.

---

Image
Normal
3D mash
Recommtaation
Nermal
View
Figure 7: GOF results without the shrinking process. Noise emerges normals and dents appear shapes.
Figure 8: GAN inversion results real images. GOF recon- struct the target images and simultaneously learn the corresponding normal maps as well 3D shapes.
pi-GAN
GOR(cur))
Image
Normal
Relighning
Image
Normal
Relighting
Figure Relighting results. Our method GOF generates desirable images under various light conditions while baseline results are far from satisfactory.
Inverse rendering. Through GAN inversion, our method is also capable inverse rendering as shown in Fig. X Given real image, GOF can reconstruct the target image successfully and realize free view synthesis by controlling the viewpoints. Besides, the recovered normal maps as well as 3D shapes pave the way for downstream tasks such relighting and editing.
Relighting. In Fig. 2 we provide the relighting results based on the learned normal maps by explicitly controlling the lighting directions. As method and baselines can't predict the corresponding albedo, the face-forwarding image is considered as the pseudo albedo. Thanks to better learned normal maps, our method GOF presents promising images under different light conditions. In contrast to ours, baseline methods like pi-GAN 66) tend to generate messy normal maps with obvious checkerboard- -iike artifacts, leading to noisy and dissatisfied relighting results.
Limitations. While training on the hair regions as in existing approaches [6] Besides, the adopted FeL-MedSTTI SIREN backbone in the generator will lead stripe artifacts the generated images especially when they are rendered only with surface points. Meanwhile, surface rendering mode will make furry cat images over-smooth and less realistic. Moreover, our method is more suitable for solid objects with only one surface.
datasets, our method GOF might present similar dents in
5 Conclusion
In this work, we propose generative occupancy fields (GOF), novel generative radiance fields for 3D-aware image synthesis. The crux of GOF a dedicated transition from the cumulative rendering in radiance fields rendering with only the surface points. Such a transition is inspired by the resemblance between the alpha values in radiance fields and the occupancy values in occupancy networks, so that we can reinterpret one as the other. In practice, such a transition is achieved during training by gradually shrinking the sampling region in the rendering process of GOF from the entire volume minimal neighboring region around the surface, where the surface located via root- finding on predicted alpha values. Thanks to the transition, surfaces learned by GOF continuously converge during the training, ensuring their compactness at the end. On three diverse datasets, GOF is shown to demonstrate great superiority in synthesizing 3D consistent images and in the meantime capturing compact surfaces, significantly broadening the application of generative radiance fields in downstream tasks.
10

---

Acknowlegements
We would like thank Eric R. Chan for sharing the codebase of pi-GAN. This work is supported by the Collaborative Research Grant from SenseTime (CUHK Agreement No. TS1712093), the General Research Fund (GRF) of Hong Kong (No. 14205719), the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
References
[1] J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio, "Generative adversarial nets," NIPS, 2014.
[2] A. Brock, Donahue, and K. Simonyan, "Large scale gan training for high fidelity natural image synthesis," in International Conference on Learning Representations, 2018.
[3] T. Karras, S. Laine, and T. Aila, "A style-based generator architecture for generative adversarial networks," in Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 4401-4410 2019.
[4] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, "Analyzing and improving the image quality of stylegan," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110 10-8119, 2020.
[5] B. Mildenhall, P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. "Nerf: Representing scenes as neural radiance fields for view synthesis," in ECCV, pp. 405-421, Springer, 2020.
[6] E. R. Chan, M. Monteiro, P. Kellnhofer, J. Wu, and G. Wetzstein, "pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis," in CVPR, 2021.
[7] K. Schwarz, Y. Liao, M. Niemeyer, and A. Geiger, "Graf: Generative radiance fields for 3d-aware image synthesis," " in NeurIPS, vol. 33, 2020.
[8] L.. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger, "Occupancy networks: Learning 3d reconstruction in function space," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4460-4470, 2019.
[9] L. Yariv, Y. Kasten, D. Moran, M. Galun, M. Atzmon, B. Ronen, and Y. Lipman, "Multiview neural surface reconstruction by disentangling geometry and appearance," Advances Neural Information Processing Systems, vol. 33, 2020.
[10] M. Oechsle, S. Peng, and A. Geiger, "Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction," arXiv preprint arXiv:2104. 10078, 2021.
[11] M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger, "Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision," in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3504-3515, 2020.
[12] Y. Jiang, D. Ji, Z. Han, and M. Zwicker, "Sdfdiff: Differentiable rendering of signed distance fields for 3d shape optimization."" " in The IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
[13] S. Liu, Y. Zhang, S. Peng, B. Shi, M. Pollefeys, and Z. Cui, "Dist: Rendering deep implicit signed distance function with differentiable sphere tracing," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2019-2028, 2020.
[14] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove, "Deepsdf: Learning continuous signed distance functions for shape representation." Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 165-174, 2019.
[15] Z. Chen and H. Zhang, "Learning implicit fields for generative shape modeling," in Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, pp. 5939-5948, 2019.
11

---

[16] K. Genova, F. Cole, D. Vlasic, A. Sarna, W. T. Freeman, and T. Funkhouser, "Learning shape templates with structured implicit functions," " in Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7154-7164, 2019.
[17] M. Michalkiewicz, K. Pontes, D. Jack, M. Baktashmotlagh, and A. Eriksson, "Implicit surface representations as layers neural networks," " in Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4743-4752, 2019.
[18] S. Saito, Z. Huang. R. Natsume, S. Morishima, A. Kanazawa, and H. Li, "Pifu: Pixel-aligned im- plicit function for high-resolution clothed human digitization." in Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2304-2314, 2019.
[19] S. Saito, T. Simon, J. Saragih, and H. Joo, "Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 84-93, 2020.
[20] V. Sitzmann, E. R. Chan, R. Tucker, N. Snavely, and G. Wetzstein, "Metasdf: Meta-learning signed distance functions," arXiv preprint arXiv:2006. 09662, 2020.
[21] V. Sitzmann, M. Zollhöfer, and G. Wetzstein, "Scene representation networks: Continuous 3d-structure-aware neural scene representations." in Advances in Neural Information Processing Systems, 2019.
[22] L. Liu, J. Gu, K. Z. Lin, T.-S. Chua, and C. Theobalt, "Neural sparse voxel fields," arXiv preprint arXiv:2007. 1571, 2020.
[23] A. Yu, R. Li, M. Tancik, H. Li, R. Ng. and A. Kanazawa, "Plenoctrees for real-time rendering of neural radiance fields," arXiv preprint arXiv:2103. 14024, 2021.
[24] T. Neff, Stadlbauer, M. Parger, A. Kurz, J. H. Mueller, C. R. A. Chaitanya, A. Kaplanyan, and M. Steinberger, "Donerf: Towards real-time rendering of compact neural radiance fields using depth oracle networks," arXiv preprint arXiv: 103.0333, 2021.
[25] P. Hedman, P. P. Srinivasan, B. Mildenhall, J. T. Barron, and P. Debevec, "Baking neural radiance fields for real-time view synthesis," arXiv preprint arXiv:2103. 14645, 2021.
[26] M. Boss, R. Braun, V. Jampani, J. T. Barron, C. Liu, and H. Lensch, "Nerd: Neural reflectance decomposition from image collections," arXiv preprint arXiv:2012.03913 2020.
[27] P. Srinivasan, B. Deng, X. Zhang, M. Tancik, B. Mildenhall, and T. Barron, "Nerv: Neural reflectance and visibility fields for relighting and view synthesis," arXiv preprint arXiv:2012.03927,, 2020.
[28] M. Atzmon, N. Haim, L. Yariv, O. Israelov, H. Maron, and Lipman, "Controlling neural level sets," arXiv preprint arXiv: 1905. 1 19911,111111 2019.
[29] P. Kellnhofer, L. C. Jebe, A. Jones, R. Spicer, K. Pulli, and G. Wetzstein, "Neural lumigraph rendering," " Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4287-4297, 2021.
[30] J. Wu, C. Zhang, T. Xue, B. Freeman, and J. Tenenbaum, "Learning probabilistic latent space of object shapes via 3d generative-adversarri modeling," " in NIPS, 2016.
[31] X. Chen, D. Cohen-Or, B. Chen, and N. J. Mitra, "Towards a neural graphics pipeline for controllable image generation," Computer Graphics Forum, vol. 40, no. 2, 2021.
[32] J.-Y. Zhu, Z. Zhang, C. Zhang, J. Wu, A. Torralba, J. Tenenbaum, and B. Freeman, "Visual object networks: Image generation with disentangled 3d representations," in NeurIPS, 2018.
[33] Y. Liao, Schwarz, I.. Mescheder, and A. Geiger, "Towards unsupervised learning of generative models for 3d controllable image synthesis," in CVPR, pp. 5871-5880, 2020.
[34] T. Nguyen-Phuoe, C. Richardt, L. Mai, Y.-.. Yang, and N. Mitra, "Blockgan: Learning 3d object-aware scene representations from unlabelled images," in Advances in Neurul Information Processing Systems 33, Nov 2020.
12

---

[35] T. Nguyen-Pharn Phuee, C. Li, L. Theis, C. Richardt, and Y.-L. Yang, "Hologan: Unsupervised learning of 3d representations from natural images." " ICCV. pp. 7588-7597, 2019.
[36] M. Niemeyer and A. Geiger, "Giraffe: Representing scenes as compositional generative neural feature fields," in CVPR, 2021.
[37] S. Wu, C. Rupprecht, and A. Vedaldi, "Unsupervised learning of probably symmetric deformable 3d objects from images in the wild," in CVPR, 2020.
[38] X. Pan, B. Dai, Z. Liu, C. C. Loy, and P. Luo, "Do 2d gans know 3d shape? unsupervised 3d shape reconstruction from 2d image gans," ICLR, 2021.
[39] M. Sahasrabudhe, Z. Shu, E. Bartrum, Alp Guler, Samaras, and Kokkinos, "Lifting autoencoders: Unsupervised learning of a dully-disentangled 3d morphable model using deep non-rigid structure from motion," ICCV Workshops, pp. 0-0, 2019.
[40] X. Li, S. Liu, K. Kim, De Mello, V. Jampani, M.-H. Yang, and J. Kautz, "Self-supervised single-view 3d reconstruction via semantic consistency," ECCV, 2020.
[41] P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang, "Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction," arXiv preprint arXiv:2106. 10689, 2021.
[42] L. Yariv, J. Gu, Kasten, and Lipman, "Volume rendering of neural implicit surfaces," arXiv preprint arXiv:2106. 12052, 2021.
[43] T. Kajiya and B. P. Von Herzen, "Ray tracing volume densities," ACM SIGGRAPH computer graphics, vol. 18, no. 3, pp. 165-174, 1984.
[44] Z. Liu, Luo, X. Wang, and X. Tang, "Deep learning face attributes in the wild," in ICCV, pp. 17730-3738, 2015.
[45] P. Paysan, R. Knothe, B. Amberg, S. Romdhani, and T. Vetter, "A 3d face model for pose and illumination invariant face recognition," in 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance, pp. 296-301, leee, 2009.
[46] W. Zhang, J. Sun, and X. Tang, "Cat head detection-how to effectively exploit shape and texture features," in ECCV. pp. 802-810 816, Springer, 2008.
13


Reviews:
[{"review_id": "z_G8cJ5C4__", "reviewer": "Reviewer_mS1W", "summary": "The paper proposes GOF, a method to improve the geometry and quality of generative methods that use a neural radiance field representation. The core idea is to modify the network's output to predict occupancy which can be used to find the surface of the rendered object. Once the surface is known, the network can be sampled in a region around the surface that shrinks during training. This accomplishes two goals, it consolidates the geometry to a surface and requires fewer samples during inference. When compared to existing approaches, the depths and normals are cleaner and the image quality metrics are marginally better.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I’ve listed my primary concerns below. Overall the paper is well written and the method is simple and straightforward. For these reasons I think this paper is marginally above the acceptance threshold.\n\na) The paper never actually demonstrates why the recovered representation is better than existing methods (other than showing a depth map). It is noted that downstream tasks like relighting could be improved, but is not demonstrated. A good portion of the paper also discusses the ability to reduce the number of samples taken during inference which could help with “applications on mobile devices”. It would have been nice to see quantitative values demonstrating the speed up of the proposed method. As the paper currently stands, the quality of the image outputs look fairly similar to pi-GAN. I think this paper would be much stronger if advantages on downstream tasks or a speedup compared to baselines were demonstrated.\n\nb) How much can we attribute the improved depth and normals on the use of occupancy compared to the included priors? It would have been nice to see qualitative examples of pi-GAN or GRAF trained with the extra priors. I am also slightly concerned about the quantitative evaluation. Since a different CNN is trained for each baseline, it seems that baselines with larger image FID will be further out of distribution when testing the CNN on the ground truth test images. How do you know that errors in the predicted test depths are a result of bad depths from the baseline versus a distribution mismatch on the CNN input?\n\nc) GOF focuses the evaluation on forward facing faces. It is argued that the smaller camera baseline causes the compared methods to “diffuse” the geometry. GRAF and pi-GAN also test on a 360 scene of synthetic cars and appear to recover fairly good geometry. For scenes with larger baselines is GOF still necessary or do existing approaches work well enough?\n\nd) The conversion of the network to output alpha instead of density forgoes the variable dictating the distance between samples (delta). While the samples are uniform, the distance does change during training because the sampling interval shrinks. Is this something that was considered and is it an issue for this model?\n\nNIT:\nLN 177 “a” -> “at”\nLN 222 “ms = ? times”", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This method is more restrictive in the types of scenes it can represent as it assumes each ray intersects with a single surface. This should be listed in the limitations.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "x9MTh8gEtzi", "reviewer": "Reviewer_P3Ee", "summary": "The authors propose a novel formulation of the 3D GAN problem using hybrid volume-surface approach. The method leads to a more accurate shape reconstruction allowing for more efficient rendering. The reconstructed image quality is also comparable or better.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Originality] \n\nThe method closely follows standard 3D GAN pipeline but introduces novel volume-surface formulation of the representation. The authors explain motivation for the technique and demonstrate its benefits.\n\n[Quality]\n\nThe technique is demonstrated adequately and the evaluation is sufficient.\n\n[Clarity]\n\nThe exposition is clear. All necessary details for reimplementation appear to be present. The network architecture in the supplemental could be described more visually for intuitive comparison to other methods.\n\n[Significance]\n\nThe paper falls within the scope of image/shape GAN methods and shares their potential application scope.\n\n[Questions, Comments and Concerns]\n\n1) The Eq. 4 replaces exponential of sum (of sigmas) by effectively a product of exponentials (as the alphas are computed by sigmoids). These two formulations are mathematically very close. Would the original NeRF formulation (Eq. 2) still be applicable if the exp(-sigma) would be used for mapping into the desired [0, 1] range before thresholding? In another words, is the key innovation the adaptive sampling around the surface alone or is the alpha formulation also necessary?\n\n2) The surface sampling pattern of GOF appears more efficient than that of GRAF. However, the training was still limited to 128x128 px resolution as in pi-GAN [6]. Why is that? Should this be listed in limitations (same as for previous work)?\n\n3) In Sup Fig. 3, the authors demonstrate interpolation in the latent space. However, is the method suitable also for inverse rendering, that is finding a fitting latent code for given output 2D image? Does the surface manifold representation provide robust gradient for such tasks?\n\n4) Minor comments:\n- L110: \"ensure the compactness of learned object surfaces.\" - What does this refer to? Size of the trained model? Area of the extracted surface manifold?\n- Eq.2 for T_i misses z in the argument\n- Figure 3 is very low-res to the point that it is hard to see differences.\n- Figure 6 - please label which prior is ablated in which panel.\n- The computational resources were promised in the appendix but I did not find them in the supplement.\n\n[Conclusion]\n\nOverall, the methods main contribution is its formulation of the volume/surface representation. The application to GAN problem does not seem to require any particularly unique features when compared to the multiview supervision as demonstrated by the concurrent work [10]. Still, it is a completely novel technique and demonstrating its efficiency in GAN setup is perfectly valid since it generally tends to be harder to train. The method produces superior surface surfaces when compared to previous volume-only based GAN techniques. The image quality is also quantitatively higher although qualitatively the method does trade-off the sampling noise artifacts to sharp boundary artifacts in some images so the visual preference may be highly subjective and vary on case-by-case basis. Weighing all these factors, I lean positive but the authors should correct their limitation discussion by moving it to the main paper and potentially adding new points.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "[Societal impact]\n\nThe method could be utilized in fast low-cost content production for 3D applications including AR/VR.\n\n\n[Limitations]\n\nSome of the results trade off blur and noise for sharp edge artifacts that are subjectively more disturbing (see the eye region in the last cat example in the video). Same in Fig 5.\n\nThe authors discuss the limitations in the supplemental which does not seem right. The discussion should clearly be in the main paper as it is essential to the message of the paper. The authors admit shape distortions in some difficult areas. The authors also mention \"stripe\" artifacts in surface based renderings. I am not sure if this is the same artifact that I observed above but I note that the sharp-edge artifacts are equally present in both the surface and volume renders (see Fig. 5).\n\nThe authors could also discuss the resolution as a potential limitation and note that the quality of the generated images is nowhere near to what 2D GANs such as StyleGAN2 can produce (which is true for all exisiting 3D GANs).\n\n", "review_text": "", "extra_scores": {"ethical_concerns": "The authors are concerned with deep fakes and environmental impact. That seems like a fitting assessment.\n\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "uAeJjuq_o6V", "reviewer": "Reviewer_eXp1", "summary": "This paper proposes a new framework for generative radiance fields, along the lines of GRAF and pi-GAN, but the authors focus on generating cleaner surfaces using an approach that is closely related to UNISURF. This method makes sense, and the generated shapes look a lot cleaner and less noisy than those generated by pi-GAN and more expressive than GRAF. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Paper strengths:\n+ generative 3D-aware methods are very interesting and challenging, the proposed framework introduces a surface-centric approach which is new\n+ the qualitative results, especially the surfaces (i.e., the focus of this work), look really good - a significant improvement over previous work\n\nPaper weaknesses:\n\n- some missing related work:\n-> Wang et al., \"NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction\" as a concurrent method to UNISURF, which can be optionally cited\n-> Yariv et al., \"Volume Rendering of Neural Implicit Surfaces\" as a concurrent method to UNISURF, which can be optionally cited\n-> Sitzmann et al., \"Scene Representation Networks\", NeurIPS 2019 as one of the very first 2D supervised neural implicit representations\n-> Kellnhofer et al., \"Neural Lumigraph Rendering\", CVPR 2021 as one of the more recent and SOTA surface-based neural rendering frameworks\n\n- method: the proposed method makes sense, but seems like a relatively simply combination of GRAF and UNISURF; the manuscript would very much benefit from a specific and detailed discussion on what's similar and different between the two\n\n- quantitative evaluation: in table 1, GOF outperforms pi-GAN only marginally; that may be okay, because the focus here is on the surface; however, I was surprised when I realized that the FID scores reported here for pi-GAN on CelebA and Cats are 15.8 and 17.6, respectively, whereas they are listed as 14.7 and 16.8 in that paper, which are both lower than the proposed method\n\n- qualitative results: the cats generated by FOG seem considerably smoother and less realistic than those of pi-GAN\n\nOverall, my assessment is that this submission takes some of the important ideas from UNISURF and applies them to a different problem setting, i.e. that of unconditional image generation. It seems very interesting that the very naive UNISURF-like progressive shrinking technique works in a GAN framework; I had my doubts that such an approach would remain stable once the updates became confined to the local region. Therefore, despite this submission being strongly influenced by the core ideas of UNISURF, I think it merits publication.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss and evaluate their limitations.", "review_text": "", "extra_scores": {"ethical_concerns": "An explicit statement on ethics considerations of this work and unconditional 3D image generation in general would be helpful.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "ozw5FPjeg3O", "reviewer": "Reviewer_3D7c", "summary": "The paper suggests a new generative model for 3D-aware image synthesis by generating radiance fields with an underlying 3D occupancy representation. This choice of representations allows the generative model to learn surfaces, which are rendered to view consistent images from arbitrary poses using volume rendering. During training, the rendering is performed using samples from a gradually shrinking sampling region around the surface, which encourages the surface to be more compact. The paper demonstrates the method's ability to synthesize images with 3D consistency and further presents their matching shape and normal.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(A) Those are the main strengths and novelty I find in the paper:\n1. The 3D surface representation results learned from unposed images are pretty impressive, based on the fact that they are learned in an unsupervised manner.\n2. The paper is well written and has a good flow. The related works section presents the method's origin story and binds the reader to the relevant area. Also, in the method section, the authors give the background for the neural radiance field naturally and easily.\n3. In the case of representing object surfaces, the motivation for more concentrated volume densities is evident. Therefore, the use of occupancy representations, which could inherently ensure deterministic surfaces, is good and intuitive for the task of generating compact surfaces (like the presented faces and cats). \n4. I appreciate the presented measure of the weights' variance, as it indicates the method compactness compared to the other baselines.\n\n(B) Following are the weaknesses I find in the presented method and the concerns which I would like to be addressed by the authors:\n1. The need for compact surfaces is clear for objects surfaces (like faces and cats), but it is not entirely trivial for me why the previous works are more prone to predict diffuse surfaces. Namely, why directly training equation 2 (which is the optimization that the basslines GRAF and pi-GAN follows) fails to maintain the surface compactness? The authors suggest that it may arise from the shape-color ambiguity, yet I see this ambiguity as another existing problem for other geometrical defects that can exist even with compact surfaces. Hence, my biggest concern that arises is the method's key contribution compare to its basslines. There are two different aspects of the presented method compared to its baseline: the 3D representation and the sampling procedure. My question is, which one of those is \"responsible\" for the surface compactness? Is it the sampling procedure around the surface that is shrinking and facilitating the weights' concentration, or is it the underlying surface representation? I would like if the authors can suggest an adequate ablation study to answer this concern or give a better intuition for why the baselines tend to diffuse densities. \n2. The authors present the evaluation of the generated images as done in standard GAN's image models, but I find the generated surfaces' evaluation to be insufficient. Since the paper's central claim is better modeling of the 3D representation, the evaluation should support that claim. I find the geometric evaluation to rely on the correctness of the CNN to predict depth maps. Nonetheless, if I understand correctly, the depth maps from the presented method, and the two other baselines are taken as the expected depth, not according to some level set as presented in the figures. A better evaluation would be to show that the generated geometry preserves similar geometry information as in the data set (e.g., mean curvature, mean geodesic distance of random points, faces are and so on). I suggest the authors present other possible evaluations for the underlying geometry or give a better motivation for their proposed evaluation.\n3. I believe the authors have an error in equation 9 (the opacity regularization) where each element should be multiplied by its probability, namely: p*log(p)+(1-p)log(1-p). This will give the entropy of the binary probability alpha, and minimizing this regularization aims to drive each sample on the ray to either 1 or 0. If there is no error here, I would like to get an explanation or intuition regarding this formula.\n4. Regarding the presented ablation study: it is not clear from the text what is the corresponding experiment for each image in figure 6. Moreover, the ablation should check the contribution of each component (regularization prior) separately. As for what I understand, the authors include results of either with and without both priors. I would appreciate it if the authors can supply an ablation of removing one regularization at a time, to determine the effect of each regularization.\n\n(C) Some other minor questions and suggestions to the authors:\n1. I wonder why it seems there is a problem with modeling the eyes geometry (more noticeable in the CelebA). I suspect it relates to the fact the light field is a bit more complicated to present, but I would like to hear the author's opinion.\n2. How did the authors extract the shape (shown in the figures) from the pi-GAN and GRAF methods? Namely, how the level set was chosen?\n3. Line 150 says that the density varies between -10 to 50, but the physical density should be positive.\n4. I suggest the authors present the concrete differences for the other baselines (pi-GAN and GRAF) in the context of the radiance field modeling and sampling procedure. It would help to understand the authors' contribution and novelty compared to existing baselines.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did address the main limitations of the method and its potential social impacts. However, those are mentioned in the supplementary only. I suggest the authors move them or add a short discussion in the main paper on the basics limitation and societal impact. Lastly, I appreciate the author's comment regarding the potential environmental effects.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received reviews from four expert reviewers in the community. All reviewers agree that the proposed hybrid volume-surface approach is novel in the context of 3D generative modeling. In the initial review, there were some concerns regarding the qualitative evaluation (reviewer mS1W), results on synthetic cars (reviewer mS1W), and evaluation on geometry (reviewer 3D7c), ablation on the regularization prior (reviewer 3D7c). After the rebuttal, the reviewers were convinced by the additional results as well as clarification. The AC reads the reviews, rebuttal, and also agrees that all the concerns raised by the reviewers have been properly addressed with the additional explanation and experimental results. The AC thus recommends to accept and encourages the authors to revise and strengthen the paper by incorporating the suggested changes and additional evaluation. 
Decision: Accept (Poster)