VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids
Authors: Katja Schwarz
Axel Sauer
Michael Niemeyer
Yiyi Liao
Andreas Geiger
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3D-aware image synthesis
sparse voxel grids
neural radiance fields
3D generative models

Abstract: State-of-the-art 3D-aware generative models rely on coordinate-based MLPs to parameterize 3D radiance fields. While demonstrating impressive results, querying an MLP for every sample along each ray leads to slow rendering.
Therefore, existing approaches often render low-resolution feature maps and process them with an upsampling network to obtain the final image. 
Albeit efficient, neural rendering often entangles viewpoint and content such that changing the camera pose results in unwanted changes of geometry or appearance.
Motivated by recent results in voxel-based novel view synthesis, we investigate the utility of sparse voxel grid representations for fast and 3D-consistent generative modeling in this paper.
Our results demonstrate that monolithic MLPs can indeed be replaced by 3D convolutions when combining sparse voxel grids with progressive growing, free space pruning and appropriate regularization.
To obtain a compact representation of the scene and allow for scaling to higher voxel resolutions, our model disentangles the foreground object (modeled in 3D) from the background (modeled in 2D).
In contrast to existing approaches, our method requires only a single forward pass to generate a full 3D scene. It hence allows for efficient rendering from arbitrary viewpoints while yielding 3D consistent results with high visual fidelity. Code and models are available at https://github.com/autonomousvision/voxgraf.

VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids
Katja Schwarz!
Axel Sauer!
Michael Niemeyer'
Yiyi Liao?
Andreas Geiger'
University of Tübingen and Max Planck Institute for Intelligent Systems, Tübingen Zhejiang University, China
Abstract
State-of-the-art 3D-aware generative models rely on coordinate-base MLPs to parameterize 3D radiance fields. While demonstrating impressive results, querying an MLP for every sample along each ray leads to slow rendering. Therefore, existing approaches often render low-resolution feature maps and process them with upsampling network to obtain the final image. Albeit efficient, neural rendering often entangles viewpoint and content such that changing the camera pose results in unwanted changes of geometry or appearance. Motivated by recent results in voxel-based novel view synthesis, we investigate the utility of sparse voxel grid representations for fast and 3D-consistent generative modeling in this paper. Our results demonstrate that monolithic MLPs can indeed be replaced by 3D convolutions when combining sparse voxel grids with progressive growing, free space pruning and appropriate regularization. To obtain a compact representation the scene and allow for scaling to higher voxel resolutions, the foreground object (modeled in 3D) from the background (modeled in 2D). contrast existing approaches, our method requires only single forward pass to generate a full 3D scene. It hence allows for efficient rendering from arbitrary viewpoints while yielding 3D consistent results with high visual fidelity. Code and models are available at https::/github..com
model disentangles
voxgraf.
Introduction
Generating photorealistic renderings of scenes high resolution is a long-standing goal in computer vision and graphics. The primary paradigm to carefully design 3D models, which are then rendered using realistic camera and illumination models. In recent years, the computer vision community has made significant headway towards reducing these design efforts by approaching content generation from data-centric perspective. Generative Adversarial Networks (GANs) [9] have emerged as powerful class of generative models for photorealistic high-resolution image synthesis [2, 16, 17, 19, 20, 36]. One benefit of these 2D models is that they can be trained with large collections of images which are readily available. However, scaling GANs to 3D is non-trivial because 3D supervision difficult obtain. Recently, 3D-aware GANs have emerged to address the gap between handcrafted 3D models and image synthesis with 2D GANs which lack 3D constraints [4, 13, 29, 30] 3D-aware GANs combine 3D generators, differentiable rendering and ad adversarial training to synthesize novel images with explicit control over the camera pose and, potentially, other scene properties like object shape and appearance.
Early 3D-aware GANs explored voxel-based 3D representations [13, 29]. To compensate for the cubic memory growth of voxel grids, HoloGAN [29] generates features on small 3D grid and uses a neural network to map 3D features to a 2D image. While such a neural renderer allows scaling to higher image resolutions, may also entangle viewpoint and generated content [37].
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Figure 1: 3D-aware image synthesis with sparse voxel grids. We investigate neural radiance fields represented as sparse voxel grids in the context of 3D-aware generative modeling. While training from unstructured image collections, our method allows for high quality image synthesis with explicit control over the camera viewpoint. In contrast to previous works, our method generates a 3D scene in a single forward pass allowing for more efficient and 3D-consistent rendering.
Consequently, early voxel-based approaches were either limited in image resolution or lacking 3D consistency. About the same time, Neural Radiance Fields (NeRF) [27] emerged in the context of view synthesis as powerful alternative 3D representation. their seminal work, Mildenhall et al. [27] represent a scene predicted color and density values are then projected to an image with differentiable volume rendering. GRAF [38] adapts NeRF's coordinate-based representation to Generative Radiance Fields (GRAF) and proposes a 3D-aware GAN using a coordinate-basse MLP and volume rendering. This propelled 3D-aware image synthesis to higher image resolutions while better preserving 3D consistency due to the pyssically-basee and parameter-fire rendering. These benefits led to the establishment of coordinate-bases MLPs new de facto standard for 3D-aware image synthesis [4,31]. While recent 3D-aware GANs [4, 6, 10] have started to attain image fidelity and resolution similar to 2D GANs, training and inference is computationally expensive the MLP must be queried at multiple points along each ray for volume rendering. However, querying 3D space densely is prohibitively costly. For example, rendering an image at resolution 2562 using 48 sample points along each ray requires to query the neural network 256 48 Rs 3M times. As a result, most recent works combine neural and volume rendering to ease computational cost at high resolutions [30]. Consequently, viewpoint and 3D content are often entangled, such that changing the camera pose might result in unwanted changes of the geometry or the appearance of the 3D scene. Further, for many downstream applications. e.g. integrating assets into physics engines, it is desirable to generate 3D content at high resolution directly. These shortcomings identify the need for a 3D representation that can be efficiently rendered at high resolution with a model that is 3D-consistent by design.
function of color and density, parameterized by coordinate-based MLP. The
Recently, there has been significant progress towards accelerated training of novel view synthesis models for single scenes by removing the MIP from the representation. In particular, DVGO [40] and Plenoxels [1] directly optimize sparse voxel grid for novel-view synthesis, demonstrating that the visual fidelity attained by NeRF [27] is not primarily attributed to its MI.P-based representation but rather to volumetric rendering and gradient-based optimization. In addition to impressive image quality, [1.40) obtain large rendering speedups due to their fast density and color queries. Taking inspiration from these works, we revisit voxel-based representations for 3D-aware GANs [1] in the context of volumetric rendering. To circumvent the cubic memory growth that limits early voxel-based approaches 29], we explore sparse voxel grids for the generative settings, see Fig. 1. We observe that sparsity is key to enable scaling the 3D representation to higher resolution and to combine with volume rendering. Specifically, we propose a 3D-aware GAN with a sparse voxel grid generator at its core. As result, our approach inherits fast rendering and trilinear interpolation while being 3D-consistent by design, separating from other recent 3D-aware GANs [3,6, 10] which require forward pass for every point along each camera ray of every view. Another difference to

---

existing 3D-aw aw-aware GANs is that sparsity is built-in feature of our representation, mitigating the need for exploiting sophisticated strategies to sample points along camera rays as [4,33,43]. Our final model achieves image fidelity similar to recent 3D-aware GANs leveraging neural rendering while generating high-resolution geometry and improving 3D consistency. During inference, our model only requires a single forward pass which takes up most of the inference time. Once the 3D) scene is generated, images can be rendered within milliseconds while existing approaches require another forward pass which is two orders of magnitude slower. We refer to our model as VoxGRAF.
2 Related Work
2D GANs. Rapid progress Generative Adversarial Networks [9] now enables photorealistic synthesis up to megapixel resolution [2, 16, 18-20, 26, 36]. While the disentangled style-space of StyleGANs [18-20] allows for control over the viewpoint of the generated images to some extent [12, 24, 39, 48]. gaining precise 3D-consistent control is still non-trite con-trivial due to its lack of physical interpretation and operation in 2D. In contrast. in this work we aim for explicit control over the camera pose by incorporating a 3D representation into the generator.
3D-Aware GANs. The first 3D-aware GANs, i.e. GANs that incorporate a 3D representation into the generator model, were voxel-based approaches. Dense grid-based approaches [13,50] are limited to lower grid resolution due to their cubic memory growth. Other works combine lower-resolution grids with neural rendering [23, 29] which scale higher resolutions, but the generated images lack 3D consistency [38]. Recently. GRAF [37] and π T-GAN combine volume rendering and goordinate-basse representations [27] allowing to scale 3D-awere GANs with physically inspired rendering high resolutions. However, dense ray marching remains computationally expensive and limits image fidelity. GIRAFFE [31] therefore proposes hybrid rendering approach. They render low-resolution feature map with ray marching and use a neural renderer to decode it into high-resolution image. Due to efficiency, this approach has been widely adopted in subsequent works [3, 10, 15,32,44, 32,44,44,,44 47,49), While some approaches try to counteract introduced inconsistencies, e.g. via dual discrimination [3] or a reconstruction loss [10], we instead propose a model that is 3D-consistent by design and can generate the 3D object at high-resolution.
As an alternative to hybrid rendering. GOF [43], ShadeGAN [33] and GRAM [6] focus on reducing the number of query points for volume rendering. While aforementioned methods aim for reducing the number of sample points as querying large MLP computationally expensive, we instead use sparse voxel grid as 3D representation. This allows us to speed up rendering without reducing the sample size as feature querying via trilinear interpolation is fast and can be efficiently implemented via custom CUDA kernels.
Sparse 3D Representations. As NeRF requires optimization time in the order of multiple days per scene, series of follow-up works [25,34, [5534,41,466] propose techniques to speed up this process. The recent works Plenoxels [1] and DVGO [40] demonstrate that sparse voxel grid representations can achieve even faster convergence and higher rendering speed. In addition to efficient rendering, sparse voxel grids enable fast trilinear interpolation when queried beyond their grid resolution. Building on this representation, our approach inherits these benefits. We remark that very recently Instant- faster rendering by combining small MLPs with multi-resolution hash
NGP [28] achieves table. Exploring this representation might be an interesting avenue for future extensions of our work. Note that all of these works focus on novel-view-synthesis for single scenes and require multi-view image supervision. Instead, we propose a generative model that trains with raw image collections and that can generate multiple novel instances at inference.
Method
We first provide the necessary background by summarizing the currently dominating paradigm for designing 3D-aware GANs which combines an MIP scene representation and volume rendering as introduced in GRAF [38]. Next, we introduce our sparse voxel-based scene representation which boosts rendering speed while retaining 3D 3D-consistency by design. We refer to our model as VoxGRAF.
3

---

{ ~ Pelie--1-05 e (00,
(K, (
G (x) 1
(1/s, mash
(i,8)
G 9:
Volume Readering
Alpha
Composition
D
(T - PD: Egrest ~ pc)
Sparse Voxel Grid
Trilinear Interpolation
Figure 2: VoxGRAF Conditioned on camera pose &. the foreground generator G maps latent code z to color values E g Rg Re and densities a E R X on a sparse voxel grid of resolution Rg- Given camera intrinsics K and camera pose g', foreground image is obtained using differentiable volume rendering [27]. The values at the sampling points along the camera rays computed by trilinearly interpolating the voxel grid [1]. The background is generated by 2D GAN G and combined with the foreground using alpha composition. The discriminator D compares the generated image I to the real image Ireal
3.1 3D-aware GANs with Coordinate- Scene Representations
Recent 3D-aware GANs typically consist of a learned coordinate- paaedd MLP as 3D generator, deterministic volume rendering step potentially combined with a learned neural renderer the 2D image domain, and a learned 2D discriminator. Let G.LP denote the 3D generator parameterized by a coordinate-bel based MLP with learnable parameters v. The 3D generator predicts a radiance field defined by color с [0, 1] and density values E I The radiance field defined any 3D point x E H33 and for any viewing direction E S2, To model different 3D scenes, the generator is additionally conditioned on an -idimennioonal latent variable E ((, 1). The inputs x and are projected to dimbe--dimeiionall features (xx E Lx and y(d) E La with a fixed positional encoding overcome the MLP's smoothness bias and model high-frequency content y(-) [27,42], Formally,
GMLP
IR 1x
R RM
R
1
(r(x).((1) z) (c,
In this paper, we challenge this paradigm and investigate sparse 3D CNN instead of a coordinate- based MLP generator, described in the next section.
The radiance field is rendered by approximating the intractable volumetric projection integral via numerical integration. First, the generator is queried at N sampling points along each camera ray yielding colors and densities {(c,, ) For each camera ray r, these points are projected to an RGB color value C, and optionally alpha mask a, using alpha composition
(R3 R
+ R
f(c of)} H.
C, c-
Σ
T T
a, ΣT T
T (1-
of - c (-of (2)
where Te and a; denote the transmittance and alpha value of sample point i along ray r and 8 |x* x 2/|12 is the distance between neighboring sample points. volume rendering has proven powerful tool for high-fidelin reconstruction, VoxGRAF retains this rendering mechanism but reduces its computational cost by leveraging sparse scene representations.
3.2 VoxGRAF: Generating Radiance Fields on Sparse Voxel Grids
Our goal is to design a 3waaw GAN based on a sparse scene representation that allows for efficient rendering. Fig. 2 shows overview over approach. In contrast to recent works [3, [336, do not use a noordinate-based MLP to parameterize the radiance field. Instead, inspired by recent work on novel view synthesis [1, 40], we generate values on sparse voxel grid using 3D convolutional neural network. Therefore, our generator requires only a single forward pass to generate a 3D scene.
To disentangle 3D content from the background we combine 3D foreground generator G with
2D background generator bg G takes camera matrix K, camera pose ç and a latent code

---

Proming
Lourned Up Upeampling
32
64
128
Occluded Vocels
= (ie (
Low-dmmitt Vonels
fiE (o. rel)
Kept Vouels
fie v. (e) T.1)
(a) Pruning
(b) Progressive Growing
Figure 3:
Pruning and Progressive Growing, A sparse representation is key for scaling voxel grids to high resolution. To sparsify the generated voxel grids, we combine density-based pruning (a)
and progressive growing (b) while training our model. See text for details.
as input and predicts colors and density values camera matrix K and pose & can be determined e.g. with an off-the-shelf pose detector as done [4]. Volume rendering yields foreground image and alpha mask. The background generator maps the latent code z to a background image which is then combined with the foreground image using alpha composition. We train model in an adversarial setting using 2D discriminator on the full image.
sparse voxel grid. For unstructured images,
Foreground Generator. Our foreground generator builds on the popular StyleGAN2 architec- ture [19] which achieves high fidelity in the 2D image domain. Conditioned on camera pose & RR.. it maps latent code color values and density values on sparse voxel grid
x IR.
3x Rg Rax Ra x R 12 Rog x Rg
(z, (c,
(3)
where fy and Rg denote the learnable parameters and the resolution of the voxel grid, respectively. Note that in contrast to most existing coordinate-basee 3D-aware GANs, see Eq. (1), we do not condition the 3D generator on the view direction (per-ray). Instead, we follow [3] and condition it on the pose & (per-image) to model directional dependencies. For rendering, compute (ct, of) via trilinear interpolation of densities and colors stored at the nearest eight vertices [1]. We use the same rendering formulation outlined in 3.1 and leverage custom CUDA kernels' for efficiency.
To generate voxel grids instead of images, we replace all 2D operations of the StyleGAN2 generator with their 3D equivalent, e.g., 3D modulated convolutions and 3D upsampling. At resolutions beyond 323, we investigate sparse convolutions [11] instead of dense ones as they can be more computationally efficient. We compare the computational efficiency of sparse convolutions? to dense convolutions for which we zero out the values of pruned voxels. For our architecture, using sparse convolutions reduces the memory consumption but due to the computational overhead for managing coordinates increases the runtime, see Table 1. To sparsify the representation, we combine progressive growing [16] with pruning [1] illustrated in Fig. 3a. Specifically, we start with training dense model at resolution 323, After sufficient training, we add the next layer of convolutions and prune its inputs based on the rendered view. Intuitively, the next layer should only operate on voxels visible in the rendered view to yield a sparse representation. Consequently, we prune voxels that are either occluded or have low density. Following Eq. (2), the rendered alpha value is a, E'' Te of Accordingly, occluded voxels (low transmittance T,) and empty voxels (low density o;) do not contribute to the final image. The pruning operator discards all voxels along camera ray for which transmittance 7" or density a is smaller than threshold Tr or To, respectively. Let V, denote the set of voxels intersecting with ray then
p V, V2
DP e (T) 7T) A (o. {T)}
(4)
where VI is the set of the retained voxels. After pruning, upsample the kept voxels via sparse transposed convolutions, After the newly-added layer is sufficiently trained, we repeat this process for the next added stage. Fig. 3b shows images and voxel grids at different resolutions. We implement this on-the-fly pruning operation using efficient custom CUDA kernels.
I We build on the kemels from http::/gihhahccm//yyy/wwwz22
We use the Minkowski Engine https//ghthbb.omm////D//A//nnnnn
5

---

In agreement with [3], we observe that it is crucial to account for view dependent effects or pose- correlated attributes in the training data, like eyes always looking into the camera. We take two measures to increase the flexibility of our model: Following [3], G is conditioned on the pose 5 which corresponds to the rendering pose Ç' in 50% of the cases and is randomly chosen otherwise. Formally, 5 ~ p=) At inference, the pose-conditioning fixed to retain 3D consistency. However, we find that pose conditioning alone is not always sufficient to account for strong cor- relations in the data. Depending on the dataset, we optionally refine the rendered image with a shallow 2D CNN with 2 hidden layers of dimension 16 and kernel size 3. While this refinement is powerful enough to model dataset biases, it is considerably less flexible than the neural rendering used in [4. 10, 30]: Our 2D CNN operates on the rendered image instead of rendered features at smaller resolution. Operating on 3 channels at full resolution allows to keep its capacity at a minimum and, due to not using any upsampling operation, results in a local receptive field.
Background Generator. We consider datasets with single object per image. Modeling only the object in 3D saves computation and is advantageous for potential downstream tasks, c.g. integrating generated assets into new environments Hence, we model the object in 3D and generate the back- ground of the image with 2D GAN. Specifically, we use the StyleGAN2 generator [19] with reduced channel size as modeling the background requires less capacity than generating the full image:
Gby I p3 Ri
(5)
We choose the same latent code z for foreground and background to allow for modeling correlations like lighting between the two. Note that, unlike the foreground generator, the background generator is not conditioned on the camera pose. As the background remains fixed when changing the camera pose, the generator is encouraged to model pose-dependent content with the foreground generator leading to disentanglement (see supp. mat. for qualitative disentanglement results).
The final image is obtained using alpha composition
i i= i mask:
i + (1
1 i's
The full generator defined as
G
TM X
X
R
X RKK
R (2,E,E', K) -
(6)
Regularization. For fast rendering, it is crucial that most generated voxels either fully opaque or empty such that early stopping and empty space skipping are effective. By regularizing the variance of the expected depth &, along each ray r, the foreground generator is encouraged to generate a single, sharp surface:
1
2 = Tha
Var(2) E,Tia
ETjas(z - 2)²
LDV ADV max(V aar()))
(8)
where is hyperparameter that defines the thickness of the surface. We find that thresholding the loss is important to avoid an empty foreground. In addition, we find that adding the grid TV regularization Lrv from [i] and fore- and background coverage regularization Lia and ang from [45] further stabilizes training (see sup. mat. for details). The full regularization term of our generator is
Lagg LDV + Lrv + L.o + Cha
Discriminator. We use the StyleGAN2 discriminator and condition it on the camera pose as proposed in [3]. Similarly, we find that conditioning guides the generator to learn correct 3D priors and canonical representation. Since rendering our sparse representation is fast, our discriminator is able to operate on the full image and does not need consider image patches as done in GRAF [38].
3.3 Training
Given images from the data distribution Pp with known camera extrinsics & and intrinsics K, and latent codes E (0, 1), we train our model using GAN objective with R RI-reuularization [26]
V V(0, Φ) = E [f(- [f(DD& (Go(z, &, (K).
(10)
+ EImpo (I, (I,
(11)

---

where f(t) = - log + exp(-t)) and À controls the strength of the RI-regularizer. Gg and De are trained with alternating gradient descent combining the GAN objective with the regularization terms:
min max
V(0, 6) + (
(12)
In practice, optimize the generator with non-saturating variant of Eq. (12) [9]. We train
approach with Adam [21] using batch size of 64 at grid resolution RG = 32, 64 and 32 at RG = 128. We use a learning rate of 0.0025 for the generator and 0.002 for the discriminator. For faster training, we first grow Ry from 32 to 128 while keeping RG at 32. Then, we alternately increase the grid resolution and the image resolution until the dataset resolution and RG 128 reached. For synthetic datasets, i.e. Carla [38], we do not add any refinement layers. Depending on the dataset, we train our models for days on Tesla V100 GPUs. Details on the network architectures can be found in the supplemental material.
4 Results
Datasets. We validate our approach standard benchmark datasets for 3D-aware image synthesis. The synthetic Carla dataset [8, 37] contains 10k images and camera poses of 18 car models with randomly sampled colors. FFHQ [19] comprises 70k aligned face images. AFHQv2 Cats [5] consists of 4834 cat faces. Following [4], we estimate camera poses for both datasets with off-the-shelf pose estimators [7. and augment all datasets with horizontal flips. Due the limited number of images in AFHQv2 and Carla, we use adaptive discriminator augmentation [17] for these datasets.
Evaluation Metrics. We measure image fidelity by calculating the Fréchet Inception Distance (FID) [14] between 20k generated images and the full dataset. For all runtime comparisons, we report times on a single Tesla V100 GPU with a batch size of 1.
4.1 Ablation Study
We investigate the sparsity of the generated voxel grids and validate the importance of the depth variance loss, see Eq. (8). Sparsity is evaluated by fusing the pruned voxel grids from 16 equally spaced camera views and reporting the number of empty voxels divided by the total number of voxels R.. Table shows results averaged over 100 instances. The depth variance loss increases sparsity from 74% to 95%, lowering the memory consumption. With the sparser representation, the times needed for scene generation Lato-hh and rendering f,r are reduced significantly. We further compare an implementation with dense instead of sparse convolutions where we zero out the values of pruned voxels. While this increases memory consumption, it reduces the runtime due to the computational overhead of managing coordinates for sparse convolutions. We prioritize faster training over memory and hence train our models with dense convolutions where we set values of pruned voxels to zero.
Sparsity
Memory
fGro-ty
1
[%]
[GB]
[ms]
[ms]
w/o LDv
74
1.4
229
W LDV
95
0.9
198
4
W LDY!
95
1.1
58
Table 1: Regularization. We compare sparsity and rendering time with and without depth vari- ance regularization LDy on FFHQ with RG 128 and R2 128. denotes an architecture with dense convolutions where values of pruned voxels are set to zero.
4.2 Baseline Comparison
Baselines. We group the baselines into two categories: (i) methods that render low-resolution features and use 2D-upsampling, i.c. a neural renderer, obtain the final image, e.g. StyleNeRF [10], and (ii) methods that render the 3D representation directly at the final image resolution, e.g. .
GAN [4]. VoxGRAF falls into the second group. We use the official code release for all methods. Further, we reference numbers reported by concurrent works GRAM [6] and EG3D [3].
Qualitative Results. Fig. 4 shows samples from multiple views for StyleNeRF, GRAM and VoxGRAF on FFHQ at resolution 250 StyleNeRF can generate additional faces or strands of hair across views, as indicated by the red frames in Fig. 4. These inconsistencies under viewpoint changes
7

---

StyleNeRF
VoXGRAF GRAM
Figure 4: Qualitative Comparison on Multi-View Consistency. compare multí-view consistency of state-of-the-art baselines and method. While StyleNeRF's powerful neural renderer can introduce multi-view inconsistencies (appearing faces, moving strands of hair), GRAM's manifold representation becomes visible in layered artifacts for steeper viewing angles. In contrast, our method leads to more multi-view consistent results.
HHHQ [19] APHQ Carla [38]
R₁
128"
R₁ 256
R₁ 256
256
Rí
1282
GIRAFFE [30]
5
GIRAFFE [3]]
31.5
16.1
StyleNeRF [10]
49
VolumeGAN [44]
9.1
7.9
EG3D [3]
27
StyleNeRF [10]
8.0
GRAF [3
219
878
EG3D [3]
4.8
3.9
--GAN [4]
154
608
GRAF [38
41
GOF [43]
199
742
T-GAN [4]
47
29.2
GRAM [6]
136
GOF [43]
69.2
54.1
29.3
VoxGRAF
58 38+3
58 + 6
GRAM [6]
17.9
18.5
26.3
VoxGRAF
9.6
9.6
6.7
Table 3: Rendering times. We report time in ms per image. Note that our method allows for separating scene generation (first number) and rendering (second number) which is useful for real-time rendering applications. *EG3D is evalu- ated on faster GPU (RTX 3090 GPU) compared to the others (Tesla V100 GPU).
Table 2: Quantitative Comparison. We com- pare against state-of-the- art methods with neu- ral rendering pipeline (first block) and without one (second block). We report FID [14] between 20k generated images and the full dataset.
are introduced by StyleNeRF's powerful neural renderer. GRAM achieves more consistent results, but its plane representation creates stripe artifacts for large viewpoint ranges. Due the shallow 2D CNN, VoxGRAF can model the dataset bias of eyes looking into the camera but otherwise achieves high 3D-consistency even under large viewing angles. Additional samples of our method and the corresponding sparse voxel grids for all datasets are provided in Fig. and Fig. 5.
Quantitative Results. Table 2 reports FID on all datasets. As expected, methods that use neural rendering with upsampling, i.e., StyleNeRF and EG3D, perform best in terms image fidelity. This is expected as neural renderer can add flexibility. But, as shown in Fig. 4, for StyleNeRF it reduces 3DDcconisteency. Among methods without a neural renderer, VoxGRAF significantly improves over T T--GAN and GOF and surpasses the current state-of-the-an approach GRAM.

---

Figure 5: Qualitative Results for Method. We show generated images resolution 256 for FFHQ [19] and AFHQ [5] and samples at resolution 128 for Carla [38].
Runtime Comparison. Lastly, we compare the rendering times at inference for methods with and methods without neural renderer. In contrast to all baselines, VoxGRAF requires only a single forward pass to generate scene, which can then be rendered from different viewpoints efficiently. Note that at inference, voxels are pruned solely based their density to amortize the rendering costs per scene. We find that this does not visibly affect the rendered images. We report the times for generating the scene and rendering GIRAFFE, is the fastest among all approaches as renders a low-resolution feature volume and uses comparably small neural networks. StyleNeRF significantly increases the neural renderer's 's size to improve image fidelity, which comes at the cost of speed compared to GIRAFFE. Yet. StyleNeRE the fastest approach for generating single image among the peeeeerrfooinggggg methods. However, potential application of 3D-aware GANs is generating novel views of a single instance real-time. In this setting, resolution 256². VoxGRAF generates novel views 167 FPS, whereas StyleNeRR runs at 20 FPS as its rendering costs are not amortized per scene.
view separately in Table 3. One of the earliest works,
Limitations and Discussion
In this work, we investigate sparse voxel grids as representation for 3D-aware image synthesis. We find that the key to generating sparse voxel grids is to combine progressive growing, pruning, and regularization to encourage a sharp surface that can be rendered efficiently. Our approach outperforms all methods that do not employ neural renderer. Instead of discarding neural rendering entirely, we find it advantageous to utilize shallow CNN for refinement. This CNN can model dataset bias but is significantly weaker than standard neural rendering approaches that upsample low resolution feature maps. Our approach can reduce the gap to models that build heavily on neural rendering, yet trade-off between 3D-consistency and image fidelity remains. Whether certain amount of neural rendering inherently needed to reach best performance is an important direction for future research. Lastly, the speed of our method depends on the sparsity of the modeled scene. Therefore, rendering times will likely increase on more complex datasets than those commonly used in literature.

---

Acknowledgments and Disclosure of Funding
We acknowledge the financial support by the BMWi in the project KI Delta Learning (project number 19A190130) the support from the BMBF through the Tuebingen AI Center (FKZ:01IS) 18039A), and the support of the DFG under Germany's 's Excellence Strategy (EXC number 2064/1 Project number 390727645). Andreas Geiger and Michael Niemeyer were supported by the ERC Start- ing Grant LEGO-3D (850533). We thank the International Max Planck Research School for In- telligent Systems (IMPRS-IS) for supporting Katja Schwarz and Michael Niemeyer. This work was supported by an NVIDIA research gift. We thank Christian Reiser for the helpful discus- sions and suggestions. Lastly, would like to thank Nicolas Guenther for his general sup- port.
References
[1] Alex Yu and Sara Fridovich-Keie M. Tancik, Q. Chen, B. Recht, and A. Kanazawa. Plenoxels: Radiance fields without neural networks. Proc. IEEE Conf. on Computer Vision and Putlern Recognition (CVPR), 2022. 2, 3, 5,
[2] A. Brock, J. Donahue, and K. Simonyan. Large scale GAN training for high fidelity natural image synthesis. In Proc. of the International Conf. on Learning Representations (ICLR), 2019.
1.3
[3] E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. D. Mello, O. Gallo, L. Guibas, J. Tremblay, S. Khamis, T. Karras, and G. Wetzstein. Efficient geometry- 3D generative adversarial networks. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR),
2022. 2, 3, 4, 5, 6,8
[4] E. R. Chan, M. Monteiro, P. Kellnhofer, J. Wu, and G. Wetzstein. Pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proc. IEEE Conf. on Compuler Vision and Pattern Recognition (CVPR), 2021. 1, 3, 6, 8
[5] Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha. Stargan v2: Diverse image synthesis for multiple domains.
In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 7, 8, 9
[6] Y. Deng, J. Yang, J. Xiang, and X. Tong. Gram: Generative radiance manifolds for 3d-aware image generation. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR),
2022. 2, 3, 4, 8
[7] Y. Deng, J. Yang, S. Xu, D. Chen, Y. Jia, and X. Tong. Accurate 3d face reconstruction with weakly-superrised learning: From single image image set. In Proc. IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR) Workshops, 2019. 7
[8] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, CARLA: An open urban driving simulator. In Proc. Conf. on Robot Learning (CoRL), 2017. 7 [9] J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS), 2014. 1,
[10] J. Gu, L. Liu, P. Wang, and C. Theobalt. Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis. Proc. of the International Conf. on Learning Representations
(ICLR), 2022. 2, 3, 6, 7,8
[11] J. Gwak, C. B. Choy, and S. Savarese. Generative sparse detection networks for 3d single-shot object detection. In Proc. of the European Conf. on Computer Vision (ECCV), 2020. 5
[12] E. Härkönen, A. Hertzmann, J. Lehtinen, and S. Paris. Ganspace: Discovering interpretable
GAN controls. arXiv.org, 2020. 3
[13] P. Henzler, N. J. Mitra, and T. Ritschel. Escaping plato's cave: 3d shape from adversarial rendering. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019. 1, 2,
[14) M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information
Processing Systems (NIPS). 2017. 7, 8
[15] K. Jo. G. Shim, S. Jung. S. Yang, and J. Choo. Cg-nerf: Conditional generative neural radiance
fields. arXiv.org, 2021. 3
[16] T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In Proc. of the International Conf. on Learning Representations (ICLR),
2018. 1, 3,
10

---

[17] T. Karras, M. Aittala, J. Hellsten. S. Laine. J. Lehtinen, and T. Aila. Training generative adversarial networks with limited data, In Advances in Neural Information Processing Systems (NeurIPS), 2020. 1,7
[18] T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, and T. Aila. Alias- free generative adversarial networks. In Advances in Neural Information Processing Systems
(NeurIPS), 2021. 3
[19] T. Karras, S. Laine, and Aila. A style-based generator architecture for generative adversarial networks. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2019. 1,
3, 6. 7. 8,
[20] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving
the image quality of StyleGAN. Proc. IEEE Conf. on Computer Vision and Pattern Recognition
(CVPR), 2020. 1,
[21] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proc. of the International Conf. on Learning Representations 2015. 7
2015.
[22] B. Lee. Cat hipsterizer, 2018. 7
[23] Y. Liao, K. Schwarz, Mescheder, and A. Geiger. Towards unsupervised learning of generative models for 3d controllable image synthesis. In Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2020. 1,
[24] H. Ling, K. Kreis, D. Li, S. W. Kim, A. Torralba, and S. Fidler. Editgan: High-precision semantic image editing. In Advances in Neural Information Processing Systems (NeurIPS),
2021. 3
[25] L. Liu, J. Gu, K. Z. Lin, T. Chua, and C. Theobalt. Neural sparse voxel fields. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 3
[26] L. Mescheder, A. Geiger, and S. Nowozin. Which training methods for gans do actually converge? In Proc. of the International Conf. on Machine learning (ICML), 2018, 3, 6 [27] B. Mildenhall,
Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing scenes neural radiance fields for view synthesis. In Proc. of the European Conf. on Computer Vision (ECCV), 2020. 2, 3,
[28] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Trans. on Graphics, 2022. 3
[29] T. Nguyen- Phuoc, C. Li, L. Theis, C. Richardt, and Y.-L. Yang. Hologan: Unsupervised learning of 3d representations from natural images. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019. 1, 3,
[30] M. Niemeyer and A. Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
1, 2, 6,
[31] M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 3.8
[32] R. Or-El, X. Luo, M. Shan, E. Shechtman, J. Park, and I. Kemelmacher. Stylesdf: High- resolution 3d-consistent image and geometry generation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022. 3
[33] X. Pan, X. Xu, C. C. Loy, C. Theobalt, and B. Dai. A shading shading-guided guided generative implicit model for shape-accurate 3d-aware image synthesis. In Advances in Neural Information Processing
Systems (NeurIPS), 2021. 3
[34] C. Reiser, S. Peng. Y. Liao, and A. Geiger. Kilonerf:
Speeding up neural radiance fields with thousands of tiny mlps. In Proc. of the IEEE International Conf. on Computer Vision (ICCV),
2021. 3
[35] A. Sauer, K. Chitta, J. Müller, and A. Geiger. Projected gans converge faster. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 1
[36] A. Sauer, K. Schwarz, and A. Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets.
ACM Trans. on Graphics, 2022. 1,3
[37] K. Schwarz, Y. Liao, M. Niemeyer, and A. Geiger. GRAF: generative radiance fields for 3d-aware image synthesis. Advances in Neural Information Processing Systems (NIPS), 2020.
3.7
[38] K. Schwarz, Y. Liao, M. Niemeyer, and A. Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 2, 3,
6, 7, 8.
11

---

[39] Y. Shi, D. Aggarwal, and A. K. Jain. Lifting 2d stylegan for 3d-aware face generation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
[40] C. Sun, M. Sun, and H.-T. Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022. 3,4
[41] T. Takikawa, J. Litalien, K. Yin, K. Kreis, C. Loop, D. Nowrouzezahrai, A. Jacobson, M. McGuire, and S. Fidler. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021. [42] M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich-Keie N. Raghavan, U. Singhal, R. Ra- mamoorthi, J. Barron, and R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[43] X. Xu, X. Pan, D. Lin, and B. Dai. Generative occupancy fields for 3d surface-aware image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 3, 4,
[44] Y. Xu, S. Peng, C. Yang. Y. Shen, and B. Zhou. 3d-aware image synthesis via learning structural and textural representations. Proc. IEEE Conf. on Computer Vision and Pattern Recognition
(CVPR), 2022. 3, 8
[45] Y. Xue, Y. Li, K. K. Singh, and Y. Lee. GIRAFFE HD: A high- high-resolution 3d-aware generative model. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022. 6 [46] A. Yu, R. Li, M. Tancik, H. Li, R. Ng, and A. Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2021.
[47] J. Zhang, E. Sangineto, H. Tang, A. Siarohin, Z. Zhong, N. Sebe. and W. Wang. 3d-aware semantic-guided generative model for human synthesis. arXiv.org, 2021. 3
[48] Y. Zhang, W. Chen, H. Ling. J. Gao, Y. Zhang, A. Torralba, and S. Fidler. Image gans meet differentiable rendering for inverse graphics and interpretable 3d neural rendering. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
[49] Zhou, L. Xie, B. Ni, and Q Tian. CIPS-3D: 3D-Aware Generator GANs Based on Conditionally-Indeeeenn
Pixel Synthesis. arXiv:org, 2021.
[50] J.-Y. Zhu, Z. Zhang, C. Zhang. J. Wu, A. Torralba, J. B. Tenenbaum, and W. T. Freeman. Visual object networks: Image generation with disentangled 3D representations. Advances in Neural
Information Processing Systems (NIPS), 2018. 3
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See supplemental material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] We will provide
code upon acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] Multiple runs are computationally infeasible for our models.
12

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We include total training time and the type of GPU we used.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you's 're
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "wDgSHSERWD", "reviewer": "Reviewer_2xja", "summary": "The paper proposes a novel 3D GAN architecture for rendering of 3D consistent objects supervised by collection of single-view images. The key innovation is the introduction of sparse voxel grid as a more computationally efficient representation when compared to the most commonly used MLPs. The authors describe how sparsity is enforced and used to reduce memory constraints of their approach. As a result, they can render medium-sized images from their voxel grid directly without need to explicit upsampling. The validation shows the method produces better images at lower computation cost than previous work and it is in some respects competitive with more recent methods published in parallel.", "questions": "11) If voxels are pruned based on visibility in \"a\" rendered view (L167), cannot they be later missing after occlusion changes due to a new viewpoint at test time?\n", "limitations": "12) The authors have mentioned the performance may drop is scene is more complex and pruning is not very effective. Other limitations I suspect is the lack of geometric smoothness in extracted shapes and the high training cost.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n[Originality] \n\n1) The method combines an established concept of NeRF-based 3D-GAN with previously described sparse voxel convolutions. The authors further describe several key tricks that are required for successful and efficient training. Many of them are inspired by similar work, but at least the loss L_D seems unique to this publication. Overall, the technical contribution is incremental but, nevertheless, it leads to a significant improvement upon SOTA in the target application (here I ignore the parallel work).\n\n\n[Clarity]\n\n2) The exposition is very clear, easy to read and I have not discovered many typos or grammar issues.\n\n3) I find problematic the way how parallel work is discussed. GRAM, StyleNERF, and EG3D are all parallel work so they do not count against contributions of this paper. I appreciate that the authors tacked the task of discussing their properties and that they included them in their result tables. However, I am not entirely sure all conclusions that they have made are fully substantiated. For instance, L264 seem to suggest that EG3D is less view consistent than the proposed method but no such effect is shown. Both methods use internal 128px representation for the radiance field and therefore, any further processing increasing the output image resolution is in principle similar for both. The authors also argue, that the advantage of their method is the possibility of faster rendering after the first frame (Table 3). However, the same could be said about EG3D as it also could precompute the StyleGAN representation and then just sample the features the same way the new paper does. In general, I find these over-extended assumptions about parallel work both not properly validated and also unnecessary. As these work are not counted as a previous work, there is no need to prove that the new method is in all or any respects better. The authors should rather focus on properties of their own work and show why it is excellent. Since the general architecture and performance appears quite similar, focusing on the possibilities of the explicit voxel representation seem like a clear goal.\n\n[Validation]\n\n4) The method is evaluated on common datasets and the results are clearly better than previous work and comparable to parallel work. The results are limited to 256px resolution with is also comparable to recent methods.\n\n5) There are no quantitative experiments done to validate the claim of better geometry and 3D consistency (L74). The authors could also attempt to measure depth and pose accuracy as done in previous work [4]. Since there is still a small CNN used for the image refinement it does not seem obvious that the method is perfectly view consistent.\n\n6) The extracted shapes appear quite noisy (Fig. 5). \n\n7) The authors only ablate the loss L_DV and only in terms of sparsity. Neither of the losses that are used for regularization (there are 4 in total) is evaluated in terms of end-to-end performance impact.\n\n8) The training is extremely costly even compared to MLP-based NeRF GANs (L224).\n\n\n[Significance]\n\n9) The paper tackles a relevant problem of 3D representation generation and it does take a path distinct from other work. While it is not clear whether the method actually is clearly better than some of the parallel work, it does clearly beat the previous baselines. The voxel representation may have practical benefits that have not been mentioned in the paper. E.g. I can imagine that they would be easier to edit in a 3D modeler. However, no such application has been shown.\n\n\n[Conclusion]\n\n10) While I see some issues with the exposition, the paper is of a high quality overall. The discussion of the performance and relation to other work can be addressed. The bigger concern is the small scope of the validation. While the most critical image synthesis quality is directly evaluated on common datasets, this leads to a total of three numbers reported in Table 2. The 3D consistency, design ablation (with exception of Table 1) and the shape quality are not directly validated in a quantifiable way. That being said, I do not suspect such experiments to significantly change the narrative of the paper so I am still inclined towards acceptance, especially if the authors can improve upon these aspects in the final version.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "m69Da5QZDRK", "reviewer": "Reviewer_xKgP", "summary": "Authors propose an approach to generate high quality images from a latent code that are multi-view consistent. The intuition is to replace the MLP that is used to parameterized radiance fields, with a hybrid approach that relies on sparse 3D convolutions to speed up the rendering time. Additionally, they propose a way to disentangle the foreground and background.", "questions": "The main question is around novelty and experiments/comparisons. I'd like authors to convince me that this paper introduces some different capability compared to [4] or that they solve the problem better. In the current form, I do not think the paper is doing a good job at convincing the reader that this method is pushing the state of art.\n\n---\nPost rebuttal: Authors made the fair point that [4] was concurrent work and there was no code release at the time of the submission. Hence I am leaning towards acceptance.", "limitations": "Yes, discussed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "++ Relevant problem for the community. Generating high quality renderings with an adversarial approach is an unsolved problem despite the recent progress\n\n++ Clarity: the paper is overall clear, authors put lots of effort to describe every details of the approach. They sufficiently discuss limitations and ethical concerns.\n\n\n-- Novelty, Contributions and Experiments. Whereas the general idea is interesting, the first thing that comes to mind once reading this paper is: why is this approach better than EG3D [4]? Indeed, they tackle the same problem and I would argue that the results of EG3D are actually better. Quantitative comparisons that author report seem to be aligned with this, and qualitative experiments are not showed (I would encourage authors to report those). According to authors, the main difference between their approach is explained at l142-l143, which is \"In contrast to recent works, we do not use a coordinate-based MLP..\". While this is true, it is not demonstrated that the proposed method is actually better. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SN1g22Q9_fM", "reviewer": "Reviewer_eE7o", "summary": "The paper introduces a 3D-aware image generation network that usess a Plenoxels-style voxel grid [1] as its scene representation. The complete network architecture consists of a sparse 3D CNN generator with novel upsampling layers that ensure sparsity of the upsampled densities. The full resolution voxel grid has size $128^3$ and stores RGB+density values for the occupied voxels. \n\nIn a parallel branch, there is a 2D GAN (following StyleGAN v2) that generates the scene background.\n\nThe sparse voxel grid is rendered at the required output resolutoin using conventional volume rendering techniques. The foreground is post-processed by a shallow (two hidden layers) refinement CNN, whose output is then composited on the the background image. \n\nAt training time, the resulting image is passed to a discriminator based on StyleGAN2. The authors describe several regularizing losses, including a novel sparsity term on the voxel densities.\n\nThe authors report both quantitative and qualitative comparisons to state-of-the-art 3d-aware GANs and demonstrate results that demonstrate a high degree of geometric consistency across viewpoints. \n\n[1] Yu et al. Plenoxels: Radiance Fields without Neural Networks. 2022", "questions": " - See previous section for questions about voxel & generator view dependence.\n - What is the speedup gained from using sparse convolutions at grid sizes > $32^3$?", "limitations": "The authors are clear about the limitations of their work. Societal impacts are adequately discussed.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a strong paper with an original technical contribution in the sparse CNN generator and the use of a sparse RGB-density voxel grid as the representation for a 3D-aware GAN. The voxel visualizations in the paper, as well as the animated sequences in the supplemental video, demonstrate the geometric plausibility of the generated representations.\n\nThe paper is very well written and the discussion of related works is thorough. The comparisons to related work are quite comprehensive, and a key ablation study is included.\n\nA particularly exciting aspect of the method is the very fast rendering times (after a slower initial generation of the voxel grid), which could allow this technique to run at interactive framerates on low-powered platforms.\n\n\n--\n\nEven though the results exhibit high geometric consistency, the visual quality of the generated still frames does not quite match that of some other methods. This limitation is acknowledged in the paper.\n\nMy main critique is around the description of viewpoint dependence. Plenoxels stores spherical harmonic coefficients in the voxels, while the proposed method seems to store RGB values without directional dependence (Eq. 3.). This is a quite significant departure from other NeRF-inspired representations, and should be made more explicit in the discussion of the method (Section 3.1).\n\nIt is somewhat surprising that the foreground generator is conditioned on the camera pose $\\xi$, since the volumetric grid can be rendered from arbitrary viewpoints $\\xi'$, even those that it was not conditioned on. Could the authors clarify for which results in the paper $\\xi = \\xi'$ ? Would it be a viable ablation to compare two animated sequences, one with $\\xi = \\xi'$ at each frame, and one where the generator was invoked only once?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept