EpiGRAF: Rethinking training of 3D GANs
Authors: Ivan Skorokhodov
Sergey Tulyakov
Yiqun Wang
Peter Wonka
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3D GANs
neural radiance fields
nerf
patch-wise training

Abstract: A recent trend in generative modeling is building 3D-aware generators from 2D image collections. To induce the 3D bias, such models typically rely on volumetric rendering, which is expensive to employ at high resolutions. Over the past months, more than ten works have addressed this scaling issue by training a separate 2D decoder to upsample a low-resolution image (or a feature tensor) produced from a pure 3D generator.  But this solution comes at a cost: not only does it break multi-view consistency (i.e., shape and texture change when the camera moves), but it also learns geometry in low fidelity. In this work, we show that obtaining a high-resolution 3D generator with SotA image quality is possible by following a completely different route of simply training the model patch-wise. We revisit and improve this optimization scheme in two ways. First, we design a location- and scale-aware discriminator to work on patches of different proportions and spatial positions. Second, we modify the patch sampling strategy based on an annealed beta distribution to stabilize training and accelerate the convergence. The resulting model, named EpiGRAF, is an efficient, high-resolution, pure 3D generator, and we test it on four datasets (two introduced in this work) at \(256^2\) and \(512^2\) resolutions. It obtains state-of-the-art image quality, high-fidelity geometry and trains \({\approx}\)2.5 faster than the upsampler-based counterparts. Code/data/visualizations: https://universome.github.io/epigraf.

EpiGRAF: Rethinking training of 3D GANs
Ivan Skorokhodov KAUST
Sergey Tulyakov Snap Inc.
Yiqun Wang KAUST
Peter Wonka KAUST
Abstract
A recent trend in generative modeling is building 3D-aware generators from 2D image collections. To induce the 3D bias, such models typically rely volumetric rendering. which expensive to employ at high resolutions. Over the past months, more than ten works have addressed this scaling issue by training separate 2D decoder to upsample low-resolution image (or a feature tensor) produced from pure 3D generator. But this solution comes at cost: not only does it break multi-view consistency (i.e., shape and texture change when the camera moves), but it also learns geometry in low fidelity. In this work, show that obtaining high-resolution 3D generator with SotA image quality possible by following a completely different route of simply training the model patch-wise. We revisit and improve this optimization scheme in two ways. First, we design a location- and scale-aware discriminator work on patches of different proportions and spatial positions. Second, we modify the patch sampling strategy based on an annealed beta distribution stabilize training and accelerate the convergence. The resulting model, named EpiGRAF, is an efficient, high resolutso,,,,n,, pure 3D generator, and we test it four datasets (two introduced in this work) at 256² and 5122 resolutions. It obtains state-of of-thh-art image quality, high-fidelity geometry and trains #2.5x faster than the upsampler- based counterparts.
Code/dataavissualizzaiiss https://ahheenommmgghbbbb.pgpgp
1 Introduction
T
We build pure NeRe-bbased generator trained in patch-wise fashion. Left two grids: samples on FFHQ 512" 25 and Cats 256 (77). Middle grids: interpolations between samples M-Plants and M-Food (upper) and corresponding geometry interpolations (lower). Right grid: background separation examples. In contrast to the upsampler-basee methods, one can naturally incorporate the techniques from the traditional NeRF literature into our generator: for background separation, we simply copy-pasted the corresponding code from NeRF++ [76].
Figure 1:
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Generative models for image synthesis achieved remarkable success in recent years and now enjoy
lot of practical applications [55] 24 While initially they mainly focused on 2D images [2] 66 251
4
recent research explored generative frameworks with partial 3D control over the underlying object in terms of texture/structure decomposition, novel view synthesis or lighting manipulation These techniques are typically built on top of the recently emerged
28
(e.g., 58 56 E 68
G 12
49)
neural radiance fields (NeRF)
38 to explicitly represent the object (or its latent features) in 3D
space.
NeRF is a powerful framework, which made possible to build expressive 3D-aware generators from challenging RGB datasets 7 12 [1. Under the hood, it trains multi-layer perceptron (MLP) F(x:; d) (c, or) to represent scene by encoding a density a E R+ for each coordinate position E I and color value c E I2 from æ and view direction d E S [N To synthesize an image, one renders each pixel independently by casting a ray r(4) o + qd (for a E R2,) from origin
E IR. into the direction d E S and aggregating many color values along it with their corresponding densities. Such a representation is very expressive but comes at cost: rendering single pixel is computationally expensive and makes it intractable to produce a lot of pixels in one forward pass. It is not fatal for reconstruction tasks where the loss can be robustly computed on a subset of pixels, but it creates significant scaling problems for generative NeRFs: they are typically formulated in GAN-based framework
[ with 2D convolutional discriminators requiring an entire image as input.
People address these scaling issues of NeRF-basee GANs in different ways. The dominating approach train a separate 2D decoder to produce a high-resolution image from a low-resolution image or feature grid rendered from a NeRF backhone 43 During the past six months, there appeared more
than a dozen of methods that follow this paradigm (e.g.. [6 L. 47 35 75 23
72
78 64))
While using the upsampler allows scaling the model to high resolution, it comes with two severe limitations: 1) it breaks the multi-view consistency of a generated object, i.e., its texture and shape change when the camera moves; and 2) the geometry gets only represented in a low resolution (x>64",, In our work, we show that by dropping the upsampler and using a simple patch-wise optimization scheme, one can build a 3D generator with better image quality, faster training speed, and without the above limitations.
Patch-wise training of NeRF-based GANs was initially proposed by GRAF [56] and got largely neglected by the community since then. The idea is simple: instead of training the generative model on full-size images, one does this on small random crops. Since the model is coordinate-based (59] 65) does not face any issues to synthesize only subset of pixels. This serves as an excellent way to save computation for both the generator and the discriminator since it makes them both operate on patches of small spatial resolution. To make the generator learn both the texture and the structure, crops are sampled to be of variable scales (but having the same number of pixels). In some sense, this can be seen as optimizing the model on low-resolution images + high-resolution patches.
In our work. we improve patch-wise training in two crucial ways. First, we redesign the discriminator by making it better suited to operating on image patches of variable scales and locations. Convolu- tional filters of a neural network learn to capture different patterns in their inputs depending on their semantic receptive fields 30, 46 That's why it is detrimental to reuse the same discriminator to judge both high-resolution local and low-resolution global patches, inducing additional burden on it to mix filters' responses of different scales. To mitigate this, we propose to modulate the discriminator's filters with a hypemetwork [16). which predicts which filters to suppress or reinforce from given patch scale and location.
Second, we change the random scale sampling strategy from an annealed uniform to an annealed beta distribution. Typically, patch scales are sampled from a uniform distribution s ~ Z[[s(t), ()), 1] 15 36. B. where the minimum scale s(t) is gradually decreased (i.c. annealed) till some iteration T from x(0) 0.9 to a smaller value s(T) (in the interval [0.125 0.5)) during training. This sampling strategy prevents learning high-frequency details early on in training and puts too little attention on the structure after st)) reaches its final value sT). This makes the overall convergence of the generator slower and less stable that's why we propose to sample patch scales using the beta distribution (eta(1, (()) instead, where 3(1) is gradually annealed from (()) R 0 to some maximum value ((T). In this way, the model starts learning high-frequency details immediately with the start of training and focuses more on the structure after the growth finishes. This simple change stabilizes the
training and allows it to converge faster than the typically used uniform distribution [56] 5 36)

---

EG3D
Ours
Figure 2: Comparing the geometry between EG3D 6 and our generator on FFHQ 512?. For each method, we computed the density field in the 512 volume resolution and extracted the surfaces using marching cubes. The geometry of our generator contains more high-frequency details (e.g.. hair strands are better separated) since it learns it in full resolution. EG3D uses the 642 rendering resolution (and 128² during the last 10% of the training) so its shapes appear over-smoothed.
We use those two ideas develop novel state-of-the-art 3D GAN: Efficient patch-informed Generative Radiance Fields (EpiGRAF). We employ it for high-resolution 3D-aware image synthesis on four datasets: FFHQ (25), Cats 1771, Megascans Plants, and Megascans Food. The last two benchmarks are introduced in our work and contain 360 renderings of photo-realistic scans of different plants and food objects (described in $ They are much more complex in terms of geometry and are well-suited for assessing the structural limitations of modern 3D-aware generators.
Our model uses pure NeRF-based backbone, that's why it represents geometry in high resolution and does not suffer from multi-view synthesis artifacts, as opposed to upsampler-based generators. Moreover, it has higher comparable image quality (as measured by FID 20) and 2.5x lower training cost. Also, in contrast to upsampler-bases 3D GANs, our generator can naturally incorporate the techniques from the traditional NeRF literature. To demonstrate this, incorporate background separation into our framework by simply copy-pasting the corresponding code from NeRI++ 766
2
Related work
Neural Radiance Fields. Neural Radiance Fields (NeRF) is emerging area 38 which combines
neural networks with volumetric rendering techniques to perform novel-view synthesis 38
76
2],
image-to-scent generation 74) surface reconstruction 45 69 iZ
and other tasks 9
17
50
In
work, we employ them in the context of 3D-aware generation from a dataset of RGB images
556 /
3D generative models. A popular way to learn 3D generative model is to train it on 3D data or in
29 This requires explicit 3D supervision and there appeared methods which train from RGB datasets with segmentation masks, keypoints or multiple object views 13
an autoencoder's latent space (c.g., [[C 70 44
Bl
39
Recently, there appeared works which train from hingle-viww and methods that extract 3D structure And recent neural rendering advancements allowed to train from purely RGB data from scratch, which became the dominating direction since then and which are typically formulated the GAN-based framework [4]
32
RGB only, including mesh-generation methods 19 5
from pretrained 2D GANs 55% 48
NeRF-based generators [56
ip
NeRF-based GANs. HoloGAN 41
generates 3D feature voxel grid which is projected on
plane and then upsampled. GRAF
56
trains noise-conditioned NeRF in an adversarial manner.
T-GAN 7 builds upon it and uses progressive growing and hyypernetwork-basee [6 conditioning

---

(00.7)
5
normaloz
M
ModDisrBlock
Convde
cont
Systa
Synchbllock
Litsrr
Linear
MedDDerrflnek
Convzd
Convid
downsomple
downsomple
Modlbeerfleek
PorEn:
Eplogue
SynthBlack 5120
Linear
real/fake
M
network
H hypernetwek
postiorall wanoding
licifer
useration
volumatric
rendering
patch scales/ffees
disribatio
Figure 3: Our generator (left) is purely NeRF-based and uses the tri-plane backbone [6] with the StyleGAN2 126 decoder (but without the 2D upsampler). Our discriminator (right) is also based on StyleGAN2, but is modulated by the patch location and scale parameters. We use the patch-wise optimization for training [56] with our proposed Beta scale sampling, which allows our model to converge x2-3 faster than the upsampler-ber based architectures despite the generator modeling geometry in full resolution (see Tab II.
in the generator. GRAM 12 builds on top of π T-GAN and samples ray points on a of learnable iso-surfaces. GNeRF 33 adapts GRAF for learning a scene representation from RGB images without known camera parameters. GIRAFFE 4 uses a composite scene representation for better controllability. CAMPARI 32) learns a camera distribution and background separation network with inverse sphere parametrization (76.. To mitigate the scaling issue of volumetric rendering, many recent works train 2D decoder under different multi view consistency regularizations to upsample low-resolution volumetrically rendered feature grid K M1 79 72 r
However, none of such regularizations can currently provide the multi-view consistency of pure-NeRF-basee generators.
Patch-wise generative models. Patch-wise training had been routinely utilized learn the textural component of image distribution when the global structure is provided from segmentation masks, sketches, latents or other sources (c.g.. 22 57
(T). Recently, there appeared works which sample patches at variable scales, in which way patch can carry global information
about the whole image. Recent works use to train generative NeRF (56), fit neural representation
in an adversarial manner 36 or to train 2D GAN on a dataset of variable resolution [5]
3 Model
We build upon StyleGAN2 P
replacing its generator with the tri-plane-based NeRF model
and using its discriminator as the backbone. We train the model on r x r patches (we use r = 64 everywhere) of random scales instead of the full images of resolution R R. Scales [ randomly sampled from a time-varying distribution s ~ (s)...
3.1 3D generator
Compared to upsampler-based 3D GANs 15
78 we use a pure NeRF 6K as our as the backbone. It consists of three components: 1) mapping network M z + w which transforms a noise vector 2 ~ g512 into the latent vector
72 79
generator G and utilize the tri-plane representation
5812 2) synthesis network S w + P which takes the latent vector w and synthesizes three 52-dimensional feature planes P = (P Py= yz. Paz) of resolution Rp Rp (i.c. P(.)- E Rt x R. x32); 3) tri-plane decoder network F P) + (c, o) E IR, which takes the space coordinate E R and tri- tri-planes P as input and produces the RGB color density value a E 24 that point by interpolating the tri- tri-plane features in the given coordinate and processing them with a tiny MLP. In contrast to classical NeRF [S), we do not utilize view direction conditioning since it worsens multi-view consistency in GANs, which are trained on RGB datasets with single view per instance. To render single pixel, we follow the classical volumetric rendering pipeline with hierarchical sampling 33 Z using ray steps in coarse and 48 in fine sampling stages. See the accompanying source code for more details.
R and

---

5
Arewaling progress: 0.0%
Arnealing progress: 26.7%
Annealing progress 13.3%
Arrweling prograss: 0.0%
Arnealing progress: 26.7%
Annesiing progress 13.3%
2.5
2.0-
Betal...3)
0.231
Anncaling progress. 40.0%
Anncaling progrese 40.0%
Betar1, 6.9)
Betafl, 1)
1.0-
0.2
0.5 0.8
3.0 0.2
0.5 0.8 3.0 0.0
0.2
0.4
0.5
0.8
1.0
Minimum scale
Minimum
Figure 4: Comparing uniform (left) and beta (middle) annealed patch scale sampling in terms of their probability density function (PDF) (for visualization purposes, we clamp the maximum density value to 5); (right) PDF of (e,a(1133). provided for completeness. Uniform distribution with annealed Smin(0) (0) = 0.9 from 0.9 to 6 mmn (T) = 0.125 does not put any attention to
details in the beginning and treats small-scale and large-scale patches equally at the end of the annealing. Beta distribution with annealed (0) x 0 to ((T) x 1, in contrast, lets the model learn high-resolution texture immediately after the training starts, and puts more focus on the structure the end.
3.2 2D scale/ocaatinn-wwrr discriminator
Our discriminator D is built top of StyleGAN2 26 Since train the model in patch-wise fashion, the original backbone not well suited for this: convolutional filters are forced to adapt to signals of very different scales and extracted from different locations. A natural way to resolve this problem is to use separate discriminators depending on the scale, but that strategy has three limitations: 1) each particular discriminator receives less overall training signal (since the batch size is limited); 2) from an engineering perspective, is more expensive to evaluate a convolutional kernel with different parameters on different inputs: 3) one can use only a small fixed amount of possible patch scales. This is why we develop novel hypernetwork-moduue architecture operate on patches with continuously varying scales.
[6. 62 discriminator
To modulate the convolutional kernels of D, we define hypernetork H (s, 6z, 6g) : (01,.... σL) as 2-layer MLP with tanh non con-eiiey the end which takes patch scale and its cropping offsets 5,, by as input and produces modulations of E (0, fu (we shift the tanh output by 1 to map into the 1-centered interval), where cru is the number of output channels in the f-th convolutional layer. Given a convolutional kernel W E _e xkx and input a 1 straightforward strategy to apply the modulation is to multiply a on the weights (depicting the convolution operation by conv2d(.) and omitting its other parameters for simplicity):
y = conv2d(W' @ @,,),
(1)
where we broadeast the remaining axes and y E Roa is the layer output (before the non-linearity). However, using different kernel weights on top of different inputs is inefficient in modern deep learning frameworks (even with the group-wise convolution trick (26). That's why we use an equivalent strategy of multiplying the weights on instead:
conv2d .x).
y
This suppresses and reinforces different convolutional filters of the layer depending on the patch scale and location. And to incorporate even stronger conditioning, we also use the projection strategy 40) in the final discriminator block. We depict our discriminator architecture in Fig 3 As we show in Tab it allows us to obtain 115% lower FID compared to the standard discriminator.
3.3 Patch-wise optimization with Beta-distributed scales
Training NeRF-based GANs computationally expensive because rendering each pixel via vol- umetric rendering requires many evaluations (e.g., in our case, 96) of the underlying MLP. For scene reconstruction tasks, it does not create issues since the typically used L2 loss 38 76 69 can be robustly computed on sparse subset of the pixels. But for NeRF-based GANs, it becomes prohibitively expensive for high resolutions since convolutional discriminators operate on dense full-size images. The currently dominating approach to mitigate this is to train a separate 2D decoder to upsample a low-resolution image representation rendered from a NeRF-baaed ML.. But this breaks multi-view consistency (i.c., object's shape and texture change when the camera is moving) and
5

---

learns the 3D geometry in low resolution (from = 16 72 to =228
(6). This is why we build upon the multi-scale patch-wise wise training scheme [56 and demonstrate that can give state-of-the-art image quality and training speed without the above limitations.
Patch-wise optimization works the following way. On each iteration, instead of passing the full-size R x R image to D, we instead input only a small patch with resolution r x r of random scale s E / R, 1J and extracted with a random offset (6z, 6y) [0, 1 - =]². We illustrate this procedure in
Fig Patch parameters are sampled from distribution:
5,62,6 8y ~ pr(s, 82, 6y) 2 (=))))((
(3)
where / is the current training iteration. In this way, patch scales depend on the current training iteration 6, and offsets are sampled independently after we know 8. As we show next, the choice of distribution (s()) has crucial influence on the learning speed and stability.
Typically, patch scales are sampled from the annealed uniform distribution 56 36 S:
(s) = smismin(t (t), 1], Smiu (t) = 1erp [1, r/R, min(t/TT ()),
(4)
where lerp is the linear interpolation function and the left interval bound smin (t) is gradually annealed during the first T iterations until it reaches the minimum possible value of r/R But this strategy does not let the model learn high-frequency details early on in training and puts little focus on the structure when 5min (4) fully annealed to r/R R (which is usually very small, e.g., r = 0.125 for a typical 642 patch-wise training on 5122 resolution). As we show, the first issue makes the generator converge slower, and the second one makes the overall optimization less stable.
To mitigate this, we propose a small change in the pipeline by simply replacing the uniform scale sampling distribution with:
(5) where B(t is gradually annealed from 8(0) to some final value B(T). Using beta distribution instead of the uniform one gives a very convenient knob to shift the training focus between large patch scales + 1 (carrying the global information about the whole image) and small patch scales - r/R (representing high-resolution local crops).
~ Beta(1, (1))()) (0)r/R))+r//
A natural way to do the annealing is to anneal from 0 to 1: at the start, the model focuses entirely on the structure, while at the end, it transforms into the uniform distribution (See Fig We follow this strategy, but from the design perspective, set (3T) to a value that is slightly smaller than I (we use (T) 0.8 everywhere) to keep more focus on the structure the end of the annealing as well. In our initial experiments, ((T) E [0.7, 1] performs similarly. The scales distributions comparison between beta and uniform sampling is provided in Fig and the convergence comparison in Fig 7
3.4 Training details
We inherit the training procedure from StyleGAN2-ADA 24 with minimal changes. The optimiza- tion is performed by Adam [ with learning rate of 0.002 and betas of 0 and 0.99 for both G and D. We use ((7)) = 0.8 for = 10000, z ~ NO, I) and set Rp = 512. D is trained with
RI regularization 37 with > 0.05. We train with the overall batch size of 64 for = 155M images
seen by D for 256 resolution and =20M for 5122. Similar to previous works 1 [2]. we use pose supervision for D for the FFHQ and Cats dataset to avoid geometry ambiguity. For this, we take the rotation and elevation angles, encode them with positional embeddings [59] 65 and feed them into a 2-layer MLP. After that, we multiply the obtained vector with the last hidden representation in the discriminator, following the Projection GAN 40 strategy from StyleGAN2-ADA, [24] We train G in full precision and use mixed precision for D. Since FFHQ has too noticeable 3D biases, we use generator pose conditioning for it [ Further details can be found in the source code.
4 Experiments
4.1 Experimental setup
Benchmarks. In our study, we consider four benchmarks: 1) FFHQ 25 in 256 and 5122 resolutions, consisting of 70,000 (mostly front-view) human face images; 2) Cats 256 777.. consisting of 9,998
'1erp(æ, y, a) = (1 a)) x + a y for x, y E IR and a E |0. 1). In practice, those methods use a very slightly different distribution (see Appx B)

---

FFHQ
Cats
M- M-Plants
M-Food
StyleNeRF
MVCGAN
GRAM
Ours
Real
Figure 5: Comparing samples of EpiGRAF and modern 3D-aware generators. Our method attains state-of-the-an image quality, recovers high-fidelity geometry and preserves multi-view consistency for both simple-shape (FFHQ and Cats) and variable-shape (M-Plants and M-Food) datasets. We refer the reader to the supplementary for the video comparisons to evaluate multi-view consistency.
(mostly front-view) cat face images; 3) Megascans Food (M-Food) 256² consisting of 199 models of different food items with 128 views per model (25472 images in total); and 4) Megascans Plants (M-Plants) 256² consisting of 1 1108 different plant models with 128 views per model (141824 images in total). The last two datasets are introduced our work to fix two issues with the modern 3D generation benchmarks. First, existing benchmarks have low variability of global object geometry, focusing entirely on a single class of objects, like human/cat faces or cars, that do not vary much from instance to instance. Second, they all have limited camera pose distribution: for example, FFHQ [25] and Cats [77] are completely dominated by the frontal and near- frortflontal views (see Appx E). That's why obtain and render 1307 Megascans models from Quixel, which photo-realistic (barely distinguishable from real) scans of real-life objects with complex geometry. Those benchmarks and the rendering code will be made publicly available.
Metries. We use FID 20 to measure image quality and estimate the training cost for each method in terms of NVidia V100 GPU days needed to complete the training process.
Baselines. For upsampler-based basclines. we compare to the following generators: StyleNeRE 15
VolumeGAN 771). MVCGAN 78] and GIR HD 72 Apart from that, we also compare to Pi-GAN Z and GRAM 12. which are non-upsampler- based GANs. To compare on Megascans, we train StyleNeRF, MVCGAN, pi-GAN, and GRAM from scratch using their official code repositories (obtained online or requested from the authors), using their FFHQ or CARLA hyperparameters, except for the camera distribution and rendering settings. We also train StyleNeRF, MVCGAN, and r-GAN on Cats 2562. GRAM 12 restricts the sampling space a set of learnable iso-surfaces, which makes it not well-suited for datasets with varying geometry.
StyleSDF 47 EG3D
7

---

Table 1: FID scores of modern 3D GANs. "f" evaluated re-aligned version of FFHQ (different
from original FFHQ (25)). Training cost is measured terms of NVidia V100 GPU days. "OOM" denotes out-of-memory error.
FFHQ
Cats 256
M-Plants 256
M-Food
Training cost
Method
Geometry constraints
256
512²
256
256²
512²
StyleNeRF 15
8.00
7.8
5.91
19.32
16.75
32--res 2D upsampler
StyleSDF 17
11.5
11.19
642-res 2D upsampler
EG3D
6
4.8"
4.71
N/A
76
128 + 2D upsampler
VolumeGAN 241
9.1
N/A
N/A
642-res 2D upsampler
MVCGAN
78
13.7
13.4
39.16
31.70
29.29
42
64
642-res 2D upsampler
GIRAFFE-HI HD 02]
11.93
12.36
N/A
N/A
162-res 2D upsampler
pi-GAN 7
53.2
OOM
68,28
75.64
51.99
none iso-surfaces
GRAM 12
13.78
OOM
13.40
188.6
178.9
56
EpiGRAF (ours)
9.71
9.92
6.93
19.42
18.15
16
24
none
pi-GAN
MVC-GAN
Ours
Figure 6: Visualizing the learned geometry for different methods. r-GAN f recovers high-fidelity shapes, but has worse image quality (see Table [ and is much more expensive to train than our model. MVC-GAN [78 fails capture good geometry because of the 2D upsampler. Our method learns proper geometry and achieves state-of-the-ar image quality. We extracted the surfaces using marching cubes from the density fields sampled on 256 grid and visualized them in Vysta (63 We manually optimized the marching cubes contouring threshold for each checkpoint of each method. We noticed that -GAN f produces lot of "spurious" density which makes.
4.2 Results
EpiGRAF achieves state-of-the-arl image quality. For Cats 256", M-Plants 256 and M-Food 256 EpiGRAF outperforms the baselines terms of FID except for StyleNeRF, performing very similar to it on all the datasets even though it does not have 2D upsampler. For FFHQ, our model attains very similar FID scores as the other methods, ranking 4/9 (including older rAN α.). which trains and evaluates different version of FFHQ and
noticeably losing only to EG3D uses pose conditioning in the generator (which potentially improves FID at the cost of multi-view consistency). We provide visual comparison for different methods in Fig
EpiGRAF is much faster to train. As reported Tab D existing methods typically train for æ1 week on 8 V100s, EpiGRAF finishes training in just 2 days for 2562 and 3 days for 512 resolutions, which 2 - 3x faster. Note that this high training efficiency is achieved without using an upsampler, which initially enabled the high-resolution synthesis of 3D-aware GANs. As to the non-upsampler methods, we couldn't 't train GRAM or T-AN on 5122 resolution due to the memory limitations of the setup with 8 NVidia V 100 32GB GPUs (i.c., 256GB of GPU memory in total).
8

---

EpiGRAF learns high-fidelity geometry. Using a pure NeRF-based backbone has two crucial benefits: provides multi-view consistency and allows learning the geometry the full dataset resolution. In we visualize the learned shapes on M-Food and M-Plants for 1) æ-GAN: pure NeRF-based generator without the geometry constraints; 2) MVC-GAN 78 an upsampler-based generator with strong multi-view consistency regularization; 3) our model. We provide the details and analysis in We also provide the geometry comparison with EG3D on FFHQ 512² Fig 2
Fig G
the caption of Fig 6
EpiGRAF easily capitalizes on techniques from the NeRF literature. Since our generator is purely NeRF based and renders images without 2D upsampler, is well coupled with the existing techniques from the NeRF scene reconstruction field. To demonstrate this, we adopted background separation from NeRF++ 77 using the inverse sphere parametrization by simply copy-pasting the corresponding code from their repo. We depict the results in Fig | and provide the details in Appx B.
4.3 Ablations
We report the ablations for different discriminator architectures and patch sizes on FFHQ 5122 and M-Plants 256 in Tab 2 Using a traditional discriminator architecture results in =15% worse performance. Using several ones (via the group-wise convolution trick (26]] results in noticeably slower training time and dramatically degrades the image quality. We hypothesize that the reason for was the reduced overall training signal each discriminator receives, which we tried to alleviate by increasing their learning rate, but that did not improve the results. A too-small patch size hampers the learning process and produces R80% worse FID. A too-large one provides decent image quality but greatly reduces the training speed. Using single scale/position best performance, outperforming the standard one by 1155% on average.
discriminator achieves the
To assess the convergence of our proposed patch sampling scheme, compared against uniform sampling on Cats 256² for T E {1000, 5000, 10000), representing different annealing speeds. We show the results for it in Fig / our proposed beta scale sampling strategy with T 10k schedule robustly converges to lower values than the uniform one with 7 = 5k or = 10k and does not fluctuate much compared to the T 1 uniform (where the model reached its final annealing stage in just Ik kilo-images seen by D).
To analyze how myper-modulation manipulates the convolutional filters of the discriminator, visualize the modulation weights a,, predicted by H, in Fig (see the caption for the details). These visualizations show that some of the filters are always switched on, regardless of the patch scale; while others are always switched off, providing potential room for pruning [1]] And 4x00 of the filters getting switched on and off depending the patch scale, which shows that H indeed learns to perform meaningful modulation.
Experiment
FFHQ 512"
M-Plants 256
Training cost
(unffomi (tetal
GRAF (with tri-planes)
24.99
beta scale sampling (T' 5k)
11.57
21.77
24
2 scale-specific D-s
10.87
21.02
28
scale-specific
21.56
43.11
28
scale/position-awwe D
9.92
19.42
24
32 putch resolution
17.44
34.32
19
6.1 patch resolution (default)
9.92
19.42
24
Seen
128 patch resolution
11.36
18.90
34
Figure Convergence com- parison on Cats 256 for dif- ferent sampling strategies.
Table 2: Ablating the discriminator architecture and patch sizes in terms of FID scores and training cost (V100 GPU days).
Patch scale
0.75
1.5
0.5
1.
0.25
0.5
Convolutional
Convoluzional
Convolutional
Convolutional
Figure 8: Visualizing modulation weights σ,, predicted by H for 2-nd, 6-th, 10-th and 14-th convolu-
tional layers. Each subplot denotes a separate layer and we visualize random 32 filters for it.

---

Limitations
Performance drop for 2D generation. Before switching to training 3D-aware generators, we spent a considerable amount of time, exploring our ideas on top of StyleGAN2 (24] for traditional 2D generation since it is faster, less error-prone and more robust to a hyperparameters choice. What observed is that despite our best efforts (see C) and even with longer training, we couldn't obtain obtain the same image quality the full-resolution StyleGAN2 generator.
Table 3:
generator in the patch-wise fashion. We tried to train longer compensate for a smaller learning signal overall (a 64 patch is 1/64 of information compared to a 512 image), but this didn't i allow to catch up. Note, however, that AnyResGAN S reaches SotA when training on 256² patches compared to 1024 images.
Trying to train a traditional StyleGAN2 R
FFHQ 5122
LSUN Bedroom 256 Training cost
Method
FID
Training cost
FID
StyleGAN2-ADA
[44
3.83
4.12
multi-scale 64 patch-wise training
7.11
6.73
4
x2 longer training
5.71
5.42
x4 longer training
4.76
24
4.31
16
A range of possible patch sizes is restricted. Tab shows the performance drop when using the 32 patch size instead of the default 642 one without any dramatic improvement in speed. Trying to decrease it further would produce 2 patches). Increasing the patch size is also not desirable since it decreases the training speed a lot: going from 642 to 128 resulted in 30% cost increase without clear performance benefits. In this way, we are very constrained in what patch size one can use.
worse performance (imagine training in the extreme case of
Discriminator does not see the global context. When the discriminator classifies patches of small scale, is forced to do so without relying on the global image information, which could be useful for this. Our attempts to incorporate it (see Appx C) did not improve the performance (though we believe we under-explored this).
Low-resolution artifacts. has some blurriness when one zooms-in into the samples. It is not well captured by FID since it always resizes images to the 299 x 299 resolution. We attribute this problem to our patch-wise training scheme, which puts too much focus on the structure and believe that it could be resolved.
While our generator achieves good FID on FFHQ 5122, we noticed that
6 Conclusion
In this work, we showed that it is possible to build a state-of-the-art 3D GAN framework without a 2D upsampler, but using a pure NeRF-based generator trained in multi-scale pitch-wise fashion. For this, we improved the traditional patch-wise training scheme in two important ways. First, proposed to scale/location-awai discriminator with convolutional filters modulated by hypernetwork depending on the patch parameters. Second, we developed a schedule for patch scale sampling based the beta distribution, that leads faster and more robust convergence. We believe that the future of 3D GANs is combination of efficient volumetric representations, regularized 2D upsamplers, and patch-wis wise training. We propose this avenue of research for future work.
Our method also has several limitations. Before switching to training 3D- 3Dawwrr generators, we spent considerable amount of time exploring our ideas top of StyleGAN2 for traditional 2D generation, which always resulted in higher FII) scores. Further, the discriminator loses information about global context. We tried multiple ideas incorporate global context, but it did not lead Next, our current pasch-wise training scheme might cause some low-res artifacts. Finally, 3D GANs generating faces and humans may have negative societal impact discussed in Appx H.
improvement.
7 Acknowledgements
We would like to acknowledge support from the SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence.
10

---

References
[1] P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas. Learning representations and generative models for point clouds. In International conference on machine learning, pages 40-49. PMLR, 2018
[2] J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R.
and P. P. Srinivasan. Mip-nerf: A multiscale representation for anti- anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855 5864, 2021.
[3] Blender Online Community. Blender- 3D modelling and rendering package. Blender Foundation,
Blender Institute, Amsterdam, 2022.
[4] A. Brock, Donahue, and K. Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1809. /006, 2018.
[5] L. Chai, M. Gharbi, Shechtman, Isola, and R. Zhang. Any-resolution training for high-resolttion image synthesis. arXiv preprint arXiv:2204.071555 2022.
[6] E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan. S. D. Mello. O. Gallo, L. Guibas, J. Tremblay,
S. Khamis, Karras, and G. Wetzstein. Efficient geoaetry-aware generative adversarial networks. In arXiv, 2021.
[7] E. R. Chan, M. Montciro, P. Kellnhofer, J. Wu, and G. Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5799-550 5009,899,22 2021.
[8] A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su. Tensorf: Tensorial radiance fields. arXiv preprint arXiv:2203.095// 2022.
[9] H. Chen, B. Hc, H. Wang, Y. Ren, S.-N. Lim, and A. Shrivastava, Nerv: Neural representations for videos. arXiv preprint arXiv:2110. 13903, 2021.
[10] Z. Chen and H. Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5939-5948, 2019.
[11] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 8789-8797, 2018.
[12] Y. Deng, Yang. J. Xiang, and Tong. Gram: Generative radiance manifolds for 3d-aware image generation. In IEEE Computer Vision and Pattern Recognition, 2022.
[13] M. Gadelha, S. Maji, and R. Wang. 3d shape induction from 2d views of multiple objects. In 2017 International Conference on 3D Vision (3DV), pages 402 411. JEEE, 2017.
[14] I. Goodfellow. J. Pouget-Abadie, M. Mirza, B. Xu. D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[15] J. Gu, L. Liu, P. Wang, and C. Thcobalt. Stylenerf: A stylc-based 3d aware generator for high-resolution image synthesis. In International Conference on Learning Representations, 2022.
[16] D. Ha, A. Dai, and Q V. Le. Hypernetworks. arXiv preprint arXiv:1609.09100 2016.
[17] Z. Hao, A. Mallya, S. Belongie, and M.-Y. Liu. GANerafit Unsupervised 3D Neural Rendering of Minecraft Worlds. In ICCV,2C20 2021.
[18] Y. He, X. Zhang, and J. Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pages 1389-1397, 2017.
[19] Henderson, Tsiminaki, and C. H. Lampert. Leveraging 2d data learn textured 3d mesh generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7498-7507, 2020.
[20] Heusel, H. Ramsauer. T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[21] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6833 2020.
[22] P. Isola. J.-Y. Zhu, T. Zhou, and A. A. Efros. image-to-imant translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 125--134, 2017.
[23] K. Jo. G. Shim, S. Jung, S. Yang, and Choo. Cg-nerf: Conditional generative neural radiance fields. arXiv preprint arXiv:21 2021.
[24] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila. Training generative adversarial networks with limited data. arXiv preprint arXiv:2006.0660 2020.
11

---

[25] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4401-4410. 2019.
[26] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 81 10-8119, 2020.
[27] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint (412,6640 2014.
[28] D. Kingma and P. Dhariwal. Glow: Generative flow with invertible 1xl convolutions. Advances in neural information processing systems, 31, 2018.
[29] A. R. Kosiorek, H. Strathmann. D. Zoran, P. Moreno, R. Schneider. S. Mokrá, and D. J. Rezende. Nerf-vae: A geometry aware scene generative model. arXiv preprint arXiv:2104.0055 2021.
[30] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[31] R. Li, X. Li, K.-H. Hui, and C.-W. Fu. Sp-gan: Sphere-guided 3d shape generation and manipulation. ACM Transactions on Graphics 40(4), 2021.
[32] X. Li, Y. Dong, Peers, and X. Tong. Synthesizing 3d shapes from silhouette image collections using multi- projection generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5535-5544, 2019.
[33] C. H. Lin, H.-Y. Lee, Y.-C. Cheng, S. Tulyakov, and M.-H. Yang. Infinitygan: Towards infinite-resolution image synthesis, arXiv preprint arXiv: 104.03963, 2021.
[34] S. Luo and W. Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR). June 2021.
[35] Y. A. Mejjati, Milefchik, A. Gokaslan, O. Wang, K. Kim, and J. Tompkin. Gaussigan: Controllable image synthesis with 3d gaussians from unposed silhouettes. arXiv preprint arXiv:2106 10.iv::2166.3222 13215, 2021.
[36] Q Meng, A. Chen, Luo, M. Wu. H. Su, L. Xu, X. He, and Yu. Gnerf: Gan-based neural radiance field without posed camera. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 6351-6361, 2021. 2021.
[37] L. Mescheder, A. Geiger, and S. Nowozin. Which training methods for gans do actually converge? In International conference on machine learning, pages 3481 -449-. PMLR, 2018
[38] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In European conference on computer vision, pages 405 421. Springer, 2020.
[39] P. Mittll, Y.-C. Cheng, M. Singh, and S. Tulsiani. AutoSDF: Shape priors for 3d completion, reconstruction and generation. In CVPR, 2022.
[40] T. Miyato and M. Koyama. egans with projection discriminator. arXiv preprint
2018.
[41] T. -Nguyen-Phuok C. Li, L. Theis, C. Richardt, and Y.-L. Yang. Hologan: Unsupervised learning of 3d representations from natural images. In The IEFE International Conference on Computer Vision (ICCV), Nov 2019.
[42] M. Niemeyer and A. Geiger. Campari: Camera-aware decomposed generative neural radiance fields. In 2021 I International Conference on 3D Vision (3DV), pages 951-961. IEEE, 2021.
[43] M. Niemeyer and A. Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11453 11464, 2021.
[44] M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3504-3515, 2020.
[45] M. Oechsle, S. Peng, and A. Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 5589-5599, 2021. 2021.
[46] C. Olah, A. Mordvintsev, and L. Schubert. hubpalllllpp22011//tttteee--i [47] R. Or-El, X. Luo, M. Shan, Shechtman, J. J. Park, and I. Gemnelmacher-Shlizeennn StyleSDF: Regh-Resolttion DD-Consistent Image and Geometry Generation. arXiv preprint arXiv:2112.11427, 2021.
Feature
visualization.
Distill, 2017.
[48] X. Pan, B. Dai. Z. Liu, C. C. Loy, and P. Luo. Do 2d gans know 3d shape? unsupervised 3d shape reconstruction from 2d image gans, arXiv preprint 1rXiv:2011.0084. 2020.
12

---

[49] X. Pan, X. Xu, C. C. Loy, C. Theobalt, and B. Dai. A bondinggguiied generative implicit model for shape- accurate 3d-aware image synthesis. In Advances in Neural Information Processing Systems (NeurIPS). 2021.
[50] K. Park, U. Sinha, J. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R. Martin-Brualla, Deformable neural radiance fields. arXiv preprint arXiv:2011.12948, 2020.
[51] T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu. Semantic image synthesis with apatiall normal- ization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2337-2346, 2019.
[52] T. Park, J. Zhu, O. Wang, J. Lu, E. Shechtman, A. Efros, and R. Zhang. Swapping autoencoder for deep image manipulation. Advances in Neural Information Processing Systems, 33:7198-7211 2020.
[53] D. Pavllo, J. Kohler, T. Hofmann, and A. Lucchi. Learning generative models of textured 3d meshes from real-world images. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV),
pages 13879-13889, October 2021.
[54] D. Pavllo, G. Spinks, T. Hofmann, M.-F. Moens, and A. Lucchi. Convolutional generation of textured 3d meshes. In Neural Information Processing Systems (NeurIPS), 2020.
[55] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 2022.
[56] K. Schwarz, Y. Liao, M. Niemeyer, and A. Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. In Advances in Neural Information Processing Systems (NeurIPS). 2020.
[57] T. R. Shaham, T. Dekel, and T. Michaeli. Singan: Learning generative model from a single natural image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4570 4580, 2019,
[58] Y. Shi, D. Aggarwal, and A. K. Jain. Lifting 2d stylegan for 3d-aware face generation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 2266-6266, 2021.
[59] V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
[60] I. Skorokhodov, S. Ignatyev, and M. Elhoseiny. Adversarial generation of continuous images. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10753-10764, 2021.
[61] I. Skorokhodov, G. Sotnikov, and M. Elhoseiny. Aligning latent and image spaces to connect the uncon- nectable. arXiv preprint arXiv:2104.06955 2021.
[62] 1. Skorokhodov, S. Tulyakov, and M. Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 3626-3636, 2022.
[63] C. B. Sullivan and A. Kaszynski. Vista:
3d plotting and mesh analysis through a streamlined interface for the visualization toolkit (VTK). Journal of Open Source Software, 4(37):1450, may 2019.
[64] F. Tan, S. Fanello, A. Meka, S. Orts-ascolano, D. Tang, R. Pandey, J. Taylor, P. Tan, and Y. Zhang. Volux-gan: A generative model for 3d face synthesis with hdri relighting. arXiv preprint 22X.v:2001.4877. 2022.
[65] M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keii, N. Raghavan, U. Singhal, R. Ramamoorthi, J. T. Barron, and R. Ng. Fourier features let networks Icarn high frequency functions in low dimensional domains. arXiv preprint arXiv:2006. 10739, 2020.
[66] A. Van den Oord, N. Kalchbrenner, L. Espeholt, O. Vinyals, A. Graves, et al. Conditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
[67] Y. Vinker, E. Horwitz, N. Zabari, and Y. Hoshen. Image shape manipulation from a single augmented training sample. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 13769-13778, October 2021.
[68] C. Wang, M. Chai, M. He, D. Chen, and J. Liao. Clip-nerf: Text-and-inen driven manipulation of neural radiance fields. arXiv preprint 1rXiv:2112.051333 2021.
[69] P. Wang, L. Liu, Y. Liu, C. Thcobalt, T. Komura, and W. Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstructoo. arXiv preprint a0Xiv:2106. 10689, 2021.
[70] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920. 2015.
[71] Y. Xu. S. Peng. C. Yang, Y. Shen, and B. Zhou. 3d-aware image synthesis via learning structural and textural representations. arXiv preprint arXi::2112.1075: 2021.
13

---

[72] Y. Xue, Y. Li, K. K. Singh, and Y. J. Lee. Giraffe hd: A high-resolution 3d-aware generative model. arXiv preprint arXiv:2203. 14954, 2022.
[73] Y. Ye, S. Tulsiani, and A. Gupta. Shelf-supervised mesh prediction in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8843 8852, June 2021.
[74] A. Yu, V. Ye, M. Tancik, and A. Kanazawa, pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4578 4587, June 2021.
[75] Zhang, E. Sangincto, H. Tang, A. Siarohin, Z. Zhong, N. Sebe, and W. Wang. 3d-aware semantic-guide guided generative model for human synthesis. arXiv preprint arXiv: 2021.
[76] K. Zhang, G. Riegler, N. Snavely, and V. Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint 27Xi::2010.04422, 2020.
[77] W. Zhang, J. Sun, and X. Tang. Cat head detection- to effectively exploit shape and texture features. In European conference computer vision, pages 802-816. Springer, 2008.
[78] X. Zhang, Z. Zheng, D. Gao, B. Zhang, P. Pan, and Y. Yang. Multi-view consistent generative adversarial networks for 3d-aware image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[79] P. Zhou, L. Xie, B. Ni, and Q. Tian. Cips-3d: A 3d-aware generator of gans based on conditionally- independent pixel synthesis. arXiv preprint arXiv:2110.09788 2021.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See and Appx A
(c) Did you discuss any potential negative socictal impacts of your work? [Yes] this in Appendix G.
We do
(d) Have you read the ethics review guidelines and ensured that your paper conforms them? [Yes] We discuss the potential ethical concerns of using our model in Appendix G.
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We provide the code/data and additional visualizations on specified in the introduction).
as
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We provide the most important training details $3.4 The rest of the details are provided in Appx B and the provided source code.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] That's s too computationally expensive and single-run results are typically reliable in the GAN field.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We report this numbers in Appx B.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We cite all the sources of the datasets which were used or mentioned in our submission.
(b) Did you mention the license of the assets? [Yes] In this work, release new datasets: Megascans Plants and Megascans Food. We discuss their licensing in Appx E.
(c) Did you include any new assets either in the supplemental material or as a URL? Yes] We provide our datasets on the project website: htts://univeroomegggiubbbb/ooppp
(d) Did you discuss whether and how consent was obtained from people whose data you 'r using/curating? [Yes] We specifyy the information on dataset collection in Appx E. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] As discussed in Appx E, the released data does not contain personally identifiable information or offensive content.
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "y2zuhLFAsQb", "reviewer": "Reviewer_bBXS", "summary": "The authors propose a framework to generate 3D images from latent code. Authors analyze in details the drawbacks of current methods, based on upsampling strategy that could affect multi-view consistency or rely on low resolution rendering. They came up with a patch based strategy similar to GRAF to improve the performance. The idea is to condition the discriminator with a hypernetwork to better make sense of the output of the convolutional filter at multiple scales. Additionally, they propose a beta distribution when sampling the scale, improving the overall convergence", "questions": "I'd like the authors to comment on the limitations I described in the Strength and Weakness section. In particular, why one could not simply rely on the EG3D triplane representation + GRAF patch based discriminator to achieve the same results proposed here. Is there any experiment showing that it's not sufficient?\n\nAlso, why one should prefer this method instead of EG3D? Seems to obtain better quantitative metrics (and qualitative metrics are not shown).", "limitations": "Yes, Limitations and Ethical concerns are described in the supplementary material", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+++ Relevance. The work is super relevant for the community, with a lot of possible applications.\n\n+++ Clarity. The paper is very well written and well motivated. Authors describe in details all the aspects of 3D generators, explaining their drawbacks and how to mitigate those. In particular, I really appreciated in the supplementary material the attempts at 2D generation and all the failing experiments the authors reported. This could be very useful for the community as GAN methods usually rely on a substantial amount of trial and errors before getting to the right combination of architecture and loss functions.\n\n--- Novelty and Experiments. Whereas this paper has many good aspects and I'd like to see it published, I have a few doubts regarding the overall novelty and the missing comparisons.\n\nIn particular, the method \"borrows\" multiple aspects from EG3D and GRAF, but comparisons are very limited. Indeed EG3D outperforms this method substantially in terms of FID, authors hint that the upsampler leads to a better FID score but worse multiview consistency (l240), but they do not provide any example of this.\n\nSimilarly, the method borrows the patch-based, multi-scale discriminator idea from GRAF, improving with the conditioning from a hypernetwork, but I am not sure there is a direct comparison with GRAF?\n\nThere are no examples of inferred geometry. Ideally authors would also show the inferred geometry and qualitatively compare with EG3D to prove that the method leads to better multi-view consistency.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_JwSNEfncHZ", "reviewer": "Reviewer_P8PQ", "summary": "The paper proposes a non-upsampler-based 3D-aware generator. To train the non-upsampler-based generator, the paper presents a patch-based discriminator. In contrast to previous patch-based schemes, the author improves patch-based discriminators in two ways. First, they adopt location and scale to modulate the discriminator (in particular, the feature maps).  Second, they modify the patch sampling strategy based on an annealed beta distribution to stabilize training and accelerate the convergence. Furthermore, they introduce two plants and food datasets, in order to evaluate 3D-aware generators. ", "questions": "Please see the weaknesses.", "limitations": "Yes, they did.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is well-written, and the experiments are comprehensive.\n\nThe patch-based discriminator studied in the paper is important for differentiating high-resolution images and would be a good contribution to the community.\n\nWeaknesses：\n\n1. The paper's main contribution is to propose a new patch-based discriminator, which is particularly useful when the generator can only render partial pixels. I would like to know whether the generator synthesizes all images or only a subset of pixels during training. Since the generator adopts a tri-plane representation, it is memory efficient. If the generator is already able to synthesize all pixels without running out of memory, it seems unnecessary to adopt a patch-based discriminator. \n\n2. I think patch-based discriminators are valuable. A more challenging setting can demonstrate the method's effectiveness, namely training pi-GAN on high-resolution images. Since pi-GAN is memory-intensive, rendering all pixels for high-resolution images is impossible during training. Maybe this setting is more suitable for demonstrating the efficacy of the patch-based discriminator. \n\n3. If possible, the authors should provide the results of the full-resolution discriminator. Since the full-resolution discriminator can see the global image, its performance can be used as an upper bound for the reference of patch-based discriminators.\n\n4. In equation 4, a simple baseline is $lerp[1, r/R, 1 - min(t/T, 1)]$. If possible, the author should compare this setting to the proposed beta strategy. \n\n5. Line 204, how to use pose supervision for the discriminator? The author can briefly introduce the details. \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "O02Rrg9lCPS", "reviewer": "Reviewer_vtTi", "summary": "This paper introduces new techniques in training NeRF-based GANs. The proposed techniques addresses the patch-based trianing issue, which blocked privious works to directly generate nerfs that renders into high quality resolution images. As a result, previous works lacks in training time, in addition to lack of view consistency because of image-based upsampling. The proposed method utilizes a hyper-network to generate filters for the patch discriminator under different resolutions, and utilizes an annealed beta distribution for samping the random scale for patch discrimination. Both disign choices are well founded and validated by the quality of the result. ", "questions": "1. Are all baseline results generated using view-independent NeRF as well?\n2. As suggested by the paper, using the beta distribution with the improved annealing improves the stability and convergence speed. Is that also the case for the hyper-network as well? Is the method robust to train given the hyper-generated patch discriminator?\n\n\n", "limitations": "The authors addressed the limitations and potential negative impact adequately. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nOriginality:\nThis paper proposes new solutions to patch-based discriminators in training NeRF-based GANs. The proposed techniques are simple to implement and effective, as demonstrated by the results. In addition, the authors validate the quality on new datasets(megascan plants&food), which further demonstrated the geometric quality of the generated nerf.\n\nSignificance:\nI think this paper would be of significance to communities interested in generative methods centered around NeRF. Though it can be said that the proposed technique is simple, but it is more a merit than a drawback. I think those solid gadgets are keys to make things work better and better.\n\nQuality:\nThis paper conducts enough experiments to support the claim, and justifications of the design choices make sense. I find the evidence in the paper convincing and the webpage of results are quite representative.\n\nClarity:\nThis paper is well written and easy to follow. Readers should be able to reimplement the method based on the inforamtion provided.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HoykHFPUhYJ", "reviewer": "Reviewer_LXyr", "summary": "This paper proposes two important techniques to train 3D generative models from 2D supervision.  \n\nExisting works adopt nerf to render 3D radiance fields into 2D images with a large memory cost. As a result, they fail to train the discriminator with high resolution and more often render 2D images with low resolution and generate high-res images by 2D CNN up-sampling. However, it brings strong view-inconsistent problems. \n\nThis paper explores patched-based discriminator. By rendering small patches, the generator can directly synthesize high-res images without any upsampling tricks. To improve the quality of patch discriminator, the paper proposes 1) a location- and scale-aware discriminator and 2) a beta distribution patch sampling strategy, both showing good improvement. It also produces quite nice and view consistent results and outperforms the baselines.\n", "questions": "I have several questions about the implementation details.\n\n1)\tAs for the rendering equation, did the authors adopt the original ray-based rendering equation (in original nerf paper), or light cone based one (in mip-nerf?) If I didn’t understand wrong, it seems the authors choose the former. In such a case, would it produce anti-aliasing problems when rendering a 64x64 patch with the scale to be 1(let’s say the original image is 256x256)? Adopting the mip-nerf seems to be the straightforward solution. Could the authors discuss this part?\n\n2)\tHow do you make real patch data for training the discriminator?  Are you cropping the real images with specific scales, then resizing them to be 64x46? I wonder do you also consider the antialiasing problem here?\n\n3)\tWhat’s the relationship between patch size and final image size? E.g., In table2 it seems for 512x512 image size, 64 patch size seems to be the best. Does it mean that for 256x256 image size we should change to 32 patch size? \n\n4)\tIt is good to see that the proposed generators can generate up to 512x512 images. I wonder what's maximum. Can it be scaled to 1024x1024? \n\n5)\tFor face and cat, the camera poses are limited to the front views.  I wonder can you do large view changes, e.g., rendering side faces, or rendering from overhead? I want to know whether the radiance field will remain view-consistent for unseen training views. On the other side, I recognize that if the training images cover 360 degrees of the object, then the radiance field doesn’t have such issues.\n", "limitations": "I appreciate that the authors honestly report that the proposed techniques don’t work in 2D cases. However, I treat it as a limitation of the paper since it undermines the generalization of the techniques.  ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe patch-based solution to solve the view-inconsistency problem of nerf-based generative models is reasonable and the two proposed techniques are quite simple but seem to work well.\nThe final FID scores are better than all the baselines with much less training time needed, and the results looks quite nice!\n\nWeakness:\nIt is very weird why the strategy cannot be applied in 2D cases, which raises concerns of generalization.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept