3D generation on ImageNet
Authors: Ivan Skorokhodov
Aliaksandr Siarohin
Yinghao Xu
Jian Ren
Hsin-Ying Lee
Peter Wonka
Sergey Tulyakov
Venue: ICLR 2023 notable top 5% (2023)
Keywords: 3d-generation
gans
generative adversarial networks
knowledge distillation
nerf
stylegan
radiance fields
volume rendering

Abstract: All existing 3D-from-2D generators are designed for well-curated single-category datasets, where all the objects have (approximately) the same scale, 3D location, and orientation, and the camera always points to the center of the scene. This makes them inapplicable to diverse, in-the-wild datasets of non-alignable scenes rendered from arbitrary camera poses. In this work, we develop a 3D generator with Generic Priors (3DGP): a 3D synthesis framework with more general assumptions about the training data, and show that it scales to very challenging datasets, like ImageNet. Our model is based on three new ideas. First, we incorporate an inaccurate off-the-shelf depth estimator into 3D GAN training via a special depth adaptation module to handle the imprecision. Then, we create a flexible camera model and a regularization strategy for it to learn its distribution parameters during training. Finally, we extend the recent ideas of transferring knowledge from pretrained classifiers into GANs for patch-wise trained models by employing a simple distillation-based technique on top of the discriminator. It achieves more stable training than the existing methods and speeds up the convergence by at least 40%. We explore our model on four datasets: SDIP Dogs $256^2$, SDIP Elephants $256^2$, LSUN Horses $256^2$, and ImageNet $256^2$ and demonstrate that 3DGP outperforms the recent state-of-the-art in terms of both texture and geometry quality. Code and visualizations: https://snap-research.github.io/3dgp.

Published as a conference paper at ICLR 2023
3D GENERATION ON IMAGENET
Ivan Skorokhodoy KAUST
Aliaksandr Siarohin Snap Inc.
Yinghao Xu* CUHK
Jian Ren Snap Inc.
Hsin-Ying Lee Snap Inc.
Peter Wonka KAUST
Sergey Tulyakov Snap Inc.
ABSTRACT
All existing 3D-from-2DD generators are designed for well-curated single-category datasets, where all the objects have (approximately) the same scale, 3D location and orientation, and the camera always points to the center the scene. This makes them inapplicable diverse, in-the-wild datasets of non-alignable scenes rendered from arbitrary camera poses. In this work, we develop 3D generator with Generic Priors (3DGP): a 3D synthesis framework with more general assump- tions about the training data, and show that it scales to very challenging datasets, like ImageNet. Our model is based on three new ideas. First, we incorporate an inaccurate off-the-shelf depth estimator into 3D GAN training via special depth adaptation module handle the imprecision. Then, we create flexible camera model and regularization strategy for it to learn its distribution parameters dur- ing training. Finally, we extend the recent ideas of transferring knowledge from pretrained classifiers into GANs for patch-wise trained models by employing a simple distillation-batt based technique on top of the discriminator. achieves more stable training than the existing methods and speeds up the convergence by at least 40%. We explore our model on four datasets: SDIP Dogs 256², SDIP Elephants 256², LSUN Horses 256², and ImageNet 256² and demonstrate that 3DGP out- performs the recent state-of-the-art in terms of both texture and geometry quality.
Code and visualizations:
http://sap.reeencc.ggttbb...dddd
INTRODUCTION
EG3D
EG3D (wide camera)
Our madel
(debut camera)
Figure 1: Selected samples from EG3D (Chan et al. 2022 and our generator trained on ImageNet 256 (Deng et al. 2009). EG3D models the geometry in low resolution and renders either flat shapes (when trained with the default camera distribution) or repetitive "layered" ones (when trained with a wide camera distribution). In contrast, our model synthesizes the radiance field in the full dataset resolution and learns high-fidelity details during training. Zoom-in for better view.
We witness remarkable progress in the domain of 3D-aeere image synthesis. The community is de- veloping new methods to improve the image quality, 3D consistency and efficiency of the generators
Work done during internship Snap Inc.

---

Published as a conference paper at ICLR 2023
tri-glanes
G
volume ronderins
LeReS
D
ResNet
lle é dictilatiion
prior
Camera
posterior
Depth Adaptor
Generator
camera
dotaset peeprocossing
Figure 2: Model overview. Left: our tripplane-based generator. To synthesize an image, we first sample camera parameters from prior distribution and pass them to the camera generator. This gives the posterior camera parameters, used to render an image and its depth map. The depth adaptor mitigates the distribution gap between the rendered and the predicted depth. Right: our discriminator receives 4-channel color-depth pair as an input. A fake sample consists of the RGB image and its (adapted) depth map. A real sample consists of real image and its estimated depth. Our two-headed discriminator predicts adversarial scores and image features for knowledge distillation.
(e.g., Chan et al. (022)) Deng et al. (2022); Skorokhodov et al. y: Zhao et al. (2022); Schwarz jtt al. (2022)) However, all the existing frameworks are designed for well-curated and aligned datasets consisting of objects of the same category, scale and global scene structure, like human or cat faces (Chan et al. 202T). Such curation requires domain-specific 3D knowledge about the object category hand, since needs to infer the underlying 3D keypoints to properly crop, rotate and scale the images (Deng et al.) 2022] Chan et al. 2022). This makes it infcasible to perform a simi- lar alignment procedure for lcrge-scale multi-category datasets that are inherently "non-alignable": there does not exist a single canonical position which all the objects could be transformed into (e.g., it is impossible to align a landscape panorama with a spoon).
To extend 3D synthesis to in-the-wild datasets, one needs a framework which relies on more univer- sal 3D priors. In this work, make step towards this direction and develop 3D generator with Generic Priors (3DGP): 3D synthesis model which is guided only by (imperfect) depth predictions from an off-the-shelf monocular depth estimator. Surprisingly, such 3D cues are enough learn reasonable scenes from loosely curated, non-aligned datasets, such as ImageNet (Deng et al. 2009).
Training 3D generator on in-the-wii wild datasets comes with three main challenges: 1) extrinsic camera parameters of real images are unknown and impossible to infer; 2) objects appear in different shapes, positions, rotations and scales, complicating the learning of the underlying geometry; and 3) the dataset typically contains lot of variation in terms of texture and structure, and is difficult to fit even for 2D generators. As shown Fig (left), state-of-the-art 3D-aware generators, such as EG3D (Chan al. 2022), struggle to learn the proper geometry in such challenging scenario. In this work, we develop three novel techniques to address those problems.
Learnable "Ball-in-Shhere camera distribution. Most existing methods utilize restricted camera model (e.g., (Schwarz et al. 2020; Niemeyer Geiger 2021b] Chan et al., (021)): ) the camera is positioned on sphere with constant radius, always points to the world center and has fixed intrinsics. But diverse, non-aligned datasets violate these assumptions: e.g., dogs datasets have images of both close-up photos of a snout and photos of full-body dogs, which implies the variability in the focal length and look-at positions. Thus, we introduce a novel camera model with 6 degrees of freedom to address this variability. We optimize its distribution parameters during training and develop an efficient gradient penalty for it to prevent its collapse to a delta distribution.
Adversarial depth supervision (ADS). generic image dataset features wide diversity of objects with different shapes and poses. That is why learning meaningful 3D geometry together with the camera distribution an ill iol-posed problem, as the incorrect scale can be well compensated by an incorrect camera model (Hartley & Zisserman, 2003), or flat geometry (Zhao et al. 2022. Chan al. 2022). To instill the 3D bias, we provide the scene geometry information to the discriminator by concatenating the depth map of a scene as the 4-th channel of its RGB input. For real images, use their (imperfect) estimates from a generic off-the-shelf monocular depth predictor (Miangolch et al. 2021). For fake images, we render the depth from the synthesized radiance field, and process it with a shallow depth adaptor module, bridging the distribution gap between the estimated and rendered depth maps. This ultimately guides the generator to learn the proper 3D geometry.
Knowledge distillation into Discriminator: Prior works found it beneficial to transfer the knowledge from off tff-tff-sheff 2D image encoders into a synthesis model (Sauer al. 2022). They typically
2

---

Published as a conference paper at ICLR 2023
utilize pre-trained image classifiers the discriminator backbone with additional regularization strategies on top /auer et al. 2021; Kumari et al.) 2022). Such techniques, however, are only appli- cable when the discriminator has an imput distribution similar to what the encoder was trained This does not suit the setup of patch-wise training (Schwarz et al. 2020) or allows to provide depth maps via the 4-th channel to the discriminator. That is why we develop more general and efficient knowledge transfer strategy based on knowledge distillation. It consists in forcing the discrimina- tor predict features of pre-trained ResNet50 (He al. 2016) model, effectively transferring the knowledge into our model. This technique has just 1% of computational overhead compared to standard training. but allows to improve FID for both 2D and 3D generators by at least 40%.
First, we explore our ideas on non-aligned single-category image datasets: SDIP Dogs 256 (Mokady et al., 2022), SDIP Elephants 256² (Mokady et al.) 2022), and LSUN Horses 256² Yu et 2015). On these datasets, our generator achieves better image appearance (measured by FID Heusel et al.] (2017)) and geometry quality than the modern state-of-the-ar 3D-aware generators. Then, we train the model on all the 1, 000 classes of ImageNet (Deng et al. 2009) showing that multi-categorical 3D synthesis is possible for non-alignable data (see Fig.
2 RELATED WORK
3D-aware image synthesis. Mildenhall et (2020) introduced Neural Radiance Fields (NeRF):
neural retwork-based representation of 3D volumes which is learnable from RGB supervision only. ignited many 3w-rware image/video generators (Schwarz
2020
Niemeyer Geiger 202
Chan et al. |021 Xue et 2022, Zhou et al. [021]] Wang et al.
2022
Gu et al. 2022 Or- El et al.
2021
Chan et al. 2022] Skorokhodov et al. | et al. 2022 Xu al.] [202] Bahmani et al,
2022 all of them being GAN-based (Goodtellow et al. 2014)) Many of them explore the techniques
to reduce the cost of 3D-aware generation for regh-uesolution data, like patch-wise training (e.g.,
Schwarz et al. (2020); [Meng et al. (2021); Skorokhodov et al. (2022)). MPI-based rendering (e.g.,
Zhao et al. )2022) or training separate 2D upsampler (e.g.. Gu et al. (2022)).
Learning the camera poses. NeRFs require known camera poses, obtained from multi-view stereo (Schönberger et al. 2016) or structure from motion (Schönberger & Frahm 2016). Alterna- tively, a group works has been introduced to either automatically estimate the camera poses (Wang) et al. 2021) or finetune them during training (Lin et al. [202]] Kuang et al.] [002)). The problem we requires learning not the camera poses from
tackling in this work is fundamentally different multi-view observations, but distribution of poses, while having access to sparse, single-view data of diverse object categories. In this respect, the work closest ours CAMPARI (Neemeyer & Geiger 2021a) as it also learns a camera distribution.
GANs with external knowledge. Several works observed improved convergence and fidelity of GANs when using existing, generic image-based models (Kumari et al. 2022 Sauer et al. [02T]]
2022 Mo et al. 2020), the most notable being StyleGAN-XL (Sauer et al.J 2022). which uses a pre-trained EfficientNet (Tan & Le 2019) followed by couple of discriminator layers. A similar technique is not suitable in our case as pre-training a generic RGB-D network on a large-scale RGB-D dataset is problematic due to the lack of data. Another notable example is Freeze et al. (220)), which proposes to distill discriminator features for GAN finetuning. Our work, on the other hand, relies
existing model for image classification.
Off-the-shelf depth guidance. GSN Veies et al., (021) also concatenates depth maps as the 4-th channel of the discriminator's 's input, but they utilize ground truth depths, which are not available for large-scale datasets. DepthGAN (Shi et al. (022) uses predictions from a depth estimator to guide the training of 2D GAN. Exploiting monocular depth estimators for improving neural rendering was also explored in concurrent work (Yu et al. (022). however, their goal is just geometry recon- struction. The core characteristic of our approach is taking the depth estimator imprecision into account by training a depth adaptor module to mitigate it (see $32)).
METHOD
We build our generator on top of EpiGRAF (Skorokhodov et al., (2022) since its fast to train, achieves reasonable image quality and does not need a 2D upsampler, relying patch-wise training in-
3

---

Published as a conference paper at ICLR 2023
stead (Schwarz al. 2020 Given a random latent code 2, our generator G produces tri tri-plane representation for the scene. Then, a shallow 2-layer MLP predicts RGB color and density a values from an interpolated feature vector a 3D coordinate. Then, images and depths are volumetrically rendered (Mildenhall et al., 2020) at any given camera position. Differently to prior works (Chan et al. 2021 Niemeyer & Geiger 202Tb) that utilize fixed camera distribution, we sample camera from a trainable camera generator C (see (3.I). We render depth and process it via the depth adaptor (see {32)), bridging the domains of rendered and estimated depth maps (Miangoleh et al. (021), Our discriminator D follows the architecture of StyleGAN2 (Karras et al. 2020a), additionally tak- ing either adapted or estimated depth as the 4-th channel. To further improve the image fidelity, we propose a simple knowledge distillation technique, that enriches D with external knowledge obtained
from ResNet (He et al. 2016) (see 3.3
The overall model architecture shown in Fig. 2
3.1 LEARNABLE "BALL-IN-SPIIRE" CAMERA DISTRIBUTION
Limitations of Existing Camera Parameterization. The camera parameterization of existing 3D generators follows an overly simplified distribution its position sampled on a fixed-radius sphere with fixed intrinsics, and the camera always points to (0, O, 0). This parametrization has only two degrees of freedom: pitch and yaw (4pps in Fig. (a)), implicitly assuming that all the objects could be centered, rotated and scaled with respect to some canonical alignment. However, natural in-the-wild 3D scenes are inherently non-al non-alignable: they could consist of multiple objects, objects might have drastically different shapes and articulation, they could even be represented only as volumes (like smoke). This makes the traditional camera conventions ill-posed for such data.
Learnable "Ball-in-pphere" Camera Distribution. We introduce a new camera parametrization which we call "Ball-in SpSheere".... Contrary to the standard one, it has four additional degrees of freedom: the field of view (Pov, and pitch, yaw, and radius of the inner sphere, specifying the look-at point within the outer sphere (Ploekat in Fig. 3 (b)). Combining with the standard parameters on the outer sphere, our camera parametrization has six degrees of freedom yp 1ppps
(Prov || |Plookat],
where || denotes concatenation.
Instead of manually defining camera distributions, we learn the camera distribution during training for each dataset. In particular, we train the camera generator network C that takes camera parameters sampled from a sufficiently wide camera prior &p and produces new camera parameters s. For class conditional dataset, such as ImageNet where scenes have significantly different geometry, we additionally condition this network on the class label e and the latent code 2, i.e. 4p = C(y'' c). For single category dataset we use 4 C(('',2)
Camera Gradient Penalty. To the best of our knowledge, CAMPARI (Niemeyer & Geiger 2021a) is the only work which also learns the camera distribution. It samples set of camera parameters from a wide distribution and passes it shallow neural network, which produces a residual Ap = o - 4p for these parameters. However, we observed that such regulariza- tion is too weak for the complex datasets we explore in this work, and leads to a distribution collapse (see Fig. Z). Note, that a similar obser- vation was also made by Gu et al.) (2022).
Ppas Plov
Ppos
flov
P tookal
(0,0.0)
(a) Standard camera model
(b) Our camera model
To prevent the camera generator from produc- ing collapsed camera parameters, we seek a new regularization strategy. Ideally, it should prevent constant solutions, while at the same time, reducing the Lipschitz constant for C, which is shown to be crucial for stable train- ing of generators (Odena et al., 2018). We can achieve both if we push the derivatives of the predicted camera parameters with respect to the prior camera parameters either one or minus one,
Figure 3: Camera model. (a) Conventional cam- era model designed for aligned datasets and uses just 2 degrees of freedom. (b) The proposed "Ball- in-Sphere" parametrization has 4 additional de- grees of freedom: field of view and the look at position. Variable parameters are shown in blue.
4

---

Published as a conference paper at ICLR 2023
RGB
nardam
image
depth
d
Figure 4: Depth adapter. Left: An example of real image with its depth estimated by LeReS (Mi- angoleh et al. (021). Note that the estimated depth has several artifacts. For example, the human legs are closer than the tail, eyes are spaced unrealistically, and far-away grass predicted to be close. Middle: depth adapter meant to bridge the domains of predicted and NeRF-rendered depth. Right: a generated image with its adapted depth maps obtained from different layers of the adapter.
arriving at the following regularization term:
0ep: 0
0cp; 0.p
Sup
where v E sp is the camera sampled from the prior distribution and yi &p is produced by the camera generator. We refer to this loss as Camera Gradient Penalty. Its first part prevents rapid camera changes, facilitating stable optimization, while the second part avoids collapsed posteriors.
3.2 ADVERSARIAL DEPTH SUPERVISION
To instill a 3D bias into our model, we develop a strategy of using depth maps predicted by an off-the-shelf estimator E (Miangoleh al. 2021), for its advantages of being generic and readily applicable for many object categories. The main idea is concatenating depth map as the 4-th chan- nel of the RGB the input of the discriminator. The fake depth maps in this case obtained with the help of neural rendering, while the real depth maps are estimated using monocular depth estimator E. However, naively utilizing the depth from leads to training divergence. This happens because could only produce relative depth, not metric depth. Moreover, monocular depth estimators are still not perfect, they produce noisy artifacts, ignore high-frequency details, and make prediction mistakes. Thus, we devise mechanism that allows utilization of the imperfect depth maps. The central part of this mechanism is a learnable depth adaptor A, that should transform and augment the depth map obtained with neural rendering to look like a depth map from E.
More specifically, we first render raw depths from NeRF via volumetric rendering:
d = ct (()))t
where ln,14 E are near/far planes, T(I) accumulated transmittance, and r(t) a ray. Raw depth shifted and scaled from the range of [tn, ty] into [-1,1] to obtain normalized depth d:
d d= d (t, + f + b)/2
t, fn
where b E [0, (t,, + 22] is an additional learnable shift needed to account for the empty space in front of the camera. Real depths are normalized into the [-1, 1] range directly.
Learnable Depth Adaptor. While d has the same range as d,, it is not suitable for adversarial supervision directly due it the imprecision of E: G would be trained to simulate its prediction artifacts. To overcome this issue, A(d) E k where h x w is a number of sampled pixels. This depth (fake d, or real d.)) is concatenated with the RGB input and passed to D.
introduce depth adaptor A adapt the depth map d,
The depth adaptor models artifacts produced by E, so that the discriminator should focus only on the relevant high level geometry. However, too powerful A would be able to fake the depth completely, and will not learn the geometry. This is why we structure as just a 3-layer con- volutional network (see Fig. 4). Each layer produces separated depth map with different levels
5

---

Published as a conference paper at ICLR 2023
EGAD
EG3D
(wide)
EpiGRAF
3DGP
Bare 3DGP
ADS PUO) m
RADS Piul Bu ADS PIA0
Real images
SDIP DoggSer
SDIP Elephant
0"
30"
0"
30" 0"
30
0F
30"
0"
30" 0"
30"
0"
30'
0"
30
(comeras (riCEOwA)
Figure 5: Qualitative multi-view comparisons. Left: samples from the models trained on single- category datasets with articulated geometry. Two views shown for each sample. Middle: ab- lations following Tab. where we change the probability of using the normalized rendered depth P(d). EG3D, EpiGRAF, and P(d) 0 do render realistic side views, due to the underlying flat geometry. Our full model instead generates realistic high-quality views on all the datasets. Right: randomly sampled real images. Zoom-in for greater detail.
of adaptation: (((2) (22) and (3) The final adapted depth d, is randomly selected from the set of {d, d' df (3)) Such design effectively learn good geometry while alleviating overfitting. For example, when D receives the original depth map d as input, provides to G a strong signal for learning the geometry. And passing an adapted depth map d to D allows G to simulate the imprecision artifacts of the depth estimator without degrading its original depth map d.
3.3 KNOWLEDGE DISTILLATION FOR DISCRIMINATOR
Knowledge from pretrained classification networks was shown to improve training stability and gen- eration quality in 2D GANs (Sauer et al. 2021 Kumari et al. 2022 Sauer et al. 2022 Casanoval cet al. 202I). A popular solution proposed by Sauer et al. (2021) 2022) is to use an off-the-shelf model discriminator while freezing most of its weights. Unfortunately, this technique is not ap- plicable our scenario since we modify the architecture of the discriminator by adding an additional depth input (see $3.2) and condition on the parameters of the patch similarly to EpiGRAF (Sko- rokhodov et al.] (022). Thus, we devise an alternative technique that can work with arbitrary architectures of the discriminator. Specifically, for each real sample, we obtain feature rep- resentations: from the pretrained ResNet (He et al., 2016) network and extracted from the final representation of our discriminator D. Our loss simply pushes as follows:
Laist lee - éll
(4)
Lais can effectively distill knowledge from the pretrained ResNet (He et al. 2016) into our D.
3.4 TRAINING
The overall loss for generator G consists of two parts: adversarial loss and Camera Gradient Penalty:
LG Ludv Σ
XupeCupps
where Ladv is the non natusttug loss (Goodfellow et al. 2014). We observe that a diverse distribu-
tion for camera origin is most important for leaning meaningful geometry, but it is also most prone to degrade to a constant solution. Therefore, we set Ay. = 0.3 for Ppas. while set X = 0.03 for
Prov and dxp, 0.003 for Ptookat- The loss for discriminator D, the other hand, consists of three

---

Published as a conference paper at ICLR 2023
parts: adversarial loss, knowledge distillation, and R1 gradient penalty (Mescheder al. 2018
LD = Ludv + Adia Edisa + R1
We use the same optimizer and hyper-paramenn as EpiGRAF. We observe that for depth adaptor, sampling adapted depth maps with equal probability is not always beneficial, and found that using P(d)= = 0.5 leads to better geometry. For additional details, see Appx B
4 EXPERIMENTAL RESULTS
Datasets. In our experiments, we use 4 non-aligned datasets: SDIP Dogs Mokady et al. 2022). SDIP Elephants (Mokady et al. 2022 LSUN Horses YYu et al. 2015), and ImageNet (Deng et al., 2009]. The first three are sing categgoy datasets that contain objects with complex articulated geometry, making them challenging for standard 3D generators. We found it useful to remove outlier images from SDIP Dogs and LSUN Horsesvia instance selection (DeVries et al.] 2020), reducing their size to 40K samples each. We refer to the filtered versions of these datasets as SDIP Dogsaok. and LSUN Horses respectively. We then validate our method on ImageNet, a real-world, multi- category dataset containing 1, 000 diverse object classes, with more than 1, 000 images per category. All the 3D generators (including the baselines) use the same filtering strategy for ImageNet, where 2/3 of its images are filtered out. Note that all the metrics are always measured on the full ImageNet.
Evaluation. We rely on FID Heusel et al. 2017) to mea- sure the image quality and 2i. which is computed on 2,048 images instead of 50k (as for FID) for efficiency. For ImageNet, we additionally compute Inception Score (IS) (Salimans et al., 2016). Note that while we train on the filtered ImageNet, we always compute the metries on the full one. There is no established protocol to evalu- ate the geometry quality of 3D generators in general, but the state-of-the-art ones tri-plane (Chan al. 2022
EG30 (wids camera)
3DGP
NFS:
NFS: 25.59
Sun et al. 2022) or MPI-based Zhao et al. (2022). and we observed that is possible to quantify their most popular geometry failure case: flatness of the shapes. For this, we propose Non-Flatness Score (NFS) which is computed as the average entropy of the normalized depth maps his- tograms. We depict its intuition in 6 and provide the details in Appx F
Figure 6: NFS for repetitive and diverse geometry with depth histograms.
4.1 3D GENERATION FOR SINGLE CATEGORY DATASETS
We show quantitative results for single category datasets in Tab. Da. where we compare with EG3D (Chan et al., 2022) and EpiGRAF (Skorokhodov et al. 2022p. EG3D was mainly designed for FFHQ (Karras et al. 2019) and uses the true camera poses inferred from real images as the camera distribution for the generator. But in our scenario, we do not have any knowledge about the true camera distribution, that is why stay as close possible the setup EG3D for, we use normal distribution for rotation and elevation angles with the same standard deviations as in FFHQ, which are equal to Syaw = 0.3 and pitch 0.155, respectively. Also, learn better geometry, we additionally trained the baselines with twice wider camera distribution: yaa = 0.6 and pach 0.3. While it indeed helped to reduce flatness, it also worsened the image quality: up to 500% as measured by FID2k. Our model shows substantially better (at least 2x than EG3D, slightly worse than StyleGAN2) FID and greater NFS on all the datasets. Low NFS indicates flat or repetitive geometry impairing the ability of the model to generate realistic side views. Indeed, as shown in Fig. (left), both EG3D (with the default camera distribution) and EpiGRAF struggle to generate side views, while our method (3DGP) renders realistic side views on all three datasets.
designed
Adversarial Depth Supervision. We evaluate the proposed Adversarial Depth Supervision (ADS) and the depth adaptor A. The only hyperparameter it has is the probability of using the non-adapted depth P(d) (see $3.2). We ablate this parameter in Tab. Dp. While FID scores are only slightly affected by varying P(d), we see substantial difference in the Non Mon-Flatness Score. We first verify that NES is the worst without ADS, indicating the lack of a 3D bias. When P(d) = O, the dis- criminator is never presented with the rendered depth d, while the adaptor learns to fake the depth,
7

---

Published as a conference paper at ICLR 2023
Table Comparisons on SDIP Dogs40k. SDIP Elephants, and LSUN Horses40k (a) includes EG3D (with the standard and the wider camera range), EpiGRAF, and 3DGP. For completeness, we also provide 2D baseline StyleGAN2 (with KD). (b) includes the ablations of the proposed contributions. We report the total training cost for prior works, our model, and our proposed contributions.
(a) Comparison EG3D, EpiGRAP 3DGP.
Dogsiok FIDx
SDIP
SDIP Elephants
LSUN
Ho NFS1
Training cost
Model
NFSt
FIDx +
NFS1
FIDx
(A 100 days)
EG3D
16.2
11.91
4.78
2.59
3.12
13.34
3.7
wide camera
24.44
5.76
17.88
19.44
25.34
3.7
EpiGRAF
25.6
3.53
8.24
12.9
6.45
9.73
2.3
3DGP (ours)
8.74
34.35
5.79
32.8
4.86
30.4
2.6
StyleGAN2 (with KD)
N/A
3.94
N/A
2.57
N/A
(b) Impact Adversarial Depth Supervision (ADS).
Bare 3DGP (w/o ADS, w/o C)
8.59
1.42
7.46
9.52
3.29
8.04
2.3
ADS, P(d) = 0.0
8.13
3.14
5.69
1.97
3.41
1.24
2.6
ADS, P(d) 0.25
9.57
33.21
6.26
33.5
4.33
32.68
2.6
ADS, P(d) 0.5
7.60
30.7
32.24
ADS, P(d)- = 1.0
12.2
27.2
12.1
26.0
8.24
27.7
2.5
pith
look-at
look-at pitch
book-s radias
yor
Your 077
Figure 7: Comparisons of regularization strategies. Left: selected generic and wide prior for each of the DoF's of camera model (top row). Without any regularization (no reg) with the residual-bee based model (residual) as in Niemeyer & Geiger (2021a), the camera generator collapses to highly concentrated distributions. In contrast, the proposed regularization (grad penalty) leads wider posterior. Right: random samples for each strategy. For (no reg) and (residual), meaningful geometry learned; only our method (grad penalty) leads good geometry.
leading to flat geometry. When P(d) = 1, the adaptor is never used, allowing D to easily determine the generated depth from the estimated depth, there is large domain gap (see Fig. 4. leading reduced FID scores. The best results are achieved with P(d) 0.5, which can be visually verified from observing Fig. 5 (middle). "bare" 3DGP, P(d) 0 and P(d) are unable to render side views, while the model trained with P(d) 0.5 features the overall best geometry and side views.
Knowledge Distillation. Here further provide insights for discriminator D using knowledge distillation strategy (see (3.3). We find that knowledge distillation provides an additional stability for adversarial training, along with the significant improvement in FID, which can be observed by comparing results of EpiGRAF in Tab. [ and Bare 3DGP Tab. Dh. Additionally, we compare different knowledge distillation strategies Appx C However, noted by |Parmar et al.) (202T), strategies which utilize additional classification networks, may provide significant boost to FID, without corresponding improvement visual quality.
"Ball-in-Spheer Camera Distribution. In Fig. 7 we analyze different strategies for learning the camera distribution: sampling the camera from the prior s' without learning, predicting the residual & sp,, and using proposed camera generator C with Camera Gradient Penalty. For the first two cases, the learned distributions are nearly deterministic. Not surprisingly. no meaningful side views can be generated geometry becomes flat. In contrast, regularization provides
8

---

Published as a conference paper at ICLR 2023
Table 2: Comparison between different generators on ImageNet 250 (note that 3DGP relies on
additional information in the form of depth supervision). Training cost is measured in A A100 days.
Method
Synthesis type
FID 4
IS 1
NFST
A100 days 4
BigGAN (Brock 2018
2D
8.7
142.3
N/A
StyleGAN-XL Sauer dl.
2022
2D
2.30
265.1
N/A
163+
ADM Dhariwal & Nichol
202T)
2D
4.59
186.7
N/A
458
EG3D Chan ct al.
2022
3D-wwrre
26.7
61.4
3.70
18.7
+ widc camera
3D-aware
25.6
57.3
9.83
18.7
VolumeGAN (Xu al.
2021
3D-wwarc
77.68
19.56
22.69
15.17
StyleNeRF (Gu et al. 2022
3D aware
56.54
21.80
16.04
20.55
StyleGAN-XL 3DPhoto (Shih et al. 2020
3D-aware
116.9
9.47
N/A
165+
Ap (Sorokkhddov et al. 2022
3D
47.56
26.68
3.93
15.9
wide camera
3D
58.17
20.36
12.89
15.9
3DGP (ours)
3D
19.71
124.8
18.49
28
sufficient regularization to C, enabling it to converge to sufficiently wide posterior, resulting into valid geometry and realistic side views. After the submission, we found a simpler and more flexible camera regularization strategy through entropy maximization, which discuss in Appx
4.2 3D SYNTHESIS ON IMAGENET
ImageNet (Deng et al. 2009) is significantly more difficult than single-category datasets. Following prior works, we trained all the methods in the conditional generation setting Brock al. (2018). The quantitative results are presented in Tab. 2 We also report the results of statc-of-ther generators for reference: as expected, they show better FID and IS than 3D generators, since they do not need to learn geometry, are trained on larger compute, and had been studied for much longer. Despite our best efforts to find a reasonable camera distribution, both EG3D and EpiGRAF produce flat or repetitive geometry, while 3DGP produces geometry with rich details (see Fig. D StyleN- eRF (Gu et al. 2022) and VolumeGAN al. (22T), trained for conditional ImageNet generation with the narrow camera distribution, substantially under-pert ander-performed in terms of visual quality. We hypothesize that the reason lies in their MLP/voxel based NeRF backbones: they have better 3D prior than tri-planes, but are considerably more expensive, which, in turn, requires sacrifices in terms of the generator's i expressivity.
2D
Training 3D generator from scratch is not the only way achieve 3D-aware synthesis: can lift an existing 2D generator into 3D using the techniques from 3D photo "inpainting". To test this idea, we generate 10k images with StyleGAN-XL (the current SotA on ImageNet) and then run 3DPhoto (Shih et al. 2020) on them This method first lifts photo into 3D via pre-trained depth estimator and then inpaints the holes with separately trained GAN model generate novel views. This method works well for negligible camera variations, but starts diverging when the camera moves more than 10" In Tab. [ we report its FID/IS scores with the narrow camera distribution: yau (0, 0.3) and pithh ~ N (T/2, The details are in Appx
CONCLUSION
In this work, we present the first 3D synthesis framework for in-the-wild, multi-category datasets, such as ImageNet. We demonstrate how utilize generic priors the form of (imprecise) monoc- ular depth and latent feature representation to improve the visual quality and guide the geometry. Moreover, we propose the new "Ball-in-Sphere' camera model with novel regularization scheme that enables learning meaningful camera distribution. On the other hand, our work still has several limitations, such as sticking background, lower visual quality compared to 2D generators, and no reliable quantitative measure of generated geometry. Additional discussion is provided in Appx A This project consumed sx NVidia A A10 GPU years in total.
We computed FID on 10k images rather than 50k (as for other generators) due to the computational costs: parallelized inference of 3DPhoto (Shih 2020) 10k images took 2 days on 8 V100s.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
Most importantly, we will release 1) the source code and the checkpoints of our generator as separate github repo; and fully pre-processed datasets used our work (with the corresponding extracted depth maps). In $ and $ and Appx B and also in the figures throughout the text, we provided complete list of architecture and optimization details needed reproduce our results. We are also open to provide any further details on our work in public and/or private correspondence.
7 ETHICS STATEMENT
There are two main ethical concerns around deep learning projects on synthesis: generation of fake content mislead people (e.g. fake news deepfakes and potential licensing, authorship and distribution abuse. This was recently discussed in the research community in the context of 2D) image generation (by Stable Diffusion Rombach et al.) (2022) and DALL-E Ramesh al. (2022) and code generation (by Github C While the current synthesis quality of our generator would still need to improved to fool an attentive human observer, we still encourage the research community to discuss ideas and mechanisms for preventing abuse in the future.
hhttps://enn http:://wwwwwiiiiiiilaaarhttttte
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Sherwin Bahmani, Jeong Joon Park, Despoina Paschalidou, Hao Tang, Gordon Wetzstein, Leonidas Guibas, Luc Van Gool, and Radu Timofte. 3d-aware video generation. arXiv preprint arXiv:2206. 144777220..4497,, 2022.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809. 11096, 2018.
Arantxa Casanova, Marlène Careil, Jakob Verbeek, Michal Drozdzal, and Adriana Komero-Soriano. Instance-conditionee gan. In Proceedings of the Neural Information Processing Systems Confer- ence, 2021.
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein, pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-awant 3D generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, al. Shapenet: An information 3d 2015.
model repository. arXiv preprint http:///gihuh..om//miccch//i
Dawson-Haggerty
trimesh.
2019. URL https: //
tri mmshhorgy
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- crarchical image database. In 2009 IEEE conference computer vision and pattern recognition, 2009.
Yu Deng, Jiaolong Yang, Jianfeng Xiang, and Xin Tong. Gram: Generative radiance manifolds for 3d-aware image generation. In IEEE Computer Vision and Pattern Recognition, 2022.
Maximilian Denninger, Martin Sundermeyer, Dominik Winkelbauer, Youssef Zidan, Dmitry Ole- fir, Mohamad Elbadrawy, Ahsan Lodhi, and Harinandan Katam. Blenderproc. arXiv preprint arXiv: 1911:/99110191111 2019.
Terrance Deiesss Michal Drozdzal, and Graham W Taylor. Instance selection for gans. Advances in Neural Information Processing Systems, 2020.
Terrance DeVries, Miguel Angel Bautista, Nitish Srivastava, Graham W. Taylor, and Joshua M. Susskind. Unconstrained scene generation with locally conditioned radiance fields. arXiv, 2021.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-988 8794, 2021.
Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In ICCV, 2021.
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanis- las Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy. Ievgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78): 1-8, 2021. URL httt::/tt wttp:pppppppp/ooeooooeeee 20 451.html
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings of the Neural Information Processing Systems Conference, 2014.

---

Published as a conference paper at ICLR 2023
Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: A style-based based 3d aware generator for high-resolution image synthesis. In International Conference on Machine Learning, 2022.
Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In Proceedings of the Neural Information Processing Systems Conference, 2017.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.025:0 2(7), 2015.
Yiwen Hua, Puneet Kohli, Pritish Uplavikar, Anand Ravi, Saravana Gunaseelan, Jason Orozco, and Edward Li. Holopix50k: A large-scale in-the-wild stereo image dataset. In CVPR Workshop on Computer Vision for Augmented and Virtual Reality, Seattle, WA, 2020., June 2020.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Tero Karras, Miik Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. arXiv preprint urXiv:2006.06676, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Ana- lyzing and improving the image quality of stylegan. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020b.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. arXiv preprint arXiv:2 12423, 2021.
Mijeong Kim, Seonguk Seo, and Bohyung Han. Infonerf: Ray entropy minimization for few-shot neural volume rendering. In CVPR, 2022.
Tachyeon Kim, Jachoon Oh, Nik Kim, Sangwook Cho, and Se- YouYoung Yun. Comparing kullback-leibler divergence and mean squared error loss in knowledge distillation. arXiv preprint arXiv:2105.088 08999 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412:61412.69880 2014.
Zhengfei Kuang, Kyle Olszewski, Menglei Chai, Zeng Huang, Panos Achlioptas, and Sergey Tulyakov. Neroie: Neural rendering of objects from online image collections. In Special In- terest Group on Computer Graphics and Interactive Techniques, 2022.
Nupur Kumari, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Ensembling off-the-shelf mod- els for gan training. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural radiance fields. In Proceedings of the IEEE International Conference on Computer Vision, 2021.
Quan Meng, Anpei Chen, Haimin Luo, Minye Wu, Hao Su, Lan Xu, Xuming He, and Jingyi Yu. Gnerf: Gan-based neural radiance field without posed camera. In Proceedings of the IEEE Inter- national Conference Computer Vision, 2021.
12

---

Published as a conference paper at ICLR 2023
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning, 2018
S Mahdi H Miangoleh, Sebastian Dille, Long Mai, Sylvain Paris, and Yagiz Aksoy. Boosting monoc- ular depth estimation models to high-resolution via content-adaptive nulti-resolution merging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
9685-9694, 2021.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing seenes as neural radiance fields for view synthesis. In European conference on computer vision, 2020.
Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze the discriminator: a simple baseline for fine- tuning gans. arXiv preprint arXiv:2002. 2020.
Ron Mokady, Michal Yarom, Omer Tov, Oran Lang, Daniel Cohen-Or, Tali Dekel, Michal Irani, and Inbar Mosseri. Self-distileed stylegan: Towards generation from internet photos. arXiv preprint 1rX1v:2202.12211, 2022.
Michael Niemeyer and Andreas Geiger. Campari: Camera-aware decomposed generative neural radiance fields. In 2021 International Conference on 3D Vision (3DV), 2021a.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021b.
Augustus Odena, Jacob Buckman, Catherine Olsson, Tom Brown, Christopher Olah, Colin Raffel, and lan Goodfellow. generator conditioning causally related gan performance? Interna- tional conference on machine learning, 2018
Roy Or-EI, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher- Shlizerman. StyleSDF: High- Righ-Resolution 3D-Consistent Image and Geometry Generation. arXiv preprint 11214.2 2021.
Gaurav Parmar, Richard Zhang, and Jun-Yan Yan Zhu. On buggy resizing libraries and surprising sub- tleties fid calculation. arXiv preprint arXiv:2104.11 1122,, 2021.
pmneila. Pymcubes. https::/gtthub..om///// cowppmmeitaa/yy 2015.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings the IEEE conference computer vision and pattern recognition, pp. 652-660, 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++:: Deep hierarchical fea- ture learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017b.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pp.
PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Proceedings of the Neural Information Processing Systems Conference, 2016.
13

---

Published as a conference paper at ICLR 2023
Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster. Advances in Neural Information Processing Systems, 34:17480-17492, 2021.
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 Conference Proceedings. 2022.
Johannes Lutz Schönberger and Jan-Michael Frahm. Structurefrrom-motion revisited. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Johannes Lutz Schönberger Enliang Zheng, Mare Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In Proceedings of the European Conference on Computer Vision, 2016.
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. Proceedings of the Neural Information Processing Systems Conference, 2020.
Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, and Andreas Geiger. Voxgraf: Fast 3d- aware image synthesis with sparse voxel grids. ARXIV, 2022.
Zifan Shi, Yujun Shen, Jiapeng Zhu, Dit- Yan Yeung, and Qifeng Chen. 3d-aware indoor scene synthesis with depth priors. 2022.
Meng-Li Shih, Shih-Yang Su, Johannes Kopf, and Jia-Bin Huang. 3d photography using context- aware layered depth inpainting. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Dong Wook Shu, Sung Woo Park, and Junseok Kwon. 3d point cloud generative adversarial net- work based on structured graph convolutions. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3859-3868, 2019.
Ivan Skorokhodov, Sergey Tulyakov, Yiqun Wang, and Peter Wonka. Epigraf: Rethinking training of 3d gans. In Proceedings of the Neural Information Processing Systems Conference, 2022.
Jingxiang Sun, Xuan Wang, Yichun Shi. Lizhen Wang, Jue Wang, and Yebin Liu. Ide-3d: In- teractive disentangled editing for high-resolution 3d-aware portrait synthesis. arXiv preprint 1555 2022.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, 2019.
Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text-and- image driven manipulation of neural radiance fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. NeRF-- Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.00 07.077 2021.
Ke Xian, Jianming Zhang, Oliver Wang, Long Mai, Zhe Lin, and Zhiguo Cao. Structure-guided ranking loss for single image depth prediction. In The IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Yinghao Xu, Sida Peng, Ceyuan Yang, Yujun Shen, and Bolei Zhou. 3d-aware image synthesis via learning structural and textural representations. arXiv preprint arXiw:2112.10759, 2021.
Yang Xue, Yuheng Li, Krishna Kumar Singh, and Yong Jae Lee. Giraffe hd: A high-resolution 3d-aware generative model. arXiv preprint arXiv:2203. 14954, 2022.
Xu Yan. Pointnet/poimtnett+ pytorch.
2019.
Wei Yin, Yifan Liu, and Chunhua Shen. Virtual normal: Enforcing geometric constraints for accu- rate and robust depth prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021.
14

---

Published as a conference paper at ICLR 2023
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.0m 03366666066,... 2015.
Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. arXiv preprint 17Xiv:2206.00665. 2022.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE con- ference on computer vision and puttern recognition, pp. 3712-3722, 2018.
Xuanmeng Zhang, Zhedong Zheng, Daiheng Gao, Bang Zhang, Pan Pan, and Yi Yang. Multi-view consistent generative adversarial networks for 3d-aware image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2022.
Xiaoming Zhao, Fangchang Ma, David Güera, Zhile Ren, Alexander G. Schwing, and Alex Colburn. Generative multiplane images: Making 2d gan 3d-aware. In Proc. ECCV, 2022.
Peng Zhou, Lingxi Xie, Binghing Ni, and Qi Tian. Cips-3d: 3d-aware generator of gans based on cundiiionally-indeeenne pixel synthesis. arXiv preprint 1rXiv:2100.09788, 2021.
15

---

Published as a conference paper at ICLR 2023
A LIMITATIONS
Lower visual quality compared to 2D generators. Despite providing a more reasonable repre- sentation of the underlining scene, 3D generators still have a lower visual quality compared to 2D generators. Closing this gap is essential for wide adaptation of 3D generators.
Background sticking. One common 3D artifact of 3DGP is gluing of the foreground and the back- ground. In other words our model predicts a single shape for both and there is no clear separation between the two. One potential cause of this artifact is the dataset bias, where most of the photos frontal, thus it is not beneficial for the model to explore backward views. Another reason is the bias of tri- tri-planes toward flat geometry. However, all our attempts to replace tri-planes with an MLP- based NeRF led to much worse results (see Appx D). Inventing different efficient parametrization may be an important direction.
No reliable quantitative measure of generated geometry. In this work we introduce Non-Flatness Score as a proxy metric for evaluating the quality of the underlying geometry. However it can capture only single failure case, specific for generators based tri-planes: flatness of geometry. Devising a reasonable metric applicable to a variety of scenarios could significantly speed up the progress in this area.
Camera generator C does not learn fine-grained control. While our camera generator is condi- tioned on the class label c, and, in theory, it should be able perform fine-grained control over the class focal length distributions (which is natural since landscape panoramas and close-up view of a coffee mug typically have different focal lengths), does not do this, as shown in Fig. [ We attribute this problem to the implicit bias of G to produce large-FoV images due to tri-planes parametrization. Tri-planes define limited volume box in space, and close-up renderings with large focal length would utilize fewer tri-plane features, hence using less generator's 's capacity. This is why 3DGP attempts to perform modeling with larger field-of-view values.
1000
Focal
900
800
ldk (sorted by the mean focal lenggth value)
Figure 8: Focal length distribution on ImageNet 256 learned by C. The blue solid line is the mean values, while lower/upper curves are 0.05 and 0.95 quantiles, respectively.
B IMPLEMENTATION DETAILS
This section provides additional architectural and training details. Also, note that we release the source code. Our generator G consists of M, S and T. Mapping network M takes noise E IR212 and class label e E 0,..., K - 1, where w R5122 Similar to StyleGAN2-ADA Karras et al. (2020a) and EpiGRAF Skorokhodoov et al. 2022), is 2-layer MLP network with LeakyReLU activations and 512 neurons in each layer. Synthesis network is a decoder network same StyleGAN2 Karras al. (2020b), except that it produce tri-plane feature features
the number of classes, and produces the style code
""",p"=.p"> E [3 A feature vector fzyz E 32 located (x. 2) E R3 is computed by projecting the coordinate back to the trinplane representation, followed by bi-linearly interpolating the nearby features and averaging the features from different planes. Following EpiGRAF (Skorokhodov et al. 2022), tri tri-plane decoder is a two- layer MLP with Leaky-Rel.U activations and 64 neurons in the hidden layer, that takes tri-plane
16

---

Published as a conference paper at ICLR 2023
Table 3: Module names glossary.
G
Generator
M
Mapping network
S
Synthesis network
T
Tri-plane decoder
C
Camera generator
A
Depth adaptor
Discriminator
E
Depth Estimator
Feature Extractor
feature fayz as input and produced the color and density (RGB, σ)) in that point. We use same volume rendering procedure as EpiGRAF (Skorokhodov et al. 2022). We also found that increasing the half-life of the exponential moving average for our G improves both FID and Inception Score. We observed this by noticing that the samples change too rapidly over the course of training. In practice, we log the samples every 400k seen images and noticed that the generator could completely change the global structure of sample during that period.
Camera generator. Camera generator consists of linear layers with SoftPlus activations, it architec- ture depicted in Fig. [ We found crucial to use SoftPlus activation and not LeakyReLU, since optimization of Camera Gradient Penalty for non-smooth functions is unstable for small learning rates (smaller 0.02). Our gradient penalty minimizes the function L [g] + 1/lg), where [g] the input/output scalar derivative of the camera generator C. The motivation is to prevent the collapse of C into delta distribution and the intuition is the following. C can collapse into delta distribution in two ways: 1) by starting to produce the constant output for all the input values (this is being prevented by the first term [g] and 2) by starting producing to for all the inputs, which are at the end converted constants since we apply sigmoid normalization on top of its outputs normalize them into a proper range (e.g., pitch is bounded in (0, T)) - this, in turn, is prevented by the second term 1/lg|- The minimum value of this function is (which is achieved when the gradient norm is constant and equals to 1), and the function itself is visualized in Fig. 9b
Adversarial depth supervision. For depth adaptor, we use three layer convolutional neural net- work with 5x5 kernel sizes (since it is shallow, we increase its receptive field by increasing the kernel size) with LeakyReLU activations and 64 filters in each layer. Additionally we use one shared convolutional layer that converts 64 x h x w features to the depth maps. We use the same architecture for D as EpiGRAF (Skorokhodov et al.) 2022). but additionally concatenate a 1-channel depth to the 3-channel RGB input. Finally E and F is pretrainined LeReS (Miangoleh et al. 2021) and ResNet50 (He et al., (016) networks without any modifications. We used the timm library to extract the features for real images.
Similarly to EpiGRAF (Skorokhodov et al. 2022) we set the R2 regularization (Mescheder et al., 2018) term A, = 0.1 and knowledge distillation term Adist 1. R, regularization helps to stabilize GAN training and is formulated as a gradient penalty on top of the discriminator's real inputs:
R1 1 V
min. D
2
The (hand-wavy) intuition is that it makes the discriminator surface more flat in the vicinity of real points, making it easier for the generator to reach them. We train all the models with Adam optimizer (Kingma & Bal 2014) using the learning rate of 2e-3 and 31 0.0. 33 0.99. Following EpiGRAF (Skorokhodov et al. (022). our model uses patch-wise training with 64 x 64-resolution patches and uses their proposed / scale sampling strategy without any modifications. We use the batch size of 64 in all the experiments, since in early experiments didn't find any improvements from using a large batch size neither for our model nor for StyleGAN2, as observed by Brock et al. (018) and [auer al. 2022).
As being said in Section we use the instance selection technique by DeVries et al. (020)) to remove image outliers from the datasets since they might negatively affect the geometry. This pro- cedure works by first extracting a 2,048-dimension multivariate gaussian distribution on the obtained dataset and removing the images, which features
feature vector for each image, then fitting a
17

---

Published as a conference paper at ICLR 2023
2
lookat
L
Plookat
4 Fov
50
+
C
frvv
5
prior
9p pos
posterior
camera
4 pos
camera
parameters
parameters
(a) Camera generator C. condition on class labels e when generating the camera position & since it might be different for different classes. And we condition it on z when generating the look-at position and field-of-view since might depend on the object shape (c.g., there is higher probability to synthesize a close-up view of a dog's 's snout rather than its tail). Each MLP consists on 3 layers with Soft- plus non-linearities.
(b) Camera gradient penalty. We structure the regularization term for C as a fuuction ((9l)) = 1//gl (see ([) and this function is visualized above. This allows prevent the collapse of C into delta distribution by either producing constant values or very large/small values (which become constant after the sig- moid normalization).
Figure 9: Camera generator architecture and visualizing its corresponding regularization term.
have low probability density. For SDIP Dogs and LSUN Horses, we fit a multi-variate gaussian distribution for the whole dataset. For ImageNet, we fit separate model for each class with addi- tional diagonal regularization for covariance, which is needed due to its singularity: feature vector has more dimensions than the number of images in a class. We refer to Vries et al. 2020 for additional details.
Also note that we will release the source code and the checkpoints, which would additionally convey all other implementation details.
C KNOWLEDGE DISTILLATION
In this section we compare our knowledge distillation strategy with projected D (Sauer al. 2021 2022), another popular technique of utilizing existing classification models for GAN training. Note that projected relies on pretrained EfficientNet (Tan & Le, 2019). it freezes most of the weights and adds some random convolutions layers that project intermediate outputs of EfficientNet to the features, that will be later processed by discriminator. Note that this strategy relies on specific architecture of Discriminator D. thus in order to adapt it for 3DGP we have rely on two dis- criminators. First one is our disciminator D with additional depth input and conditioned on patch parameters and second one is projected D. In Tab 4 we show comparison between projected D and our knowledge distillation strategy on ImageNet (Deng et al. 2009) and SDIP Dogs40k Mokady al. 2022) datasets. First we compare these strategies for 2D generator architectures: StyleGiA3-- (large) (Karras et al. 2021) and StyleGAN2 (Karras et al., 2020a). It can be observed that projected D not generic and results heavily varies depending on generator architecture. Moreover training cost of projective D is higher. Finally we test this strategy for our 3D generator, and again we can observe that this technique lead to inferior results in both visual quality and training cost.
Typically, knowledge distillation in classifiers is performed using Kullback-Leibler divergence on top of logits (Hinton et al. 2015 Kim al. 202I). rather than La distance on top of hidden activa- tions, as in our case. There are two design reasons of why we use the Zz: 1) it is more generalizable and one can transfer knowledge from other models with the same design, like CLIP (Radford al,, 2021) or Mask R-CNN (He et al.) (017); 2) our discriminator's 's task shouldn't 't be able to perform classification, that why guiding with the KL classification loss not natural. However, in Fig. 10, we provide additional exploration with other knowledge distillation objectives on top of StyleGAN2 (Karras et al. 2020b) trained for conditional ImageNet generation on the 1282 resolu- tion. One observe that KL and La objectives perform approximately the same. Also, guiding by CLIP underperforms to ResNet guidance initially, but starts to outperform after more training is performed: we hypothesize that the reason is that it is more general model. We do not CL.IP
18

---

Published as a conference paper at ICLR 2023
Table 4: Comparing our knowledge transfer strategy with the one from StylcGAN-XL
ImageNet 128 @ 10M
SDIP Dugsam 256 @ 5M
Restricts D's
Method
FID.
Training cost +
FID 4
Training cost
architecture?
StyleGAN3-t (large)
11.9
6.42
7.3
No
Projected
22.6
Yes
Knowledge Distillation
16.3
11.9
7.3
No
StyleGAN2
33.7
1.81
9.79
1.3
No
with Projected D
160.5
3.83
4.10
2.1
Yes
with Knowledge Distillation with C
20.75
1.83
2,08
1.4
No
with Knowledge Distillation with KL
25.92
1.83
2.54
1.4
No
3DGP
53.6
21.3
No
Projected
105,9
Yes
-with Knowledge Distillation
27.8
4.51
2.6
No
100
StyleGAN2 ResNet50 distil ation [KL]
StyleGAN2 ResNet50 distill ation [Zg]
StyleGAN2 distillatoon [cg]
F10
2500
5000
7500
10000
12500
15000
17500
20000
# kimgs
Figure 10: Exploring different knowledge distillation objectives. Each model is trained for ditional ImageNet generation on the 128 resolution with all other hyperparameters being the same.
guidance for generators in this paper not to give it an unfair advantage compared other models.
D FAILED EXPERIMENTS
In this section we describe several ideas that have been tried this project, which however did not work for us.
Gradient Penalty for Depth Adaptor. To prevent depth adaptor A from completely faking the depth and forcing penalty that will force gradient of A close to one. We tried it in two setups: window-to-pixe gradient and pixel-to-pixe gradient. Even with the large weight for this loss and shallow adaptor with kernel size, it didn't enforce good geometry
still provide useful signal the generator, one could employ a gradient
3DGP with Projected D. Before devoting to knowledge distillation, we spend significant resources trying to incorporate projected D in our setup. For double discrininators setting, see Appx C we tested: different weighting strategies, less frequent updates for projected D, enabling projected after geometry was already learned. We also tested single projected D, where we learned addi- tional depth encoder that can consume our depth inputs. all these experiments learned geometry was significantly inferior the setting without projected D. Most of the time geometry becomes completely flat.
MLPs instead of tri-planes. We noticed that tri-planes extremely biased towards flat generation. Therefore one natural idea would be to utilize different representations, one possible candidate is ML.Ps Chan et al. (202). However experiments with MLPs always lead to significantly inferior
19

---

Published as a conference paper at ICLR 2023
visual quality. Our hypothesis is that they just don't have enough capacity to model large scale datasets.
Few-shot NeRF regularization. We also try to improve learned geometry by incorporating ray entropy minimization loss from InfoNeRF Kim et al. (2022) with different weights. But the model always diverges. We hypothesize that this is because it is significantly more complicated to find proper geometry, if the model is stuck in local minimum where the object densities are very sharp.
Normal supervision. Another prominent idea was utilizing normal supervision, since generic net- works for normals Eftekhar et al. (2021) are also widely available and provide good results on the arbitrary data. Since computing normals requires gradient of density with respect (xr, y, 2) and pytorch does not provide second order derivatives for F grid Rl we developed custom CUDA kernel that computes second derivative for grid sample. le. Unfortunately we found that normals supervision is inferior to depth and it is much easier for the generator G to fake normals.
Preventing C collapse via variance/entropy/ v regularization. Before arriving to our cur- rent form we had been experimenting with other forms of regularization: maximizing variance with some small loss coefficient, or entropy (in the assumption that the posterior distribution gaussion), or additionally pushing mean/skewness/kurrte time, the generator was finding the ways to "cheat" the regularization and managing to produce ei- ther a delta distribution mixture of delta distributions (it very "likes" doing so since, it is able to completely flat images and cheat the geometry in this regime).
to the one of the gaussian distribution. But each
E ADDITIONAL SAMPLES
Since is much easier to visualize the samples from NeRF-based generators as videos rather than RGB images, we provide all the additional visualizations on http:://swpp.esccccc....ibbbb//
F NON-FLATNESS SCORE DETAILS
To detect and quantify the flatness of 3D generators, we propose a Non-Flltness Score (NFS), To compute NFS, sample N latent codes with their corresponding radiance fields. For each radiance field, we perform integration following Eq. 2 to obtain its depth, however, we first set the 50% of lowest density to zero. This is necessary cull spurious density artifacts.
We then normalize each depth map according to the corresponding near and far planes and compute histogram with B bins, showing the distribution of the depth values. To analyze how much the depth values are concentrated versus spread across the volume, we compute its entropy for each distribution. Averaging over N depth maps gives the sought score:
N
B
1
NFS-
exp
(8)
N
h
where ()) is the normalized number of depth values in the j-th bin of the i-th depth map (()) with N 256 and B 64. NFS does not directly evaluate the quality of the geometry. Instead, it helps to detect and quantify how flat the generated geometry is. In Fig. 6 we show examples of repetitive geometry generated by EG3D and more diverse generation produced by 3DGP along with their depth histograms and NFS.
Intuitively, NFS should provide high values to EG3D with a wide camera distribution, since its repetitive shapes are supposed to span the whole ray uniformly. But surprisingly, it does not do so and is able to relfect this repeptitiveness artifact as well, yielding low scores, as can be observed from Tables and
G INVESTIGATING THE POTENTIAL DATA LEAK FROM DEPTH ESTIMATOR
LeReS Miangolch et al. 2021) depth estimator (which was used in our work) was pre-trained on a different set of images compared to ImageNet (or other animals datasets, explored in our work),
20

---

Published as a conference paper at ICLR 2023
EG3D (wide)
EpiGRAF (wide)
3DGP
30
0
30
-30
0
30
30
0
30
Figure 11:
Random samples for random classes on ImageNet 256 (random seed of 1 for the first ImageNet classes).
21

---

Published as a conference paper at ICLR 2023
RGB
d
d
d
d
(22
3)
a
a
Figure 12: Depth maps produced by 3DGP.
22

---

Published as a conference paper at ICLR 2023
Figure 13: Depth maps on ImageNet 256 predicted by LeReS (Miangoleh et al. 202 depth estimator. See the generated depth by 3DGP in Fig. 12
23

---

Published as a conference paper at ICLR 2023
Figure 14: Random images from each pre- pre-rrinnin dataset of the LeReS MMangolehh al. 2021 depth estimator, used in our work. From top to bottom: HRWSI (Xian et al. 2020), Depth (Yin et 202T), Holopix50k (Hua al. 2020). and Taskonomy (Zamir al. 2018).
and this theoretically could help our generator achieve better results by implicitly giving it access to broader image set rather than giving only texture-less geometrie guidance. In this section, we investigate this, and conclude that it is not the case: the pre-training data of LeReS is too different from the explored datasets, containing almost no animal images while the explored datasets very animal-dominated, including ImageNet which contains 482 animal classes. Hence, LeReS has no chance in helping our generator by giving it access to additional data.
LeReS (Miangoleh et al. 2021) trains combination of 4 datasets (in fact, parts of them): 1) HRWSI (Xian et al. 2020) contains outdoor city imagery (e.g., buildings. monuments, land- scapes); 2) DiverseDepth (Yn al. 2021) contains clips from in-the-wild movies and videos; 3)
24

---

Published as a conference paper at ICLR 2023
diverse, in-the-will web images (it is the most similar to Ima- geNet terms underlying data distribution); and 4) Taskonomy (Zamir et al. 2018 contains indoor scenes (bedrooms, stores, etc.). In Fig. [4 we provide 30 random images from each dataset. In Tab. [6] we rigorously explore how many animal images does the pre-training data of LeReS con- tains. We perform this in ways: by directly counting the amount of animals with the pre-trained Mask R-CNN model He et al. (2017) and by computing FID scores between the pre-training datasets of LeReS and ImageNet animal subset. The pre-trained Mask R-CNN model from Torch Vision is able to detect 10 animal classes: birds, cats, dogs, horses, sheeps, cows, elephants, bears, zebras and giraffes. Following the official tutorial, we used threshold of 0.8. We downloaded the datasets with the official download scripts of LeReS.
Holopix50k (Hua al. 2020)
Table 5: Investigating the distribution overlap between LeReS Miangolch et al. (2021) training data and the animals subset of ImageNet.
Animal subset
FID
Dataset
#images
#images
%images
ImageNet
ImageN
HRWSI Xian et al. 2020)
18.2k
1.05k
5.75%
71.48
97.79
DevtrseDepth Yin al.
(2021
95.4k
7.9k
8.26%
90.99
122.6
Holopix50k Hua ct al.
(2020)
42k
4k
9.56
47.44
81.62
Taskonomy
Zamir et
2018
134.7k
0.5k
0.36%
135.1
154.3
All pre- pre-training data
290k
9.4k
3.79%
77.46
108.6
ImageNet
1281.2k
618.75 7k
48.3%
0.0
19.06
From Tab. 5 one can see that the pre-training data of LeReS contains just 3.79% of animal images, while ImageNet contains 48.3%. In this way, it is too far away in terms of distribution from the Animal subset ImageNet. From this, one can conclude the following: Depth Estimator has al- most never seen animals during training, since its training datasets have 600 fewer animal images than ImageNet. This means that our generator does not receive any unfair advantage in synthesiz- ing animals from using adversarial depth supervision by implicitly getting access larger set of images.
How does it affect generator's performance in synthesizing animals? Will it have unusually poor FID compared to non-animal data or compared to other methods? In Tab. 6 we report FID scores of each generator on animal non-animal subsets of ImageNet. From these scores, one can see that the trend is the same as for FID on full ImageNet. This implies that our generator performs equally well on data which constitutes the main part of our training dataset and not part of the depth estimator pre-training.
In this way. one can conclude that adversarial depth supervision helps to achieve better synthesis quality only through geometric guidance. It does not help the generator to do this by leaking the knowledge of pre-trained data from the pre-trained depth estimator.
Table 6: FID scores of different generators on 482 animal classes of ImageNet 256 (each generator was trained on the classes of ImageNet 256²). For this evaluation, we used the images only from the animal classes for both real and fake data.
Method
Synthesis type
Animal FID +
Non-Animal FID 4
FID |
StyleGAN-XL
2D
4.53
4.81
2.30
EG3D (wide camera)
3D-aware
44.55
49.14
25.6
EpiGRAF
3D
78.65
81.38
47.56
3DGP (ours)
3D
47.32
50.94
26.47
25

---

Published as a conference paper at ICLR 2023
Figure 15: Examples of real images for our rendered ShapeNet 256 dataset.
H GEOMETRY EVALUATION ON SHAPENET
In this section, we present the details and the results of an additional study, which rigorously shows that our proposed adversarial depth supervision improves the geometry quality.
H.1 RENDERING DETAILS
We take the ShapeNet dataset Chang et al. 2015), which consists of 50k models from 54 classes and render from random frontal camera positions. For this, we set the camera on sphere radius 2 and randomly choose its position by sampling rotation and elevation angles from N r/8) and (TT/2, π/ /8), respectively. We chose frontal views to be closer to the real-worke scenario: the most popular image synthesis datasets are dominated by the frontal views. Also, if the dataset is not frontal, then there less necessity in additional geometry supervision: (which does not exist in modern an-the-wild datasets). For wide camera distribution, generator can learn proper geometry on its own: see experiments on synthetic datasets with 360 degrees camera coverage (Soorokhodov al. (2022). We render just single view per model since this is also the scenario which we typically have in real datasets (e.g., in all our explored datasets, there is just single view per scene). Some of the models ShapeNet are broken (they lack the corresponding mesh files), so we discard them. In total, this gave us 51209 training images, which are visualized in Fig. 15] We used BlenderProc Denninger al. (2019) do rendering. Also, during rendering we removed the transparent elements of the meshes since they were providing aliasing issues in the depth maps.
a good enough 3D bias
H.2 EXPERIMENTS
Experimental setup. After rendering, trained our main baselines, EG3D (Chan et al. 2022 and EpiGRAF (Skorokhodov et al.] 2022). on this dataset using the ground truth camera parametriza- tion and distribution when sampling the camera poses. For 3DGP, trained in the following variants: 1) with the default adversarial depth supervision (i.e., using P(d) = 0.5); and 2-3) with the adversarial depth supervision, but with corrupted depth maps, where corruption is simulated by blurring, similar to (DeVries et al.) 2021). Also, we trained StyleGAN2 (Karras et al. 2020b) as a lower bound on FID. We disabled camera distribution learning and knowledge distillation for the experiments with 3DGP to avoid unnecessary complications:
investigate only adversarial depth supervision in this study. For all the baselines, we use white background volumetric rendering.
We compare the methods in terms of FID (Heusel al., 2017 and also Frechet Point- cloud Distance (FPD), proposed by Shu et al. (2019). This metric is an FID analog for point clouds: it uses pre-traine pre-trained PointNet (Qi al. 201 7a) to extract the features from point clouds and then computes Frechet distance between real and fake representations sets, approximating their underly- ing distribution multi-variate normal
Evaluation.
26

---

Published as a conference paper at ICLR 2023
Table Geometry evaluation for different models on ShapeNet 256²
Chang al. 2015).
Method
Synthesis type
FID
FPD J
EG3D
3D-aware
17.22
2770.5
EpiGRAF
3D
21.58
424.0
3DGP (ours)
3D
14.38
80.67
with bbuu
3D
18.33
106.9
with Oblur = 3
3D
30.37
139.5
with Oblur 10
3D
99.41
807.4
StyleGAN2
2D
5.54
N/A
We used the official codebase of Shu et al. (2019) compute the metrics. In their original work, Shu et al. (2019) used customly trained PointNet (Qi et al. 2017a) to extract the features. But we observed an issue with it: it was sometimes producing unusually high FPD scores the order of 10 even for real point clouds. This is why we extracted pointclouds features with pre-trained PointNet++ (Qi al. 2017b) model from a popular public implementation (Yan 2019).
To extract point clouds from the models, we first extracted the surfaces from them via marching cubes. For this, we sampled density fields in 256 resolution, and then used the marching cubes implementation of PyMCubes (pmneila 2015) to extract the surfaces. Following EG3D, we thresh- olded the surface for marching cubes at the density value of a = 10. Our overall pipeline identical to the original procedure of SDF extraction in the EG3D repo, but simpler in terms of implementa- tion. After that, we extracted point clouds by sampling 2,048 points on the surface for both real and fake meshes using uniform sampling from trimesh Dawson DDawson-Hageerty al. (2019).
H.3 RESULTS
The results these experiments are presented in Tab. 7 We also provide random sample exam- ples (seed 1, classes 1-8) in Fig. 16 EG3D (Chan et al. 2022) fails to recover the geometry and generates inverted shapes with hollow geometry. This is highlighted by very high FPD score. EpiGRAF (Skorokhodov et al. 2022) is able to recover shapes, but models the white background via an additional sphere, which also damages its FPD. Our method, in contrast, learns proper ge- ometry and correctly drops the background. When the depth maps corrupted with blurring, it deteriorates both its geometry quality and texture quality, which is also confirmed by the metrics and the provided samples.
I DETAILS ON THE EXPERIMENTS WITH 3D PHOTO INPAINTING
In this section, we provide the details of our experiments combining 2D generation with 3D Photo Inpainting techniques.
For a 2D generator, we chose StyleGAN-XL. (Sauer et al., 2022) since it achieves state-of-the art visual quality on ImageNet, as measured by FID (Heusel et al.) 2017]. We took the original checkpoint for 256" generation from the official repository First, generated 50,000 images with the model and computed their FID: this gave the value 2.51 vs 2.26, reported by the authors, which is negligible difference.
After that, we used the original codebase of 3DPhoto (Shih et al. 2020) to synthesize the 3D vari- ations of 10k random images. We kept all the hyperparameters the same. In 3DPhoto, there is no spherical camera parametrization, this is why we had to simulate the following way. We assumed that the sphere center lies the median depth value from the original camera position (which is (0, O, 0)), and had been rotating the camera around that point. As discussed in $4 we used the narrow camera distribution to sample the points. Namely, used the normal distributions with standard deviations of Tyaw = 0.3 and pitch 0.15 for sampling rotation and elevation angles, respectively.
27

---

Published as a conference paper at ICLR 2023
EG3D
EpiGRAF
3DGP
3DGP
Figure 16: Random samples (seed 1, classes 1-8) for ShapeNet 256 for our method and the base- lines.
28

---

Published as a conference paper at ICLR 2023
0
5
10
15
20
25
Random samples from StyleGAN-XL (Sauer et al. 2022) paired with 3D Photo Inpaint- ing (Shih et al. 2020). For large camera movements, there appear severe artifacts, like gray areas and interpolation artifacts.
Figure 17:
For negligible camera variations, this strategy produces excellent results. While it is not perfectly view consistent, it inherits statc-of-the-arr image quality from the 2D generator, which was used to synthesize the original images. But for larger camera variations, the quality becomes deteriorate very quickly. Not only noticeable inpainting artifacts start to appear, but also outpainting problem emerges. The model was not trained to perform image extrapolation or inpaint large regions and fails to fill the holes which appear in larger camera movements. We provide the visualizations for it Fig.
J ENTROPY REGULARIZATION
After the submission, we explored another strategy to regularize the camera generator and observed that it is more flexible and easier tune. Intuitively, the strategy is to maximize the entropy of each predicted camera parameter dy:
C..
H(P:))
[- log Popp),
po(P))
where pa (y:: the camera generator's distribution over yp, is differential entropy.
To do this, we used the POT package Flamary et al. (2021) to minimize the Earth Mover's Distance with the uniform distribution:
C.p EMD(po Umm,, Mp.)),
(10)
where EMD(P.Q) Q) denotes the Earth Mover's Distance between the distributions P and Q, U b] the uniform distribution on [a, b] and my,, Mp. are minimum and maximum values for given camera parameter 42:-
In practice we used 64 samples to approximate the EMD and a non-regularized Wasserstein dis- tance.
29


Reviews:
[{"review_id": "vzNmEqXeYn0", "reviewer": "Reviewer_GoQR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a new method that learns a 3D-aware generative model on unaligned image datasets like ImageNet. \nThe authors propose to use an off-the-shelf depth estimator to guide 3D generator. The estimated depth is concatenated to the real image and feed to discriminator to guide the generated image and depth from the generator. Additionally, a learnable camera model based on the ball-in-sphere geometry is proposed to support the diverse camera settings in unaligned data, which includes position, fov, and lookat parameters. \nThe results show that the method can generate realistic images with depth on ImageNet dataset. ", "summary_of_the_review": "The paper itself is an interesting read and provides a step toward making 3D-aware generative models more practical and robust. My concern is that the experiments are unfair and could lead to confusion. I am happy to raise my score if the authors could convince me about the actual performance of the model given the off-the-shelf depth estimator. ", "strength_and_weaknesses": "** Strength **\n\n- The use of depth priors is an interesting idea to guide 3D-aware generative model. \n\n\n** Weakness ** \n\n1) The use of the off-the-shelf depth estimator could, however, result in unfair comparisons as the depth estimator relies on another datasets for training. To be fair, the depth estimator has to be trained on the same dataset for the generative model, but this might not be possible on ImageNet. \nThe effect of these additional data should be properly discussed. \n\n    a) Geometry prediction of the original depth estimator should be provided. \n  \n    b) Compare a) to the geometry generated by the proposed method. \n  \n    c) Compare the geometry generated by different depth estimators, and depth estimators trained on different datasets. Again, this point is not convincing to me still because the use of the additional data. \n\n\n2) More results on ImageNet should be presented as claimed in the paper title. For example, out of the 1000 classes on ImageNet, how many categories have the authors experimented? \nTo evaluate the geometry quality, the authors might want to refer to real object datasets such as ScanObjectNN, and compare the reconstructed geometry with these objects in point cloud format with FID evaluation. \n\n    [a] Uy et al., Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data, ICCV 2019.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "s9VH0C8B1m3", "reviewer": "Reviewer_K59a", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to learn a generative model of 3D scenes from large unaligned datasets. The paper argues that previous work only produced good results for well-aligned datasets (e.g. human and animal faces), and that directly applying such methods to unaligned data did not work well. In order to address this problem, the idea is to incorporate an off-the-shelf monocular depth estimator (in this case, LeReS) into their model architecture. In their proposed model, a discriminator aims to discriminate between an true RGBD image (the result of applying LeReS to a real unaligned image) and an RGBD image rendered from their triplane image representation. The discriminator also outputs an embedding vector which aims to represent the semantic content of the image, effectively distilling a feature model (in this case, a ResNet-50) into the discriminator. Additionally, the paper uses a more flexible camera parameterization than previous work: the model predicts a field-of-view and look-at position in addition to the position of the camera on a fixed-radius sphere. \n\nThe paper performs experiments on multiple datasets including a variety of LSUN categories and the full ImageNet dataset. It shows strong results compared to previous work using GANs (e.g. EG3D, EpiGRAF, etc.). \n", "summary_of_the_review": "This paper proposes a method for learning 3D generative models from large datasets of unaligned images. The paper tackles an important \ntopic in unsupervised 3D reconstruction. Although the paper does not have any groundbreaking ideas, it combines ideas and models from the existing literature in a different manner from previous work. It tackles an important and often-overlooked problem (that of generalizing existing methods to real-world datasets) and produces relatively compelling results. \n", "strength_and_weaknesses": "### Strengths \n* This paper is an empirical work, and the empirical results are relatively strong. \n* There is no reliable quantitative measure of generated geometry, so the paper introduces Non-Flatness Score. NFS is a very simple heuristic, but it does seem to provide at least some insight into the flatness of generated geometry. \n* The NFS values (Table 2) show that for unaligned datasets, previous work (e.g. EG3D) does not learn reasonable depth values, whereas 3DGP does learn reasonable depth values.\n* The proposed camera regularization strategy is intuitive. Figure 7 is an illustrative figure in this regard. \n* The inclusion of Appendix D (Failed Experiments) is very nice to see and much appreciated. The field would benefit substantially if sections like these were included in all papers. \n\n### Weaknesses\n* The novelty of this paper may or may not be viewed as a weakness. On the one hand, there are no groundbreaking ideas; the paper uses components that have been used before and combines them to obtain strong results. On the other hand, the exact method has never been used before; nobody else has combined these components in the precise manner that is being proposed here. \n* For the ImageNet experiments, the method uses more time/compute than competing methods (28.2 days vs 18.7 days or 15.9 days; Table 2). Training for a similar amount of time as prior work would allow for a fairer direct comparison.\n* When discussing GANs with external knowledge (at the top of page 4), the paper claims that “A similar technique is not suitable in our case as pre-training a generic RGB-D network on a large scale RGB-D dataset is problematic due to the lack of data.” However, it should be possible to train a generic (2D) RGB-D network on existing 2D image datasets by converting the 2D RGB images into RGBD images using monocular depth estimation; the paper effectively uses the same. \n\n#### Minor Weaknesses\n* The method appears to be quite sensitive to hyperparameters (e.g. those on page 7). This is somewhat expected, as other GAN-based methods also tend to be quite hyperparameter-sensitive. \n\n### Questions:\n* Existing 3D monocular depth estimation models are far from perfect. Are the failure cases from the monocular depth estimation network learned and reproduced by the generative network? This is mentioned briefly on Page 6 as motivation for the depth adaptor; it would be interesting to provide some more detail on this point. \n* Do symmetries (e.g. those with respect to the position of the camera on the sphere) cause any issues with learning the camera distribution? \n* With regard to the camera distribution, it seems strange to have the same camera distance for diverse ImageNet categories such as a goldfinch and a church. One can of course partially account for this by predicting different focal lengths, but have you tried learning a more flexible distribution of camera such as the inside of a sphere or a 3D annulus? Additionally, how does the distribution of learned focal lengths for different ImageNet classes look like? \n* Given that the model learns a triplane representation, it should be possible to rotate all the way around the scene. However, all of the examples shown are front-facing views with small-to-medium camera movements. Does the model learn reasonable backsides for objects? For example, the backside of the horses shown at the bottom of Figure 5. It is not critical for the paper to reconstruct the backside of objects. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "peQ7wbphif", "reviewer": "Reviewer_pJ3E", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends the 3D GAN to more diverse and un-aligned datasets (eg., ImageNet). This problem setting change is significant and very challenging, as we have learned from 2D fields (StyleGAN3--> StyleGAN-XL). This paper introduces several new modules to adapt an existing model EpiGRAF to this new setting including 1) depth-regularized GAN, 2) learnable camera formulation, 3) a knowledge distillation loss. Results on various dataset including ImageNet are promising in both rendering and geometry quality. ", "summary_of_the_review": "This paper tries to solve a very challenging task and proposes a legitimate framework consisting several new components. The results are promising and encouraging. I'm supportive overall and my current rating is 6 but not higher because the visual quality of the rendered images are not very satisfying. Also, the introduced framework is a bit complex since it incorporates several sub-modules/networks. \n\nFor the authors: I'm sorry and please point out if I overlooked anything. Happy to chat more from here.", "strength_and_weaknesses": "**Strength**\n- The biggest limitation of existing 3D GAN is the applicable data. This paper extends the 3D GAN to a more challenging setting where large-scale un-aligned dataset is used.\n- The introduced components are very interesting and works well. \n   1. The learnable camera formulation is very useful and can be a standard thing for follow-up works in this field. I think this is very important for models object-centric images in the wild. \n   2. The motivation of using depth as regularization makes lots of sense. Even though it's not easy to leverage estimated depth data, the proposed method managed to being beneficial.\n   3. The knowledge distillation loss is particularly useful for ImageNet. \n\n**Weakness**\n\n- My biggest concern is the visual quality. From the provided annoymous link, the generated images suffer from many artifacts. Even though the \"flat\" issue is addressed compared to EG3D, the visual quality is merely improved. Both of them fall far behind the 2D generators. \n- Even though using depth is resonable, it also makes the framework more complicated. It's quite tricky to do the Adversarial Depth Supervision (ADS) and set the p(d) correct. \n- More 3D GAN baselines besides EG3D (tri-plane) could be tested. It's possible that the \"flat\" issues are caused by the tri-plane representation.  Therefore, a volume-based or MPI-based 3D GAN could have very different results.\n- The loss in Eq.1 is somewhat weird. To me, this loss is minimizing and maximizing the same thing simultaneously. Why does this loss make sense? What does it converge to? What's the optimal value for this task? \n\n**Doubts**\n\n- Is “Ball-in-Sphere” formulation good enough for any images in-the-wild? What stops us from making it more flexible? For example, why do we need the camera to stay on the hemi-sphere? Why do the loss weights for the three components (pos, fov, lookat) differ so big?\n- How are the camera priors decided? In Fig.7 most priors are relatively uniform, but pitch/look-at are more like gaussian. It's unclear to me how to set these priors. Do we need to choose different priors when dataset changed?  Also, it's also unclear to me why residue based method fails so bad.\n- Why squared loss for distillation? This is not the standard option for knowledge distillation. Does it outperform standard setting, ie., KL-divergence? \n- Why not applying a depth loss on adaptor output? For example, applying LeReS on the rendered image to get the ground-truth depth and use it to supervise the output of adaptor.\n\n**Writing**\n\n- For 2D methods (StyleGAN2 (with KD)) in Tab.1-2, it's better to change their NFS score from 0 to N/A. \n- The qualitative results from the link could be added to the paper (appendix is fine).\n- The paper should be self-contained and thus R1 gradient penalty should be given in Appendix.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fKTgur8Tp-f", "reviewer": "Reviewer_yhJP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces the first 3D generator that works on 3D datasets without alignment. To this end, conditioning on an off-the-shelf monocular depth estimation approach, the proposed method incorporates generic depth priors to facilitate 3D generation. Moreover, richer camera models are taken into account, ensuring more realistic generation results. Finally, with a distillation mechanism,  a pre-trained image classification network provides further supervision to stabilize and improve training. Experiments on standard datasets validate the effectiveness of the proposed method.\n", "summary_of_the_review": "Based on the aforementioned pros and cons, I vote for a weak accept of this paper.", "strength_and_weaknesses": "**Strength**\n- The paper is well motivated as existing methods do not work well on non-aligned data.\n\n- Using an off-the-shelf depth estimation as a generic prior is intuitive, and experiments show that such prior helps improve the generated quality.\n\n- I find this paper easy and enjoyable to read and follow.\n \n- The visual generation quality is significantly higher than compared methods.\n\n- The learnable Ball-in-Sphere camera model does have a higher than a standard with 2 dof and works for objects that are not center-aligned.  \n\n- The experiments are carefully and extensively conducted. I appreciate that failed attempts are also mentioned in the appendix, which helps the audience and following researchers better understand this work.\n\n** Weaknesses**\n\n- The usage of an off-the-shelf depth estimator may make the comparison with other depth-estimator-free methods unfair. Plus, how do the performance and generalization of the monocular depth estimator affect the performance? Will the performance improve if trained with ground truth depth (for example if it is trained on a depth estimation dataset or a synthetic 3D dataset)?\n\n- The gradient penalty is widely used and studied in GAN-based approaches, the authors simply extend that to the parameters of the camera model. How good is the method if multiple objects are in the scene? The authors imply that their model works well on \"a scene \n consisting of multiple objects\", but I cannot find any experiments verifying this claim.\n\n- The so-called knowledge distillation for discriminator is more like an engineering trick and does not provide much technical insight to me. Moreover, the authors mention that \"it can work with arbitrary architectures of the discriminator,\" but it can only work on a discriminator that produces features with the same dimension as ResNet50. I am also wondering if this L2 KD loss works better than the mostly used KL divergence loss.\n\n- This paper should compare with a 3D photo baseline [a]. Combining a vanilla 2D image generator with such 3D photo generation methods could also produce realistic 3D image synthesis results while it seems to generalize on a larger range of images.  I wonder how the performance differs for these two different generation paradigms. Will the proposed method have a higher multi-view consistency? Will the proposed fail while the 3D photo methods work better in certain situations? Thus, this line of work should also be discussed.\n\n[a] 3D Photography using Context-aware Layered Depth Inpainting. CVPR 2020.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%