WildFusion: Learning 3D-Aware Latent Diffusion Models in View Space
Authors: Katja Schwarz
Seung Wook Kim
Jun Gao
Sanja Fidler
Andreas Geiger
Karsten Kreis
Venue: ICLR 2024 poster (2024)
Keywords: 3D-aware image synthesis
diffusion model
latent diffusion
3D-aware generative model
Primary area: generative models
TL;DR: Existing 3D-aware generative models assume posed images and a shared canonical camera system between instances, making them difficult to scale to in-the-wild data. Instead, we model instances in view space and leverage monocular cues for geometry.

Abstract: Modern learning-based approaches to 3D-aware image synthesis achieve high photorealism and 3D-consistent viewpoint changes for the generated images. Existing approaches represent instances in a shared canonical space. However, for in-the-wild datasets a shared canonical system can be difficult to define or might not even exist. In this work, we instead model instances in view space, alleviating the need for posed images and learned camera distributions. We find that in this setting, existing GAN-based methods are prone to generating flat geometry and struggle with distribution coverage. We hence propose WildFusion, a new approach to 3D-aware image synthesis based on latent diffusion models (LDMs). We first train an autoencoder that infers a compressed latent representation, which additionally
captures the images’ underlying 3D structure and enables not only reconstruction but also novel view synthesis. To learn a faithful 3D representation, we leverage cues from monocular depth prediction. Then, we train a diffusion model in the 3D-aware latent space, thereby enabling synthesis of high-quality 3D-consistent image samples, outperforming recent state-of-the-art GAN-based methods. Importantly,
our 3D-aware LDM is trained without any direct supervision from multiview images or 3D geometry and does not require posed images or learned pose or camera distributions. It directly learns a 3D representation without relying on canonical camera coordinates. This opens up promising research avenues for scalable 3D-aware image synthesis and 3D content creation from in-the-wild image data.

Published as a conference paper at ICLR 2024
WILDFUSION: LEARNING AWAAR LATENT DIFFUSION MODELS IN VIEW SPACE
Katja Schwarz! ,t Seung Wook Kim ,Jun 2.02.3.4, Sanja F.d Andreas Geiger', Karsten Kreis?
University of Tübingen, "NVIDIA, "Vector Institute, 'University of Toronto
ABSTRACT
Modern learning-basd based approaches to 3D-aware image synthesis achieve high photorealism and 3D-consistent viewpoint changes for the generated images. Existing approaches represent instances in shared canonical space. However, for in-the-wild datasets a shared canonical system can be difficult to define or might not even exist. In this work, we instead model instances in view space, alleviating the need for posed images and learned camera distributions. We find that in this setting, existing GAN-based methods are prone to generating flat geometry and struggle with distribution coverage. We hence propose WildFusion, new approach 3D-aware image synthesis based on latent diffusion models (LDMs). We first train autoencoder that infers a compressed latent representation, which additionally captures the images' underlying 3D structure and enables not only reconstruction but also novel view synthesis. To learn faithful 3D representation, we leverage cues from monocular depth prediction. Then, we train a diffusion model in the 3D-aware latent space, thereby enabling synthesis of high-quality 3D-consistent image samples, outperforming recent state-of-the-ar GAN-based methods. Importantly, our 3l--awrre LDMM trained without any direct supervision from multiview images 3D geometry and does not require posed images or learned pose or camera distributions. It directly learns a 3D representation without relying canonical camera coordinates. This opens up promising research avenues for scalable 3D-aware image synthesis and 3D content creation from mn-the-wild image data. See https://afaaschaarggiibbbbbk wttp:/hhww....w/wwoiiiiiio....i. for videos of our 3D results.
INTRODUCTION
High-quality 2D and 3D content creation is of great importance for virtual reality, animated movies, gaming and robotics simulation. In the past years, deep generative models have demonstrated immense potential, enabling photorealistic image synthesis at high resolution (Goodfellow al., 2014; Karras et al., 2020; 2021; Rombach et al., 2021; Dhariwal & Nichol, 2021b; Sauer et al., 2022). Recently, 3D-aware generative models advanced image synthesis generation (Schwarz et al., 2020; Chan et al., 2021; Gu et al., 2022; Jo et al., 2021; Xu et al., 2022; Zhou al., 2021b; Zhang et al., 2021; Or-EI et al., 2022; Xu al., 2021; Pan et al., 2021; Deng al., 2022;; Xiang et al., 2023a; Schwarz et al., 2022: Chan et al., 2022). They generate images with explicit control over the camera viewpoint. Importantly, these approaches do not require 3D training data or multiview supervision, which is costly or impossible to obtain for large-scale real-world data.
fies-sonsittent, 3D-aware image
While existing 3D-aware generative models achieve high photorealism and 3D-consistent viewpoint control, the vast majority of approaches only consider single-class and aligned data like human faces (Karras et al., 2020) or cat faces (Choi et al., 2020). The reason for this is that existing methods assume shared canonical coordinate system to represent 3D objects. As consequence, they require either poses from an off-the-shelf pose estimator (Chan et al., 2022; Schwarz et al., 2022; Gu et al., 2022; Xiang et al., 2023a) or assume, and sometimes learn to refine, given pose distribution (Schwarz et al., 2020; Niemeyer & Geiger, 202 lb; Shi et al., 2023; Skorokhodov et al., 2023; 2022). In contrast, in-the-wild images typically have no clearly defined canonical camera system and camera poses or pose distributions are not available or very challenging to obtain.
We propose instead model instances in view space: Our coordinate system is viewer-centric, i.c.. it parameterizes the space as seen from the camera's 's point of view. This removes the need for camera
Part of the work done during intersship at NVIDIA.

---

Published as a conference paper at ICLR 2024
WildFusion (owrs
3DGP
Novel View Geometry
Input
Generated Images + Geometry
Figure 1: WildFusion: Left: Input images, novel views and geometry from strst-stage autoencoder. Right: Novel
samples and geometry from our second-stage latent diffusion model and 3DGP (Skorokhodov el al., 2023) for the ImageNet classes "macaw" (top), "king penguin" (middle)) "kimono" (bottom). Included videos for more results.
poses and a priori camera pose distributions. unlocking 3D-aware image synthesis on unaligned, diverse datasets. We identify crucial challenges for training in view space: For complex datasets, without a shared canonical representation, existing techniques are prone to generating poor 31) representations and the GAN-based methods struggle with distribution coverage (see Fig. 2, Table 2).
To prevent generating flat 3D representations, we leverage cues from monocular depth prediction. While monocular depth estimators are typically trained with multi-view data, we leverage an off-the-shelf pretrained model, such that our approach does not require any direct multi-view supervision for training. Recent works (Bhat et al., 2023; Eftekhar et al., 2021; Ranftl et al., 2020; Miangolch et al., 2021) demonstrate high prediction quality and generalization ability to in-the-wild data and have been successfully applied to improve 3Drrcoonstruction (Yu et al., 2022). To ensure distribution coverage on more diverse datasets, we build our approach upon denoising diffusion-based generative models (DDMs) (Ho et al., 2020; Sohl-Dickstein et al., 2015; Song et al., 2020). DDMs have shown state-of-the-art 2D image synthesis quality and offer a scalable and robust training objective (Nichol & Dhariwal, 2021; Nichol et al., 2022b; Rombach et al., 2021; Dhariwal & Nichol, 202 Ramesh et al., 2022; Saharia et al., 2022; Balaji et al., 2022; Ho et al., 2022). More specifically. we develop 3D-aware generative model based on latent diffusion models (LDMs) (Rombach et al., 2021; Vahdat et al., 2021). By training an autoencoder first and then modeling encoded data a compressed latent space, L.MS achieve an excellent trade-off between computational efficiency and quality. Further, their structured latent space can be learnt to capture a 3D representation of the modeled inputs, as we show in this work.
Our 3D-aware LDM, called WildFusion, follows LDMs' two-stage approach: First, we train powerful 3D-aware autoencoder from large collections of unposed images without multiview supervision that simultancously performs both compression and enables novel-view synthesis. The autoencoder is trained with pixel-space reconstruction losses on the input views and uses adversarial training to supervise novel views. Note that by using adversarial supervision for the novel views, our autoencoder trained for novel-view synthesis without the need for multiview supervision, in contrast previous work (Watson al., 2022; Chan al., 2023; Liu et al., 2023). Adding monocular depth cues helps the model learn a faithful 3D representation and further improves novel-view synthesis. In the second stage, we train a diffusion model in the compressed and 3D-aware latent space, which enables us to synthesize novel samples and turns the novel-view synthesis system, i.c., our autoencoder, into 3D-aware generative model. We validate WildFusion on multiple image generation benchmarks, including ImageNet, and find that it outperforms recent prate-of-the-ard 3D-aware GANs. Moreover, we show that our autoencoder is able directly synthesize high-quality novel views for given single image and performs superior compared to recent GaN-based methods, which usually require an inversion process to embed a given image into their latent space (Abdal et al., 2019; Richardson et al., 2021; Tov et al., 2021; Zhu et al., 2020; Roich et al., 2023). Further, in contrast to inversion methods, our autoencoder is trained in a single stage and does not require pretrained 3D-aware GAN as well as elaborate and often slow techniques for latent optimization.
Main contributions: (i) We remove the need for posed images and a priori camera pose distributions for 3D-aware image synthesis by modeling instances in view space instead of canonical space. (fi)
2

---

Published as a conference paper at ICLR 2024
Wildiusion (ours)
3DGP
Figure 2: Sample Diversity: Generated samples ImageNet. Rows indicate class; columns show uncurated random samples. While WildFusion generates diverse samples due to diffusion model- based framework (leff), the GAN-based 3DGP (Skorokhodov ct al., 2023) has very low intra-class diversity (mode collapse, right).
We learn powerful 3D-aware autoencoder from unposed images without multiview supervision that simultancously performs compression, while inferring a 3D representation suitable for novel-view synthesis. (iii) We show that our novel 3wD-aware LDM, WildFusion, enables high-quality 3D-aware image synthesis with reasonable geometry and strong distribution coverage, achieving state-of-the-an performance in the unposed image training setting, which corresponds to training on in-the-wild image data. Moreover, we can more efficiently perform novel view synthesis for given images than common GAN-based methods and explore promising 3d-aware image manipulation techniques. We hope that WildFusion paves the way towards scalable and robust in in-the-wild 3D-awar image synthesis.
2 RELATED WORK
We briefly provide the most relevant related works in this section, and present a comprehensive discus- sion of the theoretical fundamentals and the related literature, including concurrent works, in App. B.
Most works on 3D-aware image synthesis rely on GANs (Goodfellow et al., 2014) and focus on aligned datasets with well-defined pose distributions. For instance, POF3D (Shi et al., 2023) infers camera poses and works in canonical view space; it has been used only for datasets with simple pose distributions. such as cat and human faces. To enable training on more complex datasets, 3DGP (Skorokhodov et al., 2023) proposes an elaborate camera model and learns to refine an initial prior on the pose distribution. Specifically, 3DGP predicts the camera location in canonical coordinate system per class and sample-specific camera rotation and intrinsics. This assumes that samples within a class share a canonical system, and we observe that learning this complex distribution can aggravate training instability. Further, the approach needs to be trained heavily filtered training data. In contrast, WildFusion can generate high-quality and diverse samples even when trained on the entire ImageNet dataset without any filtering (see Sec. 4.2). Moreover, and their dual discriminator to improve view consistency (Chan et al., 2022). Note that in contrast to POF3D, 3DGP, EG3D, and the vast majority of 3D-aware image generative models, WildFusion is not a GAN. GANs are notoriously hard to train (Mescheder et al., 2018) and often do not cover the data distribution well (see mode collapse in 3DGP, Fig. 2). Instead, we explore 3D-aware image synthesis with latent diffusion models for the first time.
EG3D's triplanes
Concurrently with us, IVID (Xiang al., 2023b) trains a 2D diffusion model that first synthesizes an initial image and subsequently generates novel views conditioned on it. However, its iterative generation is extremely slow because it requires running the full reverse diffusion process for every novel view. Instead, WildFusion only runs the reverse diffusion process once to generate (latent) 3D representation. Another concurrent work, VQ3D (Sargent et al., 2023), also proposes an autoencoder architecture, but uses requence-like latent variables and trains an autoregressive transformer in the latent space whereas we train a diffusion model on latent feature maps.
WILDFUSION
Our goal is to design a 3D-aware image synthesis framework that can be trained using unposed in-the-wild images. We base framework, WildFusion, LDMs (Rombach et al., 2021; Vahdat et al., 2021) for several reasons: (i) Compared to diffusion in the original data space, they offer excellent computational efficiency due to their compressed latent space. (ii) Diffusion models use
3

---

Published as a conference paper at ICLR 2024
ExL
Iapul View
Becoustrue Jon
Po
p
SR
Lvaa Eaps
CNN
Trensformn
Vol. Runel
Blooks
Blocks
FPN
Z
CNN
tow
Earpas
Novel View
Novd View mecteee
PP
SR CNN
De
Secoad Stage
Z
Diffusion Process
Z
Learned
Fixnn
Denoieing (LNel
Upsamppe
Intermeediate
represonntation
Figure 3: WildFusion Overview: In the first stage, train autoencoder for both compression and novel-view synthesis. Feature Pyramid Network (FPN) (Lin al., 2017) encodes given unposed image I into an 3D-aware latent representation Z. constructed as a 2D feature grid. A combination of transformer blocks and a CNN then decode Z into a triplane representation. which is rendered from both the input view Po and a novel view P ne. As we model instances in view space, Po is fixed, pre-defined camera pose. The input view is supervised with reconstruction losses. Adversarial training provides supervision for novel views. In the second stage, latent diffusion model is trained on the learned latent space to obtain 3D-aware generative model.
robust objective that offers sample diversity and does not suffer from problems that plague GANs such as mode collapse. (iii) Most importantly, one can construct a latent space that can be trained to not only perform compression but also to learn a powerful 3D representation for novel view synthesis, as we demonstrate in this work. Fig. 3 shows overview over WildFusion. It consists of two training stages (Rombach et al., 2021; Esser et al., 2021). In the first stage, our new autoencoder learns compressed and abstracted latent space suitable for reconstruction and novel view synthesis from single view training data. In the second stage, latent diffusion model is trained on the latent representation from the first stage autoencoder to obtain a full generative model.
3.1 AUTOENCODER FOR COMPRESSION AND NOVEL-VIEW SYNTIEESIS
Following the LDM framework (Rombach et al., 2021), we first train an autoencoder that encodes training data into latent representations. Unlike LDMs, however, where the task of the autoencoder is simply to compress and reconstruct the inputs, our setting is more complex, as the autoencoding model must also learn a 3D representation of the data such that it can infer reasonable novel views from a single input image. The capability for novel-view synthesis will be used later by the diffusion model to perform 3D-aware image synthesis with 3D-consistent viewpoint changes. However, as no multiview explicit 3D geometry supervision is available, this novel-view synthesis task is highly ander-constrained and non-trivial to solve. To aid this process, we provide additional cues about the geometry in the form of monocular depth supervision from pre- pre-trained network (Bhat et al., 2023).
Specifically, we concatenate a given image E x3xHxW with its estimated monocular depth D RIxHX channel-wise and encode them into compressed latent representation Z exxx via an encoder. As the encoder must infer a latent representation that encodes the underlying 3D object or scene of the input image, we found it beneficial provide both and as input (see Table 4). We choose a Feature Pyramid Network (FPN) (Lin et al., 2017) architecture due to its large receptive field. For LDMs, latent space compression is crucial to train diffusion model efficiently in the second stage. At input resolution 256 x 256 pixels, we use c=4, h-w-32 32 as in Rombach et al. (2021).
The decoder predicts a feature field from the compressed latent code Z. which can be rendered from arbitrary viewing directions. The feature field is represented with triplanes. In contrast to previous works (Chan et al., 2022; Skorokhodov et al., 2022), our triplane representation constructed from the latent feature map Z instead of being generated from random noise such that it is possible to reconstruct the input image. Taking inspiration from (Lin et al., 2023; Chan et al., 2023), we process Z with combination of transformer blocks to facilitate learning global features and a CNN to increase the resolution of the features. For the transformer blocks after the CNN we use efficient self-attention (Xic et al., 2021) keep the computational cost manageable for the larger resolution feature maps. We find that this combination transformer blocks and convolutional layers achieves better novel view synthesis than using fully convolutional architecture (see Table 4).
4

---

Published as a conference paper at ICLR 2024
Next, the feature field is projected to the input view Po and novel view P via volume rendering (Kajiya & Herzen, 1984; Mildenhall et al., 2020), as described in Sec. A. For the input view, we use the same fixed pose Po for all instances. This means that we are modeling instances in view space where the coordinate system is defined from the input camera's ' point of view. Therefore, novel views can be sampled uniformly from predefined range of angles around Po- In this work, we assume fixed camera intrinsics that we choose according to our camera settings. We find that using the same intrinsics for all datasets works well in practice (for details, see Appendix). To model unbounded scenes, we sample points along rays linearly in disparity (inverse depth) instead of depth. This effectively samples more points close to the camera and uses fewer samples at large depths. Recall that these points are projected onto triplanes for inferring the triplane features. To ensure that the full depth range is mapped onto the triplanes, we use a contraction function as in (Barron et al., 2022). The contraction function maps all coordinates to a bounded range, which ensures that sampling points are mapped to valid coordinates on the triplanes (Supp. Mat. for details). We find that representing unbounded scenes with a combination of disparity sampling and contraction function improves novel view synthesis, see Table 4. We render low-resolution images flo i depth maps Blow, Doo and feature maps Flow, Flow from the feature field using volume rendering at 64 x 64 resolution, see Eq. (4), The rendered low-resolution feature maps are then processed with a superresolution CNN module (SR CNN) that increases the spatial dimensions by 4x to yield the reconstructed image I and a novel view image I. (see Fig. 3).
Training Objective. We train the autoencoder with reconstruction loss on the input view and use an adversarial objective to supervise novel views (Mi et al., 2022; Cai et al., 2022). Similar to Rombach et al. (2021), we add a small Kullback-Leibler (KL) divergence regularization term Lx on the latent space Z. The reconstruction loss Lree consists of pixel-wise loss Lpx | II, perceptual loss Lvag (Zhang et al., 2018), and depth losses Ldepth.
As our monocular depth estimation D is defined only up to scale, we first compute a scale and shift t for each image by solving a least-squares criterion for s and f, which has a closed-form solution (Eigen et al., 2014). Following Ranfil et al. (2020), we enforce consistency between rendered (2D) depth Dime and the downsampled monocular depth D tow that was estimated on the input images: Cae [|(DDlow Low --))-D(o]je weights wf of the 3D sampling points (see Eq, (4)) with the depth. Let (sD + +)) denote the index set of the k sampling points closest to the rescaled monocular depth along ray r. Then,
We further found it beneficial to directly supervise the (normalized) rendering
Σ(a- Σ
3D deptt
KEK.
Intuitively, the loss encourages large rendering weights for the points close to the re-scaled monocular depth and small rendering weights for points farther away. Note that we regularize the sum of the weights in the neighborhood K, instead of the individual weights to account for imperfections in the monocular depth. In our experiments, we use a neighborhood size of k: = 5.
In addition to the reconstruction losses on the input view, we supervise the novel views of the input image. As per-pixel supervision is not available for novel views in our setting, we follow (Mi et al., 2022; Cai et al., 2022) and use adversarial training to supervise novel views. We use a dual discriminator (Chan et al., 2022), i.e., we upsample flow and concatenate it with Ln as input to the discriminator as a fake pair (see Fig. 3). Similarly, is first downsampled simulate lower resolution image, and then is upsampled back to the original resolution and concatenated with the original I to be used as the real pair to the discriminator. Let Eg. G6 and Do denote encoder, decoder and discriminator with parameters 0, s and o, respectively. For brevity, we omit the upsampling and concatenation of the discriminator inputs in the adversarial objective
pe
V(I, ne, where f(x) log(1 exp(-x)) and X controls the RI R1-regularizer (Mescheder et al., 2018).
(-D (Eo D). Pn.))) + f(D((D) \IV) (2)
We find that an additional discriminator Ddepth on the low-resolution depth maps further improves novel view synthesis. Ddepth helps to ensure the volume-rendered Dow is realistic (see Table 4). The autoencoder and discriminators are trained with alternating gradient descent combining the adver- sarial objectives with the reconstruction and regularization terms. Implementation details in Appendix.
5

---

Published as a conference paper at ICLR 2024
In conclusion, learn latent representation suitable not only for reconstruction but also novel view synthesis, we use both reconstruction and adversarial objectives. Note, however, that this is still fundamentally different from regular GAN-like training, which most existing works on 3D-aware image synthesis rely on. Our reconstruction losses on input views prevent mode collapse and ensure stable training. This makes our approach arguably more robust and scalable. Moreover, inputs the decoder are not sampled from random noise, like in GANs, but correspond to image encodings.
3.2 LATENT DIFFUSION MODEL
The autoencoder trained as described above learns a latent representation that is (i) compressed, and therefore suitable for efficient training of a latent diffusion model, and simultancously also (ii) 3D--awree in the sense that it enables the prediction of a triplane representation from which consistent novel views be synthesized corresponding to different viewing directions onto the modeled scene. Consequently, once the autoencoder is trained, we fit a latent diffusion model on its 3D-aware latent space in the second training stage (see Fig. 3). To this end, we encode our training images into the latent space and train the diffusion model on the encoded data. Importantly, although the autoencoder produces a compact 3D-aware latent representation, it is structured as a spatial 2D latent feature grid, as in standard LDMs for 2D image synthesis. Therefore, we can directly follow the training procedure of regular LDMMs (Rombach et al., 2021) when training the diffusion model. Eventually, this allows us to train powerful 3D) generative model that can be trained and sampled efficiently in 2D latent space. Our training objective is the standard denoising score matching objective as given in Eq (3), applied in latent space.
We adopt the architecture of regular 2D image LDMs (Rombach et al., 2021) to train our model. The denoiser Fu is implemented as a 2D U-Net with residual blocks (He et al., 2016) and self-attention layers (Vaswani et al., 2017). As discussed, the autoencoder's 's latent distribution is regularized with KI. divergence loss (Kingma& & Welling, 2014; Rezende et al., 2014; Rombach et al., 2021) (see Appendix) to be roughly aligned with the standard normal distribution. However, as we enforce a very low-weighted KL loss, the distribution can have larger variance. We estimate the standard deviation of the latent space using a batch of encoded training data and use the resulting value to normalize the latent distribution to yield standard deviation close to 1 before fitting the diffusion model. We use the DDIM sampler (Song et al., 2021) with 200 steps. More implementation details in the Appendix.
4 EXPERIMENTS
The performance of LDMs is upper-bounded by the quality of the latent space they are trained on, ie.. we cannot expect to generate better novel images than what the autoencoder achieves in terms of reconstructions. Hence, powerful autoencoder is key to training a good generative model in the second stage. We first analyze the reconstruction quality of WildFusion's 's autoencoder as well as its ability to synthesize novel views (Sec. 4.1). Next, we evaluate the full WildFusion model against the ofate-of-the-ard approaches for 3D-aware image synthesis (Sec. 4.2). We provide ablation studies in Sec. 4.3. Our videos included on the project page (https://aatjjschaaazz wwtp:www........ii...e wwtpgg.iittt.ll.bee iowillffusioo// show generated 3D-aware samples with camera motions. Further results are also shown in App. E.
Datasets. While previous 3D-aware generative models mainly focus on aligned datasets like portrait images, we study a general setting in which a canonical camera system cannot be clearly defined. Hence, we use non-aligned datasets with complex geometry: SDIP Dogs, Elephants, Horses (Mokady et al., 2022; Yu et al., 2015) as well as class-conditional ImageNet (Deng et al., 2009).
Baselines. We compare against the state-of-the-art generative models for 3D-aware image synthesis, EG3D (Chan et al., 2022), 3DGP (Skorokhodov et al., 2023) and POF3D (Shi et al., 2023) as well as StyleNeRF (Gu al., 2022). 3DGP and POF3D learn camera distribution in canonical space and can be trained on unposed images. Since we also aim to compare to other models working in the same setting as WildFusion, i.e., in view space, we adapt FG3D so that can be trained in view space and without camera poses (indicated as EG3D* below); see Appendix for details. We also train another variant of EG3D* where we add a depth discriminator to incorporate monocular depth information. Note that the regular version of EG3D that relies on object poses is clearly outperformed by 3DGP (as shown in their paper (Skorokhodov et al., 2023)); hence, we do not explicitly compare to it. Evaluation Metrics. For the autoencoder, we measure reconstruction via learned perceptual image patch similarity (LPIPS) (Zhan et al., 2018) and quantify novel view quality with Fréchet Inception Distance (nvFID) (Heusel et al., 2017) on 1000 held-out dataset images. Following prior art,

---

Published as a conference paper at ICLR 2024
Imput
WildFusion (ours)
EG3D
EG3D"
Drrrch
Figure 4:
Baseline comparisons for novel view synthesis on images unseen during training. Shown are the input image and two novel views per method. Viewpoints across methods the same. Included video for more results.
Table 1:
Reconstruction and novel-view synthesis on SDIP Dogs, Elephants and Horses resolution 256". All evaluations held-out test set. We report LPIPS, novel-view FID (nvFID) and non-flatness-scor (NFS).
Dogs
Flephants
SDIP Horses
Rec.Time
ILPIPS
DvFID
NFS
JLPIPS
uvFID
1 NES
[LPIPS
4 nvFFID
NES
[d[s]
EG3D* (Chan ot al., 2022)
0.44
71.23
12.01
0.43
27.99
12.89
0.40
68.25
12.90
> 100
EG3D* Diipp
0.38
36.65
14.43
0.45
56.86
15.74
0.34
36.27
12.98
> 100
WildFusion (ours)
0.21
17.4
31.8
0.28
9.0
32.0
0.22
13.4
28.7
0.04
Table 2: 3D-aware image synthesis results on unimodal datasets. Baselines above double line require camera pose estimation; methods below work in view space.
Togs
Flcyhontt
Horsss
JFDD
"NPS
1Precision
[Recall
LFID
"NPS
1Precision
'Recall
JFID
[FIDey
TNPS
"Precision
1Recall
POFED (Shi at, 2023)
17.4
25.9
0.57
1.36
6.4
30.2
0.59
0.30
16.4
15.3
01.56
3DGP (Skorokhodow 20233
5.9
36.3
0.73
3.7
0.67
13.0
EG3D* (Chan et al, 2022)
16.3
5.8
11.8
0.60
0.29
3.0
6.8
13.3
0.59
0.31
6.7
10.2
14.1
0.57
0.23
EG3D* Pars
18.7
8.8
13.9
0.71
0.15
4.5
8.5
18.3
0.56
0.24
6.5
8.7
13.7
0.59
0.30
StyleNeRI (GU st, 2022)
12.3
79
30,0
0.65
0.34
10.0
9.1
20.1
0.53
0.17
4.5
8.1
27.2
0.68
0.35
WildPesion (ours)
12.2
52
31.7
0.56
0.38
2.9
6.5
32.3
0.70
0.34
4.3
8.8
28.8
0.70
0.37
sample camera poses around the input view Po from Gaussian distributions with = 0.3 and 0.15 radians for the yaw and pitch angles (Chan et al., 2021). We also report non-flatness-scom (NFS) (Skorokhodov et al., 2023). It measures average entropy of the normalized depth maps' histograms and quantifies surface flatness, indicating geometry quality. For the full generative models, measure NFS and evaluate FID between 10k generated images and the full dataset, sampling camera poses as for nvFID. As FID can be prone to distortions (Kynkäänniemi et al., 2023), we also show FIDCLIP, which uses CLIP features. To quantify diversity, we report Recall, and we also show Precision (Sajjadi et al., 2018; Kynkaääniemi et al., 2019). Qualitative results display a view range of 30 and 15 for azimuth and polar angles around the input view, similar to Skorokhodov al. (2023).
4.1 AUTOENCODER FOR RECONSTRUCTION AND NOVEL- VIEW SYNTHESIS
As EG3D (Chan et al., 2022) is GAN-based approach, we need to perform GAN-inversion (Chan al., 2022; Roich et al., 2023) reconstruct input images (we use the scripts provided by the authors). Quantitative results are in Table 1, qualitative comparisons in Fig. 4 (more in App. E). Compared to EG3D using GAN-inversion, WildFusion's 's autoencoder achieves superior performance on all metries and is also more efficient, since we do not require lengthy optimization process (which occasionally diverges) to embed input images into latent space. Despite its low latent space dimension of 32 32 x our autoencoder achieves both good compression and novel view synthesis. In Fig. 1, Fig. 5 and in App. E, we also show novel view synthesis results on ImageNet, which performs equally well.
4.2 3D-AWARE IMAGE SYNTHESIS WITH LATENT DIFFUSION MODELS
SDIP Datasets. We compare WildFusion against state-of-the-art 3D-aware generative models (Table 2) and provide model samples in the App. E. Our videos in the Supp. Mat. show more generated 3D-aware samples, including smooth viewpoint changes. Compared to EG3D* (Chan et al. 2022), which effectively also models instances in view-space, WildFusion achieves higher performance on all metrics, i.e., in terms of image quality (FID/FIDeLLP), 3D geometry (NFS), and diversity (Recall). This validates the effectiveness of our LDM framework this setting. When also adding depth discriminator to EG3D*, WildFusion still achieves better FID and NFS and Recall. Note that in particular on NFS and Recall, the performance gap is generally large. We also outperform StyleNeRF. We conclude that previous works that operate in view space can struggle with flat geometry and distribution coverage when training on reasonably complex, non-aligned images. In contrast, WildFusion achieves ofate-of-the-an performance in this setting.
The baselines 3DGP (Skorokhodov et al., 2023) and POF3D (Shi et al., 2023) both rely on sophisti- cated camera pose estimation procedures and do not operate in view space, which makes direct
7

---

Published as a conference paper at ICLR 2024
Input Novel View J. Coom.
Generated Images + Goometry
Figure 5: Generated 3D-aware image samples and geometry by WildFusion. Included videos for results.
Table 3: 3D-aware im- age synthesis results on class-conditional ImageNet. For WildFusion, we re-
JFID
[FIDeuP
1NFS
"Precision
"Recall
3DGP (Skorokhodov et al., 2023)
19.7
8.1
18.5
0.31
0,02
EG3D* + Dushh
111.6
27.2
20.6
0.48
0.01
StyleNeRF (Gu et al., 2022)
77.7
17.5
28.0
0.38
0.03
port results for different classifier-free guidance scales 8 (App. C.2); ex- tended results in Table 7.
WildFusion (ours), s =1
65.1
15.3
33.6
0.58
0.16
WildFusion (ours), s
35.4
11.7
33.8
0,59
0.20
WildFusion (ours), s
25.5
11.6
33.6
0.53
0.13
comparison with WildFusion difficult. Nevertheless, WildFusion performs on-par or even better on the different metrics, despite not relying on posed images or learned pose or camera distributions at all. These previous works' reliance on complex canonical camera systems represents a major limitation with regards to their scalability, which our 3D-aware LDM framework avoids. Note that in particular on Recall, WildFusion always performs superior compared to all other methods, demonstrating that diffusion- based frameworks are usually better than based ones in terms of sample diversity.
ImageNet. We find that WildFusion outperforms all baselines on NFS, Precision and Recall by large margins, for varying classifier-fre guidance scales (Table 3). The extremely low Recall scores of the GAN-based baselines indicate very low sample diversity (mode collapse). We visually validate this for 3DGP, the strongest of the three baselines, in Fig. 2: 3DGP collapses and produces almost identical samples within classes, showing virtually high-quality samples. Note that this failure of 3DGP is pointed out by the authors (see "Limitations and failure cases" at https://snap- rewpaupp-eeerrch...e www.:hh.tthhhhhhooooeoo io/3dp//). The FID metric does not accurately capture that mode collapse. While we outperform EG3D and StyleNeRE also on FID, WildFusion is slightly behind 3DGP. However, it is known that FID is questionable metric (Kynkäänniemi et al., 2023) and the qualitative results in Fig. 5 and Fig. 2 show that WildFusion generates diverse and high-fidelity images and generally learns a reasonable geometry in this challenging setting. We believe that a mode-collapsed generative model like 3DGP, despite good FID scores, is not useful in practice. Finally, note that due to limited compute resources our ImageNet model was trained only with a total batch size of 256. However, non-3D-aware ImageNet diffusion models are typically trained on batch sizes > 1000 to achieve strong performance (Rombach al., 2021; Karras et al., 2022; Kingma& & Gao, 2023). Hence, scaling our model and the compute resources would likely boost the results significantly.
diversity. In contrast, our model produces diverse,
Interpolation and Generative Resampling, In Fig. 6, we use WildFusion to perform semantically meaningful 3D-aware image interpolation between two given (or generated) images. Moreover, in Fig. 7 we demonstrate how can use our 3D-aware latent diffusion model to refine images and geometry by only partially diffusing their encodings and regenerating from those intermediate diffusion levels. These applications highlight the versatility of WildFusion and have potential use cases in 3D-aware image editing. To the best of our knowledge, this is the first time such applications have been demonstrated for such 3D-aware image generative models. See the project
8

---

Published as a conference paper at ICLR 2024
Figure 6: 3D-Aware Image In- terpolation. We encode two im- ages into latent space (far left and far right), further encode into the diffusion model's ' Gaussian prior space (inverse DDIM), interpolate the resulting encodings, and gen- erate the corresponding 3D im- ages along the interpolation path.
Figure 7: 3D-Aware Generative Image Resampling. Given an im- age (far left), we forward diffuse latent encoding for varying numbers of steps and re-generate from the partially noised encod- ings. Depending how far diffuse, obtain varying levels of generative image resampling.
page (ttps:/hattaachwarrgiibbbbb wwtp:arwwgggi....owiwwiifiiioiie for animations with viewpoint changes for these 3D-aware image interpolation and generative image resampling results.
4.3 ABLATION STUDIES
We provide a detailed ablation study in Table 4, starting from a base model, where the decoder architecture follows EG3D's generator. We gradually introduce changes and track their impact on novel view synthesis (nvFID) and geometry (NFS) (corresponding samples in App. E in Fig. 12). For computational efficiency, we perform the study at image resolution 128 and reduce the num- ber of network parameters compared to our main models. Like LDMs (Rombach et al., 2021), the initial setting is trained only with reconstruction losses. Unsurprisingly, this results in planar geometry, indicated by low NFS. This changes when discriminator on novel views is added. However, geometry becomes noisy and incomplete, indicated by high NFS and worse nvFID. We hypothesize the purely convolutional architecture is suboptimal. Hence, in the next step, we instead use a combination of convolutions and transformer blocks (ViT (Dosovitskiy et al., 2020)) in the decoder, which improves novel view quality and results in less noisy geometry. Adding the monocular depth discriminator Deepth significantly improves nvFID, and we observe even bigger improvement when tailoring the model to represent unbounded scenes with disparity sam- pling and a coordinate contraction function. Further supervision on the rendered depth (Caepas) does improve results but as it does not hurt performance either, we kept it in our pipeline. Lastly, we find that both adding depth as an input to the encoder and directly supervising the rendering weights with c3D result in slight improvements in NFS and qualitatively improve geometry.
Table 4: Ablation study on SDIP Dogs.
Model configuration
nvFID
1NFS
Base config
53.2
10.3
Do
61.1
37.0
VIT hackhone Delepiht
48.3
33.9
40.5
34,0
modeling unbounded scenes
32.8
32.4
34.6
32.0
encode depth
33.3
33.3
L3D
degth
34.0
33.7
CONCLUSIONS
We introduce WildFusion, a 3D-aware LDM for 3D-aware image synthesis. WildFusion is trained without multiview or 3D geometry supervision and relies neither on posed images on learned pose or camera distributions. Key to our framework an image autoencoder with 3D-aware latent space that simultaneously enables not only novel view synthesis but also compression. This allows us to efficiently train a diffusion model in the autoencoder's latent space. WildFusion outperforms recent state-of-the-art GAN-based methods when training on diverse data without camera poses. Future work could scale up 3D-aware LDMs to the text-conditional setting, similar to how 2D diffusion models have been applied on extremely diverse datasets (Ramesh et al., 2022; Rombach et al., 2021; Saharia et al., 2022; Balaji et al., 2022).

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
Katja Schwarz and Andreas Geiger were supported by the ERC Starting Grant LEGO-3D (850533) and the DFG EXC number 2064/1 project number 390727645. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for support- ing Katja Schwarz. Lastly, we would like to thank Nicolas Guenther for his general support.
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4432 4441, 2019.
Titas Anciukevicius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J. Mitra, and Paul Guerrero. RenderDiffusiin: Image diffusion for 3D reconstruction, inpainting and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 18
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song. Karsten Kreis, Mika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Liu, ediff-i: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211.01322 2022. 2, 9,
18
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022. 20
Miguel Ángel Bautista, Pengsheng Guo, Samira Abnar, Walter Talbott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, Afshin Dehghan, and Josh M. Susskind. GAUDI: A neural architect for immersive 3d scene generation. arXiv, 2022. 17, 18
Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth:
Zero-shot transfer by combining relative and metrie depth. arXiv, 2023. 2, 22, 23
Shengqu Cai, Anton Obukhov, Dengxin Dai, and Lue Van Gool. Pix2nerf: Unsupervised conditional p-gan for single image to neural radiance fields translation. In CVPR, 2022. 5, 19, 21
Eric R. Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. Pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In CVPR, 2021. 1, 7, 18
Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry- aware 3D generative adversarial networks. In CVPR, 2022. 1, 4, 5, 6, 7, 17, 18, 20, 21, 22, 23
Eric R. Chan, Koki Nagano, Matthew A. Chan, Alexander W. Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. GeNVS: Generative novel view synthesis with 3D-aware diffusion models. In arXiv, 2023. 2, 4, 19
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Wor Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In CVPR, 2020.
Congyue Deng, Chiyu "Max" Jiang, Charles R. Qi, Xinchen Yan, Yin Zhou, Leonidas Guibas, and diffusion as general
Dragomir Anguelov. Nerdi: Single-view nerf synthesis with image priors. arXiv preprint arXiv:22 2022a. 18
Jia Deng, Wei Dong, Richard Socher, Li jia Li, Kai Li, and Li Fei-fei. Imagenet: hierarchical image database. In CVPR, 2009. 6
A large-scale
Yu Deng, Jiaolong Yang, Jianfeng Xiang, and Xin Tong. Gram: Generative radiance manifolds for 3d-aware image generation. In CVPR, 2022b. 2022,. 1, 18
10

---

Published as a conference paper at ICLR 2024
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 118880087944 2021a. 2, 17
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), NeurlPS, 2021b.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically- damped langevin diffusion. In ICLR, 2022a. 18
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. GENIE: Higher-Order Denoising Diffusion Solvers. In Advances Neural Information Processing Systems, 2022b. 18
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 16 words: Transformers for image recognition at scale. arXiv, 2010.11929, 2020.
Yilun Du, M. Katherine Collins, B. Joshua Tenenbaum, and Vincent Sitzmann. Learning signal- agnostic manifolds of neural fields. In Advances in Neural Information Processing Systems, 2021. 18
Emilien Dupont, Hyunjik Kim, M. Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data functa: Your data point is function and you treat it like In 39th International Conference on Machine Learning (ICML), 2022. 18
Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In ICCV, 2021.
David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In NeurlPS, 2014. 5, 22
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021. 4, 20
Rafail Fridman, Amit Abecasis, Yoni Kasten, and Tali Dekel. Scenescape: Text-driven consistent scene generation. arXiv preprint arXiv:2302.01/33, 2023. 23
Ian J. Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014. 3, 18
Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis. ICLR, 2022. 1, 6. 7, 8, 18, 23
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image
recognition. In CVPR, 2016. 6
Philipp Henzler, Niloy J Mitra, , and Tobias Ritschel. Escaping plato's cave: 3d shape from adversarial
rendering. In ICCV, 2019. 18
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter, Gans trained by a two time-scale update rule converge local nash equilibrium. In NeurIPS, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 202/ Workshop on Deep Generative Models and Downstream Applications,, 2021. 17, 22, 27
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020. 2, 17
Jonathan Ho, Chitwan Saharia, William Chan, Javid Fleet, Mohammad Norouzi, and Tim Salimans.
Cascaded diffusion models for high fidelity image generation. Mach. Learn. Res., 2022. 2, 18
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6:695-709, 2005. ISSN 1532-4435. 17
11

---

Published as a conference paper at ICLR 2024
Kyungmin Jo, Gyumin Shim, Sanghun Jung, Soyoung Yang, and Jaegul Choo. Cg-nerf: Conditional generative neural radiance fields. arXiv, 2021. 1, 18
James T. Kajiya and Brian Von Herzen. Ray tracing volume densities. In ACM Trans. Graphics, 1984. 5, 17
Nikolai Kalischek, Torben Peters, Jan D Wegner, and Konrad Schindler. Tetrahedral diffusion models for 3d shape generation. arXiv preprint arXiv:2211.13220 2022. 18
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In ICLR, 2018. 21
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019. 18, 19, 21
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In CVPR, 2020. 1, 19
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In NeurlPS, 2021. 1
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Advances in Neural Information Processing Systems, 2022. 8, 22
Seung Wook Kim, Bradley Brown, Kangxue Yin, Karsten Kreis, Katja Schwarz, Daiqing Li, Robin Rombach, Antonio Torralba, and Sanja Fidler. Neuralfield-Idm: Scene generation with hierarchical latent diffusion models. In CVPR, 2023. 18
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR, 2015. 21
Diederik P. Kingma and Ruigi Gao. Understanding diffusion objectives as the elbo with simple data augmentation, 2023. 8, 22
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. ICI.R, 2014. 6, 20
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
Tuomas Kynkäääniemmi Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. In ICLR, 2023. 7,8
Yushi Lan, Xuyi Meng, Shuai Yang, Chen Change Loy, and Dai. Self-supervised geometry-aware encoder for style-based 3d GAN inversion. arXiv, 2022. 19
Yu-Jhe Li, Tao Xu, Bichen Wu. Ningyuan Zheng, Xiaoliang Dai, Albert Pumarola, Peizhao Zhang, Peter Vajda, and Kris Kitani. 3d-aware encoding for style-based neural radiance fields. arXiv, 2022. 19
Yiyi Liao, Katja Schwarz, Lars M. Mescheder, and Andreas Geiger. Towards unsupervised learning of generative models for 3d controllable image synthesis. CVPR, 2020. 18
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. arXiv preprint arXiv:2211.1044 2022. 18
Kai-En Lin, Lin Yen-Chen, Wei-Sheng Lai, Tsung-Yi Yi Lin, Yi-Chang Shih, and Ravi Ramamoorthi. Vision transformer for nerf-based view synthesis from single input image. In IEEF/CVF Winter Conference on Applications of Computer Vision, 2023, 4, 19
Tsung- Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie.
Feature pyramid networks for object detection. In CVPR, 2017. 4, 20
12

---

Published as a conference paper at ICLR 2024
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-to-3: Zero-shot one image to 3d object. arXiv preprint arXiv:2303./1325, 2023. 2, 19
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In
Proceedings of International Conference Computer Vision (ICCV), December 2015. 19
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In CVPR, 2021. 18
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do
actually converge? In Proc. of the International Conf. Machine learning (ICML), 2018. 3,5, 19, 21
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for chape-gudee generation of 3d shapes and textures. arXiv preprint arXiv:2211.07700 2022. 18
Lu Mi, Abhijit Kundu, David A. Ross, Frank Dellaert, Noah Snavely, and Alireza Fathi. im2nerf: Image to neural radiance field in the wild. arXiv, 2022. 5, 19, 21
Mahdi H. Miangoleh, Sebastian Dille, Long Mai, Sylvain Paris, and Yagiz Aksoy. Boosting monoc- ular depth estimation models to high-resolution via content- adaptive multi-rcsolution merging, In CVPR, 2021.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
5, 17, 18
Ron Mokady, Michal Yarom, Omer Tov, Oran Lang, Daniel Cohen-Or, Tali Dekel, Michal Irani, and Inbar Mosseri. Self-distilled stylegan: Towards generation from internet photos, 2022. 6
Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto Tono, Linqi Zhou, and Paul Guerrero.
3d-ldm: Neural implicit 3d shape generation with latent diffusion models. arXiv preprint
arXiv:2212.00882,, 2022. 17, 18
Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Hologan: Unsupervised learning of 3d representations from natural images. In ICCV, 2019. 18
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXhv:2212.0875/, 2022a. 18
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffuuson probabilistic models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, 2021. 2, 17
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), Proc. of the International Conf. on Machine learning (ICML), 2022b. 2, 18
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. CVPR, 2021a. 17
Michael Niemeyer and Andreas Geiger. Campari: Camera-aware decomposed generative neural radiance fields. In Proc. of the International Conf. on 3D) Vision (3DV), 2021b.
Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Park, and Ira Kemelmacher. Stylesdf: High-resolution 3d-consistent image and geometry generation. In CVPR, 2022. 1, 18
Xingang Pan, Xudong Xu, Chen Change Loy, Christian Theobalt, and Bo Dai. A shading-guided generative implicit model for shape-accurate 3d-aware image synthesis. In NeurlPS, 2021. 1, 18
Dario Pavllo, David Joseph Tan, Marie-Julic Rakotosaona, and Federico Tombari. Shape, pose, and appearance from single image via bootstrapped radiance field inversion. arXiv, 2022. 19
13

---

Published as a conference paper at ICLR 2024
Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolu- tional occupancy networks. In ECCV, 2020.
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv, 2022. 18
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents, arXiv, abs/2204.06123. 2022. 2, 18
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross datsset transfer. IEEE TPAMI, 2020. 2, 5, 22
Daniel Rebain, Mark Matthews, Kwang Moo Yi, Dmitry Lagun, and Andrea Tagliasacchi. Lolnerf: Learn from one look. arXiv,, 2021. 19
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proc. of the International Conf. on Machine learning (ICML), 2014. 6, 20
Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: a stylegan encoder for image-to-image translation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2287-2296, 2021.
Daniel Roich, Ron Mokady, Amit H. Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. ACM TOG, 2023. 7,22
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models, 2021. 1, 2, 3, 4, 5, 6, 8, 9, 17, 21, 22
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11447, 2022. 2, 18
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. Advances Neural Information Processing Systems, 31, 2018.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint XXXv:2202.000112. 2022. 17, 22
Kyle Sargent, Jing Yu Koh, Han Zhang, Huiwen Chang, Charles Herrmann, Pratul Srinivasan, Jiajun Wu, and Deqing Sun. Vq3d: Learning 3d-aware generative model on imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4240 4250, October 2023. 3, 19
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. ACM Trans. on Graphics, 2022. 1
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. GRAF: generative radiance fields for 3d-aware image synthesis. NeurlPS, 2020. 1, 17, 18
Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, and Andreas Geiger. Voxgraf: Fast 3d-aware image synthesis with sparse voxel grids. In NeurIPS, 2022. 1, 18
Zifan Shi, Yujun Shen, Yinghao Xu, Sida Peng, Yiyi Liao, Sheng Guo, Qifeng Chen, and Dit- Yan Yeung. Learning 3d-aware image synthesis with unknown pose distribution. In CVPR, 2023, 1, 3, 6. 7. 23
Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. arXiv preprint 2rXiv:2211.16677 2022. 17, 18
14

---

Published as a conference paper at ICLR 2024
Ivan Skorokhodov, Sergey Tulyakov, Yiqun Wang, and Peter Wonka. Epigraf: Rethinking training of 3d gans. arXiv, 2022. 1,4
Ivan Skorokhodov, Aliaksandr Siarohin, Yinghao Xu, Jian Ren, Hsin- Hsin-Ying Lee, Peter Wonka, and Sergey Tulyakov. 3d generation on imagenet. The Eleventh International Conference on Learning Representations, 2023. 1, 3, 7, 8, 19, 23, 26
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning. pp. 2256 2265. PMLR, 2015. 17
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Proceedings of the 33rd Annual Conference on Neural Information Processing Systems, 2019.
17
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.134555 2020. 2, 17
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG), 40(4):1-14, 2021. 2
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In NeurIPS, 2021. 2, 3, 17, 18
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, pp. 5998-6008, 2017.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Compula- tion, 1367)11611-1674, 2011. 17
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Ych, and Greg Shakhnarovich. Score ja- cobian chaining: Lifting pretrained 2d diffusion models for 3d generation. arXiv preprint arXiv:22/2.00774, 2022a. 18
Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, and Baining Guo. Rodin: generative model for sculpting 3d digital avatars using diffusion. arXiv preprint arXiv:2212.0661 2022b. 17, 18, 20
Daniel Watson, William Chan, Ricardo Martin Bruaala, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models. arXiv, 2022. 2, 19
Jianfeng Xiang, Jiaolong Yang. Yu Deng, and Xin Tong. Gram-hdi 3d-consistent image generation at high resolution with generative radiance manifolds. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 2195-2205, October 2023a. 1, 18
Jianfeng Xiang, Jiaolong Yang, Binbin Huang, and Xin Tong. 3d-aware image generation using 2d diffusion models. In Proceedings the IEEE/CVF International Conference Computer Vision (ICCV), pp. 2383-2393, October 2023b. 3, 18
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), NeurIPS, 2021. 4, 20
Jiaxin Xie, Hao Ouyang, Jingtan Piao, Chenyang Lei, and Qifeng Chen. High-fidelity 3d GAN inversion by pseudo-multi-vice optimization. arXiv, 2022.
Xudong Xu, Xingang Pan, Dahua Lin, and Bo Dai. Generative occupancy fields for surface-aware image synthesis. In NeurlPS, 2021. 1,
15

---

Published as a conference paper at ICLR 2024
Yinghao Xu, Sida Peng, Ceyuan Yang, Yujun Shen, and Bolei Zhou. 3d-aware image synthesis via learning structural and textural representations. CVPR, 2022. 1, 18
Fei Yin, Yong Zhang, Xuan Wang, Tengfei Wang, Xiaoyu Li, Yuan Gong, Yanbo Fan, Xiaodong Cun, Ying Shan, Cengiz Öztireli, and Yujuu Yang. 3d GAN inversion with facial symmetry prior. arXiv, 2022. 19
Fisher Yu, Jianxiong Xiao, and Thomas A. Funkhouser. Semantic alignment of lidar data at city scale. In CVPR, pp. 1722-1731, 2015. 6
Zchao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. NeurIPS, 2022.
Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. LION: latent point diffusion models for 3d shape generation. In NeurIPS, 2022. 17, 18
Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, and lan Reid. Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction. In CVPR, 2018.
Jichao Zhang, Enver Sangineto, Hao Tang, Aliaksandr Siarohin, Zhun Zhong, Nicu Sebe, and Wei Wang. 3d-aware semantic-guien guided generative model for human synthesis. arXiv, 2021. 1, 18
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as perceptual metric. In CVPR, 2018. 5, 21
Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In ICCV, 2021a. 18
Peng Zhou, Lingxi Xie, Bingbing Ni, and Qi Tian. CIPS-3D: A 3D-Aware Generator of GANs Based on ionditionally-Ineeeendde Pixel Synthesis. arXiv, 2021b. 1, 18
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In European conference on compuler vision, pp. 592-608. Springer, 2020. 2
16

---

Published as a conference paper at ICLR 2024
A THEORETICAL BACKGROUND
We summarize the most relevant theoretical concepts to our work in the following.
Diffusion Models (Sohl-Dickstein et al., 2015; Ho al., 2020; Song et al., 2020) create diffused inputs xx=0,x+0++ E~,(00, I) from data X~>Pdata- where ar and a, define a noise schedule, parameterized by diffusion-time т. A denoiser model Fu with parameters w is trained to denoise the perturbed data via denoising score matching (Hyvärinen, 2005; Vincent, 2011; Song & Ermon, 2019),
arg min Ex P. [[lv - Fu,(x 7 1]]
(3)
with the target V = 0t x,X (this is known as v-prediction (Salimans & Ho, 2022)). Further, Pr uniform distribution over the diffusion time т, such that the model is trained to denoise for all different times т. The noise schedule is designed such that input data is entirely perturbed into Gaussian random noise after the maximum diffusion time. An iterative generative denoising process that employs the learned denoiser Fu can then be initialized from such Gaussian noise to synthesize novel data. Classifier-free guidance can be used to amplify conditioning strength when conditioning the diffusion model on data such as classes; see Ho & Salimans (2021) and App. C.2.
Diffusion models have also been applied to 3D data (Zeng et al., 2022; Wang et al., 2022b; Bautista et al., 2022; Shue et al., 2022; Nam et al., 2022) but usually require explicit 3D or multiview super- vision. In contrast, WildFusion learns from an unstructured image set without multiview supervision.
Latent Diffusion Models (LDMs) (Rombach et al., 2021; Vahdat et al., 2021) first train a regularized autoencoder with encoder & and decoder D to transform input images I I~ Pdata into a spatially lower- dimensional latent space Z of reduced complexity, from which the original data can be reconstructed, this is, i = (I) = I. A diffusion model is then trained in the compressed latent space, with x in Eq. (3) replaced by an image's latent representation Z = E(). This latent space diffusion model can be typically smaller in terms of parameter count and memory consumption compared corresponding pixel-space diffusion models of similar performance. More diffusion model details in Appendix.
nD-Reppesentattonn for 3D-Aware Image Synthesis. 3D-aware generative models typically generate neural radiance fields or feature fields, i.e., they represent a scene by generating a color or a feature value f and a density a for each 3D point p E R (Mildenhall et al., 2020: Schwarz et al., 2020; Niemeyer & Geiger, 2021a). Features and densities can be efficiently computed from triplane representation [T sy: T Taz T T= (Chan al., 2022; Peng et al., 2020). The triplane feature is obtained by projecting p onto each of the three feature planes and averaging their feature vectors (tayg taz, t==). An MI.P then converts the triplane feature to a feature and density value [f. = MT.P(t).
Given a camera pose, the feature field is rendered via volume rendering (Kajiya & Herzen, 1984; Mildenhall et al., 2020). For that, the feature field is evaluated at discrete points pf along each camera ray r yielding features and densities {(F,, o.) For each ray r, these features are aggregated to feature fr using alpha composition
N
Ep
= af
T II o?), ==1
=-=1-exp (-0,6+), (4)
where 7i and a denote the transmittance and alpha value of sample point Pf along ray r and 8 = |p,+1 p is the distance between neighboring sample points. Similarly, depth can be rendered, see Appendix. For efficiency, a low-resolution feature map, and optionally a low-resolution image flow can be rendered instead of an image at full resolution (Niemeyer & Geiger, 2021a; la; Chan et al., 2022). The feature map is then subsequently upsampled and decoded into higher-resolution image Í.
B RELATED WORK
Here, we present an extended discussion about related work.
Diffusion Models. Diffusion models (DMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2020) have proven to be powerful image generators, yielding state-of-th art results in unconditional as well as class- and text-guided synthesis (Nichol & Dhariwal, 2021; Rombach et al., 2021; Dhariwal
17

---

Published as a conference paper at ICLR 2024
& Nichol, 2021a; Ho et al., 2022; Dockhorn et al., 2022a;b; Vahdat et al., 2021; Nichol et al., 2022b; Ramesh et al., 2022; Saharia et al., 2022; Balaji et al., 2022). However, none of these works tackles 3D-aware image synthesis.
3D Diffusion Models. There is also much literature on applying diffusion models to 3D data, e-g. 3D point clouds (Zhou et al., 202 Luo & Hu, 2021; Zeng et al., 2022) or tetrahedral meshes (Kalischek et al., 2022). Shue et al. (2022) learn a diffusion model on a triplane representation parameterizing neural occupancy field. GAUDI (Bautista et al., 2022) and 3D-LDM (Nam et al., 2022) train diffusion models in latent spaces learnt using an autodecoder framework and generate 3D scenes and 3D shapes, respectively, RODIN (Wang et al., 2022b) proposes hierarchical latent diffusion model framework to learn 3D human avatars and NF-LDM (Kim et al., 2023) trains a hierarchical diffusion model for outdoor scene generation. Dupont et al. (2022) and Du et al. (2021) treat data as functions and also explore encoding 3D signals into latent spaces, but using more inefficient meta-learning (Dupont et al., 2022) or auto-decoder (Du et al., 2021) methods. Dupont et al. (2022) also trains diffusion model on the encoded 3D data. However, all aforementioned works rely on explicit 3D or multiview supervision. contrast, our approach learns from an unstructured image collection without multiview supervision.
RenderDiifusion (Anciukevicius et al., 2023) trains a diffusion model directly on images, using triplanar 3D feature representation inside the denoiser network architecture, thereby enabling 3D-aware image generation during synthesis. However, scaling RenderDiffusion to high-resolution outputs is challenging, as it operates directly images. In fact, it considers only small image resolutions of 32x32 or 64x64, likely due to computational limitations. When trained on single-view real-world data, the paper only considers data with little pose variation (human and cat faces) and it is unclear whether the approach is scalable to diverse or high-resolution image data (moreover, no perceptual quality evaluations on metrics such FID are presented and there is no code available). In contrast, our diffusion model is trained efficiently in a low-resolution, compressed and 3D- aware latent space, while simultancously predicting high-resolution triplanes and enabling high- resolution rendering. Hence, WildFusion generates significantly higher quality 3D-aware images than RenderDiffusion and is scalable diverse datasets such as ImageNet, as we demonstrate.
Concurrently with us. IVID (Xiang et al., 2023b) trains a 2D diffusion model that first synthesizes an initial image and then iteratively generates novel views conditioned on it. However, the iterative generation is extremely slow because it requires running the full reverse diffusion process for every novel view. Further, an explicit 3D representation can only be constructed indirectly from a large collection of generated multi-view images, afterwards. Instead, WildFusion uses a fundamentally different approach and only runs the reverse diffusion process once to generate a (latent) 3D represen- tation from which multi-view images can be rendered directly and geometry can be extracted easily, At time of submission code for experimental comparisons to IVID was not available.
Optimization from Text-to-Image Diffusion Models. Another line of work distills 3D objects from large-scale 2D text-to-image diffusion models (Poole et al., 2022; Lin et al., 2022; Nichol et al., 2022a; Metzer et al., 2022; Wang et al., 2022a; Deng et al., 2022a). However, these methods follow an entirely different approach compared to 3D- and 3D-aware diffusion models and require a slow optimization process that needs to be run per instance.
3D-Aware Image Synthesis. 3D-aware generative models consider image synthesis with control over the camera viewpoint (Liao et al., 2020; Schwarz et al., 2020; Chan et al., 2021). Most existing works rely on generative adversarial networks (GANs) (Goodfellow et al., 2014) and use coordinate- based MI.s as 3D-generator (Schwarz et al., 2020; Chan et al., 2021; Gu et al., 2022; Jo et al., 2021; Xu et al., 2022; Zhou ct al., 2021b; Zhang et al., 2021; Or-El ct al., 2022; Xu ct al., 2021; Pan et al., 2021; Deng et al., 2022b; Xiang et al., 2023a; Gu et al., 2022), building Radiance Fields (Mildenhall et al., 2020) as 3D representation. Vo (Schwarz et al., 2022) and EG3D (Chan et al., 2022) proposed efficient convolutional 3D generators that require only single forward pass for generating a 3D scene. Our autoencoder uses EG3D's triplane representation and their dual discriminator improve view consistency, Early 3D-aware generative models that do not require camera poses during training and operate in view space include HoloGAN Nguyen-Phudd et al. (2019) and PlatonicGAN (Henzler et al., 2019). They are outperformed, for instance, by the more recent StyleNeRR (Gu et al., 2022), which uses a style-based architecture (Karras et al., 2019;
Neural
18

---

Published as a conference paper at ICLR 2024
2020) and proposes novel path regularization loss to achieve 3D consistency. In contrast to the aforementioned approaches, however, our generative model is not a GAN. GANs are notoriously hard to train (Mescheder al., 2018) and often do not cover the data distribution well. Instead, explore 3D-aware image synthesis with latent diffusion models for the first time.
Until recently, 3D-aware image synthesis focused aligned datasets with well-defined pose distribu- tions, such as portrait images (Liu et al., 2015; Karras et al., 2019). For instance, POF3D (Shi et al., 2023) a recent 3D-aware GAN that infers camera poses and works in a canonical view space; has been used only for datasets with simple pose distributions, such as cat and human faces. Advancing to more complex datasets, GINA-3D learns to generate assets from driving data, assuming known camera and LIDAR sensors. The two-stage approach first trains a vision transformer encoder yielding a latent triplane representation. Next, discrete token generative model trained in the latent space. We consider setting where camera information and ground truth depth are not available and train a latent diffusion model on 2D) feature maps.
To scale 3D-aware image synthesis to more complex datasets, i.e. ImageNet, 3DGP (Skorokhodov et al., 2023) proposes an elaborate camera model and learns to refine an initial prior on the pose distribution. Specifically, 3DGP predicts the camera location in canonical coordinate system per class and sample-specific camera rotation and intrinsics. This assumes that samples within class share canonical system, and observe that learning this complex distribution can aggravate training instability. Further, the approach needs to be trained on heavily filtered training data. In contrast, WildFusion can generate high-quality and diverse samples even when trained on the entire ImageNet dataset without any filtering (see Sec. 4.2).
Concurrently to WildFusion, VQ3D (Sargent et al., 2023) proposes an autoencoder architecture, but uses sequence-like latent variables and trains an autoregressive transformer in the latent space. Instead, WildFusion trains a diffusion model on latent feature maps. Another difference that VQ3D applies two discriminators on the generated images, one that distinguishes between reconstruction and training image, and another one that discriminates between reconstruction and novel view. WildFusion only applies a single discriminator to supervise the novel views and instead has an additional discriminator on the depth. At time of submission code for experimental comparisons was not available.
Novel View Synthesis. Our autoencoder is related to methods that synthesize novel views given a single input image: LoLNeRF (Rebain et al., 2021) trains an autodecoder with per- vexel reconstruction loss and mask supervision. contrast, we add an adversarial objective to supervise novel views. Mi et al. (2022) proposes similar approach but is not investigated in the context of generative modeling. Another line recent works leverage GAN inversion to generate novel views from single input images (Li et al., 2022; Cai et al., 2022; Lin et al., 2023; Xie et al., 2022; Yin et al., 2022; Lan et al., 2022; Pavllo et al., 2022) but rely on pretrained 3D-aware GANs and thereby inherit their aforementioned limitations. Several recent works (Watson et al., 2022; Chan al., 2023; Liu et al., 2023) tackled novel view synthesis with view cen-conddtioned 2D diffusion models, but are trained with explicit 3D multiview supervision. Unlike these approaches, our autoencoder also train 3D-aware generative model.
C IMPLEMENTATION DETAILS
C.I CAMERA SYSTEM
In the following, we describe the camera system we use to learn a 3D representation in view space. An overview is shown in Fig. 8. The input view Po is defined by the camera intrinsics K, and extrinsics [Ro, To]. where Ro and To denote the rotation and translation of the camera in the world coordinate system. We fix Ro and To, such that the camera located fixed radius and looks the origin of the world coordinate center. For the intrinsics Kg, we choose small, fixed field of view since most images in the datasets are cropped and perspective distortion generally small. For the experiments on all datasets, we set
01 0
0
0
55.4
0
0.5
Ro
-1
0
To
(02 2.7
Kg
0
5.4
0.5
(5)
0
-1
0
0
1.0
19

---

Published as a conference paper at ICLR 2024
Sample coordinates
Contracted coordinates
Po
lp
t
60
Linear domain
Contracted domain
Our autoencoder models objects in view space, i... it uses fixed camera pose Po render the reconstructed images. The orange line represents camera ray and i, and ( denote the near and the far plane, respectively, between which the object is located. sample coordinates which are spaced linearly in disparity (inverse depth) to better model large depth ranges. However, the triplanes that carry the features for encoding the 3D object are only defined within a normalized range (-1, 1, 1] in all directions); hence, need normalize the samples the camera ray accordingly to ensure that all coordinates Eq. (6). Sample coordinates within fixed radius r are mapped linearly to sphere radius Tin < 1, i.e., into the domain where the triplanes are defined (linear domain). Sample coordinates with norm > are contracted such that they have norm S 1 in the domain of the triplanes (contracted domain).
Figure 8: Camera System:
evaluate
camera ray the
projected onto the triplanes. Specifically, we use the contraction function in
During training, we sample novel views around Po by drawing offsets for azimuth and polar angles uniformly from [-35%, 35%] and [-15%, 15"], respectively.
As we consider unbounded scenes, we sample points along the camera rays linearly in disparity (inverse depth). Thereby, we effectively sample more points close to the camera and use fewer samples at large depths where fewer details need to be modeled. In practice, we set the near and far planes to L, = 2.25 and 1, = 5.0, respectively. During rendering, the sampled points are mapped to 3D coordinates in [-1, 1] and subsequently projected to 2D points on the triplanes (see Fig. 8). Reflecting the disparity sampling, we choose a non-linear mapping function for the 3D coordinates that places sampling points with large depth closer together. This assigns a larger area the triplanes to points close to the camera while points far away are projected onto a smaller area. Specifically, we
contraction function inspired by Barron et al. (2022) that maps points within sphere of radius r. to a sphere of radius rin. < and all points outside of the sphere to a sphere of radius 1, Let IR' denote sampled point; then, the contracted coordinate x. is calculated as
(x ( ( Tin)
if | S r otherwise
X
|x| ||| + Tin) )
[x]
We set r. = 1.3 and rin 0.8 for all experiments.
C.2 NETWORK ARCHITECTURE OBJECTIVES, AND TRAINING
First Stage: 3D-aware Autoencoder. The encoder network is feature pyramid network (FPN) (Lin et al., 2017). In practice, we use the setup from variational autoencoders (VAEs) (Kingma & 2014; Rezende al., 2014) and predict means fl and variances σ² of a normal distribution from which we sample the latent representation Z
Welling,
[n, [u,0²) = FP,())
~~
where E [exxhx (formally, we assume a diagonal covariance matrix and predict means and variances for all latent dimensions independently). We regularize the latent space by minimizing a low-weighted Kullback-Leibler divergence loss LKL between g8 (Z[[p) Nuu, o²) and standard normal distribution (O.
The decoder consists of transformer blocks at resolution of the feature map Z followed by a CNN that increases the resolution to 128² pixels. The CNN applies grouped convolutions with three groups to prevent undesired spatial correlation between the triplanes, see Wang et al. (2022b). At resolution 1282 pixels, we then add two further transformer blocks with efficient self-attention (Xie et al., 2021) to facilitate learning the correct spatial correlations for the triplanes. The triplanes have a resolution of 128 pixels.... We use the superresolution modules and dual discriminator from EG3D (Chan et al., 2022). The main task of the discriminator is to provide supervision for novel views, but can also be used to improve the details in the reconstructed views (Esser et al., 2021). We hence use 95% novel views and 5% input views when training the discriminator. For the main models in the paper, the encoder has 32M parameters. We use 8 transformer blocks in the decoder accumulating 26M
20

---

Published as a conference paper at ICLR 2024
Table Weights of losses used the training objective.
Apz
AvGG
X2D
A3D depth
AKL.
A
Xa
depth
Weight
10
10
le-4 1
10
Table Hyperparameters for our latent diffusion models.
Architecture
Training
Diffission Setup
Image shape
256 256 3
Parameterization
Diffusion steps
1000
z-shape
32 x 32 x 4
Learning rate
10
Noise schedule
Cosine
Channels
224
Batch size per GPU
64
Offset
0.008
Depth
#GPUs
4
Scale factor z
0.5
Channel multiplier
1,2,4,4 Attention resolutions
Pdrop
0.1
Sampler
DDIM
32,16,8
Steps
200
Head channels 32
1.0
parameters for the full decoder. The discriminator has 29M parameters. For computational efficiency, ablation studies (Table of the main paper) are performed with downscaled models and at an image resolution of 128² pixels instead of 2562 2562pixels. The downscaled models have reduced channel dimension; specifically, the triplane resolution is reduced to 642 pixels and the decoder uses 4 instead of 8 transformer blocks. The resulting models count 1.6M, 2.5M, and 1.8M parameters for encoder, decoder and discriminator, respectively.
The autoencoder uses Adam (Kingma & Ba, 2015) with learning rate of 1.4 10 However, for the superresolution modules and dual discriminator, we found it important to use the original learning rates from EG3D, which are x 10-3 and 1.9 x 10-3 respectively. We train all autoencoders with a batch size of 32 on 8 NVIDIA A 100-PPCIE-4OGB GPUs until the discriminator has seen around 5.5.M training images. Training our autoencoder in this setting takes around 2.5 days.
Implementation and Training. Our autoencoder is trained with a reconstruction loss on the input view and adversarial objective to supervise novel views (Mi et al., 2022; Cai et al., 2022). Similar to Rombach et al. (2021), I), we add a small Kullback-Leibler (KL) divergence regularization term L on the latent space Z, discussed above. The reconstruction loss free consists of pixel-wise loss Eye I|, a perceptual loss LyGe (Zhang et al., 2018), and depth losses C2D full training objective is as follows
deptt d The
Cree Xpz + AvcoLycG deptn Ceepth - C3D
(8)
min max [VII, nv. 0; 1b, (b) + Vdepth (I.P Aa; v. 8,49 &
+ Creo(pp P;O, vi) + AxLLxI(IP:0)
where \ạ weigh the individual loss terms (A without subscript denotes the R1 regularization co- efficient for the regular discriminator and X the RI regularization coefficient for the depth discriminator). The values for the weights are summarized in Table 5. Note that V and Vdenth denote the adversarial objectives of the regular and depth discriminator, respectively (see Eq. (5) in main paper).
Our code base builds on the official Py Torch implementation of StyleGAN (Karras et al., 2019) available https://github.. http:/ittuu..oomNNTT Sonm//Vaabs/sylleaann. EG3D (Chan et al., 2022) available at https:// cithub.com/NViigg cow//Viabs/og33d and LDM (Rombach et al., 2021) available at https//gghuub.oom/oomoo Vis/ attent-diffusin. Similar StyleGAN, the discriminator (Karras et al., 2018) and apply an exponential moving average of the autoencoder weights. Unlike Karras et al. (2019), we do not train with path regularization or style-mixing. To reduce computational cost and overall memory usage RI RI-regularization (Mescheder et al., 2018) is applied only once every 16 minibatches (also see RI regulazaiiiiti coefficients À and d in Table 5).
minibatch standard deviation layer at the end of
Second Stage: Latent Diffusson Model. We provide detailed model and training hyperparameter
choices in Table 6. We follow the naming convention from LDM Rombach al. (2021) and train the
21

---

Published as a conference paper at ICLR 2024
models for ~200 epochs for each dataset. Our LDMs trained on 4 NVIDIA A100-PCIE-40GB
GPUs for 8 hours on SDIP elephant and for 1 day on SDIP horse, dog. On ImageNet, we train a class-conditional model for ~5 days, doubling the batch size from 128 to 256. Otherwise, use the same hyperparameters and model size due to computational constraints.
Guidance. Class-conditioning is implemented through cross attention with learned embeddings, following Rombach al. (2021). We also drop the class conditioning 10% the time to enable sampling with classifier-fre free guidance (Ho & Salimans, 2021). We guidance scale of 2 in all our quantitative and qualitative results, unless indicated otherwise. The guidance scale s is defined according to the equation
(X,, c) (x,) + (x., Ex,(x))
(10)
using the noise prediction formulation (we can easily obtain the noise prediction from the v prediction, which we use for training; see Salimans Ho (2022)). the above equation, 6. (x,) denotes the unconditional score function, (x,, c) the conditional score function when conditioning on class c, and E. (x,, c) is the resulting guided score function with guidance scale s.
Compute Limitations and Further Scaling, Note that due to their noisy training objective, diffusiin models have been shown scale well with more compute and larger batch sizes (Rombach et al., 2021: Karras et al., 2022; Kingma & Gao, 2023). State-of-the-an models on regular, non-3D-award image synthesis usually use significantly larger batch sizes (> 1000) than we do. This suggests that our models in particular on the highly diverse ImageNet dataset could probably improved a lot with more computational resources.
C.3 MONOCULAR DEPTH
In our pipeline, leverage geometric cues from a pretrained monocular depth estimation net- work (Bhat al., 2023) supervise the predicted depth D from the autoencoder. Note that the predicted depth is obtained using volume rendering, similarly to Eq. (2) in the main paper
D(r)
(11)
ΣØ=1
where d; denotes the depth of sampling point along camera ray r and we is its corresponding rendering weight defined in Eq. (2) of the main paper. The monocular depth used for supervision, however, is only defined up to scale. Let D denote the depth predicted by the monocular depth estimator. We first downsample it to match the resolution of the predicted, i.c., rendered depth, which refer to as D Next, a scale and a shift t are computed for each image by solving least-squares criterion (Eigen al., 2014; Ranftl al., 2020)
(s,t) arg min 8,8
Σ
(D( +t D (r)
(12)
PER
Defining (s, and d, (D(r), T) the closed-fore solution is given by
h Σ dd
dd doo
(13)
D BASELINES
EG3D*, EG3D (Chan et al., 2022) relies on estimated camera poses which are not available for the datasets we consider in this work. Hence, we adapt it to work in view space and remove the need for camera poses. Both the generator and discriminator are originally conditioned on the camera poses. For our version, EG3D*, by sampling novel views around the input view Po as described in Sec. C. For fair comparison, we additionally train variant of EG3D* that leverages monocular depth. Specifically, equip EG3D*
remove the camera pose conditioning and model objects in view space
22

---

Published as a conference paper at ICLR 2024
with the depth discriminator from our pipeline Ddenth which compares the rescaled rendered depth with the predictions from a pretrained monocular depth estimatior (Bhat et al., 2023), see Sec. C.3 for more details on the rescaling of the depth.
We follow the training procedure from EG3D (Chan ct al., 2022) and ensure that the models train stably by tuning the regularization strength À in the adversarial objective. We use À 1 and X 2.5 for both variants on SDIP Elephants and Horses, respectively. For SDIP Dogs, we found best to use 2.5 for EG3D* and À = 1.0 EG3D* Däspeh For the depth discriminator, we Aa- 10A for all experiments. The models are trained until the discriminator has seen 10M images as we find that FID improvements are marginal afterwards. For evaluation, we select the best models in terms of FID.
Note that (Skorokhodov al., 2023) includes detailed study on training EG3D without camera poses. As 3DGP clearly outperforms EG3D in this setting, we did not train the original EG3D in canonical space but instead directly compare to 3DGP.
For inversion, we use code kindly provided by the authors of EG3D (Chan et al., 2022). The inversion performed using PTI (Roich et al., 2023) and consists of two optimization stages. In the first stage, the style code is optimized to best match the input image. In the second stage, the network parameters are finetuned to further improve reconstruction quality. We observed that the inversion occasionally diverges. For the divergent cases, we reduce the learning rate in the optimization from 10 3 to 10 finding that this stabilizes the optimization.
POF3D, 3DGP. For POF3D (Shi et al., 2023), we use the unpublished code that was kindly provided by the authors to train their model. We follow their training procedure and hyperparameter choices. For 3DGP (Skorokhodov et al., 2023), we trained the models using the publicly available code httss//github..oo//naaaa rewpa//prrrearrc//dg.pp.. which was released shortly before the submission deadline. We found that the training diverges on SDIP Elephants but, suggested by the authors, were able to restart the model from the last stable checkpoint which then converged. For SDIP Horses training diverges after around 2.5M images, even when restarting from the last stable checkpoint, so we report results on the last stable checkpoint. Both POF3D and 3DGP are trained until the discriminator has seen 10M images we observed no or only marginal improvements on FID with longer training.
StyleNeRE. We train StyleNeRF (Gu et al., 2022) using the official implementation of the au- thors https://github.commffi http.////ccccbbkrsccccrrtteeeeee
On SDIP datasets, we train until the discimi- nator has seen 20M images, Imagenet we stop training after 35M images. In both cases, we only observed marginal changes in FID with longer training.
SceneScape*. As an additional baseline, we analyzed a combination of a 2D generative model and 2D inpainting model. We base our implementation on the publicly available code of SceneScape (Frid- man et al., 2023) (https://github.com ages using the ImageNet checkpoint from LDM https://gthub..com/oooo Next, we predict the corresponding depth using ZoeDepth, i.e. using the same pretrained depth estimator as in our approach, and warp the image to a novel view. Lastly, an inpainting model fills the wholes that result from warping. We use an inpainting variant of Stable Diffusion (httss:/huggingffce htt.:/accccccocc////oooooo
git). Specifically, we generate im-
htwp/llttnt-diffuiion
wwwpinnnndiifuuusssiinnnntteeeee hatpaaaintttaablldfffiiiooootoo diffusion-inpaintinn and pro- vide the warped image, its mask, and the text prompt "a photo of <class name>" as input.
E ADDITIONAL RESULTS
Autoencoder for Compression and Novel- View Synthesis
Fig. shows further examples of WildFusion and baselines for novel view synthesis on images unseen during training (using GAN-inversion to embed the given images in latent space for EG3D* and EG3D* + Deepth). ). We see that our model generally correctly reconstructs the input object and is able to synthesize high-quality novel view. Moreover, although there are small artifacts, WildFusion also produces plausible geometry. In comparison, the baselines cannot accurately reconstruct the correct object and the geometry is often flat incorrect.
23

---

Published as a conference paper at ICLR 2024
Input
WildFusson (oure)
EG3D*
FA3R1
Figure 9: Comparison with baselines for view synthesis on images unseen during training. Shown are input image, a novel view and the geometry extracted with marching cubes. The viewpoints across methods are the same. See the included video for more results.
Fig. 13, Fig. 14, Fig. 15 and Fig. 16 show further novel view synthesis results from WildFusion's 3D-aware autoencoder. The results demonstrate our model's ability correctly generate novel views, given the encoded input view. All viewpoint changes result in high-quality, realistic outputs.
3D-Aware Image Synthesis with Latent Diffusion Models
We first compare WildFusion against the additional baseline SceneScape*. For quantitative analysis, sample novel views similar
evaluation by sampling yaw and pitch angles from Gaussian distributions with a = 0.3 and 0.15 radians, using the depth at the center the image to define the rotation center. With this approach, get FID of 12.3 on ImageNet, compared to 35.4 for WildFusion. However, as discussed in the main paper, FID only measures image quality and does not consider all important aspects of 3D-aware image synthesis, e.g. 3D consistency. fact, we observe that the inpainting often adds texture- like artifacts or more instances of the given ImageNet class. We provide some examples in Fig. 10.
To enforce consistency between novel views, we run the full pipeline of SceneScape to fuse the mesh for multiple generated images. For this setting, we sample camera poses by combining yaw angles of [00.3, 0., 0.3] and pitch angles of [-0.15, 0., 0.15] and iteratively update the mesh by finetuning the depth estimator. We show the final meshes Fig. 10 (bottom two rows). For samples we evaluated. we observe severe intersections in the mesh and generally inferior geometry to our approach. We remark that SceneScape's 's test time optimization takes multiple minutes per sample and a large-scale quantitative evaluation was out of the scope of this work. Our rotating camera movements around single object are much more challenging, e.g. due to larger occlusions, than the backward motion shown in SceneScape. We hypothesize that this causes SceneScape to struggle in our setting.
24

---

Published as a conference paper at ICLR 2024
WildFusion (ours)
SceneScape
Figure 10: We compare WildFusion against a variant of SceneSeape that combines 2D generative model with a pre-trained inpainting model. First for rows: Leftmost image is input view and next two images are novel views at +17 degree yaw angles for the two methods. We observe severe inpainting inconsistencies for the SceneScape baseline. Last rows: Leftmost image again input image, next image is novel view 177 degree yaw angle, and last image shows extracted geometry/mesh for two methods), We find that due to the inconsistencies of the inpainting model across views, the fused meshes for SceneScape have severe intersections and overall inferior geometry to WildFusion.
We also include more samples from WildFusion and 3DGP ImageNet in Fig. 11. While samples from 3DGP look very similar within class, WildFusion generates diverse samples. Fig. 17, Fig. 18, Fig. 19 and Fig. 20 show further WildFusion results for 3D-aware image synthesis leveraging latent diffusion model that synthesizes 3D-aware latent space encodings. We can see that all novel samples are high quality and the camera angle changes result in realistic view point changes of the scenes. Note that our model only trains on unposed, single view images and does not need to learn complex pose distribution because models objects in view space.
More generated results can be found in the supplementary video, including baseline comparisons and extracted geometries.
Ablation Studies.
Fig. 12 visualizes the impact of different configurations the geometry. For the base config, the model learns planar geometry which reflects in a low NFS, cf. Tab. the main paper. Adding a discriminator and the ViT backbone improve geometry significantly but the geometry remains noisy.
25

---

Published as a conference paper at ICLR 2024
WilFission (ours)
3DGP
Figure 11: Sample Diversity: Generated samples on ImageNet. Rows indicate class; columns show uncurated random samples. While WildFusion generates diverse samples due its diffusion model-based framework (leff),
the GAN-based 3DGP (Skorokhodov al., 2023) has very intra-class diversity (mode collapse, right).
Bise confii
VIT borkhenn
Detatt
mmcdliing unhumnesd scence
sucode Geptl
Reconstructions (small) and geometry for different settings in the ablation study. The geometry was extracted by applying marching cubes to the density values of the feature field. We can see an improvement in geometry, as more components are added to the model. Note that the underlying model for this experiment is very small and used only for the ablation study. It has limited expressivity.
Figure 12:
26

---

Published as a conference paper at ICLR 2024
Incorporating geometry cues in the form of monocular depth and depth discriminator Deepth helps to remove artifacts from the geometry, resulting in a lower NFS. Modeling unbounded scenes with contracted coordinates does not significantly change geometry but improves nvFID, ef. Tab. in the main paper. Further supervision on the rendered depth (C does not improve results but as input to depth result in slight improvements NFS and qualitatively improve geometry. Note how the geometry less planar when supervising the rendering weights with C3D depth- We remark that the model used in this ablation is much smaller than our main models and has limited expressivity.
does not hurt performance either, kept it in our pipeline. Lastly, both adding depth the encoder and directly supervising the rendering weights with LăD
FFI
FLID
NFS
Precisson
RRecall
65.1
15.3
33.6
0.58
0.16
1.5
45.2
12.9
33.7
0.60
0.19
$
35,4
11.7
33.8
0.59
0.20
2.5
30.2
11.0
32.7
0.59
0.19
$
28.5
10.9
32.9
0.58
0.18
5
25.5
11.6
33.6
0.53
0.13
10
29.7
13.7
33.1
0.44
0.08
Table 7: Evaluation ImageNet with different classifier-free guidance scales
We further ablate inference with different classifier-fres guidance scales (Ho & Salimans, 2021) in Table 7. For better compatibility with previous works, we also evaluated FID on 50K generated images for 3, which drops from 28.5 on 10K images to 25.3 on 50K images. For computational efficiency, report FID on 10K images on all other instances throughout the paper.
Limitations. Modeling instances in view space alleviates the need for posed images and learned camera distributions. However, it is very challenging task. This can be seen, for instance, in Fig. 9. becomes difficult to produce sharp 3D geometry for both baseline models and WildFusion, although WildFusion produces 3D-aware novel-views with high quality and still the most realistic geometry.
Furthermore, as WildFusion is trained with fixed azimuth and polar angle ranges, it is currently not possible perform novel view synthesis across the full 360 Increasing the ranges would be an interesting direction for future work. The challenge would lie in producing realistic content when largely unobserved regions become visible after large view point changes. Note, however, that the best of our knowledge currently there exist no methods that can produce 360 views when trained on the kind of datasets we are training on, which only show single view per instance, often from similar front direction.
We observed that the synthesized samples occasionally exhibited plane-like geometry artifacts. Our adversarial loss in the autoencoder, in principle, should avoid this, enforces realistic renderings from different viewpoints. We hypothesize that this is due to autoencoder favoring rare cases the simple solution of copying the image across triplanes to reduce the reconstruction loss.
Moreover, WildFusion relies on fixed camera intrinsics (see Sec. C.I), which we need to pick ourselves. However, we found that our choice worked well for all three datasets without further tuning. Hence, this is minor limitation, in particular, compared to methods that work in canonical coordinate systems and need to estimate pose for each instance (as discussed, this is not possible for many complex, non-aligned datasets). In future work, the camera intrinsics could potentially be learned.
27

---

Published as a conference paper at ICLR 2024
Figure 13: Input images (left column) and novel views from WildFusion's 3D-aware autoencoder SDIP Dogs. The results span yaw angle of 40%.
28

---

Published as a conference paper at ICLR 2024
Figure 14: Input images (left column) and novel views from WildFusion's 3D-aware autoencoder for SDIP
Elephants. The results span yaw angle of 40".
29

---

Published as a conference paper at ICLR 2024
Figure 15: Input images (left column) and novel views from WildFusion's 3D-aware autoencoder for SDIP
Horses. The results span yaw angle of 40
30

---

Published as a conference paper at ICLR 2024
SHHP BARREER
OEIDES MOROON'
Figure 16: Input images (left column) novel views from WildFusion's 's 3D-aware autoencoder ImageNet. The results span yaw angle of 40
31

---

Published as a conference paper at ICLR 2024
Figure 17: Generated images and novel views from WildFusion for SDIP Dogs. The results span yaw angle
of 4
32

---

Published as a conference paper at ICLR 2024
Figure 18: Generated images and novel views from WildFusion for SDIP Elephants. The results span a yaw
angle of 2
33

---

Published as a conference paper at ICLR 2024
1000
Figure 19: Generated images and novel views from WildFusion for SDIP Horses. The results span yaw angle
of 40
34

---

Published as a conference paper at ICLR 2024
Figure 20: Generated images and novel views from WildFusion for ImageNet. The results span a yaw angle of
40
35


Reviews:
[{"review_id": "1aXGjhx17c", "reviewer": "Reviewer_US3i", "summary": "The paper proposed a two-stage model for learning 3D-aware latent diffusion model in image view space. In the first stage, the authors learn a trasformation: (image, depth) -> latent -> triplane -> NVS images, by applying a GAN loss. Then in the second stage, the authors train a latent diffusion model, which can be directly decoded into a triplane plane NeRF. The learning of the pipeline does not require multi-view data and has shown better results than previous approaches.", "questions": "see above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is nicely presented. Charts and tables are nicely made and I found the paper easy to read through.\n\n- The two-stage training is interesting. Each step of the pipeline looks reasonable to me.\n\n- The training of the method does not require 3D or multi-view image data.", "weaknesses": "- An important work is missing in discussion/comparison. \"VQ3D: Learning a 3D-Aware Generative Model on ImageNet\", ICCV 2023. The two works are very similar and both works adopt a two-stage learning scheme. The major difference is that VQ3D applies a GAN-based method for both stages.   \n\n- I am not sensitive to the quantitative number in the main paper but I saw many NVS results in the supplementary video are distorted. Also, I did not observe a significant visual improvement over the EG3D. I would resort to opinions from other reviewers.\n\n- As the second stage is trained on a latent space obtained by the first stage training, I am concerned that the diffusion generation quality (geometry correctness and image fidelity) is bounded by the GAN-based training. So what is the benefit of introducing the second stage? Easy sampling?\n\n- Strictly speaking, the training involves a large amount of 3D data, which comes from the pre-trained single view depth estimator.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ICOEFauqBv", "reviewer": "Reviewer_HTTZ", "summary": "This paper presents a novel method address the challenge of 3D-aware image synthesis for in-the-wild images. A key difference to previous works is that it developed upon a latent diffusion model, which demonstrates more sample diversity to GAN. To enable in-the-wild generation, unlike existing methods that rely on a shared canonical space, the authors propose to model instances in view space. To enable consistent 3D representation, the diffusion model predicts an implicit 3D representation, the triplane representation. The training also leverages monocular depth estimation to further boost 3D accuracy. The experiments show that the proposed work outperforms prior art by a large margin.", "questions": "See weakness section", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors leverages latent diffusion model to address the lack of sample diversity in 3D-aware GAN.\n- They propose to represent 3D-aware image by an efficient triplane representation.\n- The training loss avoids the necessity of multi-view images of the same instance, which makes it easier to train on a much larger amount of data. \n- An extensive ablation study to support design choices.", "weaknesses": "- 1. This paper only compare with GAN-based methods. It would be more convincing if a comparison to recent diffusion-based methods (GenVS, IVID, VQ3D) is presented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2hNDGEuBQs", "reviewer": "Reviewer_D5rM", "summary": "The paper proposes a novel approach to learning a 3D generative model from unposed images in view space.\n\nThe core idea of the paper is to first pre-train an auto-encoder that takes an image and its estimated depth map as input, and encodes it into a triplane NeRF. That triplane is then used to re-render the input image and depth as well an one (or several) additional views. The input image and depth are supervised straightforwardly, while the additional view and depth is supervised via an adversarial loss.\n\nSubsequently, the authors propose to train a latent diffusion model on the recovered latent space, enabling unconditional and conditional generative modeling.", "questions": "I am overall OK with accepting this paper, as I believe that it is well-written, poses an important problem, and puts forth a reasonable baseline approach.\n\nI would be happy to increase my score if the authors could provide the baseline requested above.\n\n\n___\n\n\nI thank the authors for addressing my concerns. I think this additional baseline comparison adds to the paper! I increased my score to 8 and will happily argue for acceptance.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "- Exposure is excellent, the method is exceedingly clear. The overview figure is great.\n- The paper is well-motivated and the shortcomings of prior work are clearly highlighted.\n- Design choices are clear.\n- Baselines are appropriate.\n- Ablations are detailed and insightful.", "weaknesses": "My core complaint with this paper is that I am not quite sure why you would use this method over a simple depth-warping plus inpainting baseline.\n\nThe generated images are of somewhat low quality - they are certainly far behind anything that can be generated with any SOTA 2D generative model.\n\nFor any generated image, I could always use the same monocular depth predictor used in this paper to estimate depth, and then warp the image to a novel view. The only challenge would then be holes - which, however, one could easily inpaint, as has been demonstrated in \"SceneScape\"  (https://arxiv.org/abs/2302.01133).\n\nI would really like to see the following simple baseline:\n1. Generate an image with a 2D image generative model.\n2. Predict monocular depth with the same model you are currently using.\n3. Warp the image to a novel view using the predicted depth.\n4. Use an inpainting method such as the one used in \"SceneScape\".\n\nThe only shortcoming here is that this requires warping and in-painting at test time, to render novel views. However, one could easily merge several such in-painted views into a single mesh, which could then be rendered from novel views, similar to what SceneScape does. \n\nEven so, I *do* believe that this paper adds significantly to the literature by clearly formulating the problem, showing how prior methods fail, and producing a method that significantly outperforms prior methods. I think this will spur follow-up work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4MmCUhibVi", "reviewer": "Reviewer_CqTg", "summary": "This paper presents a method that learns 3D generation from in-the-wild images, e.g., Imagenet. Specifically, this paper proposes to first learn a 3D-aware VAE that compresses images to latent space and decodes latent into 3D representations, i.e., triplane. To facilitate 3D learning from 2D images, especially unposed in-the-wild images, authors proposes to use RGB and depth discriminator for rendered novel views. After learning the 3D-aware latent features, authors uses a standard diffusion model to sample from the latent space. Experiments are performed on Imagenet, as well as three unimodal datasets. Some advantages over GAN-based methods are shown. Ablation study is also performed to show the effectiveness of different model parts.", "questions": "IVID has shown the ability to generate 360 degree novel view synthesis. I understand that IVID uses a very different approach from this paper. But I would like to hear authors' opinion on learning 360 degree 3D generation with the current pipeline. From my point of view, there is no technical limitation, since you can freely change the novel view camera distribution sampling when training the VAE.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A new paradigm is used on the 3D-aware generation on in-the-wild images. The 3D learning happens in the first stage VAE training, where novel view RGB and depth discriminator is used. The second stage latent diffusion model samples from the trained latent space.\n2. Advantages of the new paradigm over the GAN-based methods, e.g., 3DGP and EG3D, are shown by quantitative evaluations.", "weaknesses": "1. For the first stage of 3D-aware VAE, there are previous methods that propose very similar method. For example, VQ3D and GINA-3D also encodes input images to 3D latents and decode them to the 3D representation of triplane. I would expect authors to give a more detailed explanation of how the proposed method differs from these methods.\n2. To continue on the above comment, I would also like to see the performance comparison with VQ3D on both VAE and generator, since the final generation performance largely depends on the performance of the first stage VAE. Therefore, it is important to understand if the performance improvement is from the first or the second stage.\n3. I would recommend using 50k, instead of 20k, samples to calculate FID, since it is the standard adopted by most of previous methods, e.g., StyleGAN, VQ3D. It would also allow easier comparison with previous or future work.\n4. The visual results shown in Fig. 9 is not very convincing. Flat structure can also be observed from the first two rows.\n5. Visual comparison with other methods is not enough. Firstly, I would like to see this comparison in the main paper instead of supplementary material, since it is a very important part for evaluation and can serve as reference when readers try to understand the advantage of the proposed method. Secondly, I would like to see more visual results for mode collapse of GAN-based methods, as mentioned in the third paragraph of section 4.2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)